Poll : 25 July - 07 August 2012

Analysis

In the interests of space on the Archives page, we abbreviated the question, but here's how it ran in its original form:

We're currently rebuilding our website, including a revamp of the reader rating system, where you can comment on and rate the venues listed on this site, from hotels and restaurants to nightclubs and spas. This question may seem odd, but think about it: Would you prefer a 4-point rating system, where you're forced to choose an overall positive (3 or 4) or negative (1 or 2) experience; or, do you prefer a 5-point system, where it's possible to pick a middle "neutral" 3? In other words, do you like raters to have to choose definitively plus or minus, or are you more comfortable with a scale that doesn't force your hand? Is there such a thing as a valid neutral experience? Will more people rate if we choose a 1-5 scale, or does the neutral middle diminish the value of more ratings?

The results turned out to be among the most decisive we've ever had, and we'll be heeding your input in our new design. In the meantime, here's some of your reasoning for voting as you did:

• As a software usability engineer for a large corporation, I frequently need to survey test participants and potential users, and I almost always use a 5 point Likert-style scale. I believe the Neutral option is extremely important and is an entirely valid position. Forcing participants to choose positive or negative when they’ve had a simply mediocre experience is very likely to create an artificial bias, although it’s impossible to predict whether that would be positive or negative in any given situation. “I just don’t know”, “I can’t decide” or even “I just don’t care” all provide insight as valid as “I loved it," “I liked it,” or “I hated it”. I might not be impressed enough by a restaurant to want to recommend it, but then again maybe it isn’t really bad, so I wouldn’t want to steer people away from it. So I just pick “Neutral” – “it did nothing for me, but you might like it”.
  
  
• I’m the one in the household that does all the surveys. There truly are times when you need a middle-of-the-road position because oftentimes, the way the question is worded forces you to answer one way or another but your answer thus ends up being misleading/not accurate/false. Of course, comment sections help, but surveys don’t always allow for them. For instance, being very simplistic off the top of my head, a survey question might simply ask, “Do you like this restaurant” and require you to answer with a “yes” or “no”, and you might answer “Yes” because it was “okay”, BUT in the back of your mind you are thinking yeah, I liked it BUT I would never return again or recommend it to my friends.
  
  
• I much prefer 1-5 over 1-4 for a rating system. There is nothing indecisive about giving something an average rating of 3. This will give the reader of the poll a more accurate description of the item being reviewed. If you force people to rate either positive or negative, then I think most people will go positive, which will artificially inflate the rating. What will happen is that something that is “just OK” will receive a “good” rating, which in turn will increase the expectations of poll readers, who will be disappointed when the experience the item, ultimately causing them to lose faith in the reviews in the first place. As an example, one of the major newspapers in my area uses a 1-5 star rating system for movie reviews, the other a 1-4 star system. Usually, what happens is that an “average” movie gets 3 stars in the 1-5 system, and 2.5 stars in the 1-4 system. To me, 3 out 5 stars is a better indicator than 2.5 out 4. I don’t think you have to be forced into either positive or negative. I think it’s more informative to the reader to say that something was not great, not horrible, but just OK.
  
  
• I think the answers on the page point out something more that is needed than just a single rating. Maybe a few items are needed for a restaurant: service, food quality, atmosphere, cleanliness, cost/value. Cost/value would be nice to be an actual dollar amount - big difference between $10-12 entrees and $18-19.99 entrees, but the typical, "Less than $10, $10-$20, etc." doesn't give you very much info. Thanks for everything and keep up the good work. [Ed: Firstly, thanks, and we will certainly endeavor so to do. As to your point, yes, we agree and are working toward a more sophisticated system like that, but we may not be able to incorporate that into the first version of the new site -- it may have to await Phase II, early next year.]