Posted: 5:12 p.m. Friday, Feb. 7, 2014
We all know Amazon reviews run the gamut from "insightful" to "worse than reading nothing at all," and we can cope with the poor quality. The inane reviews lend credibility to the rest, making the good ones seem culled from a vast field of "authentic" average consumers. But thanks to some of Amazon's sketchier practices -- especially a program called "Vine Voices" -- you can't even be sure of that. Here's a guide that should help you machete your way through the Amazon. I chose 20 reviews* and dissected them for content and other patterns.
2 out of 20, or 10 percent of the reviews I read, said "Vine Voices" next to the review. Vine is a six year-old invitation-only program for serious Amazon reviewers to smooth out their writing process by simply delivering them free swag. Yes, Vine (Which has nothing to do with the video sharing app) mails certain products to the homes of reviewers, and the reviewers agree to send the item back if requested, so it's just borrowed, not a gift. But apparently this rarely if ever happens. In other words, a review with "Vine Voices" next to it isn't a customer review at all.
Does it bias the reviewers? Yes, apparently it makes them harsher, according to Julie Law in Amazon's PR department. She told NPR their ratings are lower on average. Perhaps it's because these serious critics genuinely believe that with great power comes great responsibility.
I suppose the justification is that these are no different than reviews by a critic who went to a screening or a book reviewer who got their copy sent to them by the publisher, but one of the Vine Voices reviews I encountered was about a pair of socks and the reviewer wrote, "I like these socks enough that I ordered a second helping." Saying you sought out a second pair when the first was free is different from saying you bought one pair, and then bought another. It's a subtle, but meaningful distinction, and it's just one example from the two Vine reviews I read during this exercise.
It's not unethical if Amazon discloses it, and they do. They just shroud it in jargon. So spread the word about what that "Vine Voices" tag means.
A Review of the Reviews:
You'll see below that often, reviewers on Amazon are focused on providing a background that justifies their rating, rather than providing useful information. Most reviews are positive, and reviews aren't updated or given new context, even when they've been there for a decade or more.
Here are some numbers from my selection of reviews.
As far as I can tell, Amazon doesn't archive reviews of products they're still selling. The average top review in my sample was 3.8 years-old, but the oldest was from ten years ago. Still, I suppose what was true ten years ago about quality is probably still true today. The newest review was from last month.
Average length: 228 words.
Longest: 885 words.
Shortest: 20 words.
Average rating: 3.9 stars.
5 star ratings: 50%.
4 star ratings: 20%.
3 star ratings: 10%.
2 star ratings: 10%.
1 star ratings: 10%.
Eight of the twenty weren't "verified purchases." So theoretically the reviewer might have been lying about ever having had their hands on the product forty percent of the time. But were they? I doubt it.
As for content -- and I know this is highly subjective -- I tried to put myself in a buyer's position, and determine if the review left me feeling educated about the product. Twelve reviews, some of which were brief, presented valid remarks and useful descriptions. Eight were useless, mostly presenting irrelevant personal details.
Instead of making fun of a terrible review here is a review of -- at least to my mind -- about average descriptiveness:
"I've owned a few SafeTGard Adult Athletic Supporter Jockstraps for the past couple of years and this is definitely my favorite compared to other brands. I'm a big guy, and the XXL fits perfectly. The material is comfortable, and keeps everything in place where it should be. When I receive them new, I always wash it once before I wear them because out of the box they are a bit stiff. After playing football, wrestling, and rugby for over 10 years wearing all different brands of jockstraps, this is the most comfortable jockstrap i've ever worn."
I underlined everything that could be considered useful. The remark about the XXL fitting a big guy invalidates itself for the majority of buyers, the fact that it's comfortable strikes me as a minimum requirement for a three star review, and the tip about washing it before wearing it, well, seems like a no-brainer.
A Better Way?
User created lists are an underutilized feature on Amazon. A section called Listmania does have user lists, but there aren't many when you consider the size of Amazon, and they certainly don't steer users toward their list section the way some sites do. I was under the impression that Amazon lists were more of an active component of the site than they are, and I used to google search terms like "office chairs for tall people." That query gets you this page on Amazon. Disconcertingly, that is not a user list as I first thought, but some kind of pre-programmed search results page within Amazon, but for a different search query: "Best Ergonomic Office Chairs For Tall People." I no longer trust searches like that because I don't know how these pages were generated. In my worst case scenario it's just a list curated by Amazon, but that makes me a little uncomfortable.
The Wirecutter is a site that reviews products, but it's not focused on savings, and it's not altogether a substitute for Amazon because it's not, and isn't trying to be, comprehensive. The Wirecutter simply takes a category of products, and recommends one. According to the Wirecutter there's simply one best TV and it's the Panasonic ST60. There's one best soda maker, and it's the Mastrad Purefizz. Their testing process is rigorous, and it's all written about in detail. For many products, especially big, expensive purchases, The Wirecutter is a better place to do your homework than Amazon.
*The products I checked were an electric screwdriver, a litterbox, an athletic supporter, a package of highlighters, a seed spreader, a Disney branded Mr. Potato Head, a piece of window hardware, a hammock, a wastebasket, a sports bra, a mushroom guide, a microphone pop filter, a watering can, a pair of socks, a messenger bag, a book about werewolves, the Forearm Forklift, the game Call of Duty Black Ops for XBox 360, a dollhouse, and a sweater, found by querying Google for general shopping terms and choosing the first Amazon item presented to me in the organic search results. I looked only at the top review of each product, and I dissected the reviews for data and content. A machine could produce 20 random products or reviews, but I just used my own shopping habits. There you have my imprecise methodology. Your mileage may vary.