Are Amazon Book Reviews Completely Biased?

Is Amazon missing some helicopters?
Is Amazon missing some helicopters?

My ratings obsession

Anyone who knows me knows that I look at a lot of ratings before I decide to do anything. A good friend of mine once joked I wouldn’t turn left if Yelp didn’t tell me to. Have I taken it too far? – probably – I mean I’m writing a blog post on ratings.

Recently, I was looking for a new book to read when I came across Goodreads. I signed up because it has a recommendation engine that can tell  you what you might like based on books you have read and what your friends are reading.

As I browsed the ratings on Goodreads I wondered if they were any different than Amazon’s. What I saw was surprising – books had a very different ratings on Amazon vs Good Reads, even after having hundreds and thousands of ratings. So which one is better?

vs    


The Data: Goodreads versus Amazon – Ratings distribution

Based on an index of 10 books equally weighted (weighted towards business books):
These books received 15K reviews for Amazon and 315K reviews for Good Reads

Goodreads versus Amazon - Ratings distribution

How can thousands of reviews have such a different mean on one site versus the other? And why do Amazon’s ratings have such a  “U” shape compared Goodreads?

Alexa Goodreads.com
Alexa Goodreads.com
Alexa stats Amazon.com
Alexa stats Amazon.com

At first thought you might think – different audiences?

Thanks to Alexa we see it’s not age, education, or family status. The only major difference is that Goodreads skews female more heavily than Amazon. But that doesn’t explain it – why would women on average rate books lower and have a more uniform distribution in ratings. It must have something to do with the ratings system itself. It turns out it probably does (thanks to Sid for helping remind me of the bias in #1):

1) The more work one must do to submit a review, the more extremes you will see in the rating (“U” shape) On Goodreads you simply have to fill out a star rating while on Amazon you have to write a full review. Hence you’ll see many more 1s and 5s on Amazon than what might be true in the real world. So when you look at reviews on Amazon, realize in reality there are a lot of people in the middle who think the books are really 3s and 4s who didn’t review it.

2) There are penalties for dishonesty on Goodreads. Goodreads is a recommendation site. To start getting recommendations you have to rate 20 books you have read. If you aren’t harshly honest about the books you stand the risk of getting bad recommendations.
Amazon, on the other hand, is much more outward facing. In a public review of the book, you may say it was good but it wasn’t for you and settle at 4 stars, but you definitely don’t want to read something like that again so it might get 2 stars on Goodreads.
(Amazon has recommendations too – but these are much more skewed towards browsing history)

All this is to say I think Goodreads reviewers have more skin in the game and are a more representative sample. For now Goodreads = better reads.

Advertisements

2 thoughts on “Are Amazon Book Reviews Completely Biased?

  1. Two other possibilities that came to mind.. 1) Amazon is good at recommending books, so readers are more likely to rate high. and 2) If you shift the Amazon graph to the left, and break the 5 stars into two groups, maybe you get Goodread’s graph. Perhaps the criteria for stars, whether strictly enforced (as you mention) or psychological based on design / community, are different per platform.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s