When the credibility of book reviews comes into question: Part 3

Why I don’t trust Amazon reviews

Any site that allows you to both review and buy/sell a book will be biased.

It’s useful to have a network of friends or authors provide reviews for you, but these will always be positively biased, and the more this is seen on Amazon (and other sites that directly sell the book) the less these reviews will be trusted. Conversely, there are also some out there who seem to enjoy blasting indie work and will happily post negative reviews to harm their selling power.

The result of all this is an unbalanced history of ratings that make it difficult to determine how good a book actually is. I’ve discussed the question of credible book reviews before in Part 1 and Part 2 of this series but now I’m going to give some specific examples of what can happen.

1) Definitions: Take an unbiased set of ratings with an average of 3/5 (see right).  The typical shape you see is a Gaussian distribution (as discussed in part 1).

There will be an average rating – i.e. the most popular.  This is indicated by the position of the highest number of votes.

There will also be a spread – this describes how ‘undecided’ the collective group are on the rating.

For the following I will keep the spread the same and play around with the ‘average rating’.

2) Positive bias: Now, to illustrate what happens initially when most reviews are positively biased.

Take, for example a book which has largely positive ratings (80% ~ 5/5) and a small amount of ‘unbiased’ reviews (20% ~ 2/5).

Ok, so the example I show is an extreme, but it illustrates the mismatch between the two opinions.

3) Positive and negative bias: Now let’s factor in both positive (5/5) and negative (1/5) bias. 

For this example the ‘true average rating’ might lie at 3/5.  The figure on the right shows a 50:50 split between positive and negative (bias) reviews.

The result of 2 sources of bias is that it is now difficult to identify the actual average rating.

4) A real example: Lastly, just to drive this point home I’ve taken rating details from a popular indie book (where both positive and negative bias will occur) and plotted it versus what I think is probably the average rating ~3/5.

There are 2 points to note here:

1) The ‘bookstore’ (something beginning with A…) shows the behaviour we’d expect from a biased distribution.

2) The review site (think something beginning with G…) shows something closer to a ‘normal shaped’ curve.

Now I’m not saying that the latter of the two sites is completely unbiased but, because the factor of money or marketability has been mostly removed from the equation the bias in rating distribution is less obvious.  Of course a larger number of reviews also helps.

So why don’t I trust Amazon reviews?  Because I see too many examples of positive (or negative) bias.  If I really want to decide on a new book I’ll ask for recommendations from friends, or try to check ratings elsewhere.  But that’s just me…

Disclaimer 1 – The larger the number of non-biased reviews the less influence this effect will have.  So, for a well-known series, for example, this comment probably won’t apply.

Disclaimer 2 – The numbers used here are just a demonstration. The spread and mean will vary from book to book.