When the credibility of book reviews comes into question: Part 3

Why I don’t trust Amazon reviews

Any site that allows you to both review and buy/sell a book will be biased.

It’s useful to have a network of friends or authors provide reviews for you, but these will always be positively biased, and the more this is seen on Amazon (and other sites that directly sell the book) the less these reviews will be trusted. Conversely, there are also some out there who seem to enjoy blasting indie work and will happily post negative reviews to harm their selling power.

The result of all this is an unbalanced history of ratings that make it difficult to determine how good a book actually is. I’ve discussed the question of credible book reviews before in Part 1 and Part 2 of this series but now I’m going to give some specific examples of what can happen.

Example of unbiased reviews with an average of 3/51) Definitions: Take an unbiased set of ratings with an average of 3/5 (see right).  The typical shape you see is a Gaussian distribution (as discussed in part 1).

There will be an average rating – i.e. the most popular.  This is indicated by the position of the highest number of votes.

There will also be a spread – this describes how ‘undecided’ the collective group are on the rating.

For the following I will keep the spread the same and play around with the ‘average rating’.

An extreme example of positive bias2) Positive bias: Now, to illustrate what happens initially when most reviews are positively biased.

Take, for example a book which has largely positive ratings (80% ~ 5/5) and a small amount of ‘unbiased’ reviews (20% ~ 2/5).

Ok, so the example I show is an extreme, but it illustrates the mismatch between the two opinions.

Ratings example showing positive and negative bias3) Positive and negative bias: Now let’s factor in both positive (5/5) and negative (1/5) bias. 

For this example the ‘true average rating’ might lie at 3/5.  The figure on the right shows a 50:50 split between positive and negative (bias) reviews.

The result of 2 sources of bias is that it is now difficult to identify the actual average rating.

Real example of the impact of positive and negative bias 4) A real example: Lastly, just to drive this point home I’ve taken rating details from a popular indie book (where both positive and negative bias will occur) and plotted it versus what I think is probably the average rating ~3/5.

There are 2 points to note here:

1) The ‘bookstore’ (something beginning with A…) shows the behaviour we’d expect from a biased distribution.

2) The review site (think something beginning with G…) shows something closer to a ‘normal shaped’ curve.

Now I’m not saying that the latter of the two sites is completely unbiased but, because the factor of money or marketability has been mostly removed from the equation the bias in rating distribution is less obvious.  Of course a larger number of reviews also helps.

So why don’t I trust Amazon reviews?  Because I see too many examples of positive (or negative) bias.  If I really want to decide on a new book I’ll ask for recommendations from friends, or try to check ratings elsewhere.  But that’s just me…

Disclaimer 1 – The larger the number of non-biased reviews the less influence this effect will have.  So, for a well-known series, for example, this comment probably won’t apply.
Disclaimer 2 – The numbers used here are just a demonstration. The spread and mean will vary from book to book.

About indie e-books

http://indieebooks.co.uk is aimed at providing a common platform for indie writers and publishers to show-case their work. The content is kept limited to one (free) short story per author to give the reader an idea of the writing style and talent involved and to help forward them on to more substantial work.
This entry was posted in Indie Publishing, Reviews and tagged , , , , , , , , , . Bookmark the permalink.

3 Responses to When the credibility of book reviews comes into question: Part 3

  1. Siobhan says:

    Oooh I love your stats graphs. I sometimes look up reviews of books by authors I might want to commission and it’s very surprising how many of them seem to go on Amazon and review their own books (you can normally track the comments because they are SO full of praise and in the same style across a whole range of esoteric books with otherwise mediocre scores….)

  2. Pingback: Start of a catfight? Amazon vs. Goodreads | Indie Ebooks

  3. Pingback: Fifty Shades of…? | indie e-books

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s