How to fix crowdsourced restaurant reviews

In the Herald on Sunday, Lauraine Jacobs takes aim at amateur restaurant reviewers on sites like Zomato. Jacobs says the unpaid reviewers don't know what they're talking about, and are rewarded for the sheer number of notices they write rather than the quality.

I actually believe in the wisdom of the mob.

But I have a different issue with crowd-sourced reviews.

Mid-last year, I was on a panel interviewing CIO Awards finalists. One of the contenders for Executive Team of the Year was the crew from GrabOne. When the discussion turned to new initiatives, I threw in an idea of my own.

It goes like this: Personally, I always take community reviews of any restaurant or cafe (or other service provider) with a grain of salt. Friends of the owner will praise them, rivals will slag them. Yelp, which recruited "marketing consultants" to pre-populate its Auckland site, is now suing a company that sells fake reviews. And so on.

So instead off user reviews, or as a supplement to them, why not have a measure of repeat business? What percentage of people who bought Xtreme Pizza's deal-before-last came back to the trough when it next offered a deal on GrabOne?

GrabOne's MD raised his eyebrows and scribbled something down, though I've yet to see my brilliant idea implemented. Maybe there are some practical issues. But I think the broad principle is sound: people's actions speak larger than words.

I was reminded of that again this week talking to Netflix chief product officer Neil Hunt, in town to share more details ahead of his company's pending NZ launch.

Like all-comers, Netfix has crowd-sourced star-ratings for its content. There's a small army of people who "hand tag" or manually rate movies and TV series.

But Hunt says when you log onto Netflix and see a series of recommendations, it's not based on what you say you lke so much as what you actually watch and what people with similar preferences watch (when you join the service, you can do a brief survey on your tastes and favourites).

Netflix says a huge amount of work and thinking has gone into its recommendation engine (I'll have to take Hunt's word for it; I started to zone out and stare at my foot as he described slicing multidimensional matrixes to find intersection points). 

There's a lot going on here. Behavious are also watched. For example, if lots of people rewind a slice of a movie one or more times, it's a sign they're really engaged with it. They don't want to miss a moment. But the short story is: you're likely to like a new show if it gels with your previous viewing history and/or that of your peers.

And that people fib when they do their own star ratings.

Hunt says Netflix found two issues.

One is "survivor bias". That is, people who watch all of a TV series, or an entire movie, are much more likely to rate it than those who switched off. And because they watched it to the end, they're likely to have liked it. Those who abandon a show, or a film, often don't bother to rate it. 

The other is that people often rate a movie or TV series on production values, or on how worthy they think it is. Hunt gives the example of Steven Spielberg's slavery epic Amistad. People might dutifully give it five stars, but that doesn't mean they actually thought it was a good movie. And even if it did happen to be their all time favourite, that doesn't mean they want their recommendation list clogged with Hollywood "issues" films.

Another Netflix staffer gives the example of how when the service recently launched in France, there was a bit of push-back from some who saw it as an attack on culture. That's dubious, given Netflix' original series (such as House of  Cards, Orange is the New Black and Peaky Blinders) are critically acclaimed middle to highbrow fare. But more so given French Netflix viewers favourite flim turned out to be Jackass 3.

Hence the Netflix recommendation engine's emphasis on what people's actual behaviour. That's a lesson that could be learned across a lot of sectors.

ckeall@nbr.co.nz

Follow @ChrisKeall