Steam Ratings, Reviews, and The Bombing – Steam’s Histogram Puts All the Work Back on the User
This last Tuesday Valve employee Alden Kroll published a blog article addressing the issue of review bombing and their response to the problem. The article itself is very well written with an honest and transparent tone. If you haven't read it yet you should. You can find a link to the blog post from the MMORPG news article here.
Alden explains that Valve has been planning changes to User Review system for some time and cuts right to the chase diving into the one their biggest issues with their user feedback system, Review Bombing. He says, “Review bombing is where players post a large number of reviews in a very compressed time frame, aimed at lower the Review Score of a game. At the same time, they upvote each other's reviews and downvote all the other reviews.”
Valve isn’t the only party that considers this a problem. Some publishers and developers have had issues with Review Bombs including Bethesda during the Skyrim paid mod fiasco and more recently Funcom has been vocal about the issue. They aren’t the only two incidents and the practice isn’t new, but it’s becoming a bigger problem as the practice has become more established.
In the blog post Valve deconstructs the Review Bomb problem. They understand that unhappy players are voicing their opinions acknowledging those could be valid reasons for not wanting to do business with a game publisher or studio. I’ve been put off publishers before from time to time for bone-headed decisions they’ve made so I get it too. Most of us have fallen out with a product or business we’ve liked. That’s not unlike the sentiment happening here. Hello Games and No Man’s Sky is an example of a game that’s been bombed for that very reason.
The problem being, the issues surrounding review bombs are often outside the game itself, and while they could still be valid, they don’t necessarily accurately represent the state of the game or whether potential buyers will be pleased with the purchase. The entire purpose of the User Review is to allow people who’ve played the game tell people whether they should buy it or not. The point of the Review Score, which is the ratio of positive to negative reviews, is to help estimate how much a player will like the game if they buy it.
Valve looked at how Review Bombs affected scores, looked at the data, and measured score trends over time for games that have been bombed. They said that the Review Score generally recovers after a period of time, sometimes fully, but not always. From their trend analysis they concluded that Review Bombs are a temporary distortion in the long term, inferring that the voice of unhappy review bombers is drowned out in a sea of normalization.
Further they claim, “In the cases where the Review Score didn't return fully to its prior level, we believe the issue behind the review bomb genuinely did affect the happiness of future purchasers of the game, and ended up being accurately reflected in the regular ongoing reviews submitted by new purchasers. In some review bomb cases, the developers made changes in response to the community dissatisfaction, and in others they didn't - but there didn't seem to be much correlation between whether they did and what happened to their Review Score afterwards.” Valve wraps up the problem analysis by saying, “….review bombs make it harder for the Review Score to achieve its goal of reflecting the likelihood that you’d be happy with your purchase if you bought a game.”
This is where the blog post takes an unsatisfactory turn for me. There are multiple mixed messages being sent but primarily that review bombs don’t have a lasting affect or that a review score eventually normalizing doesn’t negate adverse affects or have other unpredictable consequences. There is the troubling assumption that if scores do return to normal that there wasn’t a problem, but if they don’t then there obviously was because the score didn’t normalize. This treads dangerously close to circular reasoning, if not outright crossing the line. They also assert in the article that scores trend down over time anyway and that’s because players who buy a game later don’t enjoy it as much as players who buy it earlier. Really, I’m enjoying Fallout New Vegas less than the people who played it years before me? I don’t buy into that and I hope no one else does either.
Valve looked at some solutions. They could remove scores, change the way they’re calculated, or lock them when a bomb is detected. None of those were appealing because they felt it put too much on the user to read the reviews, in the case of removing scores, or that it would make the process unwieldy for both users and Valve, in the case where they’re locked. Also, removing scores would contradict their reasoning for adding them in the first place, players demanded it, and they seem unwilling to consider whether that was a bad idea in the first place.
Instead of fixing the problem Valve decided to throw more data at the user in the form of a histogram and put all of the work figuring this out back on them. Wikipedia defines a histogram as, “…an accurate graphical representation of the distribution of numerical data.” In less fancy terms, it’s a bar graph of data over time. Histograms are great tools and are used in analytics, by professionals, to help define the quality of data and information. But histograms are only one part of an analytics toolbox to help make those decisions. So why does Valve expect end users to step into the shoes of a data analyst, using a tool they haven’t been trained in, and expect that to work well. It’s not a solution. It’s passing the buck while taking the credit for a fix.
I love analytics and data trends are interesting to me. However, I doubt it will help me figure out whether or not I like a game or if the Review Score is accurate any more than reading a few reviews and watching videos or streams of the game play. It’s much easier to do the latter, but that’s not my worry. I’m concerned what and how Valve is deciding what I see on the store page and what I might be missing because they don’t want to plaster my feed with a list of negatively review games. Am I missing out on some cool titles as a result? Are Valve and Review Scores bending the industry, shaping trends, and influencing what gets made and sold?
The best answer for this entire disaster is one I’ve heard echoed across different sites and forums. Remove the scores. They do more harm than they do good. Adding charts, while interesting, for users to decipher doesn’t resolve anything. Leave in the review comments and let people say their piece and make their recommendations, but let’s get rid of that score. Often an up or down vote is completely conditional and the reviewer has stated as such. Let the recommendation stay in the review.
The article concludes with a revelation that “personalized review scores” are coming where the algorithm is based on games the user has enjoyed in the past. I doubt the Review Score will go away anytime soon but maybe personalized review scores will eventually supersede it and hopefully soon. Maybe pressure from publishers and loss of revenue will accelerate that process. Let’s hope so. What are your concerns about this, if any? Do you feel it could be affecting the industry or what you find on your Store feed?