Sanya Weathers's MMO Underbelly: User Feedback

The most common misconception in the MMO universe is that MMO studios do not care about feedback. The industry has created this problem to some extent. There’s nothing publicly available to indicate that anyone besides a few people afflicted with OCD (*cough*) cares.

My friend Jeremy is in the middle of planning her feedback system for her new game, and is facing the same thing community people always face at this point in the process: There is very little data on best practices in terms of MMO feedback systems. There’s no research, no conversation outside specialty blogs no industry groups devoted to the concept. And studios tend to keep their own systems very, very private. Heck, there isn't much about feedback systems, period.

My instant messenger lights up when she unearths random gems like "if you ask people for information (or compel them to provide it), you should give it back to them in some familiar form that lets them see how it is useful or interesting. They will then be less inclined to resent your asking in the first place, more inclined to be curious about the relationship of their work to others’, and more open to considering its implications—especially if it helps them make arguments for more resources."

That’s a terrific example of a best practice and the underlying reasoning. It didn’t come from any MMO resource. Instead, it’s from a paper about trying to convince suspicious academics to use a web based tool.

But community specialists read that quote and think, good grief, how useful. Not to convince players to send in feedback, but to convince devs.

Let me back up and explain the way feedback reaches a development team. There are three main channels, with their own type of information:

• User initiated data. This is what we all think of when we hear “feedback.” A customer sits down, and types out some comments. Message boards, emails, and attending panels at live events are all forms of user initiated feedback.
• Usage data. This is what players are actually doing. This data is gathered from the game itself.
• Focus group data. The dev team brings in a group of players, and asks specific questions towards a specific end.

These three channels are all necessary to keep a dev team from screwing the pooch when it comes to balancing and introducing new features. Using just one channel is asking for trouble, as is looking at the data without understanding the context of the remarks.

Context can be handled by a community manager, assuming that community manager doesn’t have an axe of his own to grind. I don’t think context can be handled by a developer, a programmer, or any other member of the team – the developer is by definition close to the work, whereas the community manager is by definition part of the community. But that’s my opinion, and there’s certainly room for discussion on that point.

Why does a studio need to use three different channels to get one reliable picture?

The flaws with user initiated data are many. The feedback is limited to those members of the playerbase who are highly motivated to post. The term “silent majority” isn’t a cliché for nothing, y’all. For example, every class has its group of people screaming that their Foozle is underpowered. When no one of a particular class is screaming about their Foozle, it is drastically overpowered and the entire class should probably be removed from the game.

There was a point early in City of Heroes where I went to the forums to look something up about my class, and I noticed that the forums were eerily silent. No activity at all. My spidey senses went haywire, and just for yucks, I checked the other class forums. Indeed, they were all packed full of people complaining. Experience told me that either my class was so awful that no one was playing it at all, or I was going to get nerfed down to the atomic level. The hammer fell two patches later. Owie.

Another flaw with user feedback is that it’s almost always a one way tsunami. Hundreds of thousands of people speaking in one direction, a handful of people speaking the other way. Even if the company is dedicated to listening to and responding to user feedback, it is an enormous challenge to convince people that anyone is listening. It isn’t enough to stand up and say WE’RE LISTENING, sadly.

The MMORPG.com forums are filled with people who are convinced to their marrow that feedback is not heard, not taken into account, and not wanted. No amount of personal testimony, no proof in the form of patch notes, will ever be enough, because the feedback sent by the person posting did not get a personal reply – and worse, the next set of patch notes included something 180 degrees away from that feedback.

(Not to put too fine a point on it, but there’s an old saw about how all prayers are answered, it’s just that sometimes the answer is no. That’s true of a great deal of MMO feedback, but “no” isn’t something most people are prepared to accept.)