So, in order to understand user initiated feedback, you need usage data. How are people actually using the game? People do not report this information accurately. This is where you need black box recorders. How many people finish this quest? How often does this combat move get used as opposed to that one? What zones are they in?
Ah, zones. Long ago in a virtual world, user initiated feedback told the developers of a particular game that they hated, HATED Zone X. They constantly mocked this zone – “the golf course.” It was, admittedly, not the finest zone in the game. It was large and green without a lot of features. It has been hastily assembled at the last second. The new zone was pretty neat on paper, lots of obstacles and control elements, requiring tons of strategy.
And yet removing the golf course was, shall we say, not popular. What? Whyever so? The usage data on the old zone showed that despite the mockery, the much maligned zone was where everyone went to play if they had nothing else to do. It was the default, where a group could always be found, and you didn’t need a minimum number of people to do anything, and you could always find someone to kill. The new zone had none of those things and a layer of complexity that prevented all casual users from playing at all. Taking out the golf course was like punching the game in the head. And that’s just one example of usage data contradicting the user initiated data. Taking the two sorts of data together would have proved that while players did not love the old zone, there were elements of that zone that any replacement feature MUST provide in order to be successful.
Usage data can also tell you what a game is missing or doing wrong. If everyone has to have a particular item, the need is related to some aspect of the game that is tedious. For instance, in EQ, original recipe, about a year into its life, everyone had to have Journeyman Boots. The quest to do it was a pain because it took half a guild, required rare drop items from rare spawns (in a game where anyone could loot any dead monster), and more. But everyone still did it. Why? Because travel in EQ was tedious. J-boots didn’t let you move as fast as SOW (Spirit of Wolf), but it beat the regular walking speed. Usage data would have shown that everyone in the game who belonged to a guild had this item, which would have raised the question “why?”
Finally, the third channel for feedback is focus group testing. This is where you observe a group of people in your desired demographic interacting with the game. User initiated feedback and usage metrics don’t tell you why people do what they do. Watching people play gives you that information. You see where your own interface is preventing people from doing things, you see where they get frustrated, you see what content gets people leaning forward in their chairs. You can compare your observations with how they fill out surveys and learn whether you’re asking the right questions.
Focus group testing is sometimes hidden under the guise of “player summits” or even “blog press events.” Regular press events are not focus groups. Those are Potemkin villages lubricated with alcohol.
Anyway, these three types of feedback are all necessary to paint an accurate picture of the game. Feedback from any one source is flawed, and both developers and players can find reasons to argue and reject what they hear.
But there simply isn’t any such thing as an MMO developer that doesn’t take feedback seriously. So how can a studio communicate that to the players? And should they?
That takes us back, at long last, to my original point that there isn’t much public proof that the industry places a premium on feedback at all. In fact, you lately hear people who should know better arguing that there’s no point in making it a public process, because Blizzard doesn’t. And if it’s good enough for games with a billion players, why, it’s good enough for everyone.
Bzzt. Wrong, on several levels.
Notice that everyone trying to out-Blizzard Blizzard is face planting. World of Warcraft didn’t get a billion people playing because of their wonderful communication, but in spite of it. Anyone looking to compete in the MMO space should be looking for ways to set themselves apart, and open communication/support of the wider community is the lowest hanging piece of fruit on the tree.
And anyone who thinks Blizzard doesn’t pay close attention to feedback, particularly usage data, is someone who doesn’t read patch notes or notice expansion design.
I advocate greater transparency. Tell people how much feedback is coming in on which issues. Highlight patch notes deriving from player feedback. Post transcripts of player summits. Commit to regular public discussions. If the feedback is flying in the face of usage data, say so. None of these things can harm a dev team with their game’s concepts held firmly in mind, or do damage to a team that isn’t trying to be all things to all people. And all of these things, done publicly, will help the industry as a whole as it struggles to figure out why some things succeed, and other projects (with equally devoted developers) fail. Sharing best practices doesn’t harm competition – it enhances it by allowing products to compete on the strength of the concepts, not on the team’s ability to reinvent the wheel.