Until only a few years ago, even though the free to play business model had already exploded in the Far East, it was generally either ignored by the industry in this hemisphere, or seen as a curiosity, an afterthought, something foreign that could never really catch on here. As a result, it was pretty easy to identify the early believers since there were relatively few. 

That's how I came to meet Matt Mihaly somewhere around a decade back. He and his independent company, Iron Realms Entertainment, had launched a text MUD called Achaea, Dreams of Divine Lands in 1997. It successfully incorporated item sales, so he knew this approach was viable on a limited scale in this part of the world, and also believed it had far greater potential. Around 2006, he started work on Earth Eternal, a graphical title that will be playable in browsers. This project was spun off to a second studio, Sparkplay Media, in 2007, and is currently in closed beta.

Our conversations and communication over the years have often caused me to think things over further, so following my column last week, which focused on the dearth of meaningful data about how many people are actively playing various games, I was pleased to have the opportunity to explore some of Matt's opinions on the broader topic of metrics.

One thing he pointed out is that measurements can be tricky, especially when used for comparisons, since numbers may not be "apples to apples" even when they appear so. They're more similar in the subscription realm, but even then, there's a huge anomaly in that more than half of the 11 million or so World of Warcraft "subscribers", the ones in the Far East, pay via methods other than monthly fees. Still, despite this and the need to estimate a factor for multiple accounts, we can gauge popularity and revenue within ranges of doubt that aren't as wide as those for F2Ps.

In the latter area, Matt seems to agree with me that registration figures alones aren't particularly meaningful. In fact, he goes so far as to say they're "useless" without proper context in which to evaluate them. "If they were achieved by large marketing spends, then so what? They don't measure anything but the effectiveness and size of those programs. One can buy registrations for $2-$3 apiece. That says nothing about a game or its players. Though I admire Free Realms, this is exactly how it has neared five million registrations so quickly."

He understands that companies like to report registrations because the numbers can only increase. However, he expresses "constant amazement" that game writers, including at least some who are MMO-focused, often just "blithely" report them, which he feels implicitly attributes considerable meaning. I'm not so sure I agree since I don't believe thinking, decently informed readers are fooled. It's also very difficult to get better metrics. However, neither of these things makes the status quo truly acceptable.

Adding to the problem, using active accounts, while something of an improvement, isn't completely satisfactory either since there's no common definition for how they're measured. For some companies, it means logins within the past 30 days; for others, it might be 60. And that's not all. On Earth Eternal, it means players have reached level 4, which is one step past completing the newbie tutorial stage. All these methods introduce distortion, with the degree being dependent on how well or badly they account for churn.

What's more, even standardizing on something like "played within the last X days" wouldn't provide a complete picture. To illustrate this point, Matt offers a couple of references. One is YoVille, which has huge user numbers, perhaps more than WoW in North America, but undoubtedly far less average revenue per user. The other is his own Iron Realms. He says its ARPU exceeds WoW's by a good margin, but within a player base that's tiny by comparison. In both instances, Blizzard's title brings in a lot more money,

Then what about peak or average concurrent users? Not surprisingly, they're imperfect too. They're not good at measuring how actively people play, which Matt calls engagement. To cite an extreme case, I know of a person who is logged in to a particular MMOG almost 24/7. That means he counts as a concurrent user regardless of his actual activity level; he can even be AFK.

The key thing taking a deeper look at metrics reinforced in my mind is that worthwhile ones aren't simple. There are lots of things we can measure, including various ones we've barely or completely not mentioned such as revenue, profit, acquisition cost and customers' lifetime value. But single statistics don't furnish the complete picture. They can't. Indeed, their relative importance and relevance can change quite significantly depending what it is we want to know.

"The important thing is to recognize there's no way to identify a single axis for comparison in terms of free to play MMOs, says Matt. "You have to decide what you want to measure and then try to pick data that informs that measurement." I can only hope there will be more available as the F2P space continues to grow so that we can all understand better how it actually functions.