Case Study

RamSan Success Stories: CCP Games Problem: 17,000 concurrent users causing sluggish OLTP response

"The effect of the RamSan was immediate on both system performance and customer satisfaction," Hilmar V. Pétursson, CEO, CCP

CCP Games has created the virtual reality game, EVE Online, with true human emotions in a living and evolving world. EVE Online is the world's largest game universe with over 75,000 dedicated subscribers who play this Massively Multiplayer Online Role-playing Game (MMORPG) in a single, connected environment. Art, technology, and unbridled imagination fuse to create a persistent universe where everyone has a place with a gaming enjoyment that has no boundary. Players take on the role of spaceship pilots seeking fame, fortune, and adventure in a huge, complex, exciting, and hostile galaxy.

Players connect to the game servers via the Internet and interact in real time with other players worldwide. EVE Online is truly a massive multiplayer game, where all 75,000 plus subscribers inhabit the same game world, not split into smaller limited groups or "shards". The world of EVE Online is persistent, continuously evolving, and never sleeps. Intense graphics and sound add to the rich gaming experience, so it's no wonder that CCP Games' subscriber base continues to grow daily. CCP Games is unrelenting in its quest to maintain a fresh and challenging gaming experience to its ever-expanding universe of players. As popularity and complexity expands, so too does the need for responsiveness and performance.

Challenge: Scale to over 25,000 users with instant response times Immediate user access and application responsiveness are vital to success in the competitive online gaming market. Sluggish system response to player input is frustrating and may even result in players abandoning the game. In September 2005, EVE Online set a new record of over 15,000 concurrent users. CCP Games monitors its systems and player forums closely at all times to ensure player satisfaction. As more players came online simultaneously, the EVE Online servers were having performance degradation problems. Players were becoming frustrated as frequently accessed game features were taking up to 20 seconds to load.

The degrading performance potentially compromised CCP Games' mission of attracting and retaining customers by providing top quality online entertainment, based in part on maintaining quality of service. Performance analysis revealed that disk queues were 40 requests deep. Microsoft recommends upgrading hardware if disk queues exceed 3. Better hardware is needed if more users are accessing a server.

The EVE Online application software was running fast and scaling well to accommodate tens of thousands of simultaneous users with its 150 IBM servers. However, the system's storage was being overwhelmed with huge amounts of small data requests. Ideal RAID Storage performance is achieved when the requested data blocks are large; conversely, RAID Storage performance is greatly reduced when requested data blocks are small. Typical disk access times are 2-5 milliseconds and what was needed was a faster disk access time of 20-50 microseconds.

The Solution: RamSan-400 (Solid State Disk)

The RamSan-400 delivers 400,000 I/Os per second, has 3,000 MB of internal bandwidth with latency of less than 15 microseconds. "We did consider upgrading to faster disks, but the specs for the RamSan were so insane that we had to look into it", said Jörundur Matthíasson, Database Manager for CCP Games.

EVE Online's underlying storage bottleneck is a classic problem with Online Transaction Processing. 10,000+ users accessing account information, warping across the galaxy, buying goods from black-market free-lance smugglers and upgrading their mining frigates to assault cruisers all at the same time puts immense stress on disk-based storage. The game makes 60 million process calls per day with around 1,250 transactions per second at peak hours. The RamSan-400, based on solid-state disk (SSD) technology, is ideal for applications that have massive amounts of transactions randomly distributed across the storage media. Where disks are limited by how fast a platter can spin, SSD uses memory chips to read and write data. Transactions run as fast as the server can issue, and are not bottlenecked by grinding disk heads.

After careful evaluation, CCP Games deployed the 64-Gigabyte RamSan-400. Set up was smooth and simple. From a DBA perspective, the RamSan looked like just another disk so there was no learning curve, despite this being CCP's first experience with SSD. CCP moved the most heavily accessed data onto the RamSan. One table alone gets 8 million records added per day plus a huge amount of selects and updates.

Record number of players, customer satisfaction, and 40x performance improvement With the RamSan, EVE Online's performance and its ability to support concurrent users skyrocketed. Within four days of installation, EVE Online set a new Peak Concurrent Player record with 17,032 playing at the same time. Hilmar V. Pétursson, CEO of CCP said "After installing the RamSan our player forums quickly lit up with enthusiastic and positive comments."

"It's definitely faster. First hardware improvement that had instant results."

"Holy **** that is faster... Get more of them."

"Much much faster database calls for me...good job CCP you deserve a monster cookie!"

"Yep, I have to admit... I noticed the upgrade the moment I logged on today."

"Well I must say this was an awesome investment that CCP made..."

The RamSan eliminates virtually all database related lag in EVE Online. The game now has more concurrent users receiving their data at higher speeds than ever before. The wait queue dropped from 40 requests pending to a value so small it is not measurable. Another database performance measurement, "latches" or "total wait time", which measures the amount of time a transaction is idle waiting on another concurrent transaction to finish, went down from 25,000 to 4,000. These technical measurements are indicative of stellar performance - 40 times better than before.

"The effect of the RamSan was immediate on both system performance and customer satisfaction," said Hilmar V. Pétursson, CEO of CCP. ". Now we can focus our engineering resources on expanding our game universe and the customer experience without the limitations imposed by hard disk performance. It's going to be a nice change for our developers."