Hardware Today -- Scaling Mythical Realms With Dell

Mythic Entertainment is banking on Dell for victory in the online gaming space. The company is using combined clusters of hundreds of Dell machines to drive its flagship multiplayer online game, Dark Age of Camelot.

Last week's Dell Server Snapshot showed the vendor adding Nocona-based servers to its arsenal as it continues to couple open standards commodity computing with invincibly low prices.

This week, Hardware Today puts on its chain mail and grabs a +1 magic sword to cast a light spell in the direction of Mythic Entertainment, a company using combined clusters of hundreds of Dell machines to drive multiplayer online games including its flagship, Dark Age of Camelot, which it claims is the world's fastest growing online role-playing game. Another product soon to be added to the stable is the Imperator, a game based on an alternate Earth in which Rome never fell.

Hardware Quest

Mythic's wide-scale multiplayer online games connect client software, which individual players purchase in shrinkwrap for $9.95 or download on a trial basis, to a massive Linux back end where each player interacts with others online. Think of it like Xbox Live, but with centralized hosting and thousands of players instead of a dozen or so per game. The bulk of Mythic's revenue comes not from software sales, but in the monthly fee it charges players after the first free month.

Not surprisingly, facilitating simultaneous online game play for 30,000 to 40,000 people at any given time requires Herculean efforts.

A Mythical Creature

"To actually create these games is an intense challenge that most companies underestimate," Mythic Entertainment COO and co-founder Rob Denton told ServerWatch. He adds that most single player game companies say, "'Hey, let's take our game and make it multiplayer!' fail miserably because they grossly underestimate the challenges involved."

Mythic credits its 10-odd years of success in multiplayer games in part to its focus on back-end development in messaging, bandwidth management, and load balancing. The hardware, of course, is a critical component of these three areas.

Dell Shall be King

Mythic's vendor quest began four years ago. "We were looking for a Linux-compatible hardware solution," Denton said, "and we opted out of more expensive technologies like Sun and HP because we didn't want to license the OSes." That left a choice between Gateway rack-mounted systems and Dell racks. In the end, Dell's "best price-performance ratio" and "good track record of reliability" won out.

Mythic then standardized on Dell for its servers, laptops, and desktops, Denton said. At first, the now-retired PIII 2U PowerEdge 2550s ruled the rack. Over time, they abdicated to dual-Xeon rack servers, 2U PowerEdge 2650s, and more compact 1U PowerEdge 1750s.

Mythic currently has several hundred Dell servers in the United States; additional game hosting locations in France and South Korea have also standardized on Dell. The company has on order 40 Xeon-based PowerEdge 1850s, which are scheduled to begin shipping by October.

Denton notes that Mythic has passed on Dell's software offerings, "We look at Dell as primarily a hardware solution," he said. Thus, Mythic has avoided Dell's pre-packaged Linux solutions in favor of a customized version of Red Hat Linux. "We do software, so we're pretty good with that," he jokes.

"When you actually play our game, you join what we call a world," Denton said. Eighteen worlds live on eight Dell servers mirroring each other for redundancy purposes. A ninth server acts as a hub for those entering the game.

Seven additional servers, which Mythic refers to as World Boxes, run 100 "world tasks," Denton said. Such tasks are run across each World Box and keep the Worlds functioning as players expect. This way, should one server crash, play continues transparently.

"It's like a cluster of clusters," Denton says. Dell servers round out Mythic's more mundane functions as well. "We have an entire billing solution coded [and running] on Dell," as well as infrastructure Linux databases, he notes.

Heroic Reliability

Mythic's servers have interoperated reliably across multiple generations of Dell products, Denton said. "We have some 200 Dell servers running nonstop, some of them for over three years," he noted, with "very few hardware machine-level problems." Hot-swappable RAID Level 1 and RAID Level 5 drives handle the odd hard drive crash without service interruption.

The Dell servers have functioned so heroically that Mythic's operations staff consists of only one part-time employee. "We were expecting to need a multiperson operation staff to handle our hardware problems," Denton said. "Some [companies] might go with a Windows NT solution, and you hear these horror stories of how they need a 12-person operation staff to constantly swap out machines or deal with outages. We just don't have these problems."

For the few problems Mythic has encountered, Dell support has come through in shining armor. Even so, "over the course of three years, I can count the number of issues on one hand," Denton said.

Future Adventures

Denton is eager to see the EM64T Nocona-driven PowerEdge 1850 in actual deployment. Nocona's trifecta of better bus, memory, and processor speeds, coupled with lower power consumption, will allow more servers per rack and thus save on operating costs down the road.

Mythic casts a wizened warrior's wary eye on new server offerings. Blades for example: "They've [Dell has] come up with new technologies like their blade technology, and that looks neat, but that's not the best cost solution for us where we stand right now," Denton notes. Ditto for Opteron. Mythic will consider an Opteron offering only if such a server makes a more attractive value proposition than Dell's Xeon servers.

"We have something that works great for us right now, so why would we want to change?" Denton ponders. An advantage like dramatically lowered cost, decreased power consumption, or double processor compatibility might merit exploring newer technologies, but, "it's hard to go with something that is not proven."

Mythic is but one example of the loyal commodity customer driving Dell toward sales success in the crowded commodity space. "Of all the vendors that we tend to deal with, Dell has been one of the best," Denton notes. He goes so far as to say that Mythic's relationship with Dell has helped keep various competitive products from laying claim to Mythic's enterprise throne.

"We have a representative who answers us immediately, and gets us quotes back immediately. We get our hardware in a timely fashion, it's reliable, and it works," he said, adding, "You can file us under enthusiastic because we would not be in business right now if it weren't for Dell."