Unrest in Cyberspace

The overseers of Second Life, a complex and booming virtual world hailed by many as the first step toward an immersive 3-D Internet, attempted yesterday to calm angry cyber-citizens who have petitioned for fixes to technical bugs recently plaguing the world.

Anger managment: At a May 3 town-hall meeting in Second Life, Linden Lab chief technology officer Cory Ondrejka (right, with purple hair) told residents who are angry over frequent bugs and outages that increasing the virtual world’s stability and coping with its rapid growth “is our highest priority and the focus of the majority of our design, coding, and QA work.”

The main problem, in members’ eyes: Second Life is growing so fast that it’s straining Linden Lab’s resources to the limit, including its developers’ ability to fix old bugs and roll out new software versions that don’t introduce new problems. In a town-hall meeting yesterday inside Second Life, the company appealed for patience.

“We are working to fix bugs and enable incremental improvement,” said Cory Ondrejka, chief technology officer at Linden Lab, the venture-funded San Francisco startup that launched Second Life in 2003. The town-hall meeting was hastily arranged in response to a damning open letter published by irritated Second Life residents on April 30. “At the same time, we are building the foundations for the next-gen architecture that will radically improve our ability to scale,” Ondrejka said.

Every day, some 25,000 computer owners, plus teams from dozens of major corporations, are rushing to join Second Life. But as these new members buy virtual land, set up house for their avatars, and start in-world businesses, the strain on the Second Life “grid” is increasing. Linden Lab is adding more than 120 new servers every week, according to Ondrejka, but users say that the company still isn’t keeping up. Complaints have piled up in Second Life forums and blogs from longtime users impatient over frequent slowdowns and crashes, property that goes missing, messages that aren’t delivered, search and friend-finder functions that don’t work, purchases that aren’t completed, and poor to nonexistent customer service and technical support.

The dissatisfaction culminated this week in the open letter, which demands that Linden Lab address the bugs “immediately,” before rolling out planned features such as voice chat. More than 3,000 Second Life users have signed the letter so far.

“People feel that Linden Lab is failing them because they are paying a great deal, in some cases, for a product that is failing to work acceptably, from a company that will no longer communicate with its customers,” says one signer, a United Kingdom-based IT manager known within Second Life as Inigo Chamerberlin.

Ondrejka spent most of the hour-long meeting answering residents’ questions about the origins of the problems and explaining the steps his team of programmers plans to take to improve performance. As Ondrejka explained at the meeting and in an entry on the company’s blog, many of the problems resulted from unnoticed errors in the most recent release of the simulation software and the viewer software that users must download to their PCs. Those errors are quickly being fixed, Ondrejka said.

But the company faces a far deeper challenge, in the form of an overall software architecture that wasn’t designed to support as many people and transactions as Second Life now hosts.

The world of Second Life is divided into thousands of individual regions, or “sims,” each 65,536 square meters in area (about 16 acres). Linden Lab’s data facilities include more than 20,000 servers, each running one to four sims. The simulation software controls everything going on in its sim, from rendering the terrain and the 3-D models that make up the environment to animating members’ avatars, retrieving their inventories, performing searches, sending instant messages to members in other sims, and communicating with storage databases. If an avatar crosses from one sim into another, every bit of information about that avatar must be handed over to the new sim. The more sims Linden adds to accommodate new members, the more communication goes on between sims, and the greater the burden on each server and on the “backbone” lines connecting them.

“I don’t believe [this architecture] is scalable, at least not to the sizes I want to see it scale to,” said Zero Linden, a “studio director,” or software development manager, at Linden Lab, at a smaller meeting on May 2. (Linden Lab identifies most of its employees only by their in-world names, which always include the surname “Linden.”) But there are “major architectural changes underfoot,” he says, designed to reduce the need for constant connectivity between servers.

The new architecture, briefly referenced by Ondrejka and described in more detail by Zero Linden, would divide up responsibility for each sim into two new classes of programs. “Agent domain” programs would be solely responsible for avatars and their inventories, while “region domain” programs would simulate the environment and its physics. This way, information about an avatar would stay in one place even if the avatar itself crossed sims. “We don’t need to push responsibility around as much,” Zero Linden notes.

Zero Linden says that the company plans to roll out this new architecture sim by sim, probably starting in 2008.

But that’s too late for many members, such as Cristiano Diaz, a software developer based in Miami Beach, who wrote and publicized the open letter with help from other disgruntled Second Lifers. Like many other residents, he says that he’s saddened by the bugs and by the real economic hits reported by members who have lost inventory or suffered other disruptions to their business. “For a long time, it has felt like the promise of what Second Life can really be is being hampered by technical problems and mismanagement,” says Diaz. “It is frustrating and discouraging to watch something you are passionate about languishing because of so many problems.”

But at the town-hall meeting, Ondrejka was insistent that conditions will improve if Linden Lab developers and Second Life residents can work together to ride out the current hiccups. “Patience obviously helps, but more than that, please help the community build the pieces it needs to make Second Life a better place,” he told the audience. “If you are a programmer … come work for us. If you have bugs and can reproduce them, add them into our public JIRA [Linden Lab’s bug-tracking system]. As Bill and Ted would say, be excellent to each other.”