Linux patch becomes terminal pain

Many users of the increasingly popular Ubuntu Linux distribution found themselves thrown back to mid-1990s on Tuesday, when a botched update to the graphical X Window subsystem brought them face-to-face with the command-line terminal.

The update, pushed out to Ubuntu users Monday night, aimed to fix some hardware issues to the X Window software used by almost all Linux systems, but instead caused the graphical user interface (GUI) to fail to initialise, leaving users to deal with issuing text commands through the terminal.

By Thursday, more than 700 comments had been posted to the Ubuntu forums by affected users looking for answers, and the Linux project - managed by software and service firm Canonical - issued an apology.

"When we learned of the problem, the patch was immediately withdrawn," the group said in the mea culpa posted to its website. "Mirrors have also been disabled to ensure that the faulty patch isn't available from them. We have launched an investigation and formal quality process review to understand exactly how this happened and what corrective actions to take."

Instructions posted to the Ubuntu website allow affected users to roll back the problematic update with a few commands. The project withdrew the faulty patch early Tuesday, after about 17 hours.

"It was not clear how many people were affected," Matt Zimmerman, chief technology officer for Ubuntu, said in an interview with SecurityFocus. "The bug seems to be hardware specific."

Ironically, the incident occurred as Linux competitor Microsoft had update problems of its own. A cumulative patch to Internet Explorer published by the software giant over a week ago fixed eight vulnerabilities but introduced an exploitable security flaw into the web browser. On Thursday, Microsoft pushed out an upgrade patch that fixed the problem.

Earlier this year, Ubuntu fixed a security hole in its Linux distribution caused by the installer storing the user's main password without first encrypting the data.

Many users were willing to give a pass to the Ubuntu project, which has generally garnered rave reviews among Linux users, for the rare update issue.

"This fix worked perfectly for me," wrote a member of the Ubuntu forums identified as "Moephan." "I had no GUI for about five minutes total. Also, I didn't lose any data or anything...Stuff like this happens, even to big companies like Sun and Microsoft."

Others users criticised the project for missing the problem during quality control and predicted that the mistake would cost the project future users.

"Overall, this was an eight hour exercise, eight hours that I could afford to take out of my day, but how many others have that luxury?" wrote a forum member with the handle "Dale61". "I'm wondering how many others, particularly those in business, are still trying to find a solution to a problem that should never have happened in the first place."

Despite the distributed programming model of open source projects, catastrophic errors in patches are rare. The companies and groups behind most Linux distributions extensively test software updates, said Holger Dyroff, vice president of marketing for Novell's SUSE Linux group.

"What you try to do as a business - and this is true for Microsoft as well as the enterprise Linux distributions - you try to make sure that these issues don't happen," Dyroff said. "With thousands of customers out there - many paying for the product - it's important to invest in quality assurance."

As a project, Ubuntu checks any updates from internal and external sources and requires that a member of the team sign off on the changes. Ubuntu's Zimmerman remained hopeful that the incident would not cause users or administrators to delay patching their systems - or at least, delay any more than they already do.

"I think it is a common practice among administrators of large networks to wait before applying patches," he said. "Those administrators are very conservative and don't apply updates right away no matter who the vendor is."

The project is currently reviewing its quality checking procedures, and in the end, the incident will serve to make Ubuntu's update process better, Zimmerman said.