Health site glitches show wider problems with technology

By Craig Timberg and Lena H. Sun, The Washington Post

Friday, October 11, 2013

WASHINGTON -- Problems with the federal government's new health-care website have attracted legions of armchair analysts who speak of its problems with "virtualization" and "load testing." Yet increasingly, they are saying the root cause is not simply a matter of flawed computer code but rather the government's habit of buying outdated, costly and buggy technology.

The U.S. government spends more than $80 billion a year for information-technology services, yet the resulting systems typically take years to build and often are cumbersome when they launch. While the error messages, long waits and other problems with www.healthcare.gov have been spotlighted by the high-profile nature of its launch and unexpectedly heavy demands on the system, such glitches are common, say those who argue for a nimbler procurement system.

They say most government agencies have a shortage of technical staff and long have outsourced most jobs to big contractors that, while skilled in navigating a byzantine procurement system, are not on the cutting edge of developing user-friendly websites.

These companies also sometimes fail to communicate effectively with each other as a major project moves ahead. Dozens of private firms had a role in developing the online insurance exchanges at the core of the health-care program and its website, working on contracts that collectively were worth hundreds of millions of dollars, according to a Government Accountability Office report in June.

The result has been particularly stark when compared with the slick, powerful computer systems built for Barack Obama's presidential campaigns, which in 2008 harnessed the emerging power of social networking and in 2012 relied on aggressive data-mining efforts to identify and turn out voters. For those, the campaign recruited motivated young programmers, often from tech start-ups.

"The wizards from the campaign have no desire to contract with the federal government because it's a pain in the butt," said Clay Johnson, a veteran technologist for Democratic campaigns who pushes for procurement reform through his whimsically named start-up, the Department of Better Technology. "Is it possible to be good? Is it possible to do right by the taxpayer in this space? I'm not sure that it is."

He is one of many Obama supporters hoping to help fix the website by drawing on the collective wisdom of software developers, a mostly left-leaning group that have been analyzing healthcare.gov and sharing their thoughts in emails, blog posts and exchanges on Reddit.

Among their conclusions: Requiring all users to sign in before surfing choked the system, as did insufficient server capacity. They also noted that the website stalls if a single step in the process -- such as verifying a user's identity -- is not quickly completed.

Industry officials note that new software often is buggy, even when it is produced by respected tech firms such as Apple and Google. It's one reason that private companies prefer gradual launches and long periods of testing before starting major marketing pushes. Although it is possible to conduct "load testing" on a site in hopes of determining how it will respond to heavy demand, there is no substitute for the crush of traffic experienced by a popular system on its official launch date.

Despite warnings of looming problems from the GAO and others, federal officials expressed surprise when the website failed almost immediately, with millions of people receiving puzzling, frustrating error messages.

Federal officials have blamed the problems mainly on site usage far beyond what was anticipated, with more than 8 million people trying to use healthcare.gov in the first three days after the site was fully activated on Oct. 1.

A huge effort to remedy problems, including round-the-clock work from the contractors who built it, have eased problems. But by Wednesday -- nine days after the launch -- significant flaws and slow performance continued.

"The episode is all too typical of how government creates IT services," said Tom Lee, director of Sunlight Labs, the research arm of the Sunlight Foundation, which advocates for more government transparency. "The procurement process tends to select for firms that are good at navigating the procurement process, not providing good IT services for the dollar."

The Obama administration has declined to say how many people had successfully signed up for insurance, even as many states offered updates. The Department of Health and Human Services, which oversaw development of the site, declined to comment Wednesday.

"It's not at all uncommon for something this complicated to have problems," said Stan Soloway, president and chief executive of the Professional Services Industry Council, a Washington-based group that represents government contractors. He said it's important to know the scope of what the contractors were asked to do before rushing to judgment. "You can tell me to build a house for four kids, but then if you have 27 it's not the company's fault."

This argument generates some sympathy from outside analysts. The website needs to interact with many other systems, including those maintained by the Social Security Administration, the Department of Homeland Security and the Internal Revenue Service, to verify the identity, citizenship and income of potential applicants. It also was designed to draw from the offerings of private insurers, each with their own computer systems, rates and offerings.

John McDonough, a health policy professor at the Harvard School of Public Health, said, "The number of systems that have to align here is pretty daunting."

Aneesh Chopra, a former White House chief technology officer, said there is a common misperception that any "three kids in a garage" can put together a website.

Even in the eyes of critics, federal procurement officials were innovative in hiring a small Washington-based tech firm, Development Seed, to build what developers call the "front end" of the website. That's the part with the smiling young woman and the encouraging words "The Health Insurance Marketplace is Open!" It has worked with few glitches, earning praise for how easily it adapts to different devices, including the small screens of most smartphones.

But the "back end" -- the guts of the system, which required far more computing power and integration across other federal networks -- was built by a traditional contractor, CGI Federal, a subsidiary of a global firm based in Montreal. The company, which has an office in Fairfax, has declined to comment on the problems with the site.

Federal health officials have not yet explained why CGI was given the contract or why it was awarded on a sole-source basis. Nor is it clear which of the problems that healthcare.gov has been having are related to the work of which IT vendors -- or even whether the problems were the work of a vendor or of the HHS technical staff.

CGI is among preferred contractors who regularly do work for the HHS, said a Senate aide familiar with contracts related to the health-care law. Such firms are eligible for projects that must be done quickly because they are familiar with the agency's security standards and other aspects of the operation.

"This is going to get fixed. This is a lot of scale. They got, in 24 hours, more people trying to register than Twitter got users in 24 months," said Eric Gundersen, president of Development Seed. "Let's not all hate on the government here."