Healthcare.gov and the Gulf Between Planning and Reality

Back in the mid-1990s, I did a lot of web work for traditional media. That often meant figuring out what the client was already doing on the web, and how it was going, so I’d find the techies in the company, and ask them what they were doing, and how it was going. Then I’d tell management what I’d learned. This always struck me as a waste of my time and their money; I was like an overpaid bike messenger, moving information from one part of the firm to another. I didn’t understand the job I was doing until one meeting at a magazine company.

The thing that made this meeting unusual was that one of their programmers had been invited to attend, so management could outline their web strategy to him. After the executives thanked me for explaining what I’d learned from log files given me by their own employees just days before, the programmer leaned forward and said “You know, we have all that information downstairs, but nobody’s ever asked us for it.”

I remember thinking “Oh, finally!” I figured the executives would be relieved this information was in-house, delighted that their own people were on it, maybe even mad at me for charging an exorbitant markup on local knowledge. Then I saw the look on their faces as they considered the programmer’s offer. The look wasn’t delight, or even relief, but contempt. The situation suddenly came clear: I was getting paid to save management from the distasteful act of listening to their own employees.

In the early days of print, you had to understand the tech to run the organization. (Ben Franklin, the man who made America a media hothouse, called himself Printer.) But in the 19th century, the printing press became domesticated. Printers were no longer senior figures — they became blue-collar workers. And the executive suite no longer interacted with them much, except during contract negotiations.

This might have been nothing more than a previously hard job becoming easier, Hallelujah. But most print companies took it further. Talking to the people who understood the technology became demeaning, something to be avoided. Information was to move from management to workers, not vice-versa (a pattern that later came to other kinds of media businesses as well.) By the time the web came around and understanding the technology mattered again, many media executives hadn’t just lost the habit of talking with their own technically adept employees, they’d actively suppressed it.

I’d long forgotten about that meeting and those looks of contempt (I stopped building websites before most people started) until the launch of Healthcare.gov.

* * *

For the first couple of weeks after the launch, I assumed any difficulties in the Federal insurance market were caused by unexpected early interest, and that once the initial crush ebbed, all would be well. The sinking feeling that all would not be well started with this disillusioning paragraph about what had happened when a staff member at the Centers for Medicare & Medicaid Services, the department responsible for Healthcare.gov, warned about difficulties with the site back in March. In response, his superiors told him…

[…] in effect, that failure was not an option, according to people who have spoken with him. Nor was rolling out the system in stages or on a smaller scale, as companies like Google typically do so that problems can more easily and quietly be fixed. Former government officials say the White House, which was calling the shots, feared that any backtracking would further embolden Republican critics who were trying to repeal the health care law.

The idea that “failure is not an option” is a fantasy version of how non-engineers should motivate engineers. That sentiment was invented by a screenwriter, riffing on an after-the-fact observation about Apollo 13; no one said it at the time. (If you ever say it, wash your mouth out with soap. If anyone ever says it to you, run.) Even NASA’s vaunted moonshot, so often referred to as the best of government innovation, tested with dozens of unmanned missions first, several of which failed outright.

Failure is always an option. Engineers work as hard as they do because they understand the risk of failure. And for anything it might have meant in its screenplay version, here that sentiment means the opposite; the unnamed executives were saying “Addressing the possibility of failure is not an option.”

* * *

The management question, when trying anything new, is “When does reality trump planning?” For the officials overseeing Healthcare.gov, the preferred answer was “Never.” Every time there was a chance to create some sort of public experimentation, or even just some clarity about its methods and goals, the imperative was to avoid giving the opposition anything to criticize.

At the time, this probably seemed like a way of avoiding early failures. But the project’s managers weren’t avoiding those failures. They were saving them up. The actual site is worse—far worse—for not having early and aggressive testing. Even accepting the crassest possible political rationale for denying opponents a target, avoiding all public review before launch has given those opponents more to complain about than any amount of ongoing trial and error would have.

In his most recent press conference about the problems with the site, the President ruefully compared his campaigns’ use of technology with Healthcare.gov:

And I think it’s fair to say that we have a pretty good track record of working with folks on technology and IT from our campaign, where, both in 2008 and 2012, we did a pretty darn good job on that. […] If you’re doing it at the federal government level, you know, you’re going through, you know, 40 pages of specs and this and that and the other and there’s all kinds of law involved. And it makes it more difficult — it’s part of the reason why chronically federal IT programs are over budget, behind schedule.

It’s certainly true that Federal IT is chronically challenged by its own processes. But the biggest problem with Healthcare.gov was not timeline or budget. The biggest problem was that the site did not work, and the administration decided to launch it anyway.

This is not just a hiring problem, or a procurement problem. This is a management problem, and a cultural problem. The preferred method for implementing large technology projects in Washington is to write the plans up front, break them into increasingly detailed specifications, then build what the specifications call for. It’s often called the waterfall method, because on a timeline the project cascades from planning, at the top left of the chart, down to implementation, on the bottom right.

Like all organizational models, waterfall is mainly a theory of collaboration. By putting the most serious planning at the beginning, with subsequent work derived from the plan, the waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work. Instead, waterfall insists that the participants will understand best how things should work before accumulating any real-world experience, and that planners will always know more than workers.

This is a perfect fit for a culture that communicates in the deontic language of legislation. It is also a dreadful way to make new technology. If there is no room for learning by doing, early mistakes will resist correction. If the people with real technical knowledge can’t deliver bad news up the chain, potential failures get embedded rather than uprooted as the work goes on.

At the same press conference, the President also noted the degree to which he had been kept in the dark:

OK. On the website, I was not informed directly that the website would not be working the way it was supposed to. Had I been informed, I wouldn’t be going out saying “Boy, this is going to be great.” You know, I’m accused of a lot of things, but I don’t think I’m stupid enough to go around saying, this is going to be like shopping on Amazon or Travelocity, a week before the website opens, if I thought that it wasn’t going to work.

Healthcare.gov is a half-billion dollar site that was unable to complete even a thousand enrollments a day at launch, and for weeks afterwards. As we now know, programmers, stakeholders, and testers all expressed reservations about Healthcare.gov’s ability to do what it was supposed to do. Yet no one who understood the problems was able to tell the President. Worse, every senior political figure—every one—who could have bridged the gap between knowledgeable employees and the President decided not to.

And so it was that, even on launch day, the President was allowed to make things worse for himself and his signature program by bragging about the already-failing site and inviting people to log in and use something that mostly wouldn’t work. Whatever happens to government procurement or hiring (and we should all hope those things get better) a culture that prefers deluding the boss over delivering bad news isn’t well equipped to try new things.

* * *

With a site this complex, things were never going to work perfectly the first day, whatever management thought they were procuring. Yet none of the engineers with a grasp of this particular reality could successfully convince the political appointees to adopt the obvious response: “Since the site won’t work for everyone anyway, let’s decide what tests to run on the initial uses we can support, and use what we learn to improve.”

In this context, testing does not just mean “Checking to see what works and what doesn’t.” Even the Healthcare.gov team did some testing; it was late and desultory, but at least it was there. (The testers recommended delaying launch until the problems were fixed. This did not happen.) Testing means seeing what works and what doesn’t, and acting on that knowledge, even if that means contradicting management’s deeply held assumptions or goals. In well run organizations, information runs from the top down and from the bottom up.

One of the great descriptions of what real testing looks like comes from Valve software, in a piece detailing the making of its game Half-Life. After designing a game that was only sort of good, the team at Valve revamped its process, including constant testing:

This [testing] was also a sure way to settle any design arguments. It became obvious that any personal opinion you had given really didn’t mean anything, at least not until the next test. Just because you were sure something was going to be fun didn’t make it so; the testers could still show up and demonstrate just how wrong you really were.

“Any personal opinion you had given really didn’t mean anything.” So it is in the government; any insistence that something must work is worthless if it actually doesn’t.

An effective test is an exercise in humility; it’s only useful in a culture where desirability is not confused with likelihood. For a test to change things, everyone has to understand that their opinion, and their boss’s opinion, matters less than what actually works and what doesn’t. (An organization that isn’t learning from its users has decided it doesn’t want to learn from its users.)

Given comparisons with technological success from private organizations, a common response is that the government has special constraints, and thus cannot develop projects piecemeal, test with citizens, or learn from its mistakes in public. I was up at the Kennedy School a month after the launch, talking about technical leadership and Healthcare.gov, when one of the audience members made just this point, proposing that the difficult launch was unavoidable, because the government simply couldn’t have tested bits of the project over time.

That observation illustrates the gulf between planning and reality in political circles. It is hard for policy people to imagine that Healthcare.gov could have had a phased rollout, even while it is having one.

At launch, on October 1, only a tiny fraction of potential users could actually try the service. They generated concrete errors. Those errors were handed to a team whose job was to improve the site, already public but only partially working. The resulting improvements are incremental, and put in place over a period of months. That is a phased rollout, just one conducted in the worst possible way.

The vision of “technology” as something you can buy according to a plan, then have delivered as if it were coming off a truck, flatters and relieves managers who have no idea and no interest in how this stuff works, but it’s also a breeding ground for disaster. The mismatch between technical competence and executive authority is at least as bad in government now as it was in media companies in the 1990s, but with much more at stake.

[W]hat he fundamentally had right was the understanding that you could no longer run a country properly if the elites don’t understand technology in the same way they grasp economics or ideology or propaganda. His analysis and predictions about what would happens if elites couldn’t learn were savage and depressingly accurate.

Now, and from now on, government will interact with its citizens via the internet, in increasingly important ways. This is a non-partisan issue; whichever party is in the White House will build and launch new forms of public service online. Unfortunately for us, our senior political figures have little habit of talking to their own technically adept employees.

If I had to design a litmus test for whether our political class grasps the internet, I would look for just one signal: Can anyone with authority over a new project articulate the tradeoff between features, quality, and time?

When a project cannot meet all three goals—a situation Healthcare.gov was clearly in by March—something will give. If you want certain features at a certain level of quality, you’d better be able to move the deadline. If you want overall quality by a certain deadline, you’d better be able to simplify, delay, or drop features. And if you have a fixed feature list and deadline, quality will suffer.

Intoning “Failure is not an option” will be at best useless, and at worst harmful. There is no “Suddenly Go Faster” button, no way you can throw in money or additional developers as a late-stage accelerant; money is not directly tradable for either quality or speed, and adding more programmers to a late project makes it later. You can slip deadlines, reduce features, or, as a last resort, just launch and see what breaks.

Denying this tradeoff doesn’t prevent it from happening. If no one with authority over the project understands that, the tradeoff is likely to mean sacrificing quality by default. That just happened to this administration’s signature policy goal. It will happen again, as long politicians can be allowed to imagine that if you just plan hard enough, you can ignore reality. It will happen again, as long as department heads imagine that complex technology can be procured like pencils. It will happen again as long as management regards listening to the people who understand the technology as a distasteful act.

This entry was posted on November 19, 2013 at 2:40 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed.
Both comments and pings are currently closed.

91 Responses to “Healthcare.gov and the Gulf Between Planning and Reality”

[…] site debacle. Clay Shirky, who writes and teaches about the cultural impact of the Internet, has an interesting take on healthcare.gov, and calls out the “waterfall” approach to project management as a particular […]

good essay. the concepts in it apply to most organizations, not just government and their IT people. private firm management holds all techies and skilled labor in contempt, universally and across the board.

This is certainly true, but from my own limited experience working in a government agency, I have to agree with Fran; in many cases, internal expertise is not present on the federal side. The key piece for me was that CMS admitted they did not have enough personnel who considered themselves qualified to judge the assessments of the developers they had contracted. From the very beginning of a project, there may be only a handful of people who are capable of translating business requirements into technical specifications, writing an RFP, or assessing the information presented by potential vendors and contractors. And those who do understand the technology may have little influence or authority, or even permission to communicate directly with the contractor’s development team. The multi-year, half-billion dollar project that stands out in my mind was plagued by a lack of assertiveness on the federal side (because so few people on staff understood the technology), and a lack of responsiveness on the contractor side (because no one made it sufficiently clear to them what the system was supposed to do or why). That project is still sucking up our federal dollars, and has yielded little in terms of function. Developers and technologically-informed representatives on either side rarely met, and most feedback was exchanged between people who were neither developers nor end users, to the obvious detriment of the project.

Another problem for the federal government is that its agencies seem committed to hiring the same federal contractors over and over again, regardless of performance. I once felt fortunate to get the ear of a high-ranking superior in my agency regarding a personal project with potential ly far-reaching impact, but my heart sank when he summed up with “Just tell me what we need so I can tell [large contractor] what to build.” I was crestfallen, because not only was this the same contractor who had failed to deliver before, but it seemed like agency higher-ups had not absorbed the lesson that software development does not work that way. “The vision of “technology” as something you can buy according to a plan, then have delivered as if it were coming off a truck” is one most people with the power to command our resources still hold onto, no matter how many times they’ve watched it fail. I knew that even though I had done the research, there was little chance I’d be involved in the project directly; that would fall to more senior employees, likely in a different division altogether. So I knew that even if the project in question managed to be green-lit, it stood little chance of being successfully implemented, and would constitute yet another expensive failure.

This is not to say that the people involved are unintelligent or even doing their jobs poorly. For most of them, the expectation that they would learn to manage development of crucial technologies descended on them mid-career, or frequently, just prior to retirement. They were entrenched in positions they were well-qualified and high-performing, but have been asked to adopt an entirely new skillset upon which the future of their agencies will rest. That’s a tall order for anyone. The other issue is that the nature of federal employment, being bureaucratic far beyond typical corporate bureaucracy of the private sector, tends to put blinders on people. You have a very specific mandate as a federal employee, and relatively little freedom to innovate or question the chain of command. In many agencies, it resembles a pseudo-military structure highly dependent on hierarchies, and where thinking within only a limited scope is encouraged. This not only engenders a “cover your ass/don’t overstep” mindset among even younger hires, but also makes entering public service an unattractive option for the kinds of technologically savvy, innovation-minded workers who drive successful software development.

There was also a legacy of hiring freezes in our agency; we had very few employees between 35 and 45 years of age, largely because of a five-year hiring freeze a decade ago. And another hiring freeze several years later thinned the ranks of people under 30. This meant that the ranks of management in our agency were swelled with people over 60. We were not the only agency whose demographics were skewed by hiring freezes, which leave the imprint of past hard times on current projects.

Solving this problem will require attracting those workers to federal projects and federal work in general. That will mean giving them a kind of freedom and authority rarely found in our federal agencies today. Breaking some of the hidebound bureaucratic conventions that restrain innovation on the federal side will be the only way to implement anything like agile development or simply more responsive agency-contractor relationships. How do go about breaking that mould, though, is as thorny a problem as developing the technological solutions themselves. That, I think, will be the real challenge for high-level management going forward, since only they have the power to affect that change.

The “insulated executive” reminds me of the story told by one of the early TV news producers, about the difference between setting up for Presidential press conferences.

Truman, the product of small town and machine politics, would become alert at the hint of trouble and offer to help.

Eisenhower, in charge of the entire European Theater in WWII, would simply say, “Fix it.”

Every executive styles themselves an Eisenhower… even though they do not have his grasp of how things work. So many times I wanted to tell them, “There’s more to leadership than telling people what to do.”

About phased rollouts, a way to implement them successfully while at the same time in a politically acceptable way would be to start launching the website in a couple low-populated states and generalise the rollout at a later time.

As for learning the best way to manage tech projects, I think one of the underlying issues is the fundamental disconnect between the need for constant iterations one one hand, and the political schedule which is based on announcements on the other.

Politicians tend to aim for a big announcement that will be followed by another a couple months later. If your flagship project is not ready in time, you have no announcement to make. I’m not sure how this can be successfully reconciled with the reality of good software development.

Insightful, as always. I’d especially like to pick on your initial theme about poor intra-bureaucratic communication and the social dynamic behind it. A big-picture sort of explanation I’ve already heard is that successful execution of the healthcare.gov project was doomed because it was too “politicized,” by which I think the commentator I heard this from meant that Obama couldn’t go back to Congress for legislative adjustments before the 2012 election. But here you’re pointing to another dimension of this: project planning was too *politicized* to allow for the accurate and successful communication of project parameters (features, quality, time), too politicized to allow for the ongoing organizational learning that complex technology projects need. But I’d like to let your observations expand what “politicized” means in this context, because it’s not just about elections, it’s about power, and as you say it’s a problem that extends well beyond government circles and politics.

Certainly inefficiency arises…
* When a contractor cannot share honest assessments of the changing project needs without risking its revenue
* When an employee with critical and valuable insight cannot be heard because hearing her would mean the manager loses status
* When planners cannot change all-or-nothing requirements because support for their planning authority and/or funds for execution depend in large part on a painstakingly negotiated contract (or in this case, law) that cannot be revisited because the balance of interests and power among the negotiating parties has shifted as well
*When there’s neither commitment to nor agreement on how to collect and evaluate evidence about what’s working or not working, because “working” is defined more by the requirements of retaining or gaining power in a broader contest than by the project requirements per se

When the dust settles and history has more evidence to work with, I really can’t say I know which of these inefficiencies will have proved more damaging to the ACA roll-out or its ultimate success.

“Faced with an election that is the crystallized result of essence of policy failure, Obama decides that he…sent the wrong message. Has this man ever passed a math class? Or done any physical craft? Done anything where failure is not the result of failing to be persuasive? Perhaps not.”—Me, November 8, 2010

Fran: This story isn’t about the line-level developers but about managers and above, decision-makers, who understand technology. Most of the site was built by external contractors, not government workers. Many big companies who went public years ago are still doing innovative and exciting things, so the idea that all the good technologists are working at Snapchat or a Starbucks in SOMA is inaccurate.

In principle, this all makes perfect sense – except. Imagine what would have happened if the Republicans started to cherry-pick all the bad outcomes in those months and months of testing that should have been done. It’s entirely possible that the law would be in even worse shape by now. You can’t evaluate this situation outside the toxic context the Republicans have set up for the rollout of this law. Yes, the administration saved up all the problems. – but that may have been the right political decision.

I think where Clay errs is in describing the flawed construction of Healthcare.gov in tactical terms. The company that won the contract has been doing this for a long time, so clearly it’s good at winning contracts. It’s also responsible for some state exchanges that are, according to all reports, similarly craptacular. So the problem isn’t about something particular that was done on this contract, it was about — as he starts out by explaining — a culture that makes real-world testing irrelevant or even slightly dirty. After 5 or 10 years of marketing executives despising the technologists who break the promises they make to customers, a company like this will have weeded out any managers or senior developers who might pass bad information back up. You can have the most transparent reporting structure in the world under such a culture, and no one at the bottom will be stupid enough to use it.

Meanwhile, states who have gone with other contractors seem to have solid, well-functioning sites already (or as well functioning as you could expect with the baroque multiple-chokepoint architecture imposed by the law).

Perhaps poorly stated in an otherwise fine article, this is wildly inaccurate:

“[Now, and from now on, government will interact with its citizens via the internet, in increasingly important ways. This is a non-partisan issue; whichever party is in the White House will build and launch new forms of public service online.] Unfortunately for us, the last new technology the government adopted for interacting with citizens was the fax; our senior political figures have little habit of talking to their own technically adept employees”

For the president to say “I was not informed directly that the website would not be working the way it was supposed to,” begs some questions, such as: Did he ask anybody in the weeks, months, or years leading up to launch, if it would be working? (And were they sure?) Did he designate a trusted head-knocker from his team to haunt HHS and make damn sure it was going to work? His administrative passivity and subsequent excuse-making doubletalk (“40 pages of specs and this and that and the other and there’s all kinds of law involved…”) calls his basic competence into serious question. Or one must surmise that he just wasn’t that concerned about how well this law was implemented.

All that I will say is that the contractor (CGI) has consistently built phenomenally poor web applications. In viewing the healthcare’s web source I immediately recognized mistakes and typos that I’ve seen in inheriting other projects by them. Sometimes going with the lowest bidder — well, you get what you pay for.

“This is not just a hiring problem, or a procurement problem.” — given the poor quality of CGI’s previous work, I would consider that this is certainly a hiring problem. Or perhaps a managing problem if the proper research wasn’t conducted on each of the bidding candidates.

All good and salient points. I agree about everything you said. It is crazy to let ideology and fantasy overrule reality.

However, sadly the reality IS that fantasy and ideology and politics rule the roost in Washington DC. It’s not merely enough to just say ‘OK, what we’ve been doing clearly doesn’t work. Let’s do it another way’. That’s logical and rational thinking. Doesn’t work.

That doesn’t work in politics because the risk/reward system for politicians are NOT based on observable results or based on what WORKS. Granted, if every voter was as well informed as you or I and actually voted on issues in a clear and rational manner, then the politicians we elect will also work in rational ways to better the country and run projects in a rational manner that works. See the Nordic nations for clear examples of a democracy that works very well.

Alas, what we’ve seen time and again, is that the way the electorate elects their representatives, there’s no hope for common sense. Would we be even having this malicious debate about whether Global Warming is a global scientific conspiracy or not if the people that vote actually voted rationally? Or the birther movement? Or the Tea Party?

You cite Valve as an example of how to run a good project? Well done to Valve. But is Valve run as a democracy? Do decision makers always have to justify themselves to nay sayers who have ZERO interest in running a successful organisation, and their only intention is to tear you down, no matter the cost? If there are rogue elements like that at Valve I have zero doubt Gabe Newell will cut them loose quicker than you can say ‘Half Life 3’. Alas, the president has no such powers.

This problem with democracy in America didn’t just arrive or surface by the healthcare.gov debacle. It’s a deep rooted problem that pervades across the entire political and social spectrum. Frankly, healthcare.gov is a sideshow to the real problem we’re seeing here.

take a look at how the UK’s Government Digital Service dealt with exactly this problem: due to extraordinary leadership, vision and clarity of purpose, with a few months they turned “civil servant culture” on its head when it came to digital – and delivered products using the approaches advocated above.

It is by no means impossible, or even that deep a problem, if you have the political will and skills to make it happen.

First off, Clay, thanks for Here Comes Everybody. It changed the way my organization thought about social networking and validated what many of the younger employees were saying.

Second, next to the President’s (or governor’s or mayor’s or whatever) picture in every government office should be a copy of Alexander’s Question: “What new information would be sufficient to change your decision?” I don’t know that it’s always disdain for lower pay grades that leads to bad decisions. Sometimes it’s just the lack of courage or initiative or ability to frame the vocabulary that prevents the question being asked.

Many years ago a boss of mine disdained even touching a computer. One day he was standing impatiently over my colleague while the latter’s computer was processing a large data file. Finally he blurted out “Can’t you make it go faster?!”

I put in a few years at US Army research labs, and it was much like this. Good ideas I had got killed for political reasons and projects that were provably physically impossible are still under way after 30 years and counting.

From what I have observed in 30 years as a tech reporter, whether a project is run by the government or private interests makes no difference in terms of quality, time and features.

Government can make things work, when everyone involved is dedicated to the mission at hand. Vista, the VA’s computer system, worked well. On the other hand, the public-private system of contracting doesn’t work. AHLTA, the military’s health IT system, is garbage.

I think the President offered some serious insight in discussing the difference between his campaign insight and Healthcare.gov.

“If you’re doing it at the federal government level, you know, you’re going through, you know, 40 pages of specs and this and that and the other and there’s all kinds of law involved.”

Because of the way U.S. government contracting law works, it’s very hard to build from scratch, and you’re always going to be limited in terms of who you can go to for system integration. That needs to change.

Excellent analysis. I’m disappointed that Obama was not more active in ensuring the successful implementation of healthcare.gov; he said he was going to change Washington, but on his signature issue he employed the same corrupt, bloated engineering practices that prevent government innovation. At the very least, Sebellius should be fired immediately.

It’s also notable that they feared delaying the site because of Republican repeal efforts; when one party is dedicated to proving that government doesn’t work, it’s harder to prove the opposite.

Fran – you’re correct about the incentives for hotshot developers. And a
few hotshot developers would certainly help in getting the high-level
architecture right. But the bottom line is that any large-scale software
project will be mostly designed, coded, and tested by people who are
mostly near the median level of skill and experience. So the planning
and execution of large-scale developments has to involve developing the
right management structure and processes to allow a large number of
fairly flawed designers and programmers to produce something that
fits together and works.

That might be viewed as an argument for avoiding “large-scale” software
projects altogether, and choosing Pareto-inspired projects which can
give you 90% of the desired functionality with 10% of the code –
ideally a small enough chunk of code that a small team of hotshots in
a single location can get it done. Or it can be viewed as an argument for
careful management and extensive review and testing of every part
of the specification, design, and code at every level of integration.
Or an argument for splitting things up into independent projects even
though that introduces duplication of effort – it’s noteworthy that the
various state healthcare exchanges, solving many, but not all, of the
same problems – but with the benefits of separate management, separate
staffing, and fewer external dependencies – seem to be operating quite
smoothly.

But if the goal is to write 20M lines of code in 2 years, more or less,
then it isn’t a problem of not being able to hire hotshots. If a hotshot
can write 50K lines per year, way above average productivity, then
you’d still need 200 hotshots. And no organization can get 200 hotshots
together on a single project, no matter what they’re paying. A sprinkling
of hotshots is all you can hope for.

[…] the lines of what the family therapist told me at the bar whilst deep in his cups … “Failure is Always an Option,” from found-materials playwright, former Electronic Freedom Foundation executive and current […]

A November 2 Washington Post article on the development of the system said that there was an internal struggle over who should run the development:

“For weeks that spring, a tug of war played out inside the White House, according to five people familiar with the episode. On one side, members of the economic team and Obama health-care adviser Zeke Emanuel lobbied for the president to appoint an outside health reform “czar” with expertise in business, insurance and technology. On the other, the president’s top health aides — who had shepherded the legislation through its tortuous path on Capitol Hill and knew its every detail — argued that they could handle the job.”

Guess who won.

If you’re responsible for developing systems you should live by Murphy’s Law and the aphorism, “Without data all you have are opinions.”

Clay, I’ve been an info mgmt consultant for decades & I can tell you: You weren’t hired because “they didn’t want to listen to their tech employees,” you were hired because they DIDN’T WANT TO BE IN THE SAME ROOM W/ THEM.

@Fran: the hottest brains do tend toward the biggest reward, true. And productivity differences are huge between stars and the rest of the pack. But what makes you think that stars were required to construct healthcare.gov? This is hardly ground-breaking tech; quotidian developers can do this stuff fine, IF they are given the tools and competently managed. Mr. Shirky’s point echoes this: we know how to do this stuff! Iteration, test-driven development, customer on-site, start small and elaborate, YAGNI…you know the tools and techniques as well as I do. But trust in the workforce has to come first. “This won’t work” should be answered by “How do we fix that?”, not “it has to or you’re fired.” That kind of one-way management was proven a failure a century ago in the trenches of France, yet we’re still stuck with too much of it.

1. Time actually seems to have been an issue, in particular as it relates to another issue, sabotage. You kind of have to compare this to building something new in an active war zone. If given a full three years to build it, then testing would have been done properly. Instead, an enormously complex project is being built in a matter of months rather than years and assembled on the fly, because of court challenges, refusal by states to do their part and a hostile political/media climate. If they manage to keep this in flight, we may look back on it with Apollo 13-style romanticism, believe it or not.

2. The frame of reference here is too web-focused. This is as much about legacy systems integration, always nasty, nasty work. Under the best of circumstances there will be bugs galore. In addition, I don’t know that it’s the elite/blue collar divide we’re talking about so much as the traditional business (or in this case policy and political) versus technology divide that has characterized the thousands upon thousands of private sector IT implementation failures.

3. Even more complex, in the business world, you have the divide between business and IT. In government you have a triangulation between political staff, policy staff and support staff (whether that’s IT, finance, whatever). It’s a Bermuda Triangle, it seems.

4. It’s worth noting that some of the state exchanges (surely simpler projects by orders of magnitude) that aren’t working were also done by CGI, the prime contractor. Maybe they do suck?

I find it helpful to distinguish between mature and immature technologies. Healthcare.gov is certainly less of an engineering achievement than a Boeing 747, yet we take for granted that the 747 is going to work. Media executives stopped thinking of printers as critical because, as printing technology matured, it became reliable and the people who did it became interchangeable.

It is difficult for the technically naive to understand this, and to understand why one project is technically challenging while another project, which seems just as impressive, is relatively easy. The software engineering community doesn’t help itself by second guessing the developers when they don’t actually know how difficult a project is. You hear a lot along the lines of, “Amazon built this enormous store for less then Healthcare.gov costs. Why couldn’t they be like Amazon?”

As if using one of the most successful companies in the past 100 years as your performance standard is fair and makes sense, or that anybody has any idea whether its easier to build a store than it is to build healthcare.gov.

Non-government projects are in no way immune to these problems. I’ve been specifically excluded from participation early on in projects because management was not interested in reality and I’ve had solution architectures subverted by both management’s dearth of specific knowledge and politics. I base this opinion on my decades of software development experience both in government and industry positions and as a state employee, as a employee of a public company subcontracting to the federal government, and as an employee of a public company providing highly scalable solutions including web applications to the world’s largest carriers (ISPs and phone companies) with millions of users.

Much of what Clay identifies as the underlying and overriding issues are covered quite eloquently in the classic book (published in 1975) by Frederick Brooks Jr. called “The Mythical Man Month.” The book is just as relevant today as it was then and it is readily accessible by non-technical readers (management, sales, marketing, etc.). Although I’ve recommend it to people whom it would be of great benefit to it seems that they lack the interest in reading it or learning from it. YMMV

Your attack on waterfall method is misguided. The method is not the problem. The problem is that healthcare.gov did not use the method correctly.

In practice, waterfall doesn’t, as you say, “amount to a pledge by all parties not to learn anything while doing the actual work.” That may have happened on healthcare.gov, but that’s a poor implementation choice, not a fault of waterfall itself. McKinsey’s “Red Team” report spells out heathcare.gov’s failure to implement the method properly.

It’s true waterfall frontloads planning and design. What is wrong with laying out project constraints from the beginning? But from my experience, development, review, and testing cycles are iterative – more like a series of agile cycles.

No tenet of waterfall method says you should push a broken product to market.

Fran, your comment is an example of Shirky’s argument. Many people work at Apple and never receive public accolades. The difference is that Apple has a culture of listening to developers, designers, and engineers and not putting half baked products in the market.

The last such failure was Apple maps. Developers told the executive, Scott Forstall, maps was problematic, and that manager was let go. The Apple maps debacle is an example where the manager’s success and ego was more important than the product. And it is this same kind of selfish and willful blindness to reality that Shirky is describing in Washington and government culture.

What I find hilarious is people are complaining about the problems with Healthcare.gov but it pails in comparison to the egotistical pandering, myopia, and mismanagement of the Iraq war. Where the technical people and the analysts and the people experienced at going into situations where governments need to be rebuilt were ignored or fired when they expressed the most standard of opinions.

But this isn’t surprising when one party is categorically opposed to any kind of learning, or development of new information that may contradict their beliefs and opinions. When the primary value of those in government and the political parties is obedience and competitive advantage vs innovation and great products, the results will be disasters where learning and adaption and innovation are key to success.

Thank you. That is the clearest explanation I’ve read of the debacle. Worse, it’s not clear if it can be fixed.

I think we need a re-think on how we pay for and deliver healthcare. Obamacare as it was conceived might even be a reasonable first approximation. But when you have frustrated users who will not be able to complete enrollment according to an arbitrary government schedule, whose sensitive information seems infinitely accessible, and costs to even think about fixing it spiraling out of control, then it is time to learn something and move on.

Thank you Clay for the context and analysis. In my view as a former federal CTO and constant tech watcher your piece here is one of the best written on this topic.

I would add three short thoughts:

– In the early days of software engineering, DoD recognized many of the issues you articulate and invested heavily in mitigating issues and some of that investment was really beneficial to the emerging discipline of software engineering. The most notable example was the funding of the Carnegie Mellon Software Engineering Institute. The processes they developed and continue to coordinate, including the CMMI, are of critical importance. I have not seen any indication that any government technologists involved in managing this evolution have had any training in these approaches. I know your analysis is pointing higher up the chain than the engineers, but still this is an important point. Good leaders and managers and even political appointees should expect disciplined process from their engineers and that means, in my opinion, knowledge of things like CMMI.

– From the time I was a youth in IT I heard the elders of my community say you should never automate a bad process. In the case of Healthcare.gov there seems to be quite a bit of that. Maybe in this case there was not an option to not automate some of these very bad processes but then good engineering could have at least reduced risk of failure around them.

– I generally am very supportive of people who say they want to improve the federal acquisition system around IT, this really needs improvement. But as you point out above a real analysis of the situation reveals issues that are much more important. Meanwhile everyone who wants to avoid blame seems to be pointing at acquisition as the problem. So for me that causes a bit of a personal cognitive dissonance. I know acquisition was not the issue here, but I want IT acquisition to be reformed and am tempted to jump on the bandwagon to blame acquisition for the problem so we can make some progress here. Your well thought out analysis helps remove the temptation from me (and hopefully others) to blame acquisition rules and puts the focus where it should be, executive leadership.

[…] This excellent blog post brought that old gag to mind. Anyone with experience bidding government IT projects knows exactly why ObamaCare is collapsing. Not just the website, which is a bit player in this story. The whole thing is collapsing. This is a good insight: […]

[…] SHIRKY: Tech planning and Healthcare.gov. "The management question, when trying anything new, is “When does reality trump planning?” For the officials overseeing Healthcare.gov, the preferred answer was “Never.” Every time there was a chance to create some sort of public experimentation, or even just some clarity about its methods and goals, the imperative was to deny the opposition anything to criticize. At the time, this probably seemed like a way of avoiding early failures. But the project’s managers weren’t avoiding those failures. They were saving them up. The actual site is worse—far worse—for not having early and aggressive testing." Clay Shirky on his blog. […]

“Talking to the people who understood the technology became demeaning, something to be avoided.” I really liked this observation. I’ve recently read Thorstein Veblen’s “Theory of the Liesure Class.” He explains this phenomenon. It’s very deeply rooted in human nature — some people believe “nobility” is demonstrated by “exploits.” That is, you get your income through conquest and pillaging. Over the course of history, this has become less overtly violent and more stylized. The “nobles” are restricted to the military, government, and religion. Any other means of making a living are demeaning, polluting. Only the baseborn do actual work.

In our culture, the MOTU are the “nobles.” Executives of large corporations are the “nobles.” It is not only unpleasant to have to communicate with the underlings, it makes you dirty.

I think there’s also something from Adam Smith here (which the conservatives who love to talk about “the invisible hand” never want you to read): “The pride of man makes him love to domineer, and nothing mortifies him so much as to be obliged to condescend to persuade his inferiors. Wherever the law allows it, and the nature of the work can afford it, therefore, he will generally prefer the service of slaves to that of freemen.” Wealth of Nations, Book III, Chapter 2, Paragraph 10. I think this has been becoming more and more visible over the last thirty years.

To say the performance of CMS in managing a huge software project is unique to the government is just flat out incorrect.

Large and complex software systems usually fail in some of their functionality in their initial release and require subsequent bug fix releases. Examples abound. How many bug fix releases are provided by Microsoft on many of their products from their server technology to development tools to Word? Lots. Sometimes monthly bug fixes to server-based products that affect large corporate IT shops in their management of risk.

Corporate executives often down to IT Managers have no experience whatsoever with technology as it is currently evolving, which is moving at higher and faster rates of change. Corporate executives are especially risk aversive and will wait for others to make changes in any technology field. And the Federal Government IT environ is perhaps even worse in these regards.

Developing software is hard. Developing huge software systems as large is healthcare.gov is absurdly difficult. Managing multiple vendors delivering different products with different API’s is hairy. The multiplicities of testing such a project so many different testing scenarios needed to be started very early on. I fault the contractors in that regards. The reason you outsource is leverage the experience in building complex systems to pro’s. If they fail, don’t necessarily blame CMS. I would start with the contractors and work up from there.

All the same issues apply. And what Fran says is also true.. most of those who know how to build the backends and front ends for billions of users want to work at Snapchat, not on the ACA project, with all the rules around procurement and rules upon rules (many for very good reason, but it’s bureaucracy which many of us got into tech to avoid.. and just code our way out of..).

Route around what doesn’t work goes the ethos.. in this case, the routing around means not choosing the work, and working somewhere with awesome stock options and chance for the moon and bragging rights over everyone else.

These are all good points and overall make a good argument, but to my ear it rings hollow. I’ve been writing software professionally for nearly three decades and have experience at many kinds of organizations, from small company IT to startups to Fortune 100 government contractors. What I’ve learned is that really good technologists generally don’t want to work for the government, nor, to a lesser extent, for big government contractors. There’s no glory there and who wants the hassle when the demand for good people is so high? Would you turn down a job at a company like SnapChat to take a position where you have to work two years to move up one civil service engineering level, assuming you can jump through all the hoops needed in order to get a government job?

Assume you are a hot-shot developer in your late 20s who has the choice between a startup/well regarded technology company in Silicon Valley, or a position near DC with a large government contractor. Where would you go? I’d find it hard to think of positive reasons to go to DC. Big projects for the government usually fail or they drag on years past their delivery date. The crazy congressional budget games mean that your work could be canceled at any moment, and you’ll be re-purposed to some obscure silo project which may be even worse and laden with useless experience that you can’t point to as an advantage to a new employer.

Working on a DARPA project at a university startup is cool, but doing web forms for the government is not. Top tier talent wants something exciting with great rewards and regard from their peers. Government work just isn’t drawing them in. Is the leadership a problem? Absolutely, but the problem is deeper than that, I think. And given the current trends I imagine it is going to get worse.

“Any personal opinion you had given really doesn’t mean anything.” This is the key principle behind making anything work well — from writing an essay to building a bridge to creating a website. If it doesn’t work, throw out your preconceptions and reconceive.