NHS IT system condemned

In a jaw-dropping condemnation of the NHS National Programme for IT, the National Audit Office has exposed a white elephant in the final stages of collapse.

In what read as a final pronouncement, the NAO reported that after nine years and £2.7bn, the NHS has failed to deliver its primary aim of an electronic care record for everyone in the country.

The situation looks so dire the system might continue leeching money from the NHS for another decade if the whole scheme and all its software is not seized by state liquidators.

The NAO was confident DoH had saved the day. But it had done so by turning the systems architecture on its head. The National Programme had been all but abandoned. The NAO all but called time on it once and for all.

"There is a compelling case for the recently announced Whitehall-wide review to re-evaluate the business case for the Programme to determine what should happen now to safeguard against further loss to public value," said the NAO's Update report on the NHS IT Programme's care records systems, published today.

Yet the NAO couldn't simply move on. It was being held back by its multi-billion pound contractual obligations to failed suppliers BT and CSC.

Suppliers

Ian Watmore, the Cabinet Office head of government IT, told the Public Accounts Committee Monday the oligopoly of suppliers was to blame for the mess in government ICT. But the government couldn't simply ditch their contracts when it was so hard to prove where the bucked stopped in a court of law.

So, the NAO reported, DoH had commitment to honour its existing contracts for the failed delivery with BT and CSC. The total, with implementation and delivery fees, would cost another £4.3bn to 2016.

Might suppliers do the honourable thing now and resign, apologize, and hand what software they had over to the public?

DoH had abandoned the original scheme's aim of storing all NHS information in a single system spanning thousands of separate NHS organisations. It had effectively scrapped the system. It had opted instead for one set of rules so that disparate systems could share what information they had. This proposal was on the drawing board when the DoH first embarked on the scheme in 2003.

Oh dear

NAO had a scrappy report card for BT and CSC, who had already been paid billions but failed to deliver systems across the country.

DoH had spent £6.4bn on the entire programme to date. But masses of hospitals, ambulance crews, mental health units, community care teams and doctors surgeries were still waiting for working computer systems.

The Department was trying to handle the system requirements in a more flexible manner, in keeping with the agile mode in systems development. But NAO was pressing it to stick to the original contracted specifications as a measure of success. The new mode was "overly positive" when the cogs of the system were stuck in the mire.

DoH had permitted suppliers to deliver fewer systems without getting a commensurate reduction in price. Costs appeared still to be going up. There was no end of it and matters were only becoming more complicated. The state of the core patient system was enough to induce a mortal gag.

"Although some care records systems are in place, progress against plans has fallen far below expectations and the Department has not delivered care records systems across the NHS, or with anywhere near the completeness of functionality that will enable it to achieve the original aspirations of the Programme," said the NAO report.

Suppliers

"The Department has also significantly reduced the scope of the Programme without a proportionate reduction in costs, and is in negotiations to reduce it further still. So we are seeing a steady reduction in value delivered not matched by a reduction in costs.

"On this basis we conclude that the £2.7 billion spent on care records systems so far does not represent value for money, and we do not find grounds for confidence that the remaining planned spend of £4.3 billion will be different," said the NAO.

Deadlines had been repeatedly missed. When systems had gone in take-up of care records was poor. DoH had compromised its original requirements.

The report painted a most unpleasant picture of CSC, which had delivered care records systems to just four out of 97 hospitals in the North, Midlands and East of England.

The NAO said CSC would likely fail to deliver the rest before its contract runs out in 2016. DoH had been in dispute over its contract with CSC for 18 months, trying to claw back some of the £5bn it had promised the supplier.

DoH had meanwhile been forced into technical modifications of the one-size-fits-all software systems being delivered by the suppliers because they did not fit the slots they were being put in. Costs of these modifications were uncertain, but were £220m to date.

Delivery costs were going up, funding was going down. Additional costs were being incurred by trying to make health systems from outside the Programme compatible with those inferior systems delivered by BT and CSC.

Little was certain. BT and CSC were in such a slough they could not even agree with the Department of Health in numbers submitted to government auditors. NAO had been unable to clarify the discrepancies.

And so on...

On top of that, the NHS was facing reorganisation under the coalition government. The NAO report gave no impression these lumbering suppliers would need anything less than another 15 years after they fulfilled their current obligations to redesign the system to work in the new NHS organisational structure.

"Of the 4,715 NHS organisations in England now expected to receive a new system under the Programme, 3,197 are still outstanding. The current CSC contract alone requires delivery of 3,023 GP systems and over 160 deliveries of Lorenzo by July 2016," it said.

The ongoing saga is an embarrassment for the UK, for tax payers, for the government. Any embarrassment suffered by the scheme's suppliers has been sated by the billions paid to their shareholders, the billions more they are contracted to receive, and the hundreds of millions more still to be allotted in contracts not yet let.

It is an insult to the Cabinet Office Open Source and Software Re-use Action Plan launched in 2009 by then minister Tom Watson and it is an insult to the coalition government's repetition of its aims, both before and after the 2010 general election that brought it to power.

Were Watson and now Maude's plans to get control of public computer software ever feasible when the commercial world has such a death grip on public ICT?

Watson's plans for software re-use dared to imagine a time when public software was public property. Any work suppliers were paid to do would contribute to the public good. And that good would be stored in the form of software on which any agency, or any one from the the wealth of local health systems providers who flourished before this débâcle, could contribute improvements.

The National Programme is in such a state, the government has nothing to lose if it is serious about its Open Source Software and Re-use policy: it could become the first great public open source system.

13 Comments

Q. Are Agile methods better than "waterfall" methods for very large projects?
Q. Can a project be too big to manage effectively?
Q. Were PRINCE2 methods used and adhered to or did political interference move the goalposts?
Q. Did the management understand the requirements?
Q. Why were sub-contractors allowed to walk away without penality (seem to recall Accenture did this)

So add the 1 billion wasted in the education service on curriculum on-line, 25 million on the scrapped KS3 ICT tests and we have to ask how much money government has to lose before it wakes up to the fact that the commercial proprietary licensing model simply doesn't work for large scale public sector projects. Start in schools and make understanding the pitfalls of out-dated intellectual property concepts in a massively connected society the focus for ICT education.

I worked for BT Syntegra on this doomed project for nearly a year and by the end I felt dirty. Had I been able to afford to I would have returned the money that I had stolen from the public purse in return for what was deemed to be "work". The whole organisation was a disgrace from top to bottom and there are many stories to tell. The one that sticks in my mind was the 5 days, no less, that it took to organise the execution of 1 very simple query against a database. The query itself took 30 seconds to write and 1 second to run. On delivering the results, a flurry of emails followed to and from senior managers to congratulate me on my efforts. The delay was caused by the contractual stand-off that existed between various suppliers, each trying to safeguard it's greedy share of the pie. Of course, none of them was remotely concerned with doing the job and this attitude was endemic.

The allegations made by Dave Noble in the last comment are of the sort that one hears and are entirely believable in the context. Not that I would have any cause to question your integrity, Dave. I would have no position one way or the other. But I would be grateful if BT Syntegra and others took the opportunity to reply.

I wish I could be surprised by this report, but it seems sadly inevitable. How a national single system could possibly ever be built when it is partitioned by geography with overlap in both scope across different companies is beyond me.

The way forward has to be through the implementation of smaller systems that link together in local geographies and actually follow the needs of patients and clinicians, not a top down ordinance that is without precedent anywhere in the world. The US model of NHIN is a far more realistic and better organised system that would work well here - simply by reinforcing groups of collaborating organisations that look after patient pathways. Ideal for GP commissioning and other local services that need to join together for the continuous care of the patient.

Dont get me wrong I completely agree with Mr Bacon and his comments about the state of play. However, we as tax payers should also demand an explanation on how it is we got into this sorry mess. Why so much money has been spent with so little in return. I have spent this morning reading the NAO report which has content which is basically unbelievable. How is it possible for a government controlled (supposedly) body sign up to a contract which allows a supplier to deliver a product with no more than 700 bugs / errors!! Would you buy a care or a ready meal from such a supplier? I think not. Yet CfH is more than happy to spend our tax £ on such product and tell us its great and represents value for money. Who are the people that made / can make such decisions, to whom are they actually accountable and what consequences do they face when they get it so desperately wrong.
Its our health service, its our money and I feel its our collective responsibility to make sure we never feather the beds of large corporation shareholders in such a way again. The oversight process clear failed.
I for one want to know why and how. From its inception NPfIT attracted much criticism from within the Health Service, IT Industry, IT Academia and the press (sterling work by Mr Collins amongst a few others) and yet nobody listened nor acted. I suspect there were many gagging orders flying around to shield the likes of Mr Blair and co at the time. Why were the contracts kept private and away from scrutiny, did they adhere to the terms in SYSCON which all NHS supply should. As an NHS supplier I would welcome such scrutiny I have nothing to hide in relation to my pricing, services, bug lists, development strategies. It should all be out their and visible just like the ingredients on a tin of chicken soup. If that were the case then we would have had an upfront conversation with the likes of BT regarding acquisition of rights to supply RiO for £47 Million yet it charging the NHS way in excess of £200 million to get it into a few Trusts at £9million a site.
Somebody has to be held accountable so that this type of fiasco can not happen again. I addition the IT suppliers to the health service should also take a long hard look at themselves and their avaricious behaviour over the past years.

Dave Noble: "The one that sticks in my mind was the 5 days, no less, that it took to organise the execution of 1 very simple query against a database. The query itself took 30 seconds to write and 1 second to run..."

Can't comment on how BT Syntegra or the NHSPfIT worked, but in my experience exactly the same thing happens on in-house public sector projects, where it can take weeks to deliver a simple 10 minute task through multiple layers of bureaucracy and rigid but inefficient internal processes. When you factor in the inevitable extra restrictions (contracts, costs, SLAs, communication problems, formal request procedures etc) at the public-private sector interface, it's amazing that anything at all ever gets delivered anywhere in public sector IT.

As for open source, that is a complete red herring here. Blindly enforcing the use of OSS can mean additional costs and delays re-inventing wheels that might already be available form a low-cost but proprietary solution. And it doesn't matter what software licencing model you use if you cannot deliver anything anyway.

There are also problems with the government-mandated idea of software "re-use" in the public sector.

This is obviously a response to the history of expensive, mutually incompatible, "Big bang" silo solutions across government departments, often intended to serve much the same business needs. There are clear benefits to implementing (or simply purchasing) e.g. a standard payroll system just once, then rolling it out across different departments. Especially if ministers were actually prepared to take responsibility for telling senior civil servants to like it or lump it, instead of allowing them to strangle any hint of progress in red tape, special pleading and sheer inertia.

The problems come when trying to ensure "re-use" either for one-off specialised systems, or where the idea of "re-use" is pursued blindly to ludicrous and expensive extremes in the absence of any corresponding business requirement.

When most government IT projects struggle even to define their own requirements adequately (I have seen projects go through two entire SDLCs without ever really nailing down their own requirements), trying at the outset to address all conceivable requirements for any future "re-use" can cripple projects with grotesquely complex over-engineered solutions that still fail to deliver even basic "usability" for their primary customer, let alone the imagined cost-savings of any alleged "re-usability". This is the classic problem of premature optimisation, and it can take longer to "re-use" so-called "re-usable" code on some projects than it would take to build a customised solution from scratch.

There are certainly ways to improve software re-use in the public sector through intelligent approaches to project planning, system architecture, analysis and design, adopting existing solutions where possible, building re-usable modular components where appropriate, and creating quick and cheap bespoke solutions where necessary.

Unfortunately this kind of intelligence is conspicuously absent from most public sector projects I have ever worked on.

BT press office issued the following written statement in response to the NAO report. CSC was unavailable for comment.

BT is reviewing the National Audit Office (NAO) report. While we do not recognise some of the comments made, we will be responding fully at the Committee for Public Accounts hearing on 23 May.‪ We believe our work for the NHS is helping to deliver efficiencies and benefits. Doctors in distant locations can share accurate information more easily, electronic hospital bookings and electronic prescriptions are speeding up services.‪

On our Local Service Provider contract, BT has delivered a range of clinical information systems at 80 NHS organisations in London and the South of England serving 170,000 healthcare professionals, helping to give clinicians instant electronic access to patient information.‪ Recent changes agreed to what BT is to deliver have come about because we've been asked to deliver more complex systems at fewer locations‪, moving from centrally led to clinically led healthcare.

Our two national contacts - the Spine and N3 - continue to underpin key NHS services.‪

The Spine - the database of patient information and powerful messaging service - has seen over 30 million online hospital appointments booked.‪ It's now the second largest database in Europe, according to Gartner, and has carried 5.6 trillion messages to date.‪

N3, the secure national broadband network, is one of Europe's largest Virtual Private Networks with over 50,000 connections, connecting over a million NHS staff. According to NAO figures, it's already saved the NHS £192 million and further efficiency savings are being delivered via new transformational services such as Managed Video Conferencing and N3 Voice services.‪

These systems and services are now underpinning vital NHS services which couldn't run without them‪.

We have a political dilemma here in that using the private sector appears not to come up trumps here at all. However is it fair to put the blame on suppliers? After all they must have been invited to tender for this work and the winning tenders must have been those approved by HMG at the time.

Perhaps the expensive lesson to be learned is that at the time putting HMG IT work out to private tender was thought to lead to better results than if the work was done in-house.

Alternatively an argument could be made that this is not a failure on the part of the private sector but in the way contracts were devised by HMG at the time. Thus the Public sector failed to properly control what it had contracted the Private sector to do.

This though strays into another argument of what might have happened if the whole thing had been controlled by the Private sector.

Purely changing the software to Open Source might not achieve that much if the tendering process is not altered. One could though make an argument to state that if the collaboration and cooperation emanating from Open Source ideology could be transferred to such public sector projects then perhaps costly problems like these would be more likely to be avoided.

For example people might get involved in such projects for reasons why they do now with major Open Source projects.

I think the FOSS (Free and Open Source Software) angle should be explored here, since if the source code of what was developed was released under a Free license the buyer (UK Gov and NHS) would have a lot more control over the technology developed.

And in a case like the NHS where one needs global standards implemented from the bottom up (i.e. adopted by each NHS practice), a core technology stack that is Free would give a LOT of independence to the local NHS practices (they could accept the 'mothership' packages or develop their own). Yes their might be some fragmentation but we would probably be much better than where we are today.

Note that the requirements of delivering such technology in such Open/Free way, would force the main/code developers to have strong engineering practices (namely in the areas of application interdependencies and deployment).

But I guess the first question is: How much FOSS is already included in this project? What technologies are they using?

I have to admit that I don't know a lot of details about this project, but it would be very weird it was all 'proprietary' technology.

Also, since my specially is Application Security, is there any published information about the security reviews done to these applications? (my experience is that systems that 'struggle' to work as they were supposed to, are usually full of serious security vulnerabilities (since there is a moment where the mandate is '...just get it to work...' which usually means that 'application security' is moved even down on the priority scale))

@Dinis Cruz: "if the source code of what was developed was released under a Free license the buyer (UK Gov and NHS) would have a lot more control over the technology developed.."

Why? If you mean that by pushing an OS approach the government can ensure e.g that only GPL-compliant tools are used, why is that necessarily beneficial to the customer or taxpayer? Or do you think you can build a grand Big Bang all-encompassing NHS system based on crowd-sourcing by open source hackers in their spare time?

The NHSPfIT did not fail because of its licencing model, it failed because the software could not be delivered to do what it was supposed to do, at least not with the resources available. It was a crazy project from the start, and open source would not have changed that.

I am a keen user of FOSS products, but I do not buy into this FOSS-fundamentalism that insists there is only One True Path to software quality, or indeed to value for money.

Actually general medical practices ("Doctors' surgeries" above mostly had systems that gave reasonable service before this.

The main results for us have been a loss of status as customer, so other people are deciding what development to demand or pay for, which seems to work less well than when it was us, and a migration toward uniformity and data centres.

But only one connection per building, so distant stores are insufficiently reliable for our uses.

GPs invented most of the functioning clinical systems in the NHS, hospital doctors could no doubt devise effective automation, but don't own the computers and can't hire programmers to work with them.