Despite the efforts of the Office of Management and Budget and two federal CIOs over the past five years to cut the cost of government information systems, federal agencies are still wasting money on redundant or overlapping IT projects. That's the finding of a Government Accountability Office report to Congress published September 12.

In a spot-check of IT spending by the three federal departments with the largest IT budgets—the Defense Department, Department of Homeland Security, and Department of Health and Human Services—the GAO found that a total of $321.25 million was spent between 2008 and 2013 on projects that duplicated other efforts within those same agencies.

What's jaw-dropping about this report is that the GAO could find that much duplication of effort in only those three agencies. Considering the entire government spends about $80 billion a year on IT, $321.25 million over six years is almost a rounding error. And only a small number of the programs in these departments—two at DHS, four at DOD, and six at HHS—were found to duplicate other programs within those agencies in some way.

That's not to say $321.25 million isn't a lot of money—it would buy one and a third F-35s, after all. But there are bigger sources of waste in government IT spending than duplication of requirements.

Booking perps and fixing teeth

The Defense Department has the largest overall IT budget of any federal department—its entire IT investment for 2013 is estimated at $31.4 billion. Of the nearly 3000 IT "investments" under the DOD's purview, the GAO found two areas with duplication of effort and just over $30 million during the six-year period—in health care tracking and dental management. The DOD's CIO had already identified those systems as redundant, and all the services have consolidated down to one of the health systems. The Navy is the last to be consolidated into a single dental system—largely because it has dentists on ships, and updating software afloat requires waiting for the ship to be available for maintenance.

Meanwhile, the military services are in the midst of major efforts to consolidate much of their IT. In its 2014 budget request, DOD identified nearly $34 billion in savings over five years, and a good chunk of that total is based on moving more heavily to shared infrastructure provided by the Defense Information Systems Agency.

The DHS, which had an IT budget in 2013 of $5.7 billion, had but one pair of duplicative programs in GAO's audit—an investment of $30 million in booking systems: the Customs and Border Patrol's E3 program and Immigration and Customs Enforcement's ICE-EAGLE. According to the GAO report, DHS officials said that the two booking systems share a common back-end database but have different front ends for "mission specific workflow processes."

HHS, which had an IT budget in 2013 of $7.4 billion, was the biggest redundancy offender of the three. The department had four enterprise information security efforts totaling $256 million over the six years and two Medicare coverage determination projects with combined spending of $2 million. The security projects—three at smaller agencies within HHS (Medicare/Medicaid, Indian Health Services, and Health Resources and Services Administration), and a fourth, much larger enterprise-wide effort—had by the time of the audit been partially consolidated already, though some functions had remained separate.

Can't tell the players without a dashboard

Together, the DOD, DHS, and HHS account for more than half of the federal government's overall IT spending, according to data from the government's IT Dashboard. The dashboard—and the PortfolioStat reviews that are conducted by the federal CIO and the Office of Management and Budget with each agency CIO annually—have been effective tools in applying pressure on agencies to kill floundering IT programs. In its review of duplicate IT programs in 2010, the GAO cited improvements in the dashboard as a source of improved management of agency IT projects.

Though the IT Dashboard no longer sports the names and photos of the department-level CIOs themselves alongside their report cards (like IT baseball cards)—as they did under former US Government CIO Vivek Kundra—IT programs are one of the most transparent parts of government agencies' operations as a result of the data.

GAO's study also missed an even bigger problem—despite a push toward shared services and the cloud, there's a huge duplication of IT investment across agencies. For example, while DHS has taken point on cyber-security for the civilian side of the government, there's no central, standardized government solution for security.

But If the GAO had done this analysis 10 years ago, the results would have induced even greater rage. So thanks for the good news, GAO.

Sean Gallagher
Sean is Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat

One should point out that the report indicate there MIGHT be duplication and some these IT programs only eppeared to be duplciates of others. I read multiple articles on this same report only one so far quoted the full statement where the word might was used.

I've seen many a fiasco get started because people at 64,000 feet high saw two similar applications and start thinking "we should combine these". Just because it sounds the same doesn't mean it is the same.

By the time you get all the competing requirements settled out, two systems that would have cost $1 million apiece end up costing $10 million combined.

The F-35 has increased over 93% beyond initial cost estimates, so a bit of duplication in IT seems par for the course.

August GAO reports the F-35 total program cost now estimated at around 400 Billion, and that's assuming things work out from now on. Vanity Fair has an excellent long-form (7-page) investigative article which casts some doubts on whether things are working out at all, let alone from now on.

The thing I find most amusing about this article is that while I enjoy the context provided by the fact that the duplicated efforts is equal to around 1 1/3 the cost of an F-35 (A, B, or C?), I actually think I'd rather they waste extra money on duplicated IT efforts rather that hideously overpriced and underwhelming hangar queen.

I've seen many a fiasco get started because people at 64,000 feet high saw two similar applications and start thinking "we should combine these". Just because it sounds the same doesn't mean it is the same.

By the time you get all the competing requirements settled out, two systems that would have cost $1 million apiece end up costing $10 million combined.

I'd rather buy a fridge and an oven, and pay for both, than have some administrator tell me I have two "redundant kitchen appliances designed to control temperature of food" and try to build a combination device that heats and cools.

And if that combination device is going to be an enterprise software project, I'd run away as fast as my legs could carry me.

As a former code monkey for such projects, duplication of effort is the *least* of potential savings. You'd be better off starting with the layers upon layers of low level bureaucracy that exists solely to serve the "process" -- a process those layers are busy making more complex and inefficient because they really have nothing else to do with their time.

Those of us who liked playing around with computers when we were at school, and so took it up as a career, tend to hold on to a profound misconception of the purpose of "IT" as some noble endeavour, which is contributing to human progress. We are sadly mistaken. It is really a excuse for those who liked playing the system, learning only the curriculum and getting their jollies from team sports when they were at school, and so took up management as a career, to build empires and hoard budgets. It is perfect for this role. Unlike services, you can convince the people with money (governments, investors) that it is a form of production, like manufacturing. But, at the end of the day, there won't be a warehouse somewhere full of physical objects you can count. Hence the proliferation of all manner of flim-flam methods of accounting and development methodologies, in public-sector IT, and patents in the private sector, to pretend that the ~50% of IT projects that are pointless have provided "value" of some kind.

Only 12 out of 590 investments indicated potential duplication. I'd say that number is quite low but the one program that is sticking out from the report is HHS - Secure One costing 195.59 million which is making up the majority of the 321.25 million figure.

I'm not sure on GAO's math here because even if you remove duplicate programs you will not recover the costs of all of the programs to reach the 321.25 million figure. At best you going to save money by cutting out the cheaper programs and save that figure.

For argument's sake let's assume you combine the programs and only spend the maximum cost of the highest program. (For example, If program A costs 30 million and program B costs 10 million, you would save 10 million from de-duplication and spend 30 million).

Total savings if you remove all the duplicate programs per agency from report = 70.87 million

Now this is a far cry from the 321.25 million figure that GAO puts out. Unless the agencies are just going to cancel all these programs you not going to get to the 321.25 million figure. Anyone know how GAO is getting their 321.25 number? Am I missing something here?

I being an ex-IT government contractor find this not shocking at all. However that is not to say it isn't i'm just numb to it after years of exposure. Here's a few facts to fill in the gaps for those who haven't had to deal with this as their FT job. 1. The different pieces of the government do not play well with others and they do not like to share. Once the accountants find out they start every fire possible on who is going to pay for what and the never see "the big picture". It is borderline a pissing match "well we paid for it why should you get to use it?" and "I'm not paying my staff to do your job" and other like comments. 2. This is also the same government that basically punishes people by staying under their annual budget. We would get threatened to have less money the next fiscal year because we had money left over at the end.3. The people who get the final say, and come up with these projects, 99% of them have no IT knowledge outside of Microsoft Office applications. They see it on TV and say make it happen. I wish I were making this up. It gets better if they don't like what you have to say i.e. "This is a waste of money" "We already have something that does that but without this one feature" etc You get told essentially you are a contractor and are replaceable. 4. I'm going to stop here because i could go on for days and even provide a novella on each situation edit: for grammar

Now seriously, this is to be expected of any large organization. I've seen it in many companies, where divisions dismiss corporate projects as having pie-in-the-sky requirements or need the projects done before the large effort can be completed. So smaller, redundant projects sprout all over the place.

"You don't think they spend $20,000 on a hammer and $30,000 on toilet seat do you?"

That line amuses me because I know where each of those comes from. The gold plated hammer was actually a tactical repair toolkit designed to be used outside the sonically shielded envelope of attack submarines. Someone was tasked with making a toolkit that wouldn't make any sound when used in that role, they did so and then two toolkits were ordered for each submarine in the fleet and the research costs were spread over the number of units ordered resulting in a very high per piece cost. The kit was still a tiny, tiny fraction of the cost of losing a submarine because a tool rattled around at the wrong moment. The toilet seat was actually an entire lavatory, the planes had been flying for 50 years or so and their service lives were extended after wing inspections showed usable life left so they underwent an overhaul. Part of that overhaul was a new lavatory, which required a custom very large mold to be produced to make the single piece toilet enclosure because the original molds had obviously not been hanging around for 50 years, and again that mold was used to create a lavatory for a handful of aircraft again leading to a high per-unit cost and of course costing significantly less than developing a new airframe (the seat by the way was a cheap off the shelf unit). Both were cases where money had been prudently spent to offset much larger costs but because they had high per-unit costs some politician decided to roll them out as a soundbite. There is PLENTY of waste within the federal government, but these two "well known" examples weren't.

I have been working in the DoD for over 26 years and have watched the “creation” of IT as we know it today take place. If you are doing the math, then you will realize that my time in the military started before the explosion of civilian internet. Back in those days, the internet (a lot of us still called it DARPAnet back then, even though that was the wrong term by the late ‘80s) was mainly used to transfer files between main nodes in the defense network and also in our university/research systems. A huge amount of what became the government IT structure was truly discovery learning. This is very important to remember when assessing why things may not work so well today, from a duplicative effort standpoint. The bottom line is that many of the technologies we take for granted in the civilian side were created by and for the military just to solve problems no one had known about before.

Another key thing to remember here is the scope of the issue. There is literally no other organization on the planet that has to deal with the range of IT issues the DoD has to deal with. Besides being the largest employer in the world (by quite a lot – DoD currently employs about 3.2 million people – the only civilian company in the same league is Walmart with around 2 million employees), the DoD also has to be able to communicate on every continent, across (and under) every body of water, in the sky and in space. And, they are expected to handle that communication with 100% uptime – no one in Congress wants to hear about our critical systems going down – ever. Combine that with the fact that these systems literally have to survive combat conditions, and you begin to realize how difficult it is to form an IT “structure” for DoD that is consolidated. Should the dental IT structures be consolidated? I guess so. Should you consolidate the IT structures for controlling weapons systems? Definitely not. I am all for saving the tax payers money, but, as one poster noted above, you need to be very, very careful when you take a look at the largest organization in the world and start nit-picking the IT systems. On the other hand, we absolutely need the GAO to do this. If we don’t have someone come in from the outside once in a while and tell us where we are messed up, we can’t improve.

Unfortunately, the consolidation efforts have taken their toll. Usability is at an all-time low in military networks. Despite numerous articles about DoD adopting new technology like Blackberries, iPhones, iPads, etc, the reality is that those devices are used on maybe 1-2% of DoD networks and, most likely, you just won’t be able integrate any modern devices into DoD use. That is not to say the DoD doesn’t use smartphones and tablets, it just means that those devices are pretty much standalone or sandboxed to the point where your $1000 iPhone (the DoD doesn’t usually buy “locked” phones) is pretty much… just a phone. If you can get an iPhone, it will be locked down to the point where you cannot add music or photos or videos, no “unauthorized” apps (forget apps that cost money), no connection to the iStore, and, of course, all texts and emails will be monitored, logged and recorded (which is not that big a deal – that is the same for all DoD IT services). BYOD on military networks is not an option, for fairly obvious reasons.

Worse, though, use of the physical IT networks has gotten ludicrous. Right now, if we want to move ANY device from one physical network port to another network port (think, I want to move this printer across the room), it requires a trouble ticket and (you can’t make this up) the ticket goes to another post 300 miles away where the IT services for our region have been consolidated. The contractors who run the admin services there will look at your request in a timely manner (think 2-3 working days) and then either fly someone out to our location or send a contractor locally to come in to the room, move the device to another port, call them to authorize the port change, and then leave. This was done in the name of “consolidating services” to “save money” and “improve security”. And I am not on some small, out of the way post – I currently work on one of the largest military bases in the country – we have over 3000 personnel in my unit alone and we are the smallest unit on our post. So, now, it costs the government at least $1000 for me to move a printer across the room. Brilliant.

I've seen many a fiasco get started because people at 64,000 feet high saw two similar applications and start thinking "we should combine these". Just because it sounds the same doesn't mean it is the same.

By the time you get all the competing requirements settled out, two systems that would have cost $1 million apiece end up costing $10 million combined.

Sounds like a parody of the F35 program itself. Fit all the requirements into one overarching design.

How do we know these duplicated IT projects could be successfully combined? Does anyone in the government fully understand all these systems, and know that they are compatible?

Sounds like yet another boondoggle of spending for IT consulting: the de-duped Whatever 2.0 would cost billions and never be completed, while the legacy systems would have to be maintained and enhanced. Even if the system was completed, there would be a total scorched-earth turf war over who got to run the new system, which could get ugly. (That's all "cyber command" ever was - a huge turf war to create a new 4-star general and add a layer of management on top of all the layers that were already there.)

I've seen many a fiasco get started because people at 64,000 feet high saw two similar applications and start thinking "we should combine these". Just because it sounds the same doesn't mean it is the same.

By the time you get all the competing requirements settled out, two systems that would have cost $1 million apiece end up costing $10 million combined.

I'd rather buy a fridge and an oven, and pay for both, than have some administrator tell me I have two "redundant kitchen appliances designed to control temperature of food" and try to build a combination device that heats and cools.

And if that combination device is going to be an enterprise software project, I'd run away as fast as my legs could carry me.

Hmm I wonder if I could market a combination oven+blast chiller to any government agencies.

I've seen many a fiasco get started because people at 64,000 feet high saw two similar applications and start thinking "we should combine these". Just because it sounds the same doesn't mean it is the same.

By the time you get all the competing requirements settled out, two systems that would have cost $1 million apiece end up costing $10 million combined.

Yeah, i once saw a project for consolidating tech purchasing for NOAA. The NWS guy stood up at the meeting and said "we need to buy a $150 million weather satellite. Anybody else got that?" The meeting fell apart shortly after.

Worse, though, use of the physical IT networks has gotten ludicrous. Right now, if we want to move ANY device from one physical network port to another network port (think, I want to move this printer across the room), it requires a trouble ticket and (you can’t make this up) the ticket goes to another post 300 miles away where the IT services for our region have been consolidated. The contractors who run the admin services there will look at your request in a timely manner (think 2-3 working days) and then either fly someone out to our location or send a contractor locally to come in to the room, move the device to another port, call them to authorize the port change, and then leave. This was done in the name of “consolidating services” to “save money” and “improve security”. And I am not on some small, out of the way post – I currently work on one of the largest military bases in the country – we have over 3000 personnel in my unit alone and we are the smallest unit on our post. So, now, it costs the government at least $1000 for me to move a printer across the room. Brilliant.

I work for a government in Canada that did the exact same thing, in the name of "efficiency" and "saving money". Before they started, we'd get a call about an application support issue (which was usually someone walking over to our office and asking), and resolve it typically in a few minutes.

Now people aren't allowed to talk to us, they have to send things by email (or phone) to another office on the other side of the city. It takes those people currently 2 days (and counting) to look at the ticket, and then send it back to us because it's an applications call. At that point we can actually do something about it.

Smart users have learned to cheat the system and CC us while sending the email in, and we inevitably have the problem fixed before the new "efficient" office has even looked at it, let alone realized its something they can't do and send it to us. Less fortunate users just get to wait a few days in the name of "efficiency".

The whole thing is a total joke. This is what happens when politicians and career managers start interfering with the people who do actual work.