Software Patches Eat Government IT's Lunch

The software industry's publish-now, update-later approach exacts a huge toll on government IT leaders like Robert Jack, CIO of the U.S. Marine Corps.

Netscape co-founder and prominent tech investor Marc Andreessen famously noted that "software is eating the world." Unfortunately, it's also eating the lunch of most enterprises, including federal agencies.

For all the talk about wasteful government IT spending, little is said about the costs agencies pay to patch buggy software, a consequence of the industry's predisposition to release their wares now and fix them later. For Robert Jack, CIO of the U.S. Marine Corps, those costs aren't incidental.

"We have roughly 300,000 people, of which a third have day-to-day access to the enterprise network," Jack said at a recent forum on cybersecurity. "I have to defend the network at the desktop or end-user device. I have over 450 registered systems that are regressed to 10 significant versions. When we get a patch from a vendor, we have to go out and test that against all that."

He continued, "Think about the labor hours where I have to touch [and administer patches on] all those devices. And that's just for one patch." Every week, dozens of new vulnerabilities are catalogued by US-CERT, the government's computer emergency readiness team, offering a glimpse of the headaches Jack and CIOs like him face.

Speaking to the software industry at large, Jack said bluntly, "You're killing me."

[ As cloud and mobile proliferates, federal IT leaders should take more data-centric approach to security. Read Secure Data, Not Devices. ]

The staggering cost of software bugs is hard to nail down. However, a Cambridge University study released earlier this year estimated that finding and fixing coding problems costs software makers and the global economy $312 billion a year. That doesn't reflect what customers must also spend to patch and maintain that software on their networks.

The problem, however, goes well beyond the mechanics of software and system maintenance. It also goes to the heart of network security and the growing risks associated with unknown software vulnerabilities, Jack said. Having spent 40 years in charge of command, control, communications, computers and cyber operations for the Air Force, the Defense Department and now the Marine Corps, Jack knows the problems as well as anyone.

Software by its nature is a work in progress. While vendors can't anticipate every problem, some of which are spawned when software interacts with other software on a network, vendors are making too many calculated compromises in order to ram their products and updates into production, Jack said. But worse, they're exposing organizations and their executives to growing liabilities if something goes wrong.

Jack pointed to recent reports, which he didn't specify, indicating that 25% of hospital operating room liability lawsuits are now tied to software coding problems. Lawsuits based on software failures are also becoming a big concern for the auto industry, he said, and the issue has prompted high-level discussions within the Defense Department.

It's only a matter of time before the high-profile enterprises become targets for liability lawyers looking to exploit software mishaps, Jack warned, adding that those in positions of authority ought to consider "looking for some big-time insurance."

There's an interesting story and commentary on the importance of, and the balance between, software developers who crank out code and product managers who are responsible for the customer experience. See "Why your software development process is broken" at: http://www.informationweek.com...

This misses the point of what the Marine Corps and other large scale organizations are facing. First, the Marine Corps can't just software A and switch to software B because software A is costing more to update/fix. Second, we're talking about high costs because of the large number of users and devices that exist. The Marine Corps works with the US Navy. Together they have more than 800,000 users on the network. Simple bug fixes and version updates may not be a big deal for an organization of 1000 workers. It's a much different and more expensive matter for large organizations.

I find it interesting when people complain that they are losing money as a result of applying software fixes.

The simple answer for organizations that are losing money applying software fixes is for the organizations to discontinue using the software. Then they would they recover those "losses" -- right?

But, of course, they will nearly always "lose" even more money if they discontinue using the software. Adoption of nearly all business software products results in cost reduction of actions that can already be performed. There is often very quick adoption of new products that provide significant cost reductions because the people that purchase the products can get their money back in less than a year and then go on to get significant cost reductions year over year. However, people still gripe because they don't save even more because of product defects.

For example, Larry Ellison of Oracle said that Oracle saved a Billion dollars a year when it adopted Internet technology products for HR and other internal processes. -- I believe it. Now Oracle could have saved even more that a Billion dollars if there were no product defects. Lets suppose Oracle saved $1,000,000,000 but if there were no product defects they would have saved $1,010,000,000. Now Larry could say that Oracle lost $10,000,000 by adoption of Internet technology products --- and it would be true using the logic of the per "Software Patches Eat Government IT's Lunch" story.

Software, unlike almost all other products, is peculiar because customers pay to get fixes to defects in delivered products. Why is this? Why do customers purchase such products and then have to pay to have design defects repaired. Lets consider to companies building the same type of software products.

Lets say Company A gets the product out in 2010 where it is purchased by the Bank of America. Lets stay that Company A's product has many software defects costing BoA $10,000,000 during the next two years. Lets say Bank of America, saves $500,000,000 per year because of the adoption.

Lets say Company B gets the product out in 2012 at the same price as Company A and because of the extra two years of testing, software defects cost $0 per year and the Bank of America starts saving $500,000,000 per year at that point. What's the better deal for the Bank of America?

Did Bank of America lose $10,000,000 by adopting Company A's product or did it lose $990,000,000 by waiting to adopt Company B's bug free product?

Of course there are tradeoffs here. If the products have too many bugs then they become unfit for use. But competition typically sorts this out.

Anyway, the bottom line here is that when you hear about organizations losing money because of the cost of applying fixes for software defects, its probably not the case that the organization lost money by adopting the use of that software. They just want to save even more -- and that is good. However, sometimes it's good to remind people to consider all the costs and benefits.

Government gets a lot criticism for its bureaucratic ways, but it does offer a template, courtesy of the National Institute of Standards and Technology, for how software code can be certified to meet certain operating standards. Enterprises would be better served if there were standards software developers could agree to that, as you say, using an automated process, would give software the equivalent of a clean bill of health. That won't address the need for updates, as you say, but it would give customers more confidence in what they're buying. And that hopefully would led to marketplace choices that would reward the more responsible developers.

A bug is anything that negatively impacts user experience. That means it does not need to be a coding error. The code can be all well, but if using the application is so complicated that it will lead to many mistakes the software is buggy and needs fixing.A big problem these days are indeed the ridiculously short times to market as well as the 'agile' methods that reward not committing to anything, no planning, no documentation, and rapid releases of something. I think we all would better served by spending a few months up front to design applications and make up our minds as to what we really want, then build that. Sure, flexibility is always needed as some aspects come up while doing the development work, but it just doesn't pan out when doing everything on the fly.

Software patches are often updates, not bugs in the sense of poor code needing corrections, to bring an application into line with updates elsewhere in itself or to keep it compatible with a wider set of software. We need to get to the point where software is built in such a standard fashion that updating it is a standard, and automated, process. This is where cloud vendors start to draw away in terms of efficiency from the one-of-everything enterprise data center.

Published: 2015-03-31The build_index_from_tree function in index.py in Dulwich before 0.9.9 allows remote attackers to execute arbitrary code via a commit with a directory path starting with .git/, which is not properly handled when checking out a working tree.