Denial – Software Quality and the C-I-A of Security

Q: How many psychiatrists does it take to change a lightbulb?
A: Only one, but the lightbulb has to really want to change.

The last few decades have shown that developing software is hard and costly. Repeated surveys highlight overruns of 75% to 100%, cancellations and unsatisfactory results. These figures are well known, and haven’t changed, indicating either we don’t know how to “fix it” or aren’t willing to change. Since there are some industry segments and individual organizations that do seem to be able to deliver on-time, on-budget, there must be a method that works.

There is. But it seems the software development world, COTS and “in-house”, seems unwilling to change. Large firms like IBM and various government sponsored projects have uncovered methodologies that do work. Its not a secret, we don’t need Mulder and Scully to uncover it. There have been many books published about it. Some firms take notice and benefit, but on the whole the industry is beset by denial.

There is not one simple root cause nor one ‘silver bullet’ solution. The problem is rooted, as Steve McConnell describes in his books on professional software development, in our culture – that of the programmers, the managers and marketeers and that of of the users. It goes all the way back to our education system.

Take buffer over-runs for example. Robert Morris Snr. first described this risk in 1985; in 1988 his son released the first Internet worm which made use of this flaw. Sixteen years later it is still the main mode of remote exploit. So why do programmers still produce code that has this flaw? Why isn’t it addressed as a major blooper in all programming courses? It seems the lightbulb – er, the industry, doesn’t want to change. If we observed this behavior in an individual we would call it a major psychological problem. We’ve seen in books such as Fred Moody’s “I Sing the Body Electronic” and Alan Cooper’s book “The Inmates Are Running the Asylum” how ill disciplined software development can be, where every programmer thinks himself a “hero” and architect and managers hire “only the best”. When everyone’s ego is being played to, why should they want to change?

We all know anecdotes of marketing setting unreasonable goes and emphasizing features over quality. Scott Adams has made a good living with the Dilbert comic strip purely on the basis real stories about idiotic management. Its all part of the culture.

So what IS happening?

No-one denies that the software development process is troubled and expensive, but an obsession with a silver bullet, be it ADA, JAVA or UML, is denying the real problem. Its looking for a “fix” rather than a systematic cure. The response to the expense is to find cheaper ways to do coding without really changing the problem, for example moving things off-shore.

Although we know that fixing problems later rather than earlier is exponentially more expensive we’ve become obsessed with patching and patch management and focusing on how to reduce the cost of that rather than eliminate the need for it.

This is all part of the “denial”, refusing to make real change.

How is this relevant to security?

In many ways. The business case for better software engineering practices has always been there. Better up front planning and formal methods of project management deliver quality product for less. This has been proven in many other areas; software and IT is no exception.

Security isn’t just about keeping the hackers, Trojans and viruses out. The C-I-A principle applies to the software development, deployment and support cycle as well. It begins with the Risk Management process of the project charter, because the loss from a failed development project may easily exceed the clean-up costs of a hacker break-in – not just the over-run costs but the lost opportunity costs and the additional support costs. The failed solutions are also relevant to security – moving development and support off-shore with all the problems and risks that involves. Ultimately, we, as a society and as individuals end up paying the price, a price that is more and more a security issue – privacy & confidentiality, integrity of infrastructure, and availability of access to resources.

So what can be done about it?

Like a psychiatric patient, the software industry wants things fixed but it doesn’t actually want to change things – everyone is looking for a “silver bullet”. Like a psychiatric patient, the cure will involve making real changes which will take real effort, real commitment and time.

What do we need to do?

First, Change the Culture
The present culture is unhealthy. Get rid of the artificial polarization between “suits” and “nerds”, the spiral of features vs little regard for users’ real needs and for “quality”; the “first to market” attitude. Cultures that develop avionics and military systems that don’t have these characteristics seem to be able to deliver better – their impediment is the regulatory demand for excessive paperwork.Software is not an engineering discipline, even though it is as life-critical and business-critical as other aspects of life where engineering as a profession requiring accreditation and certification is mandated such as the building of electrical devices or repairing automobiles. Anyone can cut code and call himself a programmer. One legacy of the Dot Com Boom is a culture characterized by lack of discipline and structure, a permanent dress-down day and other self indulgences.

Next, Manage Complexity.
Code is always the end product, but it should be at the end of the process, along with testing. The architecture, the planning, the decisions, the “divide and conquer” are becoming more significant as we deal with larger, more complex systems. We have much better tools for documenting and communicating now than when Fred Brooks built OS/360 in the mid 1960s, but the lessons he documented in his book “The Mythical Man-Month” are still relevant.

Then, Manage Risk
Risk Management isn’t just for system and network administrators, it is a key part of Project Management. The Software Project Manager’s Network web site devotes most of it tools and much of its space to discussing this. Project Management is a profession just like information security, and has its own body of knowledge and accepted good practices. Many shops I’ve had to deal with as a consultant and auditor are completely ignorant of this and lack and formal project management function. Using Microsoft Project is not an acceptable alternative.

Make it relevant.
The Dot Com Boom busted when people realized a lot of it was irrelevant – toys and indulgences, but even now we find software laden with features that aren’t going to be used. And we know that the likelihood of a “bug” goes up rapidly with the size of the code and its complexity. Do you really need an embedded programming language in your word processors just to do front-office tasks like writing letters and memos? Is the fault with the end users or with the marketing process that generates a “want” rather than a “need”?

Don’t “fake it till you make it”.
Merely emulating the form of such quality processes as those used by NASA or IBM’s old Federal Services Division, playing around with UML or O-O techniques, isn’t a solution either. It may generate a lot of paperwork and result in a lot of meetings, but mimicking the form of the process without understanding its substance does not lead to effective change. The documentation and the meetings are not responsible for success. All the undue bureaucratic palaver doesn’t lead to success. Read you daily Dilbert strip.Neither does the other extreme of being unstructured, having stock options, extensive overtime and fridges full of soft drinks like Microsoft. What is necessary is understanding the processes for repeatedly producing reliable, quality software and applying them with realistic self appraisal and feedback. Its back to the psychiatrist’s couch for the lightbulb.

Will it happen?

If the last couple of decades are anything to judge by, no. But then those years have also seen a radical change in the industry, with growth far exceeding supply. We are now past the DotComBust, and saner economic forces may prevail. It seems that off-shore development and support, while being less expensive for manpower, requires a lot more management, communication and control, and may flounder on issues of security, privacy and governance. The other hopeful driver is legislative – the demand for better business controls and IT governance. The same evolution has occurred in other industries such as aviation, automotive and food and drugs.