Developing and Maintaining Secure and Reliable Software in the Real World

Monday, May 24, 2010

Fast or Secure Software Development - but you can't have both

There is a lot of excitement in the software development community around Lean Software Development: applying Japanese manufacturing principles to software development to reduce waste and eliminate overhead, to streamline production, to get software out to customers and get their feedback as quickly as possible.

Some people are going so far as to eliminate review gates and release overhead as waste: “iterations are muda”. The idea of Continuous Deployment takes this to the extreme: developers push software changes and fixes out immediately to production, with all the good and bad that you would expect from doing this.

“The other day we passed product release number 25,000 for WordPress.com. That means we’ve averaged about 16 product releases a day, every day for the last four and a half years!”

I am sure that he is not proud of their history of security problems however, which you can read about here, here, here, here, here and elsewhere.

And Facebook? You can read about how they use Continuous Deployment practices to push code out to production several times a day. As for their security posture, Facebook has "faced" a series of severe security and privacy problems and continues to run into them, as recently as last week.

I’ve ranted before about the risks that Continuous Deployment forces on customers. Continuous Deployment is based on the rather naïve assumption that if something is broken, if you did something wrong, you will know right away: either through your automated tests or by monitoring the state of production, errors and changes in usage patterns, or from direct customer feedback. If it doesn’t look like it’s working, you roll it back as soon as you can, before the next change is pushed out. It’s all tied to a direct feedback loop.

Of course it’s not always that simple. Security problems don’t show up like that, they show up later as successful exploits and attacks and bad press and a damaged brand and upset customers and the kind of mess that Facebook is in again. I can’t believe that the CEO of Facebook appreciates getting this kind of feedback on his company's latest security and privacy problems.

“Facebook are not fools of course. You don't build a business that engages every tenth adult on the planet without honing a pretty good sense for which way the wind is blowing. The company realizes that it is under no obligation to provide any real security controls to its users.

Maybe to be truly open and collaborative, you are obliged to make compromises on security and data integrity and confidentiality. Some of these Web 2.0 sites like Facebook are phenomenally successful, and it seems that most of their customers don’t care that much about security and privacy, and as long as you haven’t been foolish enough to use tools like Facebook to support your business in a major way, maybe that’s fine.

And I also don’t care how a startup manages to get software out the door. If Continuous Deployment helps you get software out faster to your customers, and your customers are willing to help you test and put up with whatever problems they find, if it gives you a higher chance of getting your business launched, then by all means consider giving it a try.

Just keep in mind that some day you may need to grow up and take a serious look at how you build and release software – that the approach that served you well as a startup may not cut it any more.

But let’s not pretend that this approach can be used for online mission-critical or business-critical enterprise or B2B systems, where your system may be hooked up to dozens or hundreds of other systems, where you are managing critical business transactions. Enterprise systems are not a game:

“I understand why people would think that a consumer internet service like IMVU isn't really mission critical. I would posit that those same people have never been on the receiving end of a phone call from a sixteen-year-old girl complaining that your new release ruined their birthday party. That's where I learned a whole new appreciation for the idea that mission critical is in the eye of the beholder.”

This is a joke right?

But seriously, I get concerned when thoughtful people in the development community, people like Kent Beck and Michael Feathers start to explore Continuous Deployment Immersion and zero-length iterations. These aren’t kids looking to launch a Web 2.0 site, they are leaders who the development community looks to for insight, for what is important and right in building software.

There is a clear risk here of widening the already wide disconnect between the software development community and the security community.

On one side we have Lean and Continuous Deployment evangelists pushing us to get software out faster and cheaper, reducing the batch size, eliminating overhead, optimizing for speed, optimizing the feedback loop.

On the other side we have the security community pleading with us to do more upfront, to be more careful and disciplined and thoughtful, to invest more in training and tools and design and reviews and testing and good engineering, all of which adds to the cost and time of building software.

Our job in software development is to balance these two opposing pressures: to find a way to build software securely and efficiently, to take the good ideas from Lean, and from Continuous Deployment (yes, there are some good ideas there in how to make deployment more automated and streamlined and reliable), and marry them with disciplined secure development and engineering practices. There is an answer to be found, but we need to start working on it together.

3 comments:

Swimlane, not just streamline, your application development efforts in new, automated ways by attacking the appsec problem on all sides. Responsibility and accountability are key indicators for any sized project at any speed.

Just look at the investments that SForce has made:http://blog.sforce.com/sforce/2010/04/introducing-forcecom-secure-cloud-development.html(note especially the link to http://force-dot-com-esapi.googlecode.com)

If you're not already developing via PaaS, I believe there are many wins to be had in these environments for both information security (especially application security) and increased velocity in application development (that would certainly support continuous deployment).

Everything has its costs, however. There may be some losses in risk management, fraud management, and litigation support in the cloud (perhaps worse in PaaS than either IaaS or SaaS). I believe that the risk assessment and information assurance guidance from ENISA on cloud computing can offset these to a large degree, although to what degree we don't have a lot of data points or trends to go on.

I am still of the opinion that adding one security bugfixer to offset the one "clueless or uncaring" appdev on any project is the way to go (unless you have a roomful of bad appdevs). You tell one appdev that it's his or her job to successfully fix all of the security related bugs during a project, potentially where some or all of the sprint deliverables (especially ones that end in an iteration demo) are lists of security controls like the ones found in the OWASP ASVS.

If that security bugfixer can work at the same pace as everyone else, the cost is moved away from other developers needing to trade-off their time and effort. There is a price attached: the cost of the security bugfixer, but it's static and scales.

My favorite part about making it one bugfixers's job is also to avoid the "writing around Fortify" problem. I see appdevs, especially teams who work really well together, decide that they are going to skip around Fortify just to speed up their development and looking like the good, supportive appdev team in the eyes of the CISO. The bugfixer can work like a dev-tester (SQE developer working before, during, and after integration), choosing their own toolchain while trying to work along with the overall appdev team.

I would think it would be funny and perhaps even sometime work well to make this role a rotating one, where the developer who produced the most security problems on the last project now has to be the one responsible for fixing security issues on the new project (with bonuses tied).

Subscribe to this blog

About Me

I am an experienced software development manager, project manager and CTO focused on hard problems in software development and maintenance, software quality and security. For the last 15 years I have managed teams building and operating high-performance financial systems.
My special interest is how small teams can be most effective in building real software: high-quality, secure systems at the extreme limits of reliability, performance, and adaptability. Software that has to work, that is built right, and built to last.
I use this blog to explore ideas and problems in software development that are important to me. To reflect and to find new answers.