Blurring the line between software development and operations

“We are a small team,” said Otto, the CEO of a promising startup in the social media space, “and we let development to deploy changes directly on production.”

“We don’t have much custom development,” said Pam, the IT manager at a small manufacturing shop in the cosmetics industry, “and we mostly do integrations with our ERP. There is no need for a staging environment.”

“We hire the best developers and our code is fully unit tested,” says Rupmeet, who manages a team of seven developers working on an analytics product in the cloud, “thus we don’t have problems deploying directly in production.”

The above is just a few examples of excuses I have heard from people when asked if they maintain a staging/pre-production environment where they test their updates before pushing changes to production. Can you really get away without staging your changes in an environment that is a close copy of production and simply pushing them to production? The answer is, “It depends.”

It depends how tightly coupled your code/systems are – would a change in one place have a trickle effect in many others?

It depends on the cost of pushing a regression into production – are you risking an outage or loss of data with the change?

It depends on the investment you’re willing to put to pro-actively prevent regressions – both manual tests and development of automation.

It depends on your user base and their threshold for BS – you might be able to pay your regression cost in instalments over time if you don’t need to do extensive testing upfront and handle issues as they come, letting the users experience some pain in the proces.

It depends on your tolerance to technical debt and willingness to pay some of it out over time or all at once.

At the end, the question how much testing is enough before pushing upgrades to production is a question of investment, loss and profit and there is no one size fits all solution. But with the availability of cloud computing and IaaS or PaaS solutions like AWS and Heroku, where you pay what you use and is relatively easy to setup a new environment, there is almost no excuse for not creating one or more staging environments and testing the changes before they go into production.

2 Responses

BigJohnsays:

Spot on Kima.
It is so easy to trip up during installation/roll out. Just recently a quick test from a colleague spotted a flaw that would have brought down one of my production databases. A frustrating day fixing it but the end result was far far better for it.

Thanks for taking the time to read and share your experience John. I am happy to hear you averted a potential disaster in your case. When running production in the cloud the incentive to prepare a cloned staging or pre-production environment that is only used when needed is there – people just need to accept that doing testing on it should be part of the regular release procedure.