We had a great time at the co-hosted PagerDuty/Threat Stack workshop in Seattle last Wednesday: “Incident Management in the 21st Century.” The event kicked off with an opening talk by Jonathan Wilkinson, VP of Product for PagerDuty. He revealed some of the new things PagerDuty is working on and demonstrated many of the interesting ways their customers are using the product and building tools on top of it, enabling them to get the right people "in the room" to handle company incidents.

Up next was the team from Blackrock Partners, who have an incredible level of experience dealing with special operations, hazardous materials, and terrorism response. They talked about the parallels between dealing with incidents such as fires, floods, and other disasters, and compared how those are handled to the ways technologists handle failures in their application and business. The great takeaways from their talk were in line with what we learned from Jonathan earlier: effective incident management requires not only the tools to enable the security team team, but also the people and processes aligned for effective resolution.

Both of these talks squared up with my own presentation on cloud security and incident response in the 21st century, as I’ve managed and built out the security operations procedures at Threat Stack over the past few years.

A theme I heard consistently throughout all of our talks, among many other side conversations throughout the day, was how security is, has, and can spread far beyond the domain of the security experts. That’s what leads me to today’s post. This evolution is not a nice-to-have; it’s a need-to-have.

Living the Problem

In the past I’ve worked with auditors and dedicated infosec teams, and more recently have been involved in the cloud security space. I’ve realized that we’re rapidly approaching an inflection point when it comes to security operations teams and the rest of the technology organization, much like what happened in 2009 when DevOps burst forth into the world.

There was the wall of confusion, with devs on one side and ops on the other.

This used to be a common image in many talks about DevOps because security teams were largely left out, or dev and ops teams just threw their work over to the security side (if they even had a security team). Even today, many of us are guilty of hitting the “snooze” button at some point or another as soon as we hear the word “security”.

But security is on everyone’s mind right now — right down to the average consumer. What’s more, the number of public security failures is rising. Think about Gogo in-flight WiFi admitting to faking SSL certificates. Or the adversaries behind ransomware, costing companies $2.3 billion through CEO email scams in the last three years alone.

It’s about time for entire companies start taking part in the overall security posture of their business. And this isn’t just a “Security Team” problem. Everyone in the business is at risk, and everyone is responsible for being part of the solution.

Security Innovations in Favor of Security Bottlenecks

The cloud is making it easier than ever to provision systems to meet security infrastructure needs — and very quickly. Speed-to-market is, of course, a major competitive advantage that many companies are leveraging through the concept of software-defined everything. Provisioning hundreds or thousands of compute instances in mere minutes is now considered an everyday activity.

Everyone wants to move fast, but in order to do so, many companies are leveraging significantly more open source software to build applications quicker and easier than ever before. But over the past few years, security researchers have been finding high severity vulnerabilities running inside much of the open source code we depend on every day (glibc, openssl, bash, curl, ssh, etc.). And often, the reach and impact of these vulnerabilities is discovered only after an incident happens.

They say “Many eyes make for secure code” but that myth is most certainly busted.

We must all be involved, making smart decisions from the get-go about the security of everything we do — every piece of code we write, every production server we touch, and so on. After all, the rate of change we see today leaves very little room for security teams to properly assess the risk in our application and infrastructure code. Especially for small teams that don't have a dedicated security person on hand, security MUST be the domain of everyone and can no longer be the bottleneck to innovation.

By integrating security best practices, workflows, and tools into dev and ops processes, we can better mitigate risks and continue our journey towards building secure, automated infrastructures. I’m thrilled that events like the one we co-hosted with PagerDuty continue to bring to light this important topic. I look forward to seeing us as an industry beginning to practice what we preach, bridging the gaps, and opening doors to security company wide.