DevOps Tip: Don't Give Developers Keys To Security

Security change controls can slow down a DevOps program. But let's break up monolithic security systems instead of giving developers more security responsibility.

Here's the latest misguided notion to come along in DevOps: Let's operationalize security so developers can handle their own security changes.

Like most bad ideas, this one arrives with good intentions. We want to make it possible for developers to make changes more quickly, without getting into drawn-out change control processes.

So why not create automated security modules that let the dev team cut the security team out of the picture? As a recovering developer myself, I'll tell you why not: because we're jerks.

Developers open ports when we need to, without thinking about the implications for the broader business. We run roughshod over regulatory compliance. We don't give a moment's thought to the other developers we work with. We care only about making our own apps the best they can be.

That kind of attitude can be a tremendous asset because it can result in great software. But give developers the keys to security? I don't think so.

If empowering developers isn't the answer, what is? How can we speed up the policy-setting process and streamline change management while making sure that each change makes sense and doesn't expose the business?

We already know that automation is part of the answer. You can push down pre-packaged rules to tell a firewall to open a series of ports, and if your syntax is correct, it'll work.

However, that's only the technical solution to the problem -- you still need a human to ensure that doing so won't expose vulnerabilities across a bunch of other apps.

Instead of trying to reduce or eliminate security's role in change control, we need to simplify it. The reason it can take six weeks to implement a change in a firewall's rules is that there may be hundreds of apps to reconcile against the change.

But with the rise of virtualized network infrastructure and SDN, we can now replace a handful of big firewalls at the edge with a lot of small, per-app firewalls with per-app policies. That way, you only have to look at a developer's request in the context of that one app. The security team can approve changes much more quickly, then let automation tools take care of the brute-force stuff.

This approach isn't limited to firewalls. As enterprise infrastructure has grown, it has become too complicated to consolidate and aggregate into just a few big chunks. We need to break it down into smaller, simpler pieces wherever we can.

That's already happening with the VPN. I see organizations that now run many little VPNs because they're easier to manage than a handful of big ones with a thousand rules no one can track. The same can apply to load balancers, proxy servers, IDS/IDP systems -- the list goes on.

As you replace big with small, and complex with simple, you can reduce the number of people needed to manage change control across your infrastructure.

This isn't a new way of thinking -- it's a lesson developers learned back in the 1970s. In his book A Quarter Century of UNIX, Peter Salus describes a key concept embraced by those early UNIX developers: to write one program that does one thing, then chain it together with other small, single-purpose programs to make a big program.

The premise of chaining individual programs is mirrored in the service chaining we see today in SDN -- each service element does one thing well in an easily understood way, and multiple services can be chained together to build apps.

After two decades of focusing on cramming more and more into integrated boxes, infrastructure guys are finally coming around to the beauty of simplicity.

Developers may be jerks, but when they're right, they're right.

Register now for Interop New York. Use the Discount Code BPIWSEP for $200 off the current price of Total Access, Conference, and 2-Day Workshop Passes. Or register for a free Expo Pass. Interact with peers, see new products and technology, and dive into hands-on workshops and educational sessions taught by independent experts.

Steve Shah is Vice President of Product Management in the NetScaler Product Group, where he drives product direction and go-to-market strategy. Before returning to Citrix, he was Principal of RisingEdge Consulting, a boutique consulting firm that specialized in strategic ... View Full Bio

I completely agree that developers should be trained to be security professionals as well. Their unit tests should include a significant amount of negative input and boundry testing at a minimum. QA processes should include static code analysis, fuzzing attacks, and full scans to validate at the system level.

Unfortunately, we aren't there yet and it doesn't take much reading on developer forums to see that we're far away from this ideal. In larger applications, end to end systems knowledge can be even harder to come by, let alone deep security know-how. So we continue to have network and application firewalls and specialized groups that focus on system / site-wide security. As long as we need those pieces, the need for enabling fast turnaround on making changes will remain.

Thanks for the candid reply. If this model becomes prominent, it seems like controllers are going to have to play a significant role for config and change management, whether as a de facto management point with a decent UI, or to feed into a larger management infrastructure. In any case, if better automation can help preserve domain expertise while also accelerating change, that seems like a win.

No "we" do not. It's time to train and vet all server developers as security professionals. There are many cases where a little less security awareness is needed (client dev, tool dev, library dev) but when a developer is implementing any kind of server project they should be tossed into the real insecure world from the very beginning. Security cannot be layered on later by someone else. Every developer on every server must start with patching, firewalls, securing the ports on whatever software they use, etc. If they cannot do that, they need to be trained or reassigned.

One of the beauties of cheap VPS is that the VPS is in the real world from the moment it is deployed. Every server developer should be able to deploy and secure one in five minutes, and spend a little extra time at each stage of development keeping it secure. For example if they add user accounts, they have to immediately add access control else their VPS out on the open internet will get hacked.

Great question, Drew. To be perfectly transparent, this is an approach that is still new and is getting fleshed out by early adopters. The key in the approach is the adoption of network controllers for heavy automation. This lets the controller take care of keeping track of all those instances, which VLANs they run on, and which apps they are tied to. Thus, when an app is moved all the VMs that made up the app plus the supporting infrastructure (firewalls, load balancers, etc.) are taken along for the ride. Ditto with handling of instantiation and decommission.

Increasingly, some notion of a configuration template or recipie (think ACI or Puppet) is used to driving configuration across each instance. The vast majority of apps use a small handful of recipies while the remaining start to leverage one-off changes. This makes the changes easy to track from a central location and state management of the device easy to do. (Unsure of the device state? Let the system get it back to a known configuration for you.)

Net-net, the goal is to make changes quick. We can either trust developers (which I clearly don't agree with) or empower the security team to be able to drive change quickly. A highly automated template driven network is one approach to empowering the security team that I believe will be the method of choice for most organizations in the future.

"Developers open ports when we need to, without thinking about the implications for the broader business. We run roughshod over regulatory compliance. We don't give a moment's thought to the other developers we work with. We care only about making our own apps the best they can be."

Steve, most developers have similar working strategies. But when doing the major changes with security, it's always better to consult with domain experts.

Developers have access problems because supporting production applications requires access to a lot of information and tools. Information and tools they don't typically have access to.

Tools like Stackify can solve these problems by giving developers complete visibility to server health, application monitoring, errors, logs, and application performance from a single dashboard. With this type of visibility in one place, they no longer need backdoor production access just to view a log file.

In many ways, the firewall and network changes that Steve is talking about flow out of data center virtualization, which allows us to break big monolithic hardware and software components into smaller, logical chunks. This process is started, by no means finished. Thanks for approaching this topic wiith a sense of humor.

This is a really interesting post, but I'd like to get your take on potential downsides of going with lots of small control points vs. a few large ones. I take your point that, for example, trying to accomodate the needs of a variety of applictaions that each has different requirements using just a few firewalls would make for a tortorous rule set. But it also seems like there could be just as many downsides with managing hundreds of firewall instances, each with their own special unicorn settings. All those instances need to be managed, kept track of, patched, decommissioned if their associated app goes away, etc.

To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.

Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.