Most people fail to consider the build environment when thinking about container security, but it is critical. The build environment is traditionally the domain of developers, and they don’t share a lot of details with outsiders (meaning Security teams). But with Continuous Integration (CI) or full Continuous Deployment (CD), we’re shooting new code into production potentially several times a day. An easy way for an attacker to hack an application is get into development or build environment – which is usually far less secure than production – and alter code or add new code to containers. And this is a difficult problem with DevOps rapidly breaking down barriers between groups, and with operations and security teams given access so they can contribute to the process. Collaboration demands a more complex and distributed working environment with more stakeholders. Better controls are needed to restrict who can alter the build environment and update code, and an audit process to validate who did what.

It’s also prudent to keep in mind the reasons developers find containers so attractive, lest you try to adopt security controls that limit the usefulness of containers. First, a container simplifies building and packaging application code — abstracting the app from its physical environment — so developers can worry about the application rather than its supporting systems. Second, the container model promotes lightweight services, breaking large applications down into small pieces, easing modification and scaling — especially in cloud and virtual environments. Finally, a very practical benefit is that container startup is nearly instant, allowing agile scaling up and down in response to demand. It is important to keep these in mind when considering security controls, because any control that reduces one of these core advantages is likely to be either rejected or ignored.

Build pipeline security breaks down into two basic areas. The first is application security, essentially testing your code and the container to ensure it conforms to security and operational practices. This includes tools like static analysis, dynamic analysis, composition analysis, scanners built into the IDE, and tools that monitor runtime behavior. We will cover these topics in the next section. The second area of concern is the tools used to build and deploy applications, including source code control, build tools, the build controller, container registires, container management facilities, and runtime access. At Securosis we often call this the “management plane”, as these interfaces — whether API or GUI — are used to set access policies, automate behaviors, and audit activity. Let’s dive into build tool security.

Securing The Build

While the problem is conceptually simple, there are many tools used for building software, nd most have several plug-ins altering how data flows, so the environment can get complicated. You can call this Secure Software Delivery, Software Supply Chain Management, or Build Server Security — take your pick as these terms are equivalent for our purpose. Our goal is to shed light on the tools and processes developers use to build application so you can better guage the threats and potential security measures to secure these system.

Following is a list of recommendations for securing platforms in the build environment to ensure secure container construction. We include tools from Docker and others to automate and orchestrate source code, building, the Docker engine, and the repository. For each tool you will select some combination of identity management, roles, platform segregation, secure storage of sensitive data, network encryption, and event logging.

**Source Code Control: ** Stash, Git, GitHub, and several variants are common. Source code control has a wide audience because it is now common for Security, Operations, and Quality Assurance to all contribute code, tests, and configuration data. Distributed access means all traffic should run over SSL or VPN connections. User roles and access levels are essential for controlling who can do what, but we recommend requiring token-based or certificate-based authentication, or two-factor authentication at a minimum for all administrative access. This is essentially good housekeeping if you are using containers or not, but the lack of transparency coupled with automated processes pushing containers into production amplifys the need to protect the build.

Build Tools and Controllers: The vast majority of development teams we spoke with use build controllers like Bamboo and Jenkins, with these platforms becoming an essential part of their automated build processes. These provide many pre-, post-, and intra-build options, and can link to a myriad of other facilities. This is great for integration but potentially complicates security. We suggest full network segregation of the build controller system(s), and locking network connections to limit what can communicate with the build controller. If you can, deploy build servers as on-demand containers without administrative access — to ensure standardization of the build environment and consistency of new containers. We recommend you limit access to the build controllers as tightly as possible, and leverage built-in features to restrict capabilities when developers need access. We also suggest locking down configuration and control data to prevent tampering with build controller behavior. We recommend keeping any sensitive data, such as ssh keys, API access keys, database credentials, and the like in a secure database or data repository (such as a key manager, .dmg file, or vault) and pulling credentials on demand to ensure sensitive data never sits on-disk unprotected. Finally, enable the build controller’s built-in logging facilities or logging add-ons, and stream output to a secure location for auditing.

Container Platform Security: Whether you use Docker, or some other tool to compose and run containers, access to the container manager is a powerful tool that controls what applications are run. As with build controllers like Jenkins, you’ll want to limit access to specific container administrator accounts. Limit network access to accept content only from the build controller system(s) Make sure the Docker client’s access is segregated between development, test and production, limiting those people/services that can launch containers into production.

Container Registry Security: We need to make special mention of container registries as there are two common mistakes that developers and IT teams make. The first is allowing anyone to add containers to the registry, regardless if they have been vetted. It becomes quite easy to select an insecure container into production. And it is common for developers to leverage open source tools and platforms to speed development, some of which are available as pre-built containers. But it’s also common for attackers to create “pre-pwned” containers loaded with malware, which will probe and attack the host and other containers. You’ll want to ensure that the registry accepts containers only from trusted sources and not just whatever some developer grabbed on a whom. Ensuring containers are vetted and that these containers were created with a digital fingerprint which can be verified before the containers are used helps protect from these problems. You’ll also want to ensure that the registries and the client require IAM credentials to limit who can control the build or add images, and that images loaded into production come from your approved registry.

Useful Links

Other Mediaops Sites

Our website uses cookies. By continuing to browse the website you are agreeing to our use of cookies. For more information on how we use cookies and how you can disable them, please read our Privacy Policy.