Blog

Disclaimer: Everything written in this article is taken from my experiences and my opinions only. This article's main audiences are, but not limited to, users of CICD (developers), engineers who wish to build and design a CICD pipeline.

INTRODUCTION

Surely, many of us have heard of the term 'CICD' and there are articles that do a far better job at trying to explain what it means and how it works than this article will. It is obvious that there are benefits of a CICD that bring seamless integration from end-to-end for the software development and deployment process. CICD saves developers a huge amount of time. In the business world where time equals money, this translate to more time can be better spent developing code to enhance software features, rather than worrying about the deployment and operations behind the scenes that brings code to life in production. From the moment code is checked-in to the moment the service is updated with the latest code from that same check-in, developers sometime feels like this process is a black box whereby they provide an input and then wait for the output to happen on the other end. This may be true but it is also where security challenges often begin. CICD is fast; security is mostly not. With the introduction of DevSecOps, we hope to change the speed and delivery of security within software. DevSecOps engineers strive to operate most security controls as part of the software being built by introducing it as a design constraint and then seeing it transparently checked by the CICD pipeline without compromising on the integrity of controls. But because security is a design constraint, it is as important for the developers to know what is going on as software is being built and deployed. Building CICD in the cloud, requires even more architecture and coordination work to reduce the volatility of the cloud environment and use the security features of the cloud to the benefit of the CICD pipeline.

Instead of an opaque pipeline:

Try a transparent pipeline:

SECURING CICD

Securing the CICD pipeline is critical. We use a method called "whitelisting" to accomplish the development of critical security controls for a workload. In this situation, CICD is another workload. Why is whitelisting CICD important, you say? Projects that are on-boarded tend to grant CICD a high level of trust and power in order for CICD to do its job. It is this convergence of trust and power that bring together a single point of failure which is commonly disregarded in most environments. CICD in its evolution from simple build tools to the almighty publishing process make it a risk for all projects. The fact that developers are providing source code into this pipeline is a good enough reason to say that developers MUST secure this pipeline. Allowing CICD with the permissions to access, execute, process and build the artifacts and then deploy them onto a production server that generates revenue? Enough said.

So what is whitelisting and how is this done? Securing the CICD pipeline and integrating security testing are essential and require a process for the mapping of these controls back to Security Operations for monitoring. On a high level, it is putting one foot into the CICD process with the intention to use it and secure it, while identifying and mitigating real risks.

LESSONS LEARNED

1. Get it working for you

As an early career engineer, setting up of a secure CICD process for a project can be technically challenging to do at first but once it is done, the reward is awesome. Because setting up can be complex and tough by nature, the only way to get CICD to even work (at least for me) is to keep asking questions to understand and set up as many environments as possible. This is the stage where developers using the service should provide feedback and communicate with the engineers building CICD because it is still fresh in their minds. Engineers building CICD should be open to feedback and track each issue down as a bug. Explore and find as many loopholes as they can, and 'write it down on a sticky note' to be dealt with instead of feared:

2. Start breaking it early - What can attackers do?

Remember the sticky note I talked about previously to write down the loopholes? Now is the time to exploit all of them. This is the stage where we think like an attack and emulate breaking the systems.

These are just some questions to ponder:

Can developers view sensitive environment variables for other projects?

Are credentials checked in and stored in plaintext?

Are some permissions too freely given so that anonymous developers could run bash scripts on all projects?

Is the build orchestrator susceptible to easy attacks?

Are developers able to delete a job from another project with ease?

Are attackers checking in their code?

What other services does CICD use and trust?

Can I exploit the bugs of any build software CICD is deploying?

CICD is usually not an omni-service on its own but contains multiple micro-services. These micro-services have to be whitelisted too. Making sure that each micro-service has been evaluated for its attack surface with appropriate controls is part of the process of whitelisting and is a task on its own. Spending some time understanding the micro-services and making sure it does not introduce security flaws into the CICD service for its use case will reduce the security risks overall.

When a development project is set up with appropriate blast radius and a swimlane in place, developers are involved in the set up process that makes their understanding of the CICD pipeline better i.e. the pipeline becomes more transparent. Teams that use cloud accounts and the code delivery process to their benefit have the ability to keep sensitive workloads safe. Having each project in their own account ensures that sufficient and correct permissions are given to each project that does not cross over to other projects. The notion of blast radius and separation of roles / permissions is important to mitigate most of the issues mentioned. This will help to limit the impact of an attack to the individual project if any were to be compromised.

3. Get it fixed

At this point, you SHOULD know the mechanics of using CICD and what does your gut tell you? You don't have to check off all the boxes on the sticky notes to get CICD to perfection, you just have to mitigate the risks with some correlation to the stage of your project and its desired outcomes.

One easy score would be to lockdown network access to any orchestrator. Orchestrators are definitely a point of infiltration when exposed to hostile environments. Do you feel comfortable letting a third party look at how your project is built? If not, then you need to ensure that only those who need to operate on the workload have access to the Continuous Delivery platforms.

In the cloud, permissions can be given to the orchestrator role and it should be locked down to just the essentials. It should be 'deny-all and only allow some'. It takes time to figure out what permissions are needed. The time required will be excessive at first just to get services to work, but subsequently, tracing down which permissions are required is as simple as looking through some of the API logs and matching those to the permissions. And events can be raised systematically to alert when an assumption and/or control might be violated.

Does your CICD provide logging, monitoring and alerting on each build, and at each stage? This proves valuable information when alerting on an un-scheduled build or potential job that was unintentionally introduced. Whether it is a detection of a bug or an intruder - these are red flags to check out. It is also worthwhile to implement a human-approval step in the process to ensure that while automation helps save time and do most of the work, humans are the decision makers behind these builds. A developer checks in code and ultimately needs to ensure the release is approved. When something isn't right, feedback should be quick and easy to understand. In the event of an attack, logs should be continuously sent to a security monitoring system in order to correlate and to trace down security issues and blast radius in case of a compromise.

CICD is also a great opportunity to integrate automated security tests within the service. Using testing tools, like gauntlt, can help increase the security and resilience of the software and the infrastructure it sits on. We are talking about testing the whole stack - not just the software application layer. It is essential that the infrastructure withstand all current attacks because it is often overlooked and software is then built on a fragile infrastructure. The best part about using the cloud is that stacks and servers are disposable and I can re-built one within minutes of updating the stack.

SUMMARY

If you are on the Security Team at your company, be the first customer of CICD: try the process, test the system, break the system, work with the CICD team to fix it. Rinse and Repeat. The end result is that you get a better, more secure product moving forward.

Although it is tough to tell other engineers to fix their product, constructive feedback is important to get this cycle going. Help to contribute and fix it - do not just say that things suck without providing constructive feedback.

PS: I am proud to say that whitelisting CICD has been a huge part of my learning and my passion to build a better, safer service for other developers. I came into this industry with a Security policy background, now I have Dev and Ops under my belt to become a true DevSecOps engineer.