Google Next Talk: Kubernetes Security Threats and Solutions

We were pleased to present at Google Cloud Next 2018 at the request of Allan Naim, a Kubernetes Engine product manager at Google. In our talk, we highlighted reference architectures for container security and technical demos of attack vectors in the ecosystem. Our talk centered around architectures for FinTech companies running on Google Kubernetes Engine (GKE), but anyone running containers and Kubernetes can leverage the findings we’ll review here.

Allan started the discussion with an overview of the Google Cloud products that retail and financial services businesses can use to build rich, tailored, easy-to-operate solutions for their customers. Alex van Boxel from Vente-Exclusive, a European flash-sale retailer, then talked about the cloud-native stack behind his company’s customized flash sales business. Many would aspire to the GitOps workflow his team has adopted!

Then, we dove into the reference architectures and demos. Wei Lien Dang, our VP of Product, began our talk by sharing concrete Kubernetes architectural patterns we’ve seen with financial services and retail customers. He covered the top security concerns we hear from our customers in the financial services industry: new tooling, emerging threats to unfamiliar components of the cloud-native stack, and the ephemerality of container infrastructure.

His key security lessons highlight the StackRox approach to Kubernetes security: work together with DevOps to get the right context throughout the container lifecycle, and use it to proactively reduce operational risk.

After that overview, I then demonstrated three attacks exploiting insecure orchestrator configuration, server-side request forgery, and application vulnerabilities. These scenarios were based on recently publicized compromises at Tesla, Shopify, and Equifax. In this post, I’ll take you through our key findings. I demonstrated how GKE and StackRox can work together to secure your workloads against them.

In this attack, a Kubernetes dashboard was left exposed to the Internet, with anonymous users able to modify the cluster’s configuration. The Kubernetes dashboard is a built-in service that can be a useful debugging tool, but it can be a source of risk if it’s configured insecurely. Over the last few Kubernetes releases, its default configuration has gotten steadily more secure, but your cluster might have been configured to give the dashboard service account overly broad Role-Based Access Control (RBAC) permissions for the Kubernetes APIs.

This type of exposure could leave your cluster vulnerable to arbitrary pod creation, and in many cases adversaries will use this opportunity to steal computing power to mine cryptocurrencies. You can apply a variety of techniques to prevent this type of compromise, including restrictions on the image registries your cluster can deploy from, or runtime detection and response to specific types of resource abuse—and, of course, you should audit your RBAC configuration to ensure this privilege elevation has not been granted.

In this example, one of Shopify’s containerized applications was vulnerable to a Server-Side Request Forgery (SSRF), leading to the leak of sensitive data from the company’s Google Kubernetes Engine clusters. As I mentioned on stage, the Shopify security team deserves a lot of credit for helping the community learn from this exposure by making the bug bounty report for this issue public.

In the demo, I showed an escalating series of information disclosures from an application that is vulnerable to server-side request forgery. I started with a simple leak of the node’s public IP address, then escalated to an enumeration of neighboring workloads via access to the kubelet’s read-only API, and finally moved on to the sensitive credentials in the cloud metadata, which more closely mirrors Shopify’s bug report.

Network policies are a powerful countermeasure, and the StackRox Container Security Platform helps you use them to effectively limit adversaries’ ability to spread through your cluster. They’re available as an opt-in feature in GKE and other providers. To specifically mitigate the access to the cloud metadata server, GKE offers Metadata Concealment; we highly recommend following the GKE team’s guidance to enable this feature.

The final scenario uses an Apache Struts vulnerability that allows remote command injection. While this particular data breach did not occur on Kubernetes or GKE, it illustrates the way that application vulnerabilities still matter in containers, especially if you haven’t deployed countermeasures or automated responses to contain compromises.

Network policies, slim images, and quick feedback on fixable vulnerabilities can help reduce the risk and mitigate the impact of this type of compromise.

If you’re running applications in containers, be sure to take advantage of all of the security features in Kubernetes, and make sure you have the visibility, detection, and response features you need to stop adversaries in their tracks. Contact us to get a personalized live demo of the StackRox Container Security Platform or if you’d like to learn more about what we presented at Google Next.