Security Center

This post has been updated to include additional statement on behalf of WeightWatchers Inc.

Lack of Security in the Cloud

Kromtech Security researchers have noticed a rise in simple security mistakes within cloud based systems that have exposed very sensitive data. It's easy to see when you look at the recent number of publicly accessible unprotected databases, administration interfaces, and storage buckets, that were disclosed by others or found by us. It seems that every time we turn around there is a massive new exposure of cloud based data. Cases of deployments compromised; customer data, corporate resources, and intellectual property exposed.

It is clear that the largest benefits of cloud systems can also be their biggest pitfalls, particularly when regarding the ease and speed of distributing changes. In today's “speed is paramount” culture, you must take a security first approach, meaning security throughout your entire process. If you don't, it will bite you.

Security must be at the top of your list however your teams function. If you are following Scrum, Agile, and DevOps ideals, in conjunction with cloud computing, it is almost a guarantee that you will rapidly introduce vulnerabilities without proper DevSecOps practices. This has proved to be a very costly step to miss.

The latest in the string of cloud based compromises was the discovery of hundreds of Kubernetes Administration consoles by RedLock, all available on the Internet and completely open, lacking any password protection. Administration consoles without any password protection? Public?

While you weigh that thought, we bring the latest case of this and how easily it lead to the compromise of their cluster.

WeightWatchers Accidentally Provides a Container Feast

Earlier this month Kromtech Security experts discovered a Kubernetes administration console belonging to WeightWatchers that was accessible over the Internet without any password protection.

This Kubernetes cluster was found on at least three IP’s with a kubelet port (10250) exposed, allowing access to all pod's specifications, including the AWS Access key (access key ID and secret access key) and several dozens S3 buckets.

To help a resource owner to close this critical security gap we had to identify who is behind these keys and cluster and to follow responsible disclosure procedure. With aws-cli tool we identified this resource affiliation to WeightWatchers and immidiately notified their security team.

What is WeightWatchers?

Weight Watchers is an American company that offers many products and services to help individuals lose and manage their weight. Headquartered in New York City, they operate globally in 30 different countries, earning approximately 1.7 Billion USD in revenue.

What is Kubernetes?

Kubernetes is an open source container orchestration tool, developed by Google, which automates the deployment, updating, scaling, and monitoring of application containers. Unlike Docker, which works on one container at a time, Kubernetes can operate thousands of containers.

In the case of Weight Watchers, the open administration interface lead to full access.

How could this happen?

Well, first, the words “public without password” and “administration Interface” should never go together. By not properly protecting the administration console Weight Watchers provided all the keys and information needed to gain full root access to their entire cluster. It was too easy.

Some thoughts:

Even if that was a test kubernetes cluster, the DevOps responsible for it has no excuse. Kubelet connection is not secure enough to be run across the internet. SSH tunnels must be used to securely put packets onto the cluster's network without exposing the kubelet's web server to the internet.

Kubernetes objects of type secret are intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image.

Keys with root access to whole infrastructure is a mortal sin and should be considered by each DevOps.

Fortunately WeightWatchers responded that their infrastructure was not compromised.

Thngs to do:

Protect your administration interfaces.

This should just be glaringly obvious. Restrict port ranges at the firewall. Force access only via secure sockets. Restrict access via PKI, where-ever possible, and restrict by IP. And, at the very least, it should be password protected. All of this should be done even if it's strictly on an internal network, reports are rife with internal machines accidentally ending up out on the Internet.

Additionally, a container should never run as root.

When one does, it can provide complete system access.

The kubelet's debug handlers /exec/ and /run/ will run code in any container on the host.

These debug handlers (--enable-debugging-handlers=true) are enabled by default and any code run in the container runs with full root capabilities (compared to docker's root with a capability bounding set).

The kubelet should serve its https with a certificate that is signed by the cluster CA.

More information on securing Kubernetes can be found in their documentation on Securing a Cluster.

Cyrptojacking and Other Malware

Redlock pointed to a rise in Cryptojacking targeting these open administration interfaces, while we did not see any evidence of this having yet happened with this particular Weight Watchers deployment, it is noted that these open administration consoles are now being targeted by attackers for Cryptomining.

As the cost of mining popular cryptocurrencies is becoming prohibitive for individuals, the less than ethical are turning to stealing resources from others. In large organizations this might go easily unnoticed.

We will soon be releasing an article specific to this threat.

We've shown before that ransomware (or other malware) quickly takes advantage of opportunity, it won't be long before more something more aggressive arrives on this scene targeting these open administration consoles.

Notification and Response

“Thanks again for responsibly disclosing your issue. We really appreciate the community working to make us all safer. We have confirmed the issue - a security group for a test cluster in our non-production account was misconfigured during testing. The issue should be resolved and keys should be revoked. We’ve also implemented some safeguards to protect against this issue from recurrence.”

We can only add that there has been no indication that the data was a non-production account.

[UPDATE FROM WEIGHTWATCHERS]

In response to your post, it bears repeating that the issue identified was related to the exposure of credentials in one non-production AWS account that is, in fact, labeled nonprod. The account was in a testing environment used only to test new services and features. To be able to test and innovate securely, we keep test environments completely separate from production environments. Our internal team and a reputable third-party security forensics team have investigated the exposed account key scope and activity, and each has independently confirmed that there was no indication that any personally identifiable information was exposed.

***

Attention - Portions of this article may be used for publication if properly referenced and credit is given to Kromtech Security Center.Do you have security tips or suggestions? Contact: security@kromtech.com