on computer science and media topics

Cloud security tools and recommendations for DevOps in 2018

Introduction

Over the last five years, the use of cloud computing services has increased rapidly, in German companies. According to a statistic from Bitkom Research in 2018, the acceptance of cloud-computing services is growing.

Cloud-computing brings many advantages for a business. For example, expenses for the internal infrastructure and its administration can be saved. Resource scaling is easier, it can be done when the appropriate capital is available. In general the level of innovation within companies is also increasing, so this can support new technologies and business models. Cloud-computing can also increase data security and data protection, because the big providers must comply with standards and obtain certification through the EU-DSGVO General Data Protection Regulation, which came into force in May 2018. There is even special cloud providers that offer such services, so these are working based on the regulations of the TCDP (Trusted Cloud Data Protection) certification.
However, not all that glitters is gold. Based on a study by KPMG Bitkom Research, 19 percent of the interviewed companies reported that data security incidents have happened in their company by the use of public cloud solutions within the last 12 months. But in my opinion, 19% is far too high, because we as experts, have the task to increase this security even more!

Figure 1: secure cloud computing

In the end, there is the question:Which areas or components within the cloud-computing technologies are affected in terms of data-security?

To be honest, there are a lot of components involved, these are:

Cloud-infrastructures:

SaaS – Software as a Service

PaaS – Platform as a Service

IaaS – Infrastructure as a Service

Devices that use cloud-services and therefore also

The user / developer / administrator (The human)

That’s many, right?

DevOps and Cloud Security

In recent years, the job title DevOps was established.
It is a composition of the words “Development” and “Operations” and it focuses on the improvement of software development. This approach should help increase the general quality of the product, and as well the speed of software delivery.
That’s what Rob England said about DevOps in 2017:

DevOps is agile IT operations delivery, a holistic system across all the value flow from business need to live software. […]

DevOps is a philosophy, not a method, or framework, or body of knowledge, or *shudder* vendor’s tool.

DevOps is the philosophy of unifying Development and Operations at the culture, system, practice, and tool levels, to achieve accelerated and more frequent delivery of value to the customer, by improving quality in order to increase velocity.

CD – Continuous Delivery: (Delivering increments with manual release of an authorized person to the production system)

CD – Continuous Deployment: (Automated delivery of increments to the production system)

But what does the DevOps approach have to do with the cloud?

David Linthicum said:

Thus, DevOps won’t have much value without the cloud, and the cloud won’t have much value without DevOps. This fact alone is being understood inside enterprises that once thought they could move to one or the other, and that no dependency existed. We’re finding that dependencies between DevOps and cloud do indeed exist.

DevOps often use tools and cloud architectures for their work. Sometimes even the operation part shifts towards the DevOps. They must ensure that the security of the product is given and that the cloud-computing technologies for that platform is used correctly.

This blog entry is intended to give a brief overview of what DevOps and administrators can do to secure their applications and their cloud-platforms. It is quite clear that not everything can be covered in this article. Therefore I always recommend, do your own extended research based on your individual topics and needs!

The here explained recommendations are focusing on the behaviour of software in general and are divided around the following topics:

First steps

Secrets management

Data encryption

Logging and monitoring

At the end of the article there is a short example about secret management with the tool git-crypt. This is intended to develop a basic understanding of the use of such secret management tools.

First steps

The first thing to say is that you ALWAYS need to use common sense. So my first recommendation: Use your brain! This also includes an extended understanding of security for employees in the IT environment and training in this area, if they are not familiar with this. It should be clear, that someone with full root access always uses strong passwords in their work and also handles their keys responsibly. No one in IT should be fooled by spam messages with phishing or other harmful links. Passwords, keys, tokens and all confidential information should never be shared within the IT department or even within the company! The use of secret management is also helpful here (see below). Each person should have personalized access. When hiring new employees, it does not hurt to check their background in advance to see if they are trustworthy. Clear and secure authorizations concepts must also be used. Not everyone should have full root access. However, humans can also be regarded as a source of errors in the area of software development. For example, an API-key accidentally uploaded to github.com in version control is also a security leak. Here are some advice’s to get this problems under control:

Repo-security-scanner: CLI tool which finds secrets (passwords, private keys) that were accidentally committed to a repository.

Gitrob: is similar to a repo security scanner. Offers the possibility to search for “sensitive” data in repositories on an organizational basis. Provides results in a web interface.

Another tool that can also search repositories is TruffleHog. It works with the search method to find strings with high entropies – which usually reflect API keys.

By using merge requests and code reviews within a project, security-critical source code passages can be identified.

Do not use public services such as Github for versioning. Gitlab software can be hosted within companies themselves.

The hardware used such as laptops, PCs, servers, etc. must be additionally protected. This succeeds through:

Use of distributions with SE-Linux / AppArmor. These work according to the MAC (Mandatory Access Control) principle. Applications may only perform activities defined for them. Only the necessary software packages should be installed. Unused programs and services must be uninstalled.

Osquery is a tool that makes operating systems searchable like databases. With the help of appropriate packages, security holes and strange behavior can be detected.

Use 2-factor authentication wherever it’s possible. Use strong passwords and encryption of hard drives. If a laptop is forgotten e.g. in a cafe, strangers cannot easily access sensitive data. The chapter Data Encryption contains detailed information about this topic.

In the architecture and conception of the server landscape/cloud and its network, individual services should be installed separately on different machines. This is achieved through firewalls, DMZ principles and various host systems. If possible, the software used, should be separated into microservices. These measures help to keep possible attack vectors small. However, if a break-in occurs, the damage is not as big, as when everything is executed on one monolithic system.

The BSI (German Federal Office for Information Security) makes the following proposals:

Security zone for cloud management, storage network and live migration, if server virtualization is used

Separate customer’s virtual machines at IaaS into security zones

The management network of the CSP (Cloud Solution Provider) should be isolated from the data network.

An important point for DevOps is to secure the CI/CD pipeline. The reason for this is, that sensitive activities such as building source code takes place here. If an attacker succeeds in hijacking such a system, he has the possibility to integrate and execute his own malicious code! The recent incident in which Eric Holmes gained access to the Homebrew repository at github.com within 30 minutes showed how extremely dangerous an unsecured Jenkins pipeline is. In addition, pipelines should be isolated in a separate network and also to other servers. If containers (Docker, RKT) are used, only secure images from trusted registries should be pulled, which also perform additional security scans!

Secrets Management

What are secrets?

The word secret has become a buzzword for keys, certificates, passwords, tokens and API-Keys in the recent years. Actually, everything that needs protection or provides access, is called a secret. But there are various software implementations, that can help you protect your data. A distinction can be made between selfhosted secret management and cloudbased secret management. However, it is not excluded that a selfhosted tool is running within a cloud architecture, like on a virtual machine. The cloudbased tools are mostly implementations provided by the cloud platform vendors themselves.

The capabilities of these tools are different. The tools could be ranked in 3 levels as shown in the figure.

Figure 2: secret levels

Level 1: Only the access to the secrets is restricted, the secrets themselves are not encrypted.

Level 2: The access is restricted again, but the secrets are encrypted.

Level 3: Like Level 2, but with additional features such as Time To Live (the duration how long a secret can be used, till it is invalid) or ACL’s (Access Control List). With this technology an user is able to assign finely granular access-rights to the various secrets in use.

Due to the complexity, it is very difficult to make a recommendation for one of the tools. It always depends on the existing architecture. In addition, you have to differentiate between the gained security and the complexity of introducing these new tools in your company.

Furthermore, an additional layer is added to the architecture of the system. For example, if the secret management fails, the deployment process of an application can also fail!

If you want to learn more about the different tools, I recommend this overview and this one.

Data encryption

There is a thing called Data at Rest. This is when data is persistently saved in a storage, like in a Hard Drive, SSD or on tapes and so on. The opposite of this is data, which is used right at the time it is needed like processed data within a CPU or stored in the RAM. This is called Data at Use.

When it comes to the encryption of data, Data at Rest is usually meant. Under the keywords “Data at Rest Encryption” you will find a lot of tooling to help you with this challenge. Encrypting data makes sense, because it helps in cases of loss or theft. If this is done the data cannot be read directly by other people without any further hacks or software. Encryption also avoids unwanted external access if a software is misconfigured. This way the attacker can access a service, but the encrypted data is worth nothing as long as he does not get the information the data is keeping of the service. A distinction is made between “data volume encryption” and software feature encryption. For example, the data of a database can be encrypted directly by using their software feature. Data volume encryption usually encrypts the entire hard disk. The best practice with volume data encryption is:

Use powerful AES/RSA algorithms

Keys should be changed regularly

Keys should be kept separate from the actual data storage

Periodic checking of encryption/decryption is essential

Only necessary data should be stored and encrypted

Some operating systems such as MacOS offer the possibility to encrypt the hard disk by default. There are also open source solutions such as LUKS (Linux Unified Key Setup), Veracrypt and many more. In addition, there are also commercial solutions such as Portworx, which can especially score in the area of encryption of container volumes such as docker volumes.

Logging and Monitoring

Logging describes the recording of all action of an application process, triggered by users and other systems, distinct the application. The Error.log of the web server Apache is a good example of logging data. If an error occurs, it can be reviewed in this file. Usually, authentication and access to services with the differentiation between the privileges of the users are also performed during logging. Normally log files are written by default for most applications. As DevOps or administrator, the challenge is to collect the log files and manage, evaluate and understand them accordingly.

Monitoring describes the monitoring of computer-services and their availability. Usually performance and availability of a service are checked in real-time, e.g. by the CPU load of a server or the availability, by the ping for an IP address. Monitoring tools offer the possibility to send automated notifications (E-Mail, SMS,…) in case of increased load or if an error occurs. But also graphical elements and the presentation of the collected data in dashboards is handled by monitoring tools.

Monitoring usually includes logging, since decisions etc. cannot be made without the previously gathered data. These two topics are therefore closely related. In the past, the following tools have become established:

The individual cloud provider offer their own monitoring and logging services. AWS calls this Cloudwatch and Google Cloud Platform also offers a variety of management tools in this area. It is also very difficult to make a recommendation here. It always depends on the experience of the employees and which architecture is used. I like Icinga2 because it is a German project and is currently in heavy development. In addition, there is solid support within the monitoring portal forum.

This Example will show a simple procedure of using git-crypt on MacOS. Notice: This is intended for a single user only. If you want to collaborate, please use an extended tutorial like this. You can also use git-crypt with symmetric keys – but then you have to make sure that the transport channel, you share the keys on, has to be secure!

Because of its easiness, homebrew (although it was recently hacked) and github.com is used in this example. You can bypass this technology by using gitlab and installing the tools by hand. Git-crypt was chosen for the example because it is simple. Furthermore you get very fast increased security and it affects almost every developer who works with git-versioning.

As you can see above – truffleHog found this “accidentally” pushed API-Key!

So now i will show you how to secure this repository.

Install Git-Crypt and GPG – encrypt the repository

If you are using a linux machine, simply install through your package manager the tools gpgan git-crypt. On MacOS first install homebrew (the missing package manager for macOS) and run the following steps for installation:

$ brew install git-crypt
$ brew install gpg

Next Step is to generate a GPG-Key and secure it with a passphrase:

$ gpg --gen-key
gpg (GnuPG) 2.2.9; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: SecureSystems
Email address: SecureSystems@test.com
You selected this USER-ID:
"SecureSystems <SecureSystems@test.com>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit?
$ O
Pass a valid passphrase and make sure you can remember it (<strong>in this case test123</strong>)

Now we can see that the “mysupersecretapi.key” is not longer a readable file. So it was also encrypted through git-crypt.

Figure 5: apikey is encrypted

If you look at the previous commits, you can still see the fake api.key! In practice, you should now clean the repository of this commit. But because this is an example, I’ll let it be.

Figure 6: old commit – secret is still readable!

As you can see, git-crypt is a powerful, easy-to-use tool if you want to secure your repositories! I strongly recommend it, as it means that no information about their secrets is publicly accessible. The used GPG keys must be backed up. In the worst case, if they are lost, you no longer have access to the data in the repository.

Summary

Increasing security within cloud computing is an extremely extensive process. There are a variety of tools and methods that can be useful. It should not be overlooked that this is a fast-growing environment. Therefore, good manpower with high expertise is necessary. Employee training is therefore essential. By so-called Gameday’s and e.g. the use of the Simian Netflix Army the DevOps team can not only be trained, but also gain more experiences. Companies should therefore take time for such actions. It is also essential that a cultural change takes place within companies. Especially in management it must be understood that the security of applications and their platforms must be prior before the speed when using new innovations and products!