Red October: CloudFlare’s Open Source Implementation of the Two-Man Rule

At CloudFlare, we are always looking for better ways to secure the data we’re entrusted with. This means hardening our system against outside threats such as hackers, but it also means protecting against insider threats. According to a recent Verizon report, insider threats account for around 14% of data breaches in 2013. While we perform background checks and carefully screen team members, we also implement technical barriers to protect the data with which we are entrusted.

One good information security practice is known as the “two-man rule.” It comes from military history, where a nuclear missile couldn’t be launched unless two people agreed and turned their launch keys simultaneously. This requirement was introduced in order to prevent one individual from accidentally (or intentionally) starting World War III.

To prevent the risk of rogue employees misusing sensitive data we built a service in Go to enforce the two-person rule. We call the service Red October after the famous scene from “The Hunt for Red October.” In line with our philosophy on security software, we are open sourcing the technology so you can use it in your own organization (here’s a link to the public Github repo). If you are interested in the nitty-gritty details, read on.

What it is

Red October is a cryptographically-secure implementation of the two-person rule to protect sensitive data. From a technical perspective, Red October is a software-based encryption and decryption server. The server can be used to encrypt a payload in such a way that no one individual can decrypt it. The encryption of the payload is cryptographically tied to the credentials of the authorized users.

Authorized persons can delegate their credentials to the server for a period of time. The server can decrypt any previously-encrypted payloads as long as the appropriate number of people have delegated their credentials to the server.

This architecture allows Red October to act as a convenient decryption service. Other systems, including CloudFlare’s build system, can use it for decryption and users can delegate their credentials to the server via a simple web interface. All communication with Red October is encrypted with TLS, ensuring that passwords are not sent in the clear.

How to use it

Setting up a Red October server is simple; all it requires is a locally-readable path and an SSL key pair. After that, all control is handled remotely through a set of JSON-based APIs.

Red October is backed by a database of accounts stored on disk in a portable password vault. The server never stores the account password there, only a salted hash of the password for each account. For each user, the server creates an RSA key pair and encrypts the private key with a key derived from the password and a randomly generated salt using a secure derivation function.

Any administrator can encrypt any piece of data with the encrypt API. This request takes a list of users and the minimum number of users needed to decrypt it. The server returns a somewhat larger piece of data that contains an encrypted version of this data. The encrypted data can then be stored elsewhere.

This data can later be decrypted with the decrypt API, but only if enough people have delegated their credentials to the server. The delegation API lets a user grant permission to a server to use their credentials for a limited amount of time and a limited number of uses.

Cryptographic Design

Red October was designed from cryptographic first principles, combining trusted and understood algorithms in known ways. CloudFlare is also opening the source of the server to allow others to analyze its design.

Red October is based on combinatorial techniques and trusted cryptographic primitives. We investigated using complicated secret primitives like Shamir's sharing scheme, but we found that a simpler combinatorial approach based on primitives from Go's standard library was preferable to implementing a mathematical algorithm from scratch. Red October uses 128-bit AES, 2048-bit RSA and scrypt as its cryptographic primitives.

Creating an account

Each user is assigned a unique, randomly-generated RSA key pair when creating an account on a Red October server. The private key is encrypted with a password key derived from the user’s password and salt using scrypt. The public key is stored unencrypted in the vault with the encrypted private key.

Encrypting data

When asked to encrypt a piece of data, the server generates a random 128-bit AES key. This key is used to encrypt the data. For each user that is allowed to decrypt the data, a user-specific key encryption key is chosen. For each unique pair of users, the data key is doubly encrypted, once with the key encryption key of each user. The key encryption keys are then encrypted with the public RSA key associated with their account. The encrypted data, the set of doubly-encrypted data keys, and the RSA-encrypted key encryption keys are all bundled together and returned. The encrypted data is never stored on the server.

Delegating credentials to the server

When a user delegates their key to the server, they submit their username and password over TLS using the delegate JSON API. For each account, the password is verified against the salted hash. If the password is correct, a password key is derived from the password and used to decrypt the user’s RSA private key. This key is now “Live” for the length of time and number of decryptions chosen by the user.

Decrypting data

To decrypt a file, the server validates that the requesting user is an administrator and has the correct password. If two users of the list of valid users have delegated their keys, then decryption can occur. First the RSA private key is used to decrypt the key encryption key for these two users, then the key encryption keys are used to decrypt the doubly encrypted data key, which is then used to decrypt the data.

Some other key points:

Cryptographic security. The Red October server does not have the ability to decrypt user keys without their password. This prevents someone with access to the vault from decrypting data.

Password flexibility. Passwords can be changed without changing the encryption of a given file. Key encryption keys ensure that password changes are decoupled from data encryption keys.

Looking ahead

The version of Red October we are releasing to GitHub is in beta. It is licensed under the 3-clause BSD license. We plan to continue to release our improvements to the open source community. Here is the project on GitHub: Red October.

Writing the server in Go allowed us to design the different components of this server in a modular way. Our hope is this modularity will make it easy for anyone to build in support for different authentication methods that are not based on passwords (for example, TLS client certificates, time-based one-time-passwords) and new core cryptographic primitives (for example, elliptic curve cryptography).

As was mentioned in the original 1.1.1.1 blog post, our policy is to never write client IP addresses to disk and wipe all logs within 24 hours. Still some folks might not want to reveal their IP address to the resolver at all. This is why we are launching a Tor hidden service for our resolver....