Note: Full administrators in Couchbase can manage certificates using the Couchbase CLI tools (as described in
ssl-manage) or REST API (as described in
Security API).

With CA-based certificates, you also achieve their simplified management and rotation without client downtime.

Note: Your prior knowledge of TLS/SSL, PKI certificates including X.509 certificates, and Certificate Authorities (CAs) is assumed for this overview.

When to use X.509 Certificates

An X.509 certificate does more than just distribute the public key: it is signed by a trusted (internal or third-party) CA and verifies the identity of the server so that clients know their information is not being sent to a rogue server.

Some common scenarios which may require use of X.509 certificates are:

In production where clients have to go through the internet.

When transferring sensitive data on the wire between application and Couchbase Server, or between data centers (XDCR).

When mandated by regulatory compliance regulations.

CA Hierarchies Supported by Couchbase

Couchbase customers can tie Couchbase Server with their CA hierarchies. The CA at the top of a hierarchy is called the root authority, or root CA. There are two types of CA hierarchies supported by Couchbase: single- or n-tier.

Single-Tier Hierarchy

In its simplest form, the single-tier hierarchy starts with a root CA.

For example, as shown in the figure above, the root CA is also the issuing CA. All certificates immediately below the root certificate inherit its trustworthiness and can be used to secure systems.

Although this is the simplest form of CA hierarchy, most enterprises use an N-tier CA hierarchy as described next.

N-Tier Hierarchy

Typically, in many production deployments, a hierarchy will have multiple CAs.

In a multi-tier hierarchy, the root CA issues certificates to the intermediate CAs, which in turn generate intermediate certificates used to sign the client certificates such as a cluster certificate:

The HTTPS specification mandates that HTTPS clients must be capable of verifying the identity of the server. This requirement can potentially affect how you generate your X.509 certificates. The HTTPS specification defines a generic mechanism for verifying the server identity, known as the HTTPS URL integrity check, which is the standard mechanism used by Web browsers.

HTTPS URL integrity check

The basic idea of the URL integrity check is that the server certificate's identity must match the server hostname. This integrity check has an important impact on how you generate X.509 certificates for HTTPS: the certificate identity (usually the certificate subject DN’s common name) must match the name of the host on which Couchbase Server is deployed.

The URL integrity check is designed to prevent man-in-the-middle attacks.

Specify the certificate identity for the URL integrity check in one of the following ways:

Using the
commonName

The usual way to specify the certificate identity (for the purpose of the URL integrity check) is through the Common Name (CN) in the subject DN of the certificate.

Using the
subjectAltName

If you deploy a certificate on a multi-homed host, however, you might find it is practical to allow the certificate to be used with any of the multi-homed host names. In this case, it is necessary to define a certificate with multiple, alternative identities, and this is only possible using the
subjectAltName certificate extension.

The HTTPS protocol also supports in host names the wildcard character *. For example, you can define the
subjectAltName as follows:

subjectAltName=DNS:*.couchbase.com

This certificate identity matches any three-component host name in the domain couchbase.com.

Note: As a best practice, try to avoid using the wildcard character in the domain name. Be sure never to do this accidentally by forgetting to type the dot (.) delimiter in front of the domain name. For example, if you specified *couchbase.com, your certificate could be used in any domain that ends with the string
couchbase.

Couchbase Cluster Certificate

The Couchbase cluster certificate is the root CA's public key ca.pem. In the configuration steps shown in the following sections, ca.pem is the CA public key that should be configured in Couchbase as the cluster certificate.

When you load the cluster certificate into Couchbase, it is first checked to make sure it is a valid X.509 certificate. Next, if the per-node certificates are not signed by the cluster certificate, a warning is shown for each node during configuration. As the per-node certificates are updated, such that they are signed by the cluster certificate, the warning for each node goes away.

Per Node Certificate

The Couchbase cluster certificate is used to sign per-node Couchbase certificates, each containing the following:

The node private key, which is named pkey.key as shown in the configuration steps below.

The node public key certificate file, which is named pkey.pem as shown in the configuration steps below.

The certificate chain file based on the supported CA hierarchy, This file is named chain.pem as shown in the configuration steps below.

Table 1. Private and public keys you need to have

Key name

Description

Server-side files

ca.pem

Root CA public key or the cluster certificate.

int.pem

Intermediate public key. There can be one or more intermediate public keys in the hierarchy.

pkey.key

Node private key per node (private key of the node). Each node in the cluster must have its private key.

pkey.pem

Node public key (public key of the node). Each node in the cluster must have its public key.

chain.pem

Concatenated chain file (chain file). This file contains the node public key and the intermediate public keys that signed first the node key (pkey.pem) and then each other.

All the keys and certificates are generated in a directory named SSLCA, which can be located anywhere on your machine.

The generated private node key (
pkey.key) and chain certificate (
chain.pem) must be posted in a specific place that is in the certificate trust path (such as
/Users/<username>/Library/Application\ Support/Couchbase/var/lib/couchbase/inbox/ on MacOSX).

Do you have one or more nodes in the cluster?

With one node, you will generate one node directory inside the directory SSLCA that will contain the private node key (pkey.key) and the certificate chain file (chain.pem). The node public key (pkey.pem) is included in the chain file.

With multiple nodes, you need to add an appropriate number of node directories with distinctive names, such as node-sales, node-hr, or whatever your situation requires.

Do you have one or more intermediate CAs in your trust path?

With only one CA, create one directory named
int. If you have multiple intermediate CAs, be sure to name them in a way that will allow you to stack them properly in the chain file, such as
int1,
int2, and so on.

This order will show that the intermediate CA closest to the node (which signed the node certificate) has the higher number, or in the sample below
int2.

For example:

Configure X.509 Certificates using openSSL

Here are the steps to help you generate X.509 certificates:

Create a top-level directory called SSLCA in your user folder and three types of sub-directories:

When a client application tries to verify a certificate signed by the intermediate CA, it must also verify the intermediate certificate against the root certificate. To complete the chain of trust, create a certificate chain to present it to the application.

In a chain file, the lowest certificate goes first, then the intermediate certificates in proper order.

Important: Do not include in your chain file the root CA’s public key certificate and intermediate certificates that are not in the trust path.

Note: The root certificate is not included in this chain.

To create the certificate chain file, concatenate the intermediate and root certificates together, beginning with the lowest one in the chain.

If you have only one intermediate CA in your trust path, use this command:

cat pkey.pem ../int/int.pem > chain.pem

If you have more intermediate CAs in your trust path, keep in mind that the order of certificates in an SSL Certificate Chain file is important:

Rotating X.509

Certificate rotation is needed when a certificate expires, if you are considering moving from an old CA authority to a new CA authority, there is a change in the policy of the certificates issued by the CA, or in the case of a widespread breach of security occurs in the system.

You need to have a plan in place to renew the CA well before it expires. X.509 certificate rotation in Couchbase is an online operation and does not require a node or cluster restart. You can be reassured that the application will have continued access to Couchbase without getting hit with a downtime during the rotation operation.

How to Rotate an X.509 Certificate in Couchbase

Generate a new certificate.

Before you rotate a certificate, you need to generate a new certificate.

Typically, your Certificate Authority (CA) will give you a self-service option to re-issue certificates. If this is not the case, you can manually regenerate your new X509 certificate.

Renew the root CA certificate

The root certificate authority (CA) is the topmost CA in a CA hierarchy. Its validity period is typically longer, between 10 and 20 years.

Note: When you renew the root CA, you have the option of reusing its existing private key. If you keep the same private key on your root CA, all certificates can continue to validate successfully against the new root; all that's required of you is to trust the new root.

During the development process these external tools might come in handy for verifying and debugging SSL traffic:

openssl: OpenSSL command line tool

wireshark: Network traffic analyzer

nmap: Sophisticated security scanner

Revert from the X.509 to the self-signed certificate

If you configured Couchbase to use X.509 certificates, and you want to go back to the self-signed certificates, you can do this by regenerating the self-signed cluster certificate using CLI or REST.

Warning: Moving from CA certificates to self-signed certificates will cause application downtime because you need to reconfigure the self-signed cluster certificate on the client machines after self-signed certificate regeneration.

Troubleshooting X.509

This section lists the error messages connected to the configuration of cluster and node certificates in Couchbase.

Cluster CA Certificates

Here are some error messages you might encounter when configuring the cluster CA certificate and the suggested corrective actions:

Table 2. Error messages when configuring cluster CA certificates

Couchbase Error Message

Description

Suggested User Action

Certificate should not be empty

This error message can occur if the request body of the certificate is empty.

Open the certificate file, and verify whether it is empty or not. The certificate file should be readable using openssl or via online SSL tools such as sslcheker .

Certificate is not valid at this time

This error message can occur if the certificate has expired, or is not yet valid.

Verify whether the certificate validity dates (begins on, and expires on) are currently valid corresponding to the server clock time.

Malformed certificate

This error message can occur due to many reasons - an extra space in the certificate digest body, incorrect certificate format, and so on.

Use a properly configured certificate, and make sure it’s readable using openssl. It should look as follows: Certificate begins with

-----BEGIN CERTIFICATE-----

and ends with

-----END CERTIFICATE-----

on a new line with no spaces before or after.

Only one certificate per request is allowed

Appears when the file contains more than one key or certificate.

Open the .pem file, and make sure that it has only a single certificate digest (such as single BEGIN CERTIFICATE,END CERTIFICATE pragmas).

Encrypted certificates are not supported

This error message can occur if you are trying to load a certificate that is encrypted. Verify by opening the certificate file. If you see something like shown below, you will know your certificate is encrypted.:

-----BEGIN RSA PRIVATE KEY-----

Couchbase does not support encrypted certificates. Decrypt the certificate with openssl before loading the certificate in Couchbase.

openssl rsa -in privateKey.pem -out newPrivateKey.pem

Invalid certificate type: ~s

Appears when a header other than BEGIN CERTIFICATE has been found.

Open the certificate file, and verify whether it is a valid certificate. The certificate file should be readable using openssl or via online SSL tools such as sslchecker.

Node Certificates

Here are some error messages you might encounter when configuring the node certificate and the suggested corrective actions:

Table 3. Error messages when configuring node certificates

Couchbase Error Message

Description

Suggested User Action

Cluster CA needs to be set before setting node certificate

This error can occur when your cluster is still using the self-generated certificate, and you are attempting to configure a node certificate.

Set up the cluster CA certificate before configuring the per node certificate.

Incorrectly configured certificate chain. <Error>

Denotes an invalid certificate in the chain file when configuring Couchbase.

Chain file should contain a sequence of PEM (base64) encoded X.509 certificates ordered from leaf to and including the intermediate certificate authorities.

Unable to read private key file <Path>. <Error>

<Error> is one of the file read errors.

Make sure that you have copied an unencrypted version of the private key file to the inbox folder on the Couchbase node.

Unable to read certificate chain file <Path>. <Error>

<Error> is one of the file read errors.

Make sure that you have copied an unencrypted version of the chain file to the inbox folder on the Couchbase node.

Invalid private key type: <Type>

The private key has an unsupported header.

Make sure that you use a valid private key file.

Provided certificate doesn't match provided private key

The certificate doesn't recognize the message signed with a private key.

Be sure that you use a complete key pair

Encrypted keys are not supported

The private key is encrypted.

Couchbase does not support encrypted keys. You should decrypt the private key with OpenSSL before loading the certificate in Couchbase.

Provided private key contains incorrect number of entries

The private key is a chain of entries.

The private key file should contain a single key digest.

Malformed or unsupported private key format

The private key cannot be used.

Open the key file, and verify whether it is a valid private key. The certificate file should be readable using openssl.

File does not exist

The file is missing, does not exist.

Add the missing file.

Missing permission for reading the file, or for searching one of the parent directories

You don't have the proper permissions to read the file or to search its parent directories.