Overview

OpenShift Container Platform uses certificates to provide secure connections for the
following components:

masters (API server and controllers)

etcd

nodes

registry

router

You can use Ansible playbooks provided with the installer to automate checking
expiration dates for cluster certificates. Playbooks are also provided to
automate backing up and redeploying these certificates, which can fix common
certificate errors.

Possible use cases for redeploying certificates include:

The installer detected the wrong host names and the issue was identified too late.

The certificates are expired and you need to update them.

You have a new CA and want to create certificates using it instead.

Checking Certificate Expirations

You can use the installer to warn you about any certificates expiring within a
configurable window of days and notify you about any certificates that have
already expired. Certificate expiry playbooks use the Ansible role
openshift_certificate_expiry.

JSON Report

There are two top-level keys in the saved JSON results: data and summary.

The data key is a hash where the keys are the names of each host examined and
the values are the check results for the certificates identified on each
respective host.

The summary key is a hash that summarizes the total number of certificates:

examined on the entire cluster

that are OK

expiring within the configured warning window

already expired

For an example of the full JSON report, see /usr/share/ansible/openshift-ansible/roles/openshift_certificate_expiry/examples/cert-expiry-report.json.

The summary from the JSON data can be easily checked for warnings or expirations
using a variety of command-line tools. For example, using grep you can look
for the word summary and print out the two lines after the match (-A2):

If available, the jq tool can also be used to pick out specific values. The
first two examples below show how to select just one value, either warning or
expired. The third example shows how to select both values at once:

Redeploying Certificates

Use the following playbooks to redeploy master, etcd, node, registry, and router
certificates on all relevant hosts. You can redeploy all of them at once using
the current CA, redeploy certificates for specific components only, or redeploy
a newly generated or custom CA on its own.

Just like the certificate expiry playbooks, these playbooks must be run with an
inventory file that is representative of the cluster.

In particular, the inventory must specify or override all host names and IP
addresses set via the following variables such that they match the current
cluster configuration:

openshift_hostname

openshift_public_hostname

openshift_ip

openshift_public_ip

openshift_master_cluster_hostname

openshift_master_cluster_public_hostname

The playbooks you need are provided by:

# yum install atomic-openshift-utils

The validity (length in days until they expire) for any certificates
auto-generated while redeploying can be configured via Ansible as well. See
Configuring Certificate Validity.

OpenShift Container Platform CA and etcd certificates expire after five years. Signed OpenShift Container Platform certificates expire after two years.

Redeploying All Certificates Using the Current OpenShift Container Platform and etcd CA

The redeploy-certificates.yml playbook does not regenerate the
OpenShift Container Platform CA certificate. New master, etcd, node, registry, and router
certificates are created using the current CA certificate to sign new
certificates.

This also includes serial restarts of:

etcd

master services

node services

To redeploy master, etcd, and node certificates using the current
OpenShift Container Platform CA, run this playbook, specifying your inventory file:

Redeploying a New or Custom OpenShift Container Platform CA

The openshift-master/redeploy-openshift-ca.yml playbook redeploys the OpenShift Container Platform CA
certificate by generating a new CA certificate and distributing an updated
bundle to all components including client kubeconfig files and the node’s
database of trusted CAs (the CA-trust).

This also includes serial restarts of:

master services

node services

docker

Additionally, you can specify a
custom CA certificate when redeploying certificates instead of relying on a CA
generated by OpenShift Container Platform.

When the master services are restarted, the registry and routers can continue to
communicate with the master without being redeployed because the master’s
serving certificate is the same, and the CA the registry and routers have are
still valid.

To redeploy a newly generated or custom CA:

Optionally, specify a custom CA. The certfile that you specify as part of
the custom CA parameter, openshift_master_ca_certificate, must contain only
the single certificate that signs the OpenShift Container Platform certificates. If you have
intermediate certificates in your chain, you must bundle them into a different
file.

To specify a CA without intermediate certificates, set the following variable
in your inventory file:

# Configure custom ca certificate
# NOTE: CA certificate will not be replaced with existing clusters.
# This option may only be specified when creating a new cluster or
# when redeploying cluster certificates with the redeploy-certificates
# playbook.
openshift_master_ca_certificate={'certfile': '</path/to/ca.crt>', 'keyfile': '</path/to/ca.key>'}

To specify a CA certificate that is issued by an intermediate CA:

Create a bundled certificate that contains the full chain of intermediate
and root certificates for the CA:

# Configure custom ca certificate
# NOTE: CA certificate will not be replaced with existing clusters.
# This option may only be specified when creating a new cluster or
# when redeploying cluster certificates with the redeploy-certificates
# playbook.
openshift_master_ca_certificate={'certfile': '</path/to/ca.crt>', 'keyfile': '</path/to/ca.key>'}
openshift_additional_ca=intermediate/certs/ca-chain.cert.pem

After running this playbook, you must regenerate any service signing certificate or key pairs
by deleting existing secrets that contain service serving certificates or removing and re-adding
annotations to appropriate services.

Redeploying Registry or Router Certificates Only

The openshift-hosted/redeploy-registry-certificates.yml and
openshift-hosted/redeploy-router-certificates.yml playbooks replace installer-created
certificates for the registry and router. If custom certificates are in use for
these components, see
Redeploying Custom
Registry or Router Certificates to replace them manually.

Redeploying Registry Certificates Only

To redeploy registry certificates, run the following playbook, specifying your
inventory file:

Redeploying Custom Registry or Router Certificates

When nodes are evacuated due to a redeployed CA, registry and router pods are
restarted. If the registry and router certificates were not also redeployed with
the new CA, this can cause outages because they cannot reach the masters using
their old certificates.

The playbooks for redeploying certificates cannot redeploy custom registry or
router certificates, so to address this issue, you can manually redeploy the
registry and router certificates.

Redeploying Registry Certificates Manually

To redeploy registry certificates manually, you must add new registry
certificates to a secret named registry-certificates, then redeploy the
registry:

Switch to the default project for the remainder of these steps:

$ oc project default

If your registry was initially created on OpenShift Container Platform 3.1 or earlier, it may
still be using environment variables to store certificates (which has been
deprecated in favor of using secrets).

Run the following and look for the
OPENSHIFT_CA_DATA, OPENSHIFT_CERT_DATA, OPENSHIFT_KEY_DATA environment
variables:

$ oc env dc/docker-registry --list

If they do not exist, skip this step. If they do, create the following ClusterRoleBinding:

Redeploying Router Certificates Manually

To redeploy router certificates manually, you must add new router certificates to a secret named router-certs, then redeploy the router:

Switch to the default project for the remainder of these steps:

$ oc project default

If your router was initially created on OpenShift Container Platform 3.1 or earlier, it might
still use environment variables to store certificates, which has been
deprecated in favor of using service serving certificate secret.

Run the following command and look for the
OPENSHIFT_CA_DATA, OPENSHIFT_CERT_DATA, OPENSHIFT_KEY_DATA environment
variables:

router.pem is the file that contains the concatenation of the
certificates that you generated.

Redeploy the router:

$ oc rollout latest dc/router

When routers are initially deployed, an annotation is added to the router’s
service that automatically creates a
service serving certificate secret named router-metrics-tls.

To redeploy router-metrics-tls certificates manually, that service serving certificate can be triggered to be recreated by deleting the secret, removing and re-adding annotations to the router service, then redeploying the router-metrics-tls secret: