Migrating a Key Trustee KMS Server Role Instance to a New Host

Note: This migration procedure applies only to the Key Trustee KMS Server, and does not apply to the
Key Trustee Server. Do not attempt this migration on the Key Trustee Server.
In some cases–for example, after upgrading your servers–it is desirable to migrate a Key Trustee KMS Server role instance to a new host. This procedure describes how to move a Key Trustee KMS proxy
service role instance from an existing cluster host to another cluster host. The security and performance requirements for the KMS proxy are based on providing a dedicated system to the role, and not
shared with CDH or other services. The KMS proxy represents a service that must be:

secure

isolated from non-administrator access

maintained as a system with a higher level of isolation and security requirements compared to other cluster nodes

Assumptions and Requirements

Warning: Do not attempt this migration procedure on any Key Trustee KMS version lower than 5.16.0.

The following assumptions and requirements apply during the migration of a Key Trustee KMS server role instance to a new host as described in this procedure:

Complete the steps one node at a time (migrate to the first new node, verify, then repeat the steps to migrate to second new node, verify, and so on).

The sequence of restarts indicated throughout the steps are critical to successfully completing the migration without data loss. Do not skip any of the
steps.

As required for any KMS service that is configured for HA, Zookeeper must be deployed as a service (true by default). Refer to Adding a Service for details about how to add services.

Review and examine TLS and Kerberos configuration requirements: the new KMS nodes must be ready with a Java Keystore and Truststore that present the correct host certificates while
also trusting the Key Trustee Server. If the custom Kerberos keytab retrieval script is in use for Kerberos integration, it is important to have those keytabs ready and ingested before proceeding.
Refer to Using a Custom Kerberos Keytab Retrieval Script for details.

For this use case/procedure, assume that the existing KMS proxy host instances are named:

Run the Add Role Instances wizard for the Key Trustee KMS service (Key Trustee KMS service > Actions > Add Role Instances).

Click Select hosts and select the checkbox for the host to which you are adding the new Key Management Server proxy service role instance. Click
OK and then Continue.

On the Review Changes page of the wizard, confirm the authorization code, organization name, and settings, and then click Finish.

Select and start the new KMS instance (Actions for Selected > Start).
Important:

The initial startup of the KMS instance may fail with the following error message:

java.io.IOException: Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.

If this occurs, it indicates that the KMS attempted to verify the Key Trustee private key has been synchronized with the new instance, but was unable to because that synchronization has not yet taken
place. This is expected behavior at this point in the process. Proceed to the next step, and the new KMS instance will come up when the KMS service is restarted after the synchronization.

Verify that a Kerberos HTTP principal has been created for that specific host role instance in the Security configuration interface (Administration > Security > Kerberos Credentials)).

For example, in this scenario the existing KMS Kerberos principal is HTTP/ktkms01.example.com@EXAMPLE.COM, and you must verify the new host principal
HTTP/ktkms03.example.com@EXAMPLE.COM has been created before continuing. If you cannot verify the new host principal, then click Generate Missing
Credentials on the Kerberos Credentials tab to generate it before continuing.

Synchronize the Key Trustee KMS private key. In this use case you log into the original working KMS instance host, which is ktkms01.example.com, and
synchronize the keys from your existing KMS host ktkms01.example.com to the new host to which you are migrating, ktkms03.example.com. Copy the private key over the network by running the following command as a privileged (root) user on the original Key Trustee KMS host:

Replace the host name (here we are using ktkms03.example.com) with the hostname of the Key Trustee KMS host to which you are migrating.
Warning: It is very important that you perform this step. If the KMS private key is not copied from the existing instance to the new instance,
then the encryption keys will be inaccessible when the old KMS instance is removed from the cluster.

To verify that the Key Trustee KMS private keys successfully synchronized, compare the MD5 hash of the private keys. On each Key Trustee KMS host, run the following command:

$ md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg

If the outputs are different, contact Cloudera Support for assistance. Do not attempt to synchronize existing keys. If you overwrite the private key and do not have a backup, any keys
encrypted by that private key are permanently inaccessible, and any data encrypted by those keys is permanently irretrievable. If you are configuring Key Trustee KMS high availability for the first
time, continue synchronizing the private keys.

Run the steps in Validating Hadoop Key Operations. Perform the check multiple times to exercise the
Load balanced KMS nodes properly. If a command fails during this test, stop immediately, halt the cluster, and contact Cloudera Support.

Re-run the steps in Validating Hadoop Key Operations. Perform the check multiple times to exercise the
Load balanced KMS nodes properly. If a command fails during this test, stop immediately, and halt the cluster.

Repeat these steps for any additional KMS node migrations you wish to perform. So in the use case shown here, we would repeat the steps to migrate the ktkms02.example.com host to the ktkms04.example.com host.

If this documentation includes code, including but not limited to, code examples, Cloudera makes this available to you under the terms of the Apache License, Version 2.0, including any required
notices. A copy of the Apache License Version 2.0 can be found here.