Building Hashicorp Vault in OCI - Part II

Building Consul in OCI

Now that we have defined the IAM and network resources that Vault depends on, it's time to start building Consul nodes, which we will use as the backend for Vault.

In order to build Consul, and completely automate the bootstrap, we will take advantage of some OCI and Terraform features. In OCI, we will leverage the internal DNS so that we can render a static configuration file with the hostnames for the cluster pre-populated. In order for this to work, the dns_label must be defined at both the VCN and subnet levels, and in addition, you must set the hostname_label on the oci_core_instance resource. All of this together allows us to leverage the OCI internal DNS service, and resolve the hostnames for the consul nodes as: $instance_hostname_label.$subnet_dns_label.$vcn_dns_label.oraclevcn.com

In addition to using these attributes for our resources in our VCN, we need to do some standard configuration and open the necessary ports in both the system's firewall, and in the seclist definitions for our VCN. Finally, we also need to define the Consul configuration to leverage these attributes.

These include DNS and UI access, but the default behavior is to bind these services to the loopback, so they are not actually accessible by default. They are included in the seclist for convenience.

Create Nodes

To build consul, we need somewhere to put it. Based on the recommended architecture, we will use 5 Consul servers spread across 3 subnets as our storage backend for Vault. We'll define a map variable to allow us to spread the servers in the desired placement while doing a simple count = 5 for the oci_core_instance resource.

Here's the Terraform config for the Consul servers. I leverage the output from the remote state for network and common to get the subnet and compartment, respectively. Then we define some metadata, including defining ssh_authorized_keys (so we can log in) and the user-data, which is how we will configure the Consul cluster. I also defined some freeform tags, which we can use to label and organize our resources within OCI. I also added some outputs for convenience (the OCID of the instances and public IPs).

Cloud Init

We'll use cloud-config-data style cloud-init, passed to the instances via user-data for the actual configuration of the instances.

There is a lot going on here. This userdata creates the consul user and group, writes the config files we've embedded in base64, and then runs a series of commands. These commands configure the consul user (some parameters are not available via cloud-config-data) install consul, ensure correct permissions, configure the system firewall, and finally register, enable and start the consul service unit via systemd.

This is where we are leveraging the internal DNS; the retry_join parameter in the config file is using a list of internal DNS names based on the $instance_hostname_label.$subnet_dns_label.$vcn_dns_label.oraclevcn.com hostname specification discussed earlier. This, along with bootstrap_expect = 5 parameter allows the nodes to discover each other and bootstrap the cluster without any user interaction at all.

While we have ui = true defined, the default behavior is to bind on the client service address, which defaults to 127.0.0.1. This means they are not accessible without establishing a tunnel to the host (we're going to use this for Vault). For convenience, they can be enabled by appending the following to /etc/consul.d/consul.hcl and restarting the service once the nodes are created:

"client_addr": "0.0.0.0"

This only really would need to be done on one node, if you wanted to check out the UI for debugging or play around with the DNS interface, which is why I left it unexposed by default (again, using this to store secrets).

This will be refined in later posts in this series.

Summary

All of this adds up to another simple terraform apply, run from within the vault directory, which deploys the nodes we will use for Consul. The cloud-config-data we pass in user-data then configures the nodes post-boot, and as the nodes configure themselves and install consul, they ultimately discover each other and form the Consul cluster we will use as our Vault storage backend.