Advanced VMware vSphere Template Orchestration

Template management is a critical facet of infrastructure management and traditionally one of the more challenging operations there is. The advent of tools like HashiCorp Packer have provided administrators with the ability declaratively automate the template creation process.

Automation is great but in most organizations a single piece of automation is part of a larger process that includes tickets and potentially other hand offs.

In this example we’ll walk through creating a process for creating VMware vSphere machine templates in a fully automated fashion with deep integrations with other tools for documenting and testing the created template.

The code for this example can be found in the following GitHub repositories.

Tools

StackStorm is an open source orchestration tool with numerous integrations with third-party tools for simplifying the creation of complex orchestration workflows. StackStorm will drive the orchestration in this example and handle all of the hand offs.

Jenkins is a continuous integration (CI) server typically used for testing and building code. In our example Jenkins will be used for validating our deployed code. This shows how the workflow could incorporate tests that may already exist that are created by the applications developers or QA team. This helps to reduce potential issues created by base template updates

Terraform is used to provision the virtual machine that is used for validating that the new template doesn’t break any downstream applications.

Stages

The high level stages of the orchestration displayed in the image below are detailed below along with the StackStorm code for each stage.

Create Incident

This first stage of the workflow is to create an incident in ServiceNow for the new template. The goal of this stage is integrating with ServiceNow to provide a record of what is being done as part of the automated template creation process. The StackStorm code below shows that we’re creating a new record in the incident table and providing some metadata about the incident such as the description and the submitter.

Once the record has been created in the incident table we are storing the sysid of the ServiceNow incident in the incident_id variable for use later when we resolve the incident. Additionally we’re specifying the next stage of the workflow which cleaning up any existing clones of our code git repo.

The screenshot below displays a new incident created by the orchestration to track the work performed during this process and align with change management processes.

Clone Github Repo

The second stage of the workflow cleans up any existing copy of the Github repository and clones the Github repository that contains the Packer template code as well as the Terraform code used for testing the newly created template.

Build Template

The third stage of the workflow builds the VMware vSphere template from an ISO image using HashiCorp Packer and the Jetbrains vSphere Packer plugin.

The Packer JSON file is hosted in the cloned GitHub repository and the variable file that contains sensitive information is stored on the filesystem of the StackStorm host. This data could be retrieved in a more secure fashion such as retrieved from HashiCorp Vault or StackStorm’s encrypted data storage.

Provision Test Stack

The fourth stage of the workflow provisions a VMware vSphere virtual machine based on the previously created template and configures it using Puppet via the Terraform Puppet Provisioner.

In a real world example this might be an entire application stack owned by another team that consumes the template. This provides tighter integration to help prevent breaking changes from being propogated.

Validate Test Stack

The fifth stage of the workflow is run some validation tests against the stack. These tests might be tests provided by teams that own applications that are built on the base template. In this example StackStorm executes a defined job in Jenkins that runs some ServerSpec tests.

The screenshot below displays the console output of the Jenkins job that was run to perform the validation of the test virtual machine provisioned by Terraform.

Destroy Test Stack

The sixth stage of the workflow is to tear down the virtual machine used for testing the updates to the base template. Since Terraform was used to provision the virtual machine we’ll just run a Terraform Destroy to tear it down.

Fetch Template Metadata

The seventh stage of the workflow is to fetch the metadata for the template such as the hardware specifications to input them into the CMDB in ServiceNow. The Packer JSON is just being outputted via the cat command but a custom script for parsing file or fetching the data directly from vCenter would provide a more robust solution. The output is being stored in the template_data variable.

Create CMDB Entry

The eighth stage of the workflow is creating a record in the CMDB for the new vSphere template. The payload contains metadata about the template that we defined in the previous stage and made available to this stage with the publish declaration.

The screenshot below displays the CMDB record for the template created by the orchestration.

Close Incident

The nineth and final stage of the workflow is to close or resolve the incident in ServiceNow. This stage closes the loop on the change in ServiceNow with the template successfully being created. In the example we’re passing the incident_id variable that was returned from the very first stage in the workflow along with providing the state of “6” which is resolved and the required fields of code and notes.