Managing VMware vSphere environment with Go and Lua by using Gru orchestration framework

If you haven’t heard about Gru yet - Gru is a concurrent
orchestration and configuration management framework written
in Go and Lua
as the DSL language,
which allows you to manage your environment in an idempotent manner.

You can read more about the project at
Github, where you can also find
various code examples and use cases.

In this post we will see how we can use Gru for automating the
management of VMware vSphere environment by using the recently
introduced features in the project.

The resources we will talk about in this post are Datacenter,
Cluster, ClusterHost, Host, DatastoreNfs and VirtualMachine.

In order to keep things simple and not repeat ourselves we will first
create a Lua table, which will
contain the credentials used to authenticate against the remote
VMware vSphere API endpoint.

---- The credentials to the remote VMware vSphere API endpoint--credentials={username="root",password="r00tp4ssw0rd",endpoint="https://vc01.example.org/sdk",insecure=true,--> Needed if the vCenter is using a self-signed certificate}

The very first thing that we are going to automate is the creation of the
vSphere Datacenter. In order to do that we will use the Datacenter
resource as shown in the code below.

The next thing we need to take care of in our VMware vSphere
environment is the cluster. The code below takes care of
creating a vSphere Cluster and also enables the DRS service for it.

---- Manage the VMware vSphere Cluster--cluster=vsphere.cluster.new("MyCluster")cluster.endpoint=credentials.endpointcluster.username=credentials.usernamecluster.password=credentials.passwordcluster.insecure=credentials.insecurecluster.state="present"cluster.path="/MyDatacenter/host"cluster.config={enable_drs=true,drs_behavior="fullyAutomated",}cluster.require={dc:ID()}--> The cluster depends on the datacentercatalog:add(cluster)

One thing to note in the code above is that the Cluster resource
depends on the Datacenter resource we’ve created previously, which
will ensure proper evaluation and processing of resources in the resulting
resources DAG graph.

Once we have the cluster ready we can now add some
compute resources to it by adding an ESXi host to the cluster.

The code below shows how to add an ESXi host a vSphere cluster.

---- Add an ESXi host to the VMware vSphere Cluster--host=vsphere.cluster_host.new("esxi01.example.org")host.endpoint=credentials.endpointhost.username=credentials.usernamehost.password=credentials.passwordhost.insecure=credentials.insecurehost.state="present"host.path="/MyDatacenter/host/MyCluster"host.esxi_username="root"host.esxi_password="esxi_p4ssw0rd"host.ssl_thumbprint="ssl-thumbprint-of-host"host.require={cluster:ID()}--> The ESXi host depends on the clustercatalog:add(host)

Make sure to provide the correct SSL thumbprint of your ESXi host.
This post in virtualGhetto
provides details on how to extract the SSL thumbprint of an
ESXi host.

Gru also provides a resource for managing various configuration
settings of ESXi hosts such as DNS and lockdown mode settings.
For more information about the various settings that you can
manage on ESXi hosts make sure to check the
Host
resource.

Another resource that we will use is the DatastoreNfs one,
which allows you to manage NFS datastores on ESXi hosts.

The following example shows how to mount an NFS datastore on our
ESXi host.

---- Mount an NFS datastore on our ESXi host--datastore=vsphere.datastore_nfs.new("vm-storage01")datastore.endpoint=credentials.endpointdatastore.username=credentials.usernamedatastore.password=credentials.passworddatastore.insecure=credentials.insecuredatastore.state="present"datastore.path="/MyDatacenter/datastore"datastore.hosts={"/MyDatacenter/host/MyCluster/esxi01.example.org",}datastore.nfs_server="nfs01.example.org"datastore.nfs_path="/storage/vm-storage01"datastore.mode="readWrite"datastore.require={host:ID()}--> The datastore depends on the ESXi hostcatalog:add(datastore)

The last resource that we will see in this post is the
VirtualMachine resource.

Using the Lua-based DSL of the project we will create a few
Virtual Machines.

---- Manage VMware vSphere Virtual Machines--names={"kevin","bob","stuart"}--> You know these guys, right?for_,nameinipairs(names)dovm=vsphere.vm.new(name)vm.endpoint=credentials.endpointvm.username=credentials.usernamevm.password=credentials.passwordvm.insecure=credentials.insecurevm.state="present"vm.path="/MyDatacenter/vm"vm.pool="/MyDatacenter/host/MyCluster"vm.datastore="/MyDatacenter/datastore/vm-storage01"vm.wait_for_ip=truevm.power_state="poweredOn"vm.template_config={use="/MyDatacenter/vm/Templates/centos-7-x86-64-template",}vm.require={host:ID(),datastore:ID()}--> The VM depends on the ESXi host and datastorecatalog:add(vm)end

Before we run the code we have written so far, let us first have a
look at the resulting resource dependency graph.

In order to do that we will use the gructl graph command, e.g.

$ gructl graph site/code/vsphere.lua | dot -Tpng-O

The resulting DAG graph looks like this.

What is worth mentioning about Gru is that it is good in executing
operations concurrently. Looking at the above dependency graph we can
see that if all dependencies are satisfied by the time we reach the
VirtualMachine resources then we could actually create our
Virtual Machines concurrently.

Being able to execute operations like this in concurrent way can
greatly reduce the time required for standing up a new
environment.

What is interesting to note in the output above is that Gru is
actually creating the Virtual Machines concurrently.

Since all dependencies have been satisfied by the time we have
reached the Virtual Machine resources and the VirtualMachine
resource type itself is concurrent this allows Gru to schedule such
operations for concurrent execution.

This example shows one of the key features of the project - it’s
ability to execute operations concurrently.

Hopefully this introduction gave you a nice overview on how to start
managing your virtual infrastructure using Gru.

Make sure to also check the
project at Github for more
information and other code examples.