vmtoolsd

I was fortunate to have been given early access to the VMFork (Instant Clone) PowerCLI cmdlets to help provide early feedback and usability improvements before it was released to customers. Having spent some time with the Fling, I have learned a thing or two about how Instant Cloning works and some of the caveats or gotchas while creating the customization scripts that are used as part of the Instant Clone workflows. I wanted to put together a quick reference on some of my findings as well as well as other recommendations from Engineering who have worked closely with the Instant Clone feature.

The idea is to have this as a living document which I will update as new tips and tricks are identified.

Best Practices

Ensure VMware Tools is installed inside the guestOS and also good time to ensure you are running the latest

Both Pre/Post Customization scripts are uploaded to /var/tmp by the Enable-InstantCloneVM cmdlet

Do not delete Child VMs directly on ESXi, manage it through vCenter Server. There is currently a known issue in which deleting Child VMs will also delete the Parent VM's disk

Additional custom variables can be passed to the post-customization script by adding to the -ConfigParams array of variables.

An example could be passing in two custom properties called "foo" and "bar" which would look like:

@{foo = "val1";bar ="val2"}

To retrieve the variable "foo" and "bar" from within the post-customization script, you would issue the following commands:

A Forked Child VM will also have a duplicate MAC Address which needs to be updated as it is not automatically picked up.

You can either manually set it by retrieving the guestinfo.fork.ethernet0.address with the post-customization script.

An easier way would be to reload it based on the guestOS type. On a Linux system, you can use the modprobe command like the following (Submited by George Hicken):

modprobe -r vmxnet3;modprobe vmxnet3

A Forked Child VM may also have identical kernel entropy pools which means semi-predictable RNG, possibly including TCP sequence numbers (Submited by George Hicken)

A Forked Child VM's system clock may also be out of date (until you call hwclock --hctosys or similar) which can cause problems with ordering of file timestamps (Submited by George Hicken)

Shared host keys if you are using a PKI system or identical asset identifiers in the case of Windows and any sort of AD infrastructure would also need to be either removed prior or updated after a Child VM is created (Submited by George Hicken)

Instant Cloning Nested ESXi has been a bit tricky due to a known issue with the VMware Tools for Nested ESXi. I have found that manually preparing the guest prior to Instant Cloning has yield better results. For more information on how to Instant Clone Nested ESXi, check out the blog post here

Powering off the Parent VM means that the VM is no longer quiesced and this also means that new Child VMs can not be instantiated until all existing Forked Child VMs have been powered off and the Parent VM has been re-quiesced

If you plan on downloading or installing additional software packages on the Parent VM, it is recommended that you perform that operation directly in the VM and not within the pre-customization script. I have noticed that if pre-customization takes too long, the quiesce operation eventually fails even though the operations within the pre-customization script executed successfully.

To ensure Forked Child VMs do not contain duplicate disk ID's from Parent VM such as setting up a VSAN environment using Instant Clone Nested ESXi, add the disks after Forked Child VMs have been created.

When you hard reset or power off on a child VM it will respawn from the parent, soft resets will not respawn (Submitted by Alan Renouf)

Troubleshooting

Instant Clone guestOS logs are stored in /var/tmp/quiesce.log

Consider enabling tracing within your customization scripts. An example of this for a shell script is using

set -x

Add additional echo or print statements like Start/Stop of certain sections like Pre/Post which can aide in reviewing the Instant Clone logs as seen in the screenshot above

For Instant Cloning Nested ESXi guestOSes, I recommend taking a snapshot after you have prepared the guest and removed any system specific information. This allows you to quickly revert back to a known state for ease of debugging. I found this to be very useful to be able to start back a known clean state while developing the customization scripts for Instant Cloning Nested ESXi

A known issue that is mentioned in the documentation of the Instant Clone cmdlets is after enabling a ParentVM for Instant Cloning, is that it is no longer available for migration to another ESXi host. The reason for this is that after powering off the VM, the "parentEnabled" boolean flag is still set to "true" which prevents the migration. Currently, there is not a work around but hopefully this will be resolved in a future update of the cmdlets. You can see this by running the following PowerCLI snippet:

I have received several questions about this in the last couple of weeks regarding the process of upgrading VMware Tools for running Nested ESXi 5.x and 6.0 when the physical ESXi host has been upgraded to ESXi 6.0. Instead of individual replies, I thought I would share this quick tip. First off, VMware Tools for Nested ESXi provides a very specific set of capabilities for Nested ESXi guests as shown below:

Allows the nested ESXi VM to be cleanly shut down or restarted when performing power operations with the vSphere Web/C# Client or vSphere APIs.

Executes scripts that help automate ESXi guest OS operations when the guest’s power state changes.

Supports the Guest Operations API (formally known as the VIX API).

Unlike traditional VMware Tools which may provide updated capabilities with each new release, VMware Tools for Nested ESXi exposes only a subset of those capabilities which has not changed between ESXi 5.x and 6.0. This is an important fact to be aware of because you may see "Unsupported older version" for the VMware Tools status in the vSphere Web/C# Client and this is perfectly fine and expected.

Here is a screenshot of a Nested ESXi 5.5 VM with VMware Tools installed running on top of an upgraded physical ESXi 6.0 host:

In this scenario, the VMware Tools status will be reported as "Unsupported older version" because the version of VMware Tools does not match the latest version of VMware Tools included with ESXi 6.0. However, you should not be alarm as the expected functionality listed above will continue to work without any problems and you can just ignore the UI warning. The only way to get rid of this warning is to upgrade the Nested ESXi VM to ESXi 6.0 which I go over in more details below. I know upgrading may not be an option if you still wish to run ESXi 5.x, but as far as I know, there will not be an update to VMware Tools VIB for ESXi 6.0 as it is now pre-installed with ESXi 6.0.

Here is a screenshot of the same VM which has now been upgraded to ESXi 6.0 running on top of an upgraded physical ESXi 6.0 host:

In this case, the VMware Tools status will be reported as "Unsupported older version" because the version of VMware Tools does not match the latest version of VMware Tools included with ESXi 6.0. However, because VMware Tools now comes pre-installed with ESXi 6.0. We can easily remedy this by removing the VMware Tools VIB we installed for ESXi 5.x by running the following ESXCLI command and then rebooting:

esxcli software vib remove -n esx-tools-for-esxi

Once the ESXi host has rebooted, the VMware Tools that is pre-installed with ESXi 6.0 will automatically start up if it detects it is running as a VM. If you now look at your vSphere Web/C# Client, you will see that VMware Tools status shows current and is also the default behavior if you are running Nested ESXi 6.0 VM on top of physical ESXi 6.0 host.

With VMware Tools being pre-installed with ESXi 6.0 and only loaded when it detects it is being run as a VM, you no longer need to worry about manually installing additional VIBs get the benefits of having VMware Tools installed for your Nested ESXi VMs.

I recently noticed a trend of questions from various users about extracting specific bits of information such as the version of ESXi that's running or the MoRef ID of the VM, all while within the guestOS. I had already written an article about this topic awhile back called How to extract host information from within a VM? but it seems like there is still a continued interest to easily obtain this information about the underlying vSphere infrastructure from within the guestOS.

I came across another method while researching a different topic which is to use the vApp property of a VM called the OVF runtime environment. This is a feature that has been around since the early days of vSphere 4.x, if not earlier which provides a mechanism to retrieve some of this information as well as custom properties. The OVF feature provides users with the capability to pass in any type of metadata information such as application start up parameters, network configuration, password management, etc. directly into the guestOS for flexible guest customization.

The OVF runtime environment is only available while the VM is powered and it can be accessed from either VMware Tools or from a special ISO that is mounted to the guestOS CD-ROM device. To enable the OVF runtime environment, you just need to perform these two simple steps:

2. Under the OVF Settings, specify the how you want to access the OVF environment by either VMware Tools or ISO Image transport:

If you are interested in enabling this using the vSphere API, take a look at the vAppConfig property. You can enable OVF runtime environment on a running VM, but the changes will not go into effect until the VM has been completely powered off and then powered back on, a guestOS reboot will not suffice.

After the settings have been applied, go back into the OVF Settings for a powered on VM and you now should be able to click on the "View" button which will show you the OVF runtime environment for that particular VM.

For a vanilla VM, you should see three basic things:

The vCenter MoRef of the VM (vApp property is only available with vCenter and not on a standalone ESXi host)

The hypervisor name and version

The network (Portgroup/Distributed Portgroup) the VM is connected to

To retrieve this same information from within the guestOS you have two options:

VMware Tools

Run the following command:

vmtoolsd --cmd "info-get guestinfo.ovfenv"

Below are two examples for both a Linux and Windows VM:

ISO Image

Mount the CD-ROM in the guestOS and inside you will find the ovf-env.xml file which contains the OVF runtime environment information.

As I mentioned earlier, you can also create custom key/value properties which can then be accessed by the guestOS VM. If you go to Advanced section and click on the Properties button, you will be able to add your own custom properties. Below is a screenshot of adding three customer properties called: application_owner, startup_option and system_owner just to give you an idea of the possibilities.

The OVF runtime environment does not just stop with vSphere, but it is also available to VMs running in vCloud Director. The only requirement is that vCloud Director VMs has the vApp options enabled at the vSphere layer as noted earlier. Below is a screenshot for a VM that has been imported into vCloud Director from vSphere and you will notice some additional "vCloud Director" specific properties.

Even though the OVF runtime environment makes it easier to retrieve some of these "underlying" infrastructure details, I am still interested in some of the use cases where you would need to know the hypervisor version or MoRef ID from within the guestOS. If you are currently requiring this or other bits of information, please leave a comment about your particular use case.

From time to time, I see this question come up asking how one might be able to extract a certain piece of information from either ESX(i) or the management APIs (vSphere API) from within a virtual machine. The simple answer is you can not, by default the guest operating system has no idea of the underlying hypervisor nor does it have the access to the management APIs. This of course, assumes you are following VMware's best practices in isolated and segregating off your management network from your virtual machine network.

Having said that, there are certain bits of information that you can extract about your ESX(i) host from within the guestOS using some of the utilities that is installed with VMware Tools. The first utility is called VMware Toolbox command which can be found on both UNIX/Linux and Windows systems that have tools installed.

UNIX/Linux - /usr/bin/vmware-toolbox-cmd

Windows - C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe

This utility provide some information about ESX(i) and guestOS configuration including basic resource statistics.

One interesting sub-command is the resource statistics about both ESX(i) and guestOS such as speed of the hypervisor's CPUs or the current balloon, swap, memory limit, memory reservations, cpu limit or cpu reservations.

Here is an example of retrieving ESX(i) CPU speed.

As I mentioned before, you do not have access to the management network that your ESX(i) are on and that also means you do not have access to the vSphere APIs. What if you wanted to to associate a specific piece of information from ESX(i) and be able to access that piece of information from the guestOS? You can do so with the vmtoolsd (VMware Tools Daemon) utility.

UNIX/Linux - /usr/bin/vmtoolsd

Windows - C:\Program Files\VMware\VMware Tools\vmtoolsd.exe

This utility has an option called --cmd that allows you to run various commands including one that allows you to extract guest information using the "info-get" parameter. This guestinfo can be set using two methods:

1) Hard-coded within the virtual machine's .vmx configuration file2) In VMX memory while a virtual machine is running within ESX(i)

Option 1 is pretty straight forward, you can add a custom attribute that has the following format:

guestinfo.[variable] = [value]

e.g.guestinfo.provision.date = "01/11/2011"

You can use the vSphere Client to add this custom attribute while the virtual machine is powered off or you can manually edit the .vmx configuration file. If you use the latter, you will need to reload the VM's configuration, you can use vim-cmd vmsvc/reload.[vmid] to do so. One the VM is powered on, you can extract this piece of information, here is an example:

Option 2 is actually pretty interesting as the configuration of the custom guestinfo variable is not permanently stored. What I mean by this is the configuration is only persisted while the VM is running. This is interesting because if you have runtime information that potentially could change, this allows you to dynamically present this information to the guestOS. One use case could be set for management VMs such vCenter running in a VM and you wanted to know at any given time which ESX(i) host is currently managing it. This can be trivial with vCenter access, but what if the service is offline but the virtual machine is still running? Using the classic ESX Service Console command vmware-cmd you can loop through all powered on VMs and set some custom variables for example the current hostname of the ESX host managing the VM and version it is currently on.

Note: Notice, you do not need to append the keyword guestinfo, you just need to specify the variable name. Once you are in the guestOS, to access the custom variable, you will need to specify the full name which should be "guestinfo.[VARIABLE]"

As you can see this can be tedious to set for each and every VM, so let's automate this using a script that will go through all powered on VMs and set two custom variables called "hypervisor.hostname" which will set the current hostname of the ESX host and "hypervisor.build" which will retrieve the build of your ESX. To further automate this since you might have multiple ESX hosts running in a DRS cluster, you would store this script in a shared storage (VMFS and/or NFS) and setting up a cronjob that would run at some interval to always provide the latest information to the guestOS.

Here is an example of a shell scrip that does exactly that:

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

#!/bin/bash

IFS=$'\n'

forVM in$(vmware-cmd-l);

do

VM_STATE=$(vmware-cmd"${VM}"getstate|awk-F"= "'{print $2}')

if["${VM_STATE}"=="on"];then

echo"Setting info for ${VM}"

vmware-cmd"${VM}"setguestinfo hypervisor.hostname"$(hostname)"

vmware-cmd"${VM}"setguestinfo hypervisor.build"$(vmware -v)"

fi

done

unset IFS

You will want to save this script to your ESX host and make sure the script has executable permissions for it to execute. You will also create a cronjob based on how often you want the script to run and this needs to be configured for every ESX host that you would like to take part in this update.

Here is an example of running the script every hour:

0 * * * * /vmfs/volume/shared-storage/setGuestInfo.sh > /tmp/out

Note: I redirect the output to /tmp/out to ensure that script is in fact working and once you have, you can remove that all together.

Here is an example of extracting these two custom variables on both a Linux and Windows VM:

As I mentioned earlier, option 2 does not persist these custom variables with respect to the VM's configuration file. The variable configuration only takes affect when the VM is powered on and once it is powered off, the information is discarded. To support ESXi, you can use the various vSphere SDKs (vSphere SDK for Perl, PowerCLI, etc) but what you will find is that the behavior of setting this attribute is similar to that of the local vmware-cmd on classic ESX, it does not append the entry into the .vmx configuration file.

Here is a vSphere SDK for Perl script called addVMAdvParamOption.pl that can be used to guestinfo parameter for a given VM. Here is an example of this script:

If you are using vMA, you can also use the vCLI's vmware-cmd utility to set this variable. The format is slightly different than that of the classic ESX's vmware-cmd. Here is an example of this:

Notice, the variable name must include "guestinfo." keyword before specifying the variable name.

If you wanted to persist these configurations, you will need to adjust your provision workflow to append the variables into the VM's .vmx configuration file.

Primary Sidebar

Search this website

Author

William Lam is a Staff Solutions Architect working in the VMware Cloud on AWS team within the Cloud Platform Business Unit (CPBU) at VMware. He focuses on Automation, Integration and Operation of the VMware Software Defined Datacenter (SDDC).