How to automatically log all VM configuration changes using a vCenter Server Alarm?

If you followed my previous blog post on How to audit VM reconfigurations and see what exactly changed, you may have concluded that historical VM configuration events may potentially be unavailable by the time you need to perform an audit. The reason for this is that the VM Events tables may have rotated out depending on the retention policy of your vCenter Server Database. This is where we can take advantage of the powerful vCenter Server Alarm feature which would allows us to capture every single VM reconfiguration and store this information outside of the vCenter Server. This not only allows you to reduce the amount of data stored in the vCenter Server Database but it also allows you to efficiently archive this data into a data warehouse or Big Data platform that provides more advanced analytics and reporting capabilities.

Building on top of our previous script, I have created a slightly modified version called Get-VMConfigChangesFromAlarm.ps1. The main difference is instead of passing in a VM object to look for past configuration changes, the script can now pull in information from a triggered VmReconfigureEvent and log the specific changes associated with that VM reconfiguration. The way in which it does this is by querying the following two environmental variables which are set when the vCenter Server alarm is triggered and below is an example of their values:

Alarm Environmental Variable

Value

VMWARE_ALARM_TRIGGERINGSUMMARY

Event: VM reconfigured (9322)

VMWARE_ALARM_TARGET_ID

vm-125

In the first variable, what we care about is the eventId which is associated with the VM reconfiguration. What I have found is you have to take this ID and subtract 1 to get the event which actually contains the reconfiguration information that we want. The second variable provides us with the MoRef ID of the VM that was configured. Using these two pieces of information, we can then perform an event lookup to pull out the configuration changes that were made to that particular VM.

Here are the steps for creating the vCenter Server Alarm:

Step 1 - Ensure that the latest PowerCLI 6.0 Release 1 is installed on your vCenter Server. I will assume that this is how you wish to execute the PowerCLI script. If not, other options can include sending an SNMP trap to vRealize Orchestrator and have it perform the execution of the script.

Step 2 - Create a new Alarm and fill in the general settings as shown in the screenshots below.

Step 3 - Add the "VM reconfigured" trigger with the status set to "Unset", this will ensure you do not have a notification icon when the alarm is activated.

Step 4 - Select "Run a command" as the action and then paste the full path to where a "wrapper.bat" script will be located in the vCenter Server.

Step 5 - Create the "wrapper.bat" script on the vCenter Server system with the following (adjust the paths to fit your environment):

The following snippet will execute the PowerCLI script and any console output will be re-directed to the alarm.log file (which can be helpful in troubleshooting errors in the script itself).

Step 6 - Download the Get-VMConfigChangesFromAlarm.ps1 script and place it on the vCenter Server system and ensure it aligns with the path of the wrapper.bat script. You will also need to edit the script to update the vCenter Server credentials as well as the log file in which the VM configurations will be stored. Currently it will just append to a file called alarm.txt.

If everything was configured successfully, you can now test the alarm by simply editing one of your VMs. If you click under Monitor->Events for the VM, you should see the alarm being triggered and that the script was executed.

If we take a look at our alarm.txt file, we should hopefully see the VM reconfiguration details logged like the following:

Logging the output to a file is just one way of storing this data. I am sure some of you may want to modify the script to forward to a remote syslog collector like vRealize Log Insight for example or directly storing it into a Big Data platform. I will leave that as an exercise for my readers but hopefully this give you idea on how you can automatically archive all VM reconfigurations along with the what, when and who details for auditing and/or reporting purposes in the future.

Lastly, I wanted to give a big thanks to Jonathan Medd who helped me with a problem I was running into when trying to get a PowerCLI script to execute when using a vCenter Server Alarm. It turned out that I had the vCenter Server service configure to run as a local admin and switching it over to a Domain Account resolved my problem.

Haha sorry, was too early in the morning for me to post rational thoughts.

What I meant was: Based on your section that said: “Ensure that the latest PowerCLI 6.0 Release 1 is installed on your vCenter Server. I will assume that this is how you wish to execute the PowerCLI script. ”

Since we use the vcsa, we obviously can’t install powerCLI on it. Our only Windows machine is our VUM machine. How do we configure the alerts to use a machine other than vcenter itself to trigger scripts (since it looks like it uses C:\…. as it’s path). Unc path or something maybe?

1) Have a script which would then remote to a Windows system to run the PowerCLI script passing in the variables that would be parse from the alarms, this behavior is the same regardless of Windows VC/VCSA

2) Implement your own script that would run on the VCSA using something like pyvmomi (vSphere SDK for Python) or rbvmomi (vSphere SDK for Ruby) that does exactly what the PowerCLI script does which is reading the environmental variables and performing the query.

I’ve configured your scripts and I can make them work manually they look great! I do have a problem running them from the Alarm. I can debug out that it is receiving the env variables correctly. I can see the EventID is always +1 and I see your script subtracts 1 which is good probably because it sees this as two events. The change event and the alarm event. I can see that the powershell script runs and the values are correct, just not seeing any data $report. I’m not sure if the $tFilter.EventChainID is being set correctly not sure how to verify that. Just doesn’t seem to pull any data. The vCenter service is using a domain account so it looks good just doesn’t report anything unless I manually run it.

thanks for sharing this wonderful script.
But I’m curious when using Get-VMConfigChangesFromAlarm.ps1.
This script seems not able to log the Network Label changing event.
When I change a VM’s network label, the “Get-VMConfigChanges.ps1” logged the event normally.
Should I customize the script somewhere?

Dear William,
This is very useful. If I want to trigger something with VCSA when a VM configuration changes what should I do? Do you have a PERL equivalent of this to run in VCSA? Appreciate your guidance.

Primary Sidebar

Search this website

Author

William Lam is a Staff Solutions Architect working in the VMware Cloud on AWS team within the Cloud Platform Business Unit (CPBU) at VMware. He focuses on Automation, Integration and Operation of the VMware Software Defined Datacenter (SDDC).