Recovering VMs after a vmware’s Purple Screen of Death (PSOD)

I had an interesting case a while ago. One of our test ESXI hosts running ESXi version 5.5 has crashed taking down number of test environments with it.

All the attempts to bring the host back to life was in vain, as each reboot was giving us a Purple screen of Death. We needed these test environments up and running ASAP, and due to time limit on hands, it was decided to :

keep the current VMFS datastore and install partition intact,

Install ESXI 5.5 from scratch onto a USB flash drive, and

Re-create the vSwitches

Re-import the VMs into inventory

Re-import and start up the vCenter

Login to vCenter and bring up the test environment back online

Luckily this test server- a Cisco UCS C220- had its CIMC enabled, and IP configured for remote access. So, I was able to connect to the hosts’s remote management panel (CIMC) and install the new Esxi via Virtual KVM.

Once the Esxi installer boot up and ran an initial scanning it recognized the previously used partitions, as well as the newly added Lexar USB flash drive , which we will be using as our new install base. USB flash drive I chose to use was formatted with FAT32, so I got a disk selection warning. After that the rest of the install goes as usual, choose a default language, choose a root password, and let the setup do its work. Once the ESXi installs successfully it’s time to reboot, and re-attach previously VMFS formatted drives containing our test Environment VMs. Don’t forget that you now have a new boot device, in our case it’s the Lexar USB drive, which we need to boot from at the moment .

After about 30 mins we are in back in business, once we are booted into the new environment Now let’s re-add the host back to vCenter server, and configure a minimal networking settings to move the VMs off to a better server. Browse the datastores and confirm that the VMs are visible and still there, it’s time to import and confirm that the VMS are bootable or not.

Select a VM, open the folder, right click on the vmx file, and “Add to inventory”. Now, there were several VMs each with several snapshots deep, and I was not able to bring them up, and did not have spare time on hand to spend on its recovery troubleshooting. So it’s not all so rosy when it comes to recovering snapshotted VMs after hosts PSOD.