How To Read Dumps – ESX Crash Dumps That Is

by bunchc on February 11, 2009

About thirty years ago in the jungle in South Korea I was spending some time living as a monk. One of the things I learned from these monks, was the ancient art of Dump reading. Yes! That’s right, I can tell the future by reading the finer texture and smell of a dump.

Ok, while not true (I’m naught by 26) and I can’t tell the future by reading dumps. I can tell you, however, that reading ESX dumps would be conducive to your future.

What Makes A Dump?

Lots and lots of fiber in your diet. That… and PSOD’s (Purple Screens of Death). They’ll generate an ESX kernel dump and drop a crash dump file into the /root/ directory, named something like: ‘vmkernel-zdump-<reversed date>.#.#.#’

This file is created on the first reboot following your psod and is generated from the contents of your VMKCORE partition, you did make a VMKCORE partition, right? It’s the one labeled ‘fc’. Can’t find it? Sure? Did you look in your sock drawer? Ok… well in that case “vmkdump -d /dev/sda5″ where /dev/sda5 is the output from esxcfg-dumppart -l

I Have My Dump, Now What?

So you can do a few things. First is to generate a support bundle and send it off to VMware for analysis (which you should do anyways). However, if you’re like me, and can’t wait, from the service console you can do the following:

The last instruction was E1000PollTxRing then E1000_PollRings then BH_Check then VMKCall and finally VMKVMMEnterVMKernel

Based on the name of the last instruction, this host probably crashed due to some type of packet or frame corruption in the Intel E1000 driver in the VM that was running with world ID 1169 in vmm0 named ‘notthemama’.

Thanks for playing along. If you have questions hit me up in the comments or on twitter @cody_bunch