Extracting Windows VM crash dumps

This post aims to provides an overview of the many options for extracting Windows crash dump files and their equivalents out of Windows virtual machines. The assumption is that a Windows instance has just crashed with a Blue Screen Of Death (BSOD) and a dump is deemed helpful to diagnose the issue. Note that most of the tips below apply to physical machines as well but the examples here focus on a QEMU/KVM virtual machine.

1. Copy MEMORY.DMP after reboot

This is straightforward as long as MEMORY.DMP has actually been generated and there is a simple way to copy it out of the VM. Note that when Windows is showing the BSOD screen and “dumping memory to disk”, it is not creating a crash dump file yet. To increase the chances that the memory dump file will be persisted, Windows writes it in the page file first. Then, on first reboot, it tries to extract it into a separate file, usually %SystemRoot%\MEMORY.DMP.

A few things can go wrong here. First, if the system got super messed up by whatever caused the crash, the VM may not be able to boot again. No boot means no MEMORY.DMP and we’re out of luck. There may also not be enough free disk space to create the crash dump. This would, again, mean no MEMORY.DMP for us. Note that freeing disk space after the fact won’t help as the dump in the page file is already partially overwritten by the time the user gets to do anything about the disk space situation. Unless of course the disk can be accessed offline from the host or another VM. If you’re willing to do this, it is not necessary to reboot though (see below). Here’s an older blog post with a detailed description of the Windows behavior and useful tips.

2. Extract the page file

Little known is the fact that the page file itself can be fed into windbg and used in lieu of MEMORY.DMP (thanks to Vadim Rozenfeld for enlightening me!). The page file will be larger than MEMORY.DMP but it doesn’t require the reboot mentioned above. A Windows VM crashed and doesn’t boot back up? No problem, just extract the page file:

Note that you have to know or guess (or let Volatility guess) the version of Windows running in the VM, sometimes down to the Service Pack / build level. If all works well, the resulting MEMORY.DMP can be opened in windbg as usual. You just won’t see any bug check code and BSOD parameters (duh!) and probably no context either.

4. Push dump data out of the VM programmatically

This is advanced and more of a note to self, as I hope to turn it into a tool at some point. Windows offers two kernel APIs of interest.

KeRegisterBugCheckReasonCallback with KbCallbackDumpIo
In theory, this should allow a kernel driver to be invoked by Windows as the BSOD is happening and memory is being dumped to the page file. The driver is given a pointer to the memory blocks being dumped and can push this data out of the VM using, for instance, a virtual serial port. Quite understandably, the callback code running on BSOD is very limited in what it can do. Memory allocations are obviously out of the question as is everything else that might end up allocating. What’s worse is that it’s not clear if the addresses passed to the callback are virtual or physical. The first part of the dump (header) is indicated with virtual addresses. The second part (body) starts with a few virtual pages and is then followed by physical pages indicated with physical addresses. Secondary data is again virtual. I might be missing something but apparently I’m not the only one observing this. As it stands now, this API is not usable without ugly and fragile heuristics.

KeInitializeCrashDumpHeader
This call produces the crash dump header, a 4096 byte data structure at offset 0 of MEMORY.DMP. And it works, as long as the physical memory layout that Windows works with is identical to that of whoever is producing the actual physical memory data to be appended to the header. Which sadly doesn’t seem to be the case with QEMU’s dump-guest-memory. So on one hand, having a valid and correct header saves us from the guesswork that Volatility has to do (no need to mess with Volatility “profiles”). But it’s still necessary to at least understand, or better patch the header to adapt it to the guest memory dump. Here’s a partial description of the dump header layout: computer.forensikblog.de/en/2006/03/dmp-file-structure.html.

Advertisements

Share this:

Like this:

LikeLoading...

Published by ladipro

Ladi Prosek is a software engineer at Red Hat, working on virtualization and focusing primarily on KVM/QEMU Windows guest support. He can be contacted at lprosek <at> redhat.com.
View all posts by ladipro