KVM with kvmtool on ARMv7

In this third part about KVM on ARMv7 I use kvmtool as the user-space part of the hypervisor. This lightweight hypervisor allowed me to run up to 72 virtual machines… This does not really serves any purpose other than demonstrating what is possible :-). See the video in the end how that looked like.

In 2011 Pekka Enberg announced kvmtool (native Linux KVM tool). Initially it was meant to live within the Linux kernel source tree, but it ended up in a separate git repository on kernel.og.. In between it has been ported to several architectures too, including ARM and ARM64. Its binary usually goes with the name lkvm. It strictly depends on KVM and is otherwise kept rather lean.

Also like in the last post I built a guest root file system using Yocto/OpenEmbedded’s poky-tiny distribution and the qemuarm machine. During that process I found yet another issue with the generated Busybox’ inittab and initramfs. With this fix in place, and the changes to the qemuarm machine to use appropriate tuning and console (see below), the resulting cpio.gz ends up at around 700kB.

The Kernel has been built outside of OpenEmbedded using this defconfig. I tried to keep the configuration quite minimal. One difference to the Kernel I used for Qemu is that I also enabled 8250/16550 and compatible serial support (SERIAL_8250, SERIAL_8250_CONSOLE and SERIAL_OF_PLATFORM). kvmtool by default emulates a 16550A UART.

The --params option allows to pass parameters to the kernel, e.g. “console=ttyS0 debug” can be helpful to get the kernel boot log. The option --console virtio can be used to use a para-virtualized serial console (in this case user-space needs to start a console on /dev/hvc0).

kvmtool is quite neat and has lots of interesting options especially for development and testing. There is not a lot of documentation, but the help commands lkvm help and lkvm help COMMAND are quite usable.

Run many machines using kvmtool and KVM

This was my main purpose for trying out kvmtool: My hope is to run more virtual machines than I could do with Qemu in my previous post… In this experiment, I use lkvm with a serial console (the processes stdio) and start each lkvm instance in its own screen session:

Like I did when trying to start lots of machine using Qemu, I tried making use of KSM (Kernel Samepage Merging) again. Initially, enabling KSM did not take any effect at all. After digging a bit through kvmtool’s source code a bug turned up which did not annotate the pages correctly using madvise.

This video shows how I was able to start 72 VMs using kvmtool on a Toradex Colibri iMX7 module with 512MiB of RAM. Halfway in the video I enable KSM. After enabling KSM, the system needs a while to go through the pages and merge the same pages. So I ended up starting a bunch of machines and monitor the memory usage repeatably. One could probably automate the process using a script, but I left that as an exercise for somebody else :-). After starting the 73nd VM (at the very end of the video), the OOM (Out-Of-Memory) killer kicked in which somehow also locked up my ssh shell. The system was still accessible via serial console and the other 72 VMs continued to work fine: