There are many companies shipping products based on ARMv8 today, including AMD, Samsung, NVIDIA, Cavium, Apple, Broadcom, Qualcomm, Huawei, and others. What makes ARM unique is the differentiation between vendors; while a majority of features are based on the ARM reference processor and bus architectures, additional custom features may be introduced or reference features may be omitted at the vendor’s discretion. This results in various profiles that are tailored to specific products, making it extremely difficult for another architecture to compete against such a wide selection of choices. As ARMv8 matures, it’s moving out of mobile and embedded devices into Industrial IoT (Internet of Things), network routers, wireless infrastructure, and eventually, the cloud.

For those of you not familiar with KVM, it stands for Kernel Virtual Machine. It’s a Linux Kernel hypervisor module. KVM is the primary open-source-based hypervisor, and is the default for Openstack, a popular cloud computing software platform that’s used to deploy a wide variety of services.

The Future Uses of ARM

ARM added hardware virtualization extensions first in the ARMv7 architecture, and also included them in ARMv8. For the most part, extensions for both architectures are compatible to the extent that ARMv7 and v8 share a lot of common ARM KVM code. Given that ARM is used extensively in the embedded world, this adds a whole new dimension to virtualization. One key application is Industrial IoT, where hardware is developed with future expansion in mind and typically only requires new software upgrades to add new features.

Virtualization is a perfect solution for such situations, and cloud provisioning of such devices would help with remote servicing, debugging, legacy support through emulation, and many other things. In particular, virtualization and ARM KVM are an ideal match given the majority of embedded devices run on Linux and ARM processors; additionally, with virtualization extensions, ARM KVM works well in embedded environments. This concept make so much sense that Intel, Freeescale and WindRriver have all presented their own vision of IoT device virtualization.

ARM has also continued making progress in the network hardware market. For example, Cavium produces processors based on ARMv8 for large, Software-Defined Networking (SDN) capable routers, and Broadcom is headed in that direction as well after unveiling a new ARMv8 processor. Currently, several features such as Unified Extensible Firmware Interface (UEFI), and Advanced Configuration and Power Interface (ACPI) support are in development for the server market. Prior to ARM-KVM, only proprietary micro-kernel virtualization vendors were available on ARM, and these were often complicated solutions with high levels of vendor lock in. Today, Linux runs on wide-ranging products from embedded devices to servers and now ARM KVM is reaching full maturity. A device with as little as 32MB of flash memory is enough to run several ARM-KVM guests.

What Makes ARM Virtualization so Great?

ARM hardware virtualization extensions span all processor features, and the primary focus is on limiting Guest exits, or at least minimizing exit intervals. The following list summarizes a few of the ARM virtualization extensions:

Second stage tables – Also known as nested page tables, these relieve the hypervisor from managing the guest shadow page tables in the software.

Clock source and timer virtualization – Allows the guest to schedule its own timers and directly access a clock source. The OS uses this to measure the passage of time or to schedule timer events. This is one of several areas where ARM has improved over x86.

Event Trapping – Programs the CPU events that should be trapped to the hypervisor. For example, access control to privileged registers can be used to prevent the guest from reprogramming the host registers.

Exit reason and instruction decoding on Guest exit – The former accelerates resolving of all exits, and the latter is typically used for accelerating Guest IO accesses.

ARM has taken a completely different approach to managing the Guest and Host context. As shown below, on the x86 architecture, a Virtual Machine Control Structure (VMCS) is implemented which, among other important fields, contains the Guest and Host state. On entry to a Guest, the Host state is saved and the Guest is restored; on an exit, the opposite occurs.

ARM, on the other hand, has decided to add a HYP mode and an exception level 2 for ARMv7 and v8 respectively. This extra mode is more privileged than the secure mode, but less privileged than the KVM Host SVC or EL1 mode. With HYP mode, transition to and from the Guest and Host occurs through this extra mode. For a world switch (guest to hypervisor context switch) the approach of x86 and ARM differs considerably:

ARM (left) has improved how guest exits are handled considerably over other approaches like x86 VMCS.

ARM performs a VM enter world switch in software, saves Host context on HYP stack and restores Guest from KVM vCPU structure, and the reverse on a VM exit. This allows future software optimization based on the features a processor supports.

It’s quite possible that some exits can be handled in HYP mode; this is an ultra-light, inexpensive exit. In fact, some benchmarks on the KVM-ARM mailing list show millions of such exits with no apparent performance degradation. The key is that in HYP mode, only a portion of the state is saved to handle the exit and resume the Guest. An example is discussed in this article, which shows this behavior while running massive compilations.

ARM Virtualization Comes of Age

Over the past several months the open source community has made great progress in enabling ARM KVM for various markets and closing the gap with x86. The following are some key features that have been pushed upstream:

Vitual IO (virtio) – The most widely used Input/Output (I/O) in cloud environments. ARM supports virtio block, network, and memory balloon devices based on virtio-mmio transport (as opposed to virtio-pci). Virtio-pci is the next step once a reference a PCI controller that supports PCI cycles has been developed.

Live, cold migration support, and image save/restore – These features are essential for zero downtime maintenance, high availability, and load balancing. This applies to all environments including embedded devices, cloud, and specific NFV elements.

Transparent huge pages – This is essential for near native performance. While it’s been supported for sometime now, during live migration a new feature is introduced to dissolve huge pages during the migration and coalesce after the migration completes. This enables the rapid migration of memory loads that were previously not possible.

GDB debug – This is for debugging the guest.

Additionally, the community is hard at work to support UEFI, ACPI, and GUID Partition Table (GPT) partitioned image boot. There is also ongoing work on Libvirt to support ARM KVM machine model, as well as additional Openstack integration. Finally, future work requires support for hardware profiling support, in the Guest and Host simultaneously, as well as Asynchronous Page Fault support which allows the Guest to execute another thread during a second stage page fault.

We’ll continue to follow these developments, so stay tuned for future articles that focus on our community activities and ARM KVM adoption as the technology continues to mature.

Share this:

Author: Mario Smarduch

Mario has been involved with Linux Kernel for past 15 years and virtualization for past 5 years. His work has been mostly on the product side, interfacing with Open Source communities as needed.
View all posts by Mario Smarduch

Hi Mario
When I was searched for kernel virtualization, i got your blog in google.
I am interested to learn how virtualization is implemented in linux kernel..
Several times searched internet but no success on clarity.
How page tables are maintained , for para virtualization and Hw asited paging.. If you don’t mind , please let me know

Hi Sameer,
it’s a hard topic to find information on – you’ll find a lot of admin/cloud stuff but little on this. Most of the material is buried in QEMU (the device emulation part), hw extensions – cpu, interrupt, io – hw manual pages, KVM which is the linux kernel hypervisor and surrounding software like bootloader, device tree/acpi.

Covering ‘how virtualization is implemented in linux kernel’ is quite a job – it pulls in QEMU, HW extensions – cpu/interrupt/io – and that would require a lot of blogs.

Regarding paging – in short KVM keeps track of 2nd stage or nested page tables for whole VM Guest Physical Address (aka as Intermediatte Physical Address – IPA). Guest physical acesses resolve to a fake IPA range defined for the machine model guest is compiled against, 2nd stage faults are resolved to host physical pages by KVM. There are other details like mmu notifiers and so on. In short more detail wold require a blog. Regarding PV-paging – I don’t deal much with that, it’s most likely Xen, but as HW assist became more efficient, with feature like large intermediatte page table walk cache (cache 1st stage table entries to 2nd stage) – not sure what the role of PV-paging support is.