Virtual machine monitors (VMM) emulate most guest access to interrupts and the advanced programmable interrupt controller (APIC) in a virtual environment. They also virtualize all guest interrupts. These activities require the exit and reentry of the virtual machines (VM), but they are time consuming and are a major source of overhead. in order to minimize that effect, the latest Intel(R) Xeon(R) processors, such as the Intel(R) Xeon(R) E5-2600 v2 Product Family, emulate those activities in the hardware. This new feature is called APIC virtualization (APICv). More information can be found at [6] and in chapter 29 of [7].

Figure 1 - VM-VMM interaction with and without APICv

Figure 1 shows that all virtualized activities relating interrupts and APIC to and from the guest OS have to go through VMM in systems without APICv; however, in systems with APICv, they are executed in the hardware, not in the VMM. This way all activities can stay inside the VM, thus eliminating the need to issue the "VM exit" command; this change results in reduced overhead and increased I/O throughput.

In his blog we describe how we tested this new feature to determine if it improves throughput and how it affects CPU utilization of the system.

Terminology

Advanced Programmable Interrupt Controller (APIC) is programmable interrupt controller (PIC) that can handle interrupts from multiple processors. More information about PIC can be found here[2].

Fread is reading a file using the function fread(). More information about this can be found here[8] in the download documentation link.

Fwrite is writing a file using the function fwrite(). More information about this can be found here[8] in the download documentation link.

Guest Operating System (OS) is an OS that runs in a VM. More information about the guest OS can be found here[5].

Pread is reading from a file at a given offset.

Pwrite is writing to a file at a given offset.

Random Read is reading from a file with access being made to random locations within the file. More information about it can be found here8] in the download documentation link.

Random Write is writing to a file with access being made to random locations with the file. More information about it can be found here[8] in the download documentation link.

Virtualization is a way to run multiple independent virtual operating systems on a single physical computer. More information about it can be found here[1].

Virtual Machine (VM) is a piece of computer software, firmware or hardware that creates and run VMs. More information about it can be found here[4].

The Test

Iozone v3.42 was chosen to test the file I/O performance of a system. The beta version of the 64-bit Enterprise edition of Redhat* 7 was used with kvm* installed on a system with two pre-production Intel(R) Xeon(R) E5-2697 v2 processors and 64GB of RAM. We created twelve VMs, and each VM had the same OS on it: beta version of 64-bit Enterprise edition of Redhat 7.

On each VM, we configured Iozone to perform the test on a 100MB file with record size of 64KB.

The following steps were used to collect file I/O activities and CPU utilization:

1) Enable APICv feature in kvm

2) Turn on only one VM

3) Ssh to execute Iozone an all running VMs

4) At the main console, run the command sar (system activity report) to collect the percentage CPU utilization.

5) Increase the number of running VMs by one and repeat the above steps starting from step 3

6) Repeat until all VMs created are running

7) Disable APICv feature in kvm

8) Repeat the above steps starting from step 3 through step 7

Note that the results can vary from system to system depending on many factors like the record size, file size, type of data disk and so on.

Figure 2 shows the CPU utilization increase ranging from 0.26% to 4.98% when compared to systems with and without APICv features. This makes sense in systems with APICv. They have requests activities served within VMs without going out to VMM. The result is less overhead. This translates into more requests being ready to execute within the same time frame, compared to those of systems without APICv resulting in more CPU activity. Further analysis showed reduced iowait time in systems with APICv enabled, compared to systens without APICv as can be seen in figure 2a. Iowait is the portion of time the CPU is idle during which the system has an outstanding disk I/O request. In summary, even though the CPU utilization increased with APICv enabled, this should not be interpreted as a negative: more work was being done, and less time was being spent waiting for I/O to complete.

~~Figure 2a - Percentage of iowait with and without APICv.

~~Figure 3 – Throughput of random read with and without APICv
With APICv the random-read throughput improved, ranging from 0.29% - 2.73%

~~Figure 4 – Throughput of random write with and without APICv
With APICv the random-write throughput increase ranged from 0% - 8.51%

~~Figure 5 – Throughput of pread with and without APICv
With APICv the pread throughput improved, ranging from 0.52% - 9.3%

~~Figure 6 – Throughput of pwrite with and without APICv
With APICv the pwrite throughput improved, ranging from 0.8% - 3.44%

~~Figure 7 – Throughput of fread with and without APICv
With APICv the fread throughput improved, ranging from 0.08% - 1.39%

~~Figure 8 – Throughput of fwrite with and without APICv
With APICv the fwrite throughput increase ranging from 0.73% - 4.18%

Conclusion
Through our synthetic experiments, even with a fairly minimal number of concurrent virtual machines, we saw improved read and write performance. Additionally, because the CPU spends less time dealing with system calls, it is freed up to do more computation-intensive work. APICv will help improve I/O throughput with less overhead since APIC-related activities are done in the hardware and not in VMMs.