Enabling Nested Virtualization for VM Instances

This document describes how to enable support for
nested virtualization
on Compute Engine VM instances. It also covers the basic steps of
starting and configuring a nested VM.

Nested virtualization adds support for
Intel VT-x
processor virtualization instructions to Compute Engine VMs. Using
nested virtualization, you start a VM instance as normal on
Compute Engine and then install a KVM-compatible hypervisor on the VM
instance so you can run another VM instance on top of that hypervisor. Other
hypervisors, such as Hyper-V, ESX, and Xen are not currently supported. You can
use nested virtualization on any Linux VM instance running on a Haswell or newer
platform.

Nested virtualization is ideally suited for VM-based applications and workloads
where converting or importing your VM images to Compute Engine is not
feasible. For example, you can use nested virtualization to build a disaster
recovery solution for an on-premises workload running on KVM-based virtual
machines, that can seamlessly failover to VMs running on Compute Engine
without the extra time or orchestration needed to convert your KVM-based VM to a
native Compute Engine image. Another workload that could be
well-suited for nested virtualization is a software-validation framework that
needs to test and validate new versions of a software package on numerous
versions of different KVM-compatible OSes. Running nested VMs removes the
need to convert and manage a large library of Compute Engine images.

How nested virtualization works

Compute Engine VMs run on top of physical hardware (the host server)
which is referred to as the L0 environment. Within the host server, a
pre-installed hypervisor allows a single server to host multiple
Compute Engine VMs, which are referenced to as L1 or native VMs on
Compute Engine. When you use nested virtualization, you install
another hypervisor on top of the L1 guest OS and create nested VMs,
referred to as L2 VMs, using the L1 hypervisor. L1 or native
Compute Engine VMs that are running a guest hypervisor and nested VMs
can also be referred to as host VMs.

Nested Virtualization Diagram

Restrictions

Nested virtualization can only be enabled for L1 VMs running on Haswell
processors or later. If the default processor for a zone is Sandy Bridge or
Ivy Bridge, you can use minimum CPU selection
to choose Haswell or later for a particular instance. Review the
Regions and Zones
page to determine which zones support Haswell or later processors.

Nested virtualization is only supported for KVM-based hypervisors running on
Linux instances. ESX and Xen hypervisors are not supported.

Nested virtualization does not currently support Windows instances.

Tested KVM versions

Google runs basic nested virtualization boot and integration tests using the
following Linux distros and kernel/KVM versions on the Compute Engine
instance:

CentOS 7 with kernel version 3.10

Debian 9 with kernel version 4.9

Debian 8 with kernel version 3.16

RHEL 7 with kernel version 3.10

SLES 12.2 with kernel version 4.4

SLES 12.1 with kernel version 3.12

Ubuntu 16.04 LTS with kernel version 4.4

Ubuntu 14.04 LTS with kernel version 3.13

If you are having trouble running nested VMs on distributions and kernel/KVM
versions not listed here, reproduce your issue using one of the above
environments as the guest operating system on the
host Compute Engine instance before reporting the issue as a bug.

Performance

Even with hardware-assisted nested virtualization, there will be a performance
penalty for the nested VMs themselves and any applications or workloads
running inside them. While it is impossible to predict the exact performance
degradation for a given application or workload, expect at least a 10% penalty
for CPU-bound workloads and possibly much more for I/O bound workloads.

Enabling nested virtualization on an instance

You can enable nested virtualization using the API or gcloud component. To
enable nested virtualization, you must create a custom image with a special
license key that enables VMX in the L1 or host VM instance and then use that
image on an instance that meets the
restrictions for nested virtualization. The license key does
not incur additional charges.

Create a boot disk from a public image or from a custom image with an
operating system. Alternatively, you can skip this step and apply the license
to an existing disk from one of your VM instances.

gcloud

Use the gcloud command-line tool to create the disk from the boot
disk image of your choice. For this example, create a disk named
disk1 from the debian-9 image family:

Check that nested virtualization is enabled by running the following
command. A non-zero response confirms that nested virtualization is
enabled.

grep -cw vmx /proc/cpuinfo

Starting a nested VM

You can start a nested VM in many different ways. This section provides an
example of starting a nested VM using
qemu-system-x86_64
on an L1 VM running Debian. If you are having trouble running nested VMs using
methods other than this documented process, reproduce your issue using this
process before reporting the issue as a bug.

Configuring a nested VM to be accessible from outside the host VM

You can set up an instance with multiple network interfaces or set up an
instance using an alias IP so that VMs outside the
host VM can ping the nested VM.

The following sample procedure sets up your host and nested VM so that the
nested VM is accessible from other VMs on the same network using alias IPs.
This procedure is intended for an L1 VM running Debian.

Create a VM with nested virtualization enabled but make sure to include an
alias IP range and support for HTTP/HTTPS traffic. For example:

On your host VM, set up iptables to allow forwarding from your host VM to
your nested VM. For example, if you want to use an alias IP of 10.128.0.13
to forward traffic to your hosted VM at 192.168.122.89:

Troubleshooting

Google runs basic nested virtualization boot and integration tests using
specific Linux distros and kernel/KVM versions on the Compute Engine
instance. Additionally, these tests use a specific process. Before reporting
the issues as a bug, reproduce those issues using the following items:

If you did not run screen before each nested VM session, you will not be able
to detach from the nested VM. Log into the host VM in another terminal and kill
the process, then run screen on the host VM before you start a nested VM.

My iptables rules aren't forwarding traffic to my nested VM.

iptables resolve rules from top to bottom, make sure your rules are higher
priority than other rules.

Check that there are no conflicting rules intercepting your packets.

Consider flushing your iptables:

Note: This disables your firewall. You should only do this if you want to
start over configuring your firewall.