Jailhouse

Jailhouse treats all inmates as opaque binaries, and although it
provides a small framework to develop them faster, the only thing it
needs to know about the inmate image is its base address. Jailhouse
expects an inmate entry point at 0xffff0 (which is different from
the x86 reset vector). apic-demo.bin is a standard demo inmate that comes
with Jailhouse, and the inmate's framework linker script ensures that if
the binary is mapped at 0xf0000, the entry point will be at the right
address. apic-demo is just a name; it can be almost anything
you want.

Finally, start the cell with:

sudo tools/jailhouse cell start apic-demo

Now, switch back to the terminal from which you run QEMU. You'll see that lines
like this are being sent to the serial port:

apic-demo is purely a demonstrational inmate. It programs the APIC timer
(found on each contemporary CPU's core) to fire at 10Hz and measures
the actual time between the events happening. Jitter is the difference between
the expected and actual time (the latency), and the smaller it is, the
less visible (in terms of performance) the hypervisor is. Although this
test isn't quite comprehensive, it is important, as Jailhouse targets
real-time inmates and needs to be as lightweight as possible.

Jailhouse also provides some means for getting cell
statistics. At the most basic level, there is the sysfs interface under
/sys/devices/jailhouse. Several tools exist that pretty-print this
data. For instance, you can list cells currently on the system with:

sudo tools/jailhouse cell list

The result is shown in Figure 3. "IMB-A180" is the root cell's
name. Other cells also are listed, along with their current states and
CPUs assigned. The "Failed CPUs" column contains CPU cores that triggered
some fatal error (like accessing an unavailable port or unassigned memory
region) and were stopped.

Figure 3. Jailhouse cell listing—the same information is available
through the sysfs interface.

For more detailed statistics, run:

sudo tools/jailhouse cell stat apic-demo

You'll see something akin to Figure 4. The data is updated periodically (as
with the top utility) and contains various low-level counters like the number of
hypercalls issued or I/O port accesses emulated. The lifetime total and
per-second values are given for each entry. It's mainly for developers,
but higher numbers mean the inmate causes hypervisor involvement more
often, thus degrading the performance. Ideally, these should be close
to zero, as jitter in apic-demo. To exit the tool, press Q.

Figure 4. Jailhouse cell statistics give an insight into how cells
communicate with the hypervisor.

What's the reason for this? Remember the apic-demo cell had
the "running/locked" state
in the cell list. Jailhouse introduces a locked state to prevent changes
to the configuration. A cell that locks the hypervisor is essentially
more important than the root one (think of it as doing some critical job at
a power plant while Linux is mostly for management purposes on that
system). Luckily, apic-demo is a toy inmate, and it unlocks Jailhouse after
the first shutdown attempt, so the second one should succeed. Execute
the above command one more time, and apic-demo should disappear from
the cell listing.

Now, create tiny-demo cell (which is originally for tiny-demo.bin, also
from the Jailhouse demo inmates set), and load 32-bit-demo.bin into it
the usual way:

Look at com2.txt in the host (the same directory you started QEMU
from). Not only does this show that cells can be re-used by the inmates
provided that they have compatible resource requirements, it also proves
that Jailhouse can run 32-bit inmates (the hypervisor itself and the
root cell always run in 64-bit mode).

When you are done with Jailhouse, you can disable it with:

sudo tools/jailhouse disable

For this to succeed, there must be no cells in
"running/locked" state.

This is the end of our short trip to the Jailhouse. I hope you enjoyed your
stay. For now, Jailhouse is not a ready-to-consume product, so you may
not see an immediate use of it. However, it's actively developed and
somewhat unique to the Linux ecosystem, and if you have a need for real-time
application virtualization, it makes sense to keep a close eye on its
progress.

Jailhouse for Real

QEMU is great for giving Jailhouse a try, but it's also possible to test
it on real hardware. However, you never should do this on your PC. With
a low-level tool like Jailhouse, you easily can hang your root cell where
Linux runs, which may result in filesystem and data corruption.

Jailhouse comes with a helper tool to generate cell configs, but usually
you still need to tweak the resultant file. The tool depends on Python;
if you don't have it on your testing board, Jailhouse lets you collect
required data and generate the configuration on your main Linux PC
(it's safe):

The configuration tool reads many files under /proc and /sys (either collected
or directly), analyzes them and generates memory regions, a PCI devices
list and other things required for Jailhouse to run.

Post-processing the generated config is mostly a trial-and-error
process. You enable Jailhouse and try to do something. If the system
locks up, you analyze the serial output and decide if you need to grant
access. If you are trying to run Jailhouse on a memory-constrained
system (less than 1GB of RAM), be careful with the hypervisor memory area,
as the configuration tool currently can get it wrong. Don't forget to
reserve memory for Jailhouse via the kernel command line the same way
you did in QEMU. On some AMD-based systems, you may need to adjust
the Memory Mapped I/O (MMIO) regions, because Jailhouse doesn't support AMD
IOMMU technology yet, although the configuration tool implies it does.

To capture Jailhouse serial output, you'll likely need a serial-to-USB
adapter and null modem cable. Many modern motherboards come with no
COM ports, but they have headers you can connect a socket to (the cabling
is shown in Figure a). Once you connect your board to the main Linux PC,
run minicom or similar to see the output (remember to set the port's baud rate
to 115200 in the program's settings).