Documentation and guides

These notes detail the process of setting up a Xen system on a Cubieboard2 (or Cubietruck).
They are based on the Xen ARM with Virtualization Extensions/Allwinner documentation, but try to collect everything into one place.
I'm trying to document the exact steps I took (with the wrong turns removed); some changes will be needed for other systems.

TL;DR: There is now a script available that generates an image with Xen, Ubuntu dom0, and the OCaml tools installed. Just run the make instructions at mirage/xen-arm-builder and copy the resulting image onto an SDcard and boot up your Cubieboard2 or Cubietruck (password mirage/mirage which you should change as the first thing you do). The script is kept more up-to-date than the instructions on this page.

The remainder of this guide covers:

Installing U-Boot, Xen and a Linux Dom0

Running a Linux DomU

Running a FreeBSD DomU

Running a Mini-OS DomU

Warning: OUT OF DATE

These instructions no longer work, as some of the Git branches referred to no longer exist. Instead, please look at the scripts in the mirage/xen-arm-builder repository.
They are based on these instructions, but are kept up-to-date.

Glossary

ARMv7
: Version 7 of the ARM architecture, which adds optional virtualisation support.

ARM Cortex-A7
: A 32-bit RISC CPU designed by ARM. It implements ARMv7, including the virtualisation extensions.

SPL (Secondary Program Loader)
: "SPL is a very small bootloader able to fit into the very small amount of memory (128kB) available inside the CPU. SPL does autodetection and sets up memory accordingly. After memory is set up, SPL loads u-boot, and the booting proceeds as usual."

Initial testing

This step can be skipped, but it's useful to check that everything works.

Download Cubieez, a Debian 7.1 image that works on the Cubieboard2.
Write to microSD card with:

dd if=cubieez_1.0.0_A20.img of=/dev/mmcblk0

You will need to hook up the serial console on the Cubieboard, being careful not to connect up one of the pins.
One good cable that is known is work can be bought here.
Follow the instructions at the Sunxi/TTL page to connect up 3 of the cables to the right pins.

Connect the USB serial cable and run "screen" to monitor it:

screen -h 10000 /dev/ttyUSB0 115200

Note: I tried using "minicom" first, but for some reason keyboard input didn't work with that.

Insert microSD card in board and connect power. You should see boot messages in screen:

Remark: You may have noticed that the above .cmd files allocate a
rather large amount of memory to dom0 (look at the dom0_mem
parameter). This is needed to compile large libraries like
Core.
However, if you use autoballoon=on in
/etc/xen/xl.conf,
xl will automatically reduce the amount of memory assigned to dom0
to free memory for new domains. An OCaml daemon
squeezed, currently in
development (and based on
xenopsd), will dynamically
move memory between dom0 and VMs to satisfy their needs.

Building Linux

Get my Linux Git tree, master branch. This fork has a few extra patches we need.

cd ../..
git clone https://github.com/talex5/linux.git
cd linux

Configure:

make ARCH=arm multi_v7_defconfig
make ARCH=arm menuconfig

Here are the settings I used (check it works with just these settings and whether they're all actually necessary):

The first bit "U-Boot SPL" is the SPL running, setting up the RAM and loading the main U-Boot.
The "spl: not an uImage at 1600" warning is harmless. It looks at offset 1600 first, and then tries 80 next and succeeds.

The "bad CRC" warning is just because we didn't specify an environment file.

You must also install these packages and avahi-utils on your
computer.

You probably want to give your Cubieboard a nice name. Edit
/etc/hostname and replace the existing name with the one of your
choice — cubie2 for the following.
(You should also change linaro-developer in /etc/hosts
to cubie2.)
For the changes to take effect, you can either reboot or run
hostname cubie2 followed by /etc/init.d/avahi-daemon restart.
You should now be able to connect with e.g., from your computer

ssh root@cubie2.local

To see the list of Avahi services on your network, do avahi-browse
-alr. If you do not see your Cubieboard, check that its network is
up: doing

ip addr show

at the Cubieboard root prompt, should output some information
including a line starting with br0:
<BROADCAST,MULTICAST,UP,LOWER_UP>. If it doesn't try

brctl addbr br0

If you get add bridge failed: Package not installed, you forgot to
include Ethernet bridging in your kernel (it is included with the
recommended .config file above so this should not happen).

Kill the network and shut down:

ifdown eth0
mount -o remount,ro /
halt -f

Remove the init=/bin/bash from boot.cmd and put the new boot.scr into mmcblk0p1. Then boot again.
You should now be able to ssh in directly.

FreeBSD guest

I created a VM on my laptop and installed FreeBSD from FreeBSD-10.0-RELEASE-amd64-bootonly.iso. I then used that to cross-compile the Xen/ARM version. Your build VM will need to have at least 4 GB of disk space.

Note: I tested with the xen-arm branch, but the xen-arm-v2 branch has some useful fixes.

Note: Installing Git using FreeBSD using ports on a clean system is very slow, uses a lot of disk space, requires many confirmations and, in my case, failed. So I suggest cloning the repository with your main system and then transferring the files directly to the FreeBSD build VM instead.

In the build FreeBSD (note: the build takes several hours; you might want to assign multiple CPUS to your VM and use -j here):

If you try to start the domain with Debian's version of xl, you'll get Unable to find arch FDT info for xen-3.0-unknown.
To fix this, you need to rebuild the Xen toolstack with these two patches (I applied them to the stable-4.4 branch):

https://patches.linaro.org/22228/

https://patches.linaro.org/22227/

Build it using the ARM build guest:

cd xen/tools
./configure --prefix=/opt/xen-freebsd
make
make install

Transfer /opt/xen-freebsd to dom0 and you can then start the FreeBSD domain with: