Warning: This document describes an old release. Check here for the current version.

Install VMM Software

On this page, you will install the Nimbus VMM software
workspace-control and any of its dependencies
(including Xen or KVM as well as libvirt, if these are not present
already).

workspace-control contains scripts that will help you test that your Xen/KVM/libvirt installation will work with the programmatic access patterns that Nimbus uses.

You should use one VMM node for this exercise. When things are working end to end, you can propagate the configuration and dependencies to all of the other nodes on your cluster (assuming that they are roughly homogeneous).

Before automating a VMM node with Nimbus software, it needs to be a VMM node. So the first thing we need to do is make sure that all the virtualization and networking dependencies are present.

The workspace service currently manages Xen
or KVM VMs. It interfaces with both
of these systems using a library called
libvirt.

This guide will explain how to install the Nimbus workspace control
software on top of both Xen 3.x and KVM. The instructions are intermingled
so it is recommended that you first read our notes page for the
hyper-visor that you intend to use:

There are many steps in this section which are common to both KVM and
Xen. However there are also some slight differences mixed in with each
step. We will visually distinguish the two with the following styles.

This is information needed for Xen

This is information needed for KVM

The reader can then key in on the information specific to their cloud.

Basic Dependencies

ebtables requires kernel support on the VMM machine. If your kernel does not include these for some reason one place to find the options to enable are under Networking :: Networking options :: Network packet filtering :: Bridge Netfilter Configuration :: Ethernet Bridge Tables

There are differences in lower libvirt versions that have caused problems for some Nimbus users. If you take a chance with this and see an error about "createXML", you've probably got a very old version.

Python bindings for libvirt

An important step is to make sure that your libvirt installation supports the libvirt "python bindings". This is how the Nimbus software actually interacts with libvirt.

In some cases, the bindings (which come with a normal libvirt installation) are not installed. On some systems you need to trigger this to be there by installing the "python-dev" package before installing libvirt. In order to check you have the right install, Nimbus includes a script in the workspace-control download which you will run later (discussed below).

Our next step in this guide will be to create a unix account for workspace-control and then install it. This will give us access to some scripts we can use to further verify the system is ready to be automated with Nimbus.

Bridged networking

You need to get a basic VM running with bridged networking. Using NAT networking for guest VMs will not be sufficient.

privileged account - Pick an account to make privileged (using the sudo program).

In this guide (especially in the command samples) we will refer
to an account of this type named nimbus with terminal
prompts like "nimbus $"

Note: this is not a privileged account in the sense that root is a
privileged account. This is a regular account that you will
give powers to during the setup process. It is an account that
will be privileged because it will have control over the hyper-visor.

superuser account - The root account is necessary to install
dependencies on the VMM nodes (Xen/KVM, ebtables, etc.) and also to
install the Nimbus agent that lives on the VMM nodes (workspace
control).

In this guide (especially in the command samples) we will refer
to an account of this type named root with terminal
prompts like "root #"

Note that root is not needed on the service node, just here on the VMM.

OK, you've got your non-root user nimbus created, now we need to allow it to control VMs using libvirt.

First, try out a libvirt command as root (it will not change anything if it succeeds or fails).

root # virsh -c 'xen+unix:///' list

root # virsh -c 'qemu:///system' list

Make sure this works as root before we try to enable the nimbus user to do this.

If that does not work, you will need to enable Xen to work with domain
sockets. In the /etc/xen/xend-config.sxp file,
activate the xend-unix-server configuration by
including yes. Then uncomment the
xend-unix-path setting and note the value (for
example /var/lib/xend/xend-socket).
Note that using Xen with the HTTP server method instead of this domain socket method is typically not secure.

Once it works as root, try as the nimbus user:

nimbus $ virsh -c 'xen+unix:///' list

nimbus $ virsh -c 'qemu:///system' list

You may hit an error about failing to connect to the hypervisor.

error: unable to connect to '/var/run/libvirt/libvirt-sock': Permission denied
error: failed to connect to the hypervisor

Why? Because you are not root. Check out the permissions on /var/run/libvirt/libvirt-sock

libvirtd needs to be restarted for any changes to take effect. Validate your changes by checking the permissions with "ls" again.

Make sure the group change has taken effect with the nimbus user (using the "groups" command within the shell, you may need to relogin).

Try again:

nimbus $ virsh -c 'xen+unix:///' list

nimbus $ virsh -c 'qemu:///system' list

If you can still not do this basic list command as the nimbus user, look into your distribution's libvirt configurations and also don't hesitate to contact the workspace-user mailing list with problems.

Download the "Control Agents" tar file from the download page, and untar it. This archive contains both workspace-control and the workspace pilot. For this configuration we are using workspace-control. You need to copy it to the /opt/nimbus destination directory like so:

Now make sure the permissions are set up for running via sudo safely. Note the chown -R nimbus part, this is crucial for letting workspace-control do a lot of its work as the non-root user nimbus (only calling out to root via sudo to perform specific priviliged tasks).

Test the basic dependencies

Before moving on with a VM launching test, first we should confirm the basic assumptions about the environment. The workspace-control package contains a script that will examine the system.

Run the following command:

nimbus $ ./sbin/test-dependencies.sh

If this script reports you have a problem, look into your distribution's
libvirt support (especially the python bindings that were discussed earlier) and don't hesitate to contact the
workspace-user mailing list with
problems.

Now you have created a kernel called default that workspace-control will launch paravirtualized VMs with. The default kernel and initrd to use for VMs is controlled by the authz_kernels configuration in /opt/nimbus/etc/workspace-control/kernels.conf (by default the authorized kernel is "default").

It is a minimal Debian VM with very little installed. It runs an SSH server when it boots. If you have console access to it, you will be able to log in to it using the root account with password root.
It also runs the Nimbus context agent on boot, so you can use it to test contextualization.

Workspace control must be configured to use the correct libvirt hypervisor. This is done in
/opt/nimbus/etc/workspace-control/libvirt.conf. The default hypervisor
is Xen. If you are using KVM you'll need to edit the vmm property
in this file.

Notice virbr0. This value must match the value for
default: virbr0 in the
[defaultbridge] section of that configuration file
/opt/nimbus/etc/workspace-control/networks.conf.
In that same file you must also change the value
xen-ebtables-config.sh to
kvm-ebtables-config.sh.

Before we can do a live test, we need to configure workspace-control to use sudo privileged properly.

Using the visudo command, add the necessary sudo policies.
These policies reflect need to use the user that will be running workspace-control ("nimbus") and the correct full paths to the libexec tools. See /opt/nimbus/etc/workspace-control/sudo.conf for all the details and sample rules.

Seeing "ERROR" is a good thing here. If you see a password prompt, type in the nimbus user's password which hopefully is a "first time use" warning from sudo. Then try again. The goal is to perform this command from the nimbus account without needing any password.

You may need to comment out any "requiretty" setting in the sudoers policy, like so:

#Defaults requiretty

The commands run via sudo are not using a terminal and so if you have "requiretty" enabled, this can cause a failure.

In a previous section of this guide, you configured DHCPd and the network addresses to give to guest VMs.

The central IaaS services container wrote out the dhcpd.entries file that will be used for the DHCPd server configuration. In the same directory a sample file for workspace-control was written out, we will use this to test a VM launch via libvirt with your DHCPd server running. The file you need to obtain is called control.netsample.txt

Transfer this file from the central node to anywhere on the VMM node: $NIMBUS_HOME/services/var/nimbus/control.netsample.txt

We will use it in the next step. The example location on the VMM will be /tmp/control.netsample.txt.

This may need to be repeated to solve any problems that are being mentioned. Anytime you run libvirt-xml.sh, you should pair it with destroy-control-test.sh before trying again.

The output on stderr that looks like "libvir: Test error : Domain not found" is normal. This is an artifact of the tools scanning for information.

Having a problem? If the problem is not obvious from the output, the error and debug logs can be found like so:

nimbus $ ./sbin/cat-latest-logfile.sh

If putting the culprit error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Attaching this debug log and including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

Now the VM will boot. If it gets the correct network address from DHCP you should be able to ping it and even log in over SSH.

First find the IP address we are expecting it to get:

nimbus $ grep ip /tmp/control.netsample.txt | awk '{print $2}'

Then give it a ping:

nimbus $ ping -c 1 192.168.0.2

If this does not work, the next step will be to log in via the console and troubleshoot what happened. You can log in using console to the nimbus-z2c image using the root account with password root

nimbus $ virsh -c 'xen+unix:///' console control-test

nimbus $ virsh -c 'qemu:///system' console control-test

Console permissions

You may need to be root to run a virsh console subcommand because the nimbus user may not have access to the /dev/pts device being used for console access.

Does it not get an IP address from DHCPd? Is DHCPd running?

Did it get some other IP address from DHCPd? Your DHCPd may be misconfigured, perhaps it is not giving specific MAC addresses specific IP addresses. Perhaps you did not load it with the current dhcpd.leases entries from Nimbus.

If you get an error like "There is a VM running with this handle but there
is no record of it in the program" then you may have been starting with
virsh but not through
libvirt-xml.sh. The way to correct this is to
run virsh destroy control-test directly.

Seeing an error from a component, for example in the libvirt logs? If putting the error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

When all is well, you should be able to log in as root over SSH. Trying this from off the node is the best final test.

The final test will be to launch the same file with the same parameters, but using workspace-control.sh just as the service will. We will use a wrapper program called control-test.sh to invoke workspace-control.sh.

If you have a control-test VM running at the moment, shut it down using the kill script:

If the exit code is zero, we are good to go. SSH again to test for sure. Run the destruction script and you can move along to the next section in this guide.

./sbin/destroy-control-test.sh

Having a problem? If the problem is not obvious from the output (this is not a program usually oriented towards humans), the error and debug logs can be found like so:

nimbus $ ./sbin/cat-latest-logfile.sh

If putting the culprit error into a search engine does not produce any obvious answers, don't hesitate to contact the workspace-user mailing list. Describe your situation and any errors you are seeing. Attaching this debug log and including version information (of the VMM and libvirt and your OS) can often be helpful and reduce the amount of emails that need to occur.

Once you've gotten everything working, proceed to the next page where you will configure communication between
the service node and the VMM node: SSH Setup.