Setting Up a Headless VirtualBox VM in Ubuntu Server

In the past week, I’ve been asked to reinstall Windows XP onto a PC which has recently suffered a hard drive failure. One of the problems with the continuing use of this OS (which celebrates its tenth birthday this year) is its antique selection of included drivers and inability to load from anything except a floppy disk at install time – press F6, etc…

Thankfully, we have nLite, which helps the process of slipstreaming drivers written this century into an installation and creating custom media for loading the now ancient XP onto machines with such advanced features as…SATA.

This is exactly the issue I faced with this particular machine, but without a Windows machine on which to run nLite (which is sadly not available for Mac OS X or Linux) I was pretty stuck. For my own work, I never have any need to use Windows, however there is one killer application for Windows which means I do sometimes have use for it once or twice a year – repairing other Windows computers. A virtual machine is the perfect solution for this problem.

In this tutorial, I’ll show you how to set up VirtualBox on a headless Linux server and access the VM from any internet-connected computer.

(These instructions are written for Ubuntu Server 10.04, however they should work with other versions of Ubuntu and be similar to other distributions as well)

Ingredients

Oracle VirtualBox 4.0

Operating system installation media

Microsoft Remote Desktop for Mac (or equivalent)

Step 1: Installing VirtualBox

Visit Oracle’s VirtualBox download page on your client machine. There is a wide range of supported Linux distributions and versions, so find the one you are running. Right-click the appropriate link and select ‘Copy Link Address’

Open an SSH connection to your server and navigate to the folder you wish to download the installer to (for example /home/user/downloads).

Use wget and the address you have copied to the clipboard to download the installation files from Oracle. For example:

Step 2: Configuring the Host

You will need to make sure your user account is in the group ‘vboxusers’. For example, if your username was ‘randymarsh’:

sudo adduser randymarsh vboxusers

The ‘vboxusers’ group will need to have the correct kernel permissions. To set this, you will need to edit ‘/etc/udev/rules.d/40-permissions.rules’ (using "sudo nano /etc/udev/rules.d/40-permissions.rules") and add the following line:

KERNEL=="vboxdrv", GROUP="vboxusers", MODE="0660"

Step 3: Configuring the Virtual Machine

We’re now ready to configure the new VirtualBox VM. The following command uses ‘VBoxManage’ to create a new VM with the name ‘WinXP’ – this can be anything you like. The OS is of type ‘WindowsXP’ – you need to change this to match the type of the guest OS. Finally, the VM is registered with VirtualBox.

VBoxManage createvm -name WinXP –ostype WindowsXP -register

Next, we need to enter some configuration information for the new VM. In the following command, we use VBoxManage to modify the virtual machine ‘WinXP’. ‘–memory’ is allocated in megabytes, in this case 1024MB. PAE and ACPI are enabled, and the DVD drive is set as the first boot device. VirtualBox allows you to configure up to 9 virtual ethernet devices (nic1-9). In this command, we specify that ‘–nic1’ is going to operate in a bridge with the physical ethernet interface ‘eth1’. You will need to change ‘eth1’ to whatever the primary network device is on your machine (usually eth0).

We now need to create an IDE channel to connect our virtual drives to.

VBoxManage storagectl WinXP --name IDE0 --add ide

Before attaching the virtual hard drive, we first need to create it. In this command, we are creating a .vdi file (virtual hard disk image) of size 12000MB. As usual, you will need to replace ‘WinXP’ with the name of your VM.

Okay, now we can attach the new drive image to the IDE channel we created in step 3. Again, alter ‘WinXP’ to match the name of your VM and the path to the .vdi image to match the path you set in step 4.

The second device we need attached to our IDE controller is the virtual DVD drive we will be using to install the operating system. We also need to specify the ISO image of our install media which is to be mounted – in this example it is /home/randymarsh/windowsxp.iso.

Finally, we need to configure VRDE. VRDE is the protocol VirtualBox uses to forward the display of a VM running on a headless system to a GUI-inclined client machine. It is compatible with Microsoft’s RDP remote desktop protocol, however rather than using Windows’ (or other guest OS’) RDP server, VirtualBox takes care of things, which means you can forward the virtual BIOS, boot sequence and, crucially, installation process to a client machine before it has booted to OS. The following configuration uses no security authentication and the default RDP port of 3389. This is fine if you’re going to use the VM casually and only very occasionally, but will not do for servers or anything ‘mission-critical’.

Step 4: Running and Connecting to the Virtual Machine

The VM is now fully configured, VRDE is enabled and it’s time to fire it up for the first time. At the command prompt, enter the following to start the machine:

VBoxHeadless -startvm WinXP &

The host will present you with a couple of lines of copyright information and confirmation that the VRDE server is listening on the assigned port, and then return the command line. It is now time to load your RDP software and connect to the VRDE server. As my client machine is running Mac OS X, I use Microsoft Remote Desktop Connection for Mac (available here). To connect using this software, input the IP address of the host server and press connect.

If everything has gone to plan, you should be viewing the startup sequence of the virtual machine. The DVD drive should be configured as the first boot device, and you can now proceed to install an OS. Remember to install Oracle’s VirtualBox Guest Additions (VBoxGuestAdditions) to get full functionality and optimal performance from your VM after OS installation is complete.

After all this, you should have a very useful VM at your disposal. And remember, with port forwarding correctly configured you will be able to access the machine from any computer with an RDP client!

Absolutely awesome! Many years ago we used to run a Windows-based answering machine for our home office. Then I switched to Mac and the alternatives were few and far between. We did the mgetty option for several years, but the issue there is lack of support for common hardware and when it breaks, it’s difficult to get the thing back up and running whereas the Windows app we used was flawless.

I’ve almost got Windows installed in my VM… and if I can get my three modems connected, this will be pure gold.

Thanks so much for the detailed instructions. Just a word of caution to others… copy/paste on some of the commands didn’t work for me. The –‘s were getting converted to some smart dash that made some of the commands fail.