Aufs for VMware Appliance
Junjiro R. Okajima
o Introduction
----------------------------------------------------------------------
A recent trend for VMware and Xen tends to consume much disk
space. For instance, a typical VMware appliance is distributed as a
8GB disk image and you need such large disk space for each
virtualization even if the filesystem image in it actually consumes
less than 1GB.
Additionally every virtual machine has the same files mostly. When you
construct three virtual servers from one VMware appliance, you will
need 3 x 8GB plus alpha disk spaces. More and more virtual servers,
more and more disks are needed. The "plus alpha" means the part of
differences between servers.
But the actual necessary files are single sharable system files, 1GB
for example, and the part of differences.
With AUFS you can share the common part of virtual servers, and stores
the server specific part individually.
The basic approach is such like this.
- extract the actual filesystem image from VMware appliance which we
call it "common system image"
- construct an NFS-exportable aufs with an empty writable branch and
the readonly "common system image"
- boot a virtual server with PXE and nfsroot, mount the exported aufs
as its root filesystem
- so you need two or three server softwares but they can live in a
single host
+ NFS server
+ TFTP server
+ DHCP server (optional)
Finally you can save much of disk space,
3 x (8GB + diff) > 8GB + 3 x diff
In this document, I will describe the sample steps to build such
environment.
o VMware appliance
----------------------------------------------------------------------
Here is an assumption.
+ you already have VMware environment.
you can get and start any VMware appliance, and know how to
customize it.
In this sample, we use debian appliance where you can get
http://mirror.o-line.net/vmware/debian-4.0r1-netinst.7z
- unpack it
$ 7z x debian-4.0r1-netinst.7z
:::
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst-f001.vmdk
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst-f002.vmdk
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst-f003.vmdk
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst-f004.vmdk
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst-f005.vmdk
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst.vmdk
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst.nvram
Extracting debian-4.0r1-netinst/vmware-0.log
Extracting debian-4.0r1-netinst/vmware.log
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst.vmx
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst.vmxf
Extracting debian-4.0r1-netinst/debian-4.0r1-netinst.vmsd
Extracting debian-4.0r1-netinst
:::
- check the disk partitions
$ file debian-4.0r1-netinst-f00*.vmdk
debian-4.0r1-netinst-f001.vmdk: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 15952482 sectors; partition 2: ID=0x5, starthead 0, startsector 15952545, 819315 sectors, code offset 0x48
debian-4.0r1-netinst-f002.vmdk: data
debian-4.0r1-netinst-f003.vmdk: data
debian-4.0r1-netinst-f004.vmdk: data
debian-4.0r1-netinst-f005.vmdk: data
It shows that there are two disk partitions, one is from 63 sector
and has 15952482 sectors, the other is from 15952545 sector and has
819315 sectors. The latter is swap space, so let's ignore it here.
- extract the filesystem image
$ cat debian-4.0r1-netinst-f00?.vmdk | dd ibs=512 skip=63 count=15952482 obs=32m of=etch.ext3
$ fsck.ext3 -nf etch.ext3
$ sudo mount -o ro,loop etch.ext3 /mnt
$ df /mnt
Now you get the ext3 filesystem image which is stored in the VMware
appliance. In this sample, we create a brand new ext2 fs-image and
duplicate the system files since it will be an unchanged readonly fs
and ext3's journaling feature is unnecessary.
$ dd if=/dev/zero of=etch.ext2 bs=1M count=1k
$ mkfs -t ext2 etch.ext2
$ sudo mount -o loop ./etch.ext2 /tmp/w
$ sudo chown 0:0 /tmp/w
$ sudo rsync -aqHSEx --numeric-ids /mnt/* /tmp/w
$ df /tmp/w
Let's make sure that our files are all fine.
$ cd /mnt
$ sudo find . -printf '%M %n %U %G %s %t %p %l\n' |
> sort -k 11 |
> awk '
/^l/ {print $1, $2, $3, $4, $5, $11, $12; next}
/^d/ {print $1, $2, $3, $4, $6, $7, $8, $9, $10, $11, $12; next}
{print}'
>| /tmp/l1
$ cd /tmp/w
$ sudo find . -printf '%M %n %U %G %s %t %p %l\n' | sort -k 11 >| /tmp/l2
$ diff -u /tmp/l[12]
$ sudo umount /mnt
Now we have the base system under /tmp/w.
Next let's customize the common part.
- customize the common part
In order to disable the root and swap partition on the local (virtual)
disk, edit fstab. The file will be such like this.
$ sudo vi /tmp/w/etc/fstab
proc /proc proc defaults 0 0
#/dev/sda1 / ext3 defaults,errors=remount-ro 0 1
#/dev/sda5 none swap sw 0 0
/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
In this sample, we use 192.168.1.2 as local DNS server.
$ sudo vi /tmp/w/etc/resolv.conf
nameserver 192.168.1.2
In this debian VMware appliance, the server to retrieve the packages
is registered in /etc/apt/sources.list, and it still has CD-ROM
entry. Let's delete it. Of course, you can customize the package
server as you like.
$ sudo vi /tmp/w/etc/apt/sources.list
#deb cdrom:[Debian GNU/Linux 4.0 r1 _Etch_ - Official i386 NETINST Binary-1 20070820-20:21]/ etch contrib main
deb http://debian.lcs.mit.edu/debian/ etch main
deb-src http://debian.lcs.mit.edu/debian/ etch main
deb http://security.debian.org/ etch/updates main contrib
deb-src http://security.debian.org/ etch/updates main contrib
If you don't live within the timezone of EST, it is better to change
the timezone too.
$ sudo chroot /tmp/w tzconfig
Now we finished the customization of the "common system
image". Unmount it and keep the filesystem image.
$ sudo umount /tmp/w
o AUFS on NFS server
----------------------------------------------------------------------
In this sample, I will not describe the generic NFS server
issues. Refer to the other documents if you need.
Building and loading the aufs module is easy, but you should enable
CONFIG_AUFS_EXPORT and some other configurations. It purely depends
on your kernel version, and is described in other documents in
aufs. Please refer them too.
Here is a sample step to mount and export aufs.
$ mkdir /tmp/ro /tmp/rw /tmp/u
$ sudo mount -o ro,loop etch.ext2 /tmp/ro
$ sudo mount -t aufs -o br=/tmp/rw:/tmp/ro=rr none /tmp/u
$ sudo exportfs -i -o rw,async,no_subtree_check,no_root_squash,fsid=999 \*:/tmp/u
Please refer to the other manual if you want to know how to use
exportfs command. In this sample, I just inform you some important
things.
- export it with remote root user can write
- specify fsid since aufs has no own real disk device
o New VMware appliance
----------------------------------------------------------------------
In this sample, we create a new VMware appliance which doesn't have
any local disk. So it will never swap-out. Of course you can create
local disk and use it as swap area if you want.
Here we just create a new virtual server by VMware Server Console. I
hope you know it already and I won't describe here.
But this sample uses its MAC address in next section to set-up the
TFTP/DHCP server.
o TFTP/DHCP server
----------------------------------------------------------------------
This part is entirely depends upon your environment. On my test
environment, I used dnsmasq and tftpd-hpa tools. Here is what I did.
- /etc/dnsmasq.conf
# assign 192.168.1.99 for a vmware debian
dhcp-range=192.168.1.99,192.168.1.99,12h
dhcp-host=00:1a:2b:3c:4d:5e,192.168.1.99
# boot it with pxe
dhcp-boot=pxelinux.0,jrodns,192.168.1.2
# gateway
dhcp-option=3,192.168.1.1
# dns server
dhcp-option=6,192.168.1.2
The MAC address here is created by VMware Server Console. You may
not need it in your environment.
Now restart dnsmasq daemon.
Fortunately, the VMware appliance we use here supports nfsroot and
we just need to pass some parameters to boot it. Also you don't need
to use MAC address as a filename of configuration. You can freely
customize your TFTP server. The important thing is booting the
VMware appliance with nfsroot.
- tftpd-hpa
$ cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot
$ mkdir /var/lib/tftpboot/pxelinux.cfg
$ vi 01-00-1a-2b-3c-4d-5e
default linux
#prompt 1
timeout 600
label linux
kernel auware/etch/vmlinuz-2.6.18-5-686
append initrd=auware/etch/initrd.img-2.6.18-5-686 root=/dev/nfs rootfstype=nfs nfsroot=192.168.1.102:/tmp/u,v3 ip=dhcp noresume
- copy the kernel and initramfs image from the vmware appliance to
/var/lib/tftpboot/auware
$ mkdir -p /var/lib/tftpboot/auware/etch
$ cp -ip /tmp/ro/boot/vmlinuz-2.6.18-5-686 \
/tmp/ro/boot/initrd.img-2.6.18-5-686 \
/var/lib/tftpboot/auware/etch
Of course you don't need to use DHCP if you don't want. In this
case, you will change the kernel parameters too.
----------------------------------------------------------------------
Now our preparations are all completed.
Boot your virtual server from VMware Server Console. It will boot with
nfsroot and the all modification will be put under /tmp/w on NFS
server.
Enjoy!