Post navigation

While working on creating the next version of the pmsApp OVA, I ran into some issues so I thought I would try to make a small guide to creating an OVA here. Mostly for my own benefit.

I installed version 0.2 of the pmsApp and updated it to get the latest software versions. When I then tried to create a new OVA it turned out to be quite a lot larget than the original. I needed to make the image smaller.

I started out by clearing the yum cache and uninstalling the old kernel, that shrinked the used space to around the same as before the update. I found this guide as to how to prepare a VM to be a template, that freed up a bit more space.

Using VirtualBox, i created an OVA from the VM. Now the problem is that an OVA created with VirtualBox can not be imported into ESXi. Fortunately an OVA is actually just a tar archive so I was able to extract the .ovf file and the vmdk’s without issues. After that it is a queston of editing the ovf and changing the hardware version from virtualbox-2.2 to vmx-Y where Y is the vmware hardware version wanted. I plan on going with vmx-8.

Now the problem is assemling the OVA again, to do that, I used vmware’s ovftool, but it wants a .mf file to ensure that the files are not corrupted. I found this article, explaining how to create the .mf file.

To have everything in one place, here is what I have done to the VM to make it into an OVA:

Share this:

This not an update of the original Poor Mans Storage Appliance, but rather a new take on it. This time it is built through DRBD and ZFS. DRBD is designed for mirroring over network and the zfs filesystem is able to do compression and deduplication although deduplication is not recommended unless you have lots and lots of memory. I started out with Debian 7.5 (wheezy), but it comes stock with drbd 8.3. Most guides I could find on configuring and using drbd is for drbd 8.4. By using backports, I was able to get drbd up to version 8.4, but then I had trouble getting zfs to work correctly. After some trial and error I decided that it was too much hassle and instead opted for Ubuntu. Ubuntu 14.04 LTS comes with drbd 8.4 and zfs installs without any issues. I started with a VM on each of my two physical hosts. Each VM has a virtual disk for the OS and a virtual disk for the shares storage. The following resources were used as reference:

After installing Ubuntu Server, configuring static IP and ensuring host names resolve (either through your own DNS server or /etc/hosts files on both VMs), I started installing software (everything needs to be done on both VMs).

Install the software. drbd to distribute data between nodes, ntp to ensure time is synchronized, zfs to create filesystem capable of doing compression and deduplication, nfs to export the distributed storage and heartbeat to manage the two nodes in a cluster.

Depending on which command is used, you should see the device being ready or syncing, but you should see that now one node is primary and the other node is secondary. (cat /proc/drbd) Now we have a device on which we can create a filesystem (/dev/drbd0). To enable us to choose compression and deduplication we will use zfs. The filesystem will only be created on the primary node.

The zfs documentation says that after the zpool is created, a zfs should be created inside that pool. I have opted not to do that for 3 reasons. The zpool functions as a filesystem, I do not neet more that one filesystem in my pool and if a zfs is created inside the zpool, zfs will try to mount it during boot. If you want compression enabled on the filesystem:

sudo zfs set compression=on myredundantfilesystem

If you want deduplication enabled on the filesystem:

sudo zfs set dedup=on myredundantfilesystem

Now we need to ensure that the mountpoint for the zfs filesystem exists on both nodes:

# Create directory for mounting zfs if it does not exist
if [ ! -d /myredundantfilesystem ]; then
sudo mkdir /myredundantfilesystem
fi

On the primary node the directory should already exist and the zfs filesystem should be mounted. Now we can edit /etc/exports on both nodes to ensure that it can export the zfs filesystem to clients. The export should look something like this:

Most options are “normal” for nfs exports, but the fsid is there to tell the client that it is the same filesystem on both nodes. Finally we need clustering to maintain a failover relationship between the nodes. We have already installed heartbeat, now we need to configure it. (it should be done on both nodes) Before we start configuring heartbeat, we need to disable the nfs server during boot.

sudo update-rc.d -f nfs-kernel-server remove

First we will create the /etc/ha.d/ha.cf file, it should look something like this.

The parameter deadtime tells Heartbeat to declare the other node dead after this many seconds. Heartbeat will send a heartbeat every keepalive number of seconds. Next we will protect the heartbeat configuration by editing /etc/ha.d/authkeys, it should look something like this:

auth 3
3 md5 my_secret_key

Set permissions on the file

sudo chmod 600 /etc/ha.d/authkeys

Next we need to tell heartbeat about the resources we want it to manage. It is done in the file /etc/ha.d/haresources and it should look something like this:

In this example node1 will be primary, but if it fails, the other node will take over. We need a “floating” IP address as our clients needs a fixed IP to connect to. Heartbeat needs to ensure that the drbd device is active on the primary node. zfsmount is a script I have created in /etc/init.d, more about in a short while Lastly, heartbeat should start the nfs server. In haresources there is a reference to the zfsmount script. This is a script I have created my self to ensure that both the drbd device and the zfs filesystem gets activated on the secondary node in the event of primary node failure. The script should be located in /etc/init.d and accept at least the start and stop parameters. My script looks like this:

Some trial and error as well as a lot of looking through /var/log/heartbeat-debug log file helped me create this script. The script must exit with success if it is called with stop even though it has already been stopped.

Initially I thought that it would be sufficient to make the node primary (drbdadm primary myredundantstorage) during a failover, but it seems that the zfs driver does not necessarily recognize that there is a zfs filesystem just because the device becomes available. That is why the import is done. The import will fail if the filesystem is already recognized, but that does not matter in this script, it will still try to mount the filesystem.

I hope this will be helpful for other people as well. It took me a long time to find anything dealing with drbd and zfs at the same time. If you have any questions, don’t hesitate to contact me. ( jimmy at dansbo dot dk )

Btw. I can say that this setup performs a hell of a lot better than zfs and gluster on the same hardware.

Share this:

This is it. I have not been able to find the time or the energy to keep this project updated. The domain pmsapp.org will expire on the 2. of March this year and I do not plan on renewing it. I am sorry to see the project die, but lack of interest and a strained schedule on my part forces me to stop this.

If you are interested in this project or need assistance in setting up your own storage appliance, you can reach me at pmsapp at dansbo.dk.

Share this:

Just wanted to let everyone know that this project is still alive. I am continuing work on creating a web interface to ease deployment of a pmsApp storage cluster. Unfortunately it will probably be some time before version 0.3 with the new web interface is released.

In the meantime I am still hoping for a nice logo for the pmsApp project, with a bit of luck something will have been submitted before the release of the next version.

I would like to thank manyrootsofallevil for his great work in performance testing the pmsApp and at the same time ask for input on how to increase performance. If you have any suggestions or tips on how to increase performance on software RAID created from iSCSI targets, please let us know.

Share this:

A new version (0.2) has just been uploaded. This version adds a web interface to help do the initial network configuration. It also allows NTP and iSCSI configuration without having to dig around configuration files. Have a look at the guide and get the new appliance from the download page.

Share this:

A friend of the pmsApp project – A.Z. has done some failover testing. This is what he has found.

As long as there is no I/O taking place while the cluster is failing over, all is ok. If there is I/O then the results are mixed.

SCP stopped and would not restart

A script I wrote (writing the current date to a file every second) is unable to write to the file, but when the cluster comes back up, it continues on its merry way.

The problem seems to be that an ongoing session is not moved from the failed node to the node that takes over. A new session needs to be established. I am looking in to the possibility of having the sessions transferred from the failing node to the new active node. So far all I have found, is an article on how to create a failover nfs cluster. This is however with drbd, but that should not matter much.

A big thank you to A.Z. for testing the pmsApp. Looking forward to your performance tests.