Anyone using Gluster?

I'm waiting on two new servers to come in for virtualization testing. Each one will have..

2 X Intel Xeon E564064GB Memory4 X 2TB drives

They will be used with RHEL6.x and virsh/KVM to host about six other CentOS/RHEL6.x VM's. Just light apache/MySQL and an IPA server. What I would like to do is install Gluster on both systems and use it on top of the partition/path I will store the VM's in. I'm hoping to define a directory path on the Gluster mounted area as the storage pool in virsh. My thinking is that this will allow me to do live migrations from one system to another bypassing the need for any external storage or using DRBD to replicate block devices. So, am I crazy to think this is possible?

I would Also like to know if anyone has tried or uses Redhat's commercial version of Gluster(Virtual Storage Appliance)

I used the Gluster .rpms to install Gluster as a shared file system for KVM with CENTOS 5 on two test servers and tested live migrations of VMs between them. This was using 3.1 .rpms from gluster.com before Red Hat merged the site into theirs, so my information is a little out of date.

The migrations worked quite well, I got about 3 seconds downtime (measured with HTTP requests to a web server on the VM) migrating a VM between servers and no detectable outage migrating it back.

I did have some trouble with failing physical nodes in and out of the cluster. Sometimes when I brought a node back online it appeared to be copying all of the contents from the remaining node and sometimes the whole cluster would hang. I may have been doing it wrong.

Performance was tolerable for VMs (I was only running gigabit ethernet for transport) but I tested it with an application using many small files and performance was pretty awful.

One of our customers tested the supported virtual appliance and was not impressed. However, this was just after they'd released the appliance (long before the Red Hat acquisition) so things have probably changed considerably.

I have high hopes for Gluster in the future, I hope it becomes part of RHEL proper and the whole supported VM thing goes away. I know we have a bunch of customers that would like to see the back of GFS.

I've been subscribed the the users list and the variety of problems people hit is astounding.

If you're in an "enterprise" setting, some kind of fully-supported GlusterFS configuration would maybe be OK.

Otherwise, use it and hope it doesn't break in some weird way.

E.g. a while back I learned (from reading the users list) that if a brick goes down in a 'replicated' filesystem, you'll have a full downtime while the thing re-replicates. Which might be a while if it's a big brick.

E.g. a while back I learned (from reading the users list) that if a brick goes down in a 'replicated' filesystem, you'll have a full downtime while the thing re-replicates. Which might be a while if it's a big brick.

I have been testing gluster 3.2.5 on 4 KVM virtual machines. Two are CentOS 6.2 and two are CentOS 5.8. The two CentOS 6.2 VM's are bricks for a two pair replicated volume. I have all four set as bricks for a Distributed-Replicate with double replication. When I down one of the CentOS 6.2 VM's I can read and write to the replicated volume from a fifth VM mounting the volume using the glusterFS client just fine. So maybe it was a problem with an earlier release. 3.2.5 is the version Redhat sells commercially.

I have not had time to test gluster on my new servers, but once I do I will report back here.

RHEL6.2 server(only virtual host group) install on both systems with the following disk layout. 100GB Linux RAID1 across all four drives for OS(ext4). 1GB swap partition on all four disks. Linux RAID10 across all four drives for the rest(3.4TB) with LVM on top. 200GB logical volume formatted ext4 mounted to /gluster-test. Both systems have the first, second, and third NIC bridged for KVM guests to use on my different VLAN's. Fourth NIC is connected to each system directly via cat6 patch cable. The fourth NIC has a non routable private TCP/IP network used for live VM migration and glusterFS cluster communications.

* I Cannot get virt-manager to use it at all for storing VM's. Here is what I tried.

1. I first tried adding the mounted volume as a simple directory based store. When I create a VM and begin an install it errors out saying it cannot open the disk image. I googled the error and apparently others have tried this too with the same results.

2. Virt-manager allows for the selection of glusterfs in the network filesystem storage pool type. However, this does not work at all.

3. copied over an existing VM image and tried to start it. It fails with the same error.

Tests using NFS to mount Gluster volume

Mounted gluster-test volume using NFS from localhost to /vm-store on each system.

* Virt-manager works just fine when using the volume as a directory based storage pool. I installed two RHEL6.2 VM's(20GB images, 4GB of ram, 2 CPU's) at the same time. Cloned one VM twice to get a total of four VM's running at the same time. Had two doing software updates at the same time with good performance.

* Live migration also seems to work just fine. It only takes a few seconds to move the 4GB contents in memory over. There is of course a short pause in network activity as the new routes through the switch are found. I never saw a pause more than 30 seconds before ssh sessions to the VM's started responding again.

More testing and thoughts.

I still need to test how well gluster self heals after I down the main host and start the guests on the backup host.

I would also like to test this with a pair of 10GB NICS or some infiniband.. I don't have access to either at the moment for these two nodes.

Setting up Gluster is a lot easier to setup than DRBD for doing high availability. When this matures I think GlusterFS will give KVM a huge advantage over VMware and hyper-V for people who want to do two node high availability VM hosts.

I would love, love, LOVE to see a step-by-step walkthrough on setting that scenario up, DMC. L-O-V-E.

BTW, have you tried iSCSI and Gluster? You ought to be able to at least try setting up KVM images using iSCSI as well. It would be interesting to see if it performs better doing an iSCSI export rather than doing filesystem-level imaging...

I would love, love, LOVE to see a step-by-step walkthrough on setting that scenario up, DMC. L-O-V-E.

BTW, have you tried iSCSI and Gluster? You ought to be able to at least try setting up KVM images using iSCSI as well. It would be interesting to see if it performs better doing an iSCSI export rather than doing filesystem-level imaging...

I will try to post my step by step notes tomorrow.

I tinkered with the idea of using the FUSE drivers to mount the volume and export raw files as block devices via iSCSI back to the host itself. That is the only way I see it working. However, That makes the setup quite a bit more complicated and I doubt it would perform any better than NFS due to the extra overhead. Incase you are unfamiliar with how Gluster works. Gluster sits on top of a normal formatted partition like ext3/4, XFS. You need to mount the Gluster volume via NFS or the FUSE driver for the realtime replication to work. Here is the different layers assuming I wanted to use iSCSI.

OK, back with another update on my testing progress.. I have been to busy with other work projects to get my testing done. Good thing too because they released glusterfs 3.3 beta3 last week. I started my testing from scratch with the new beta3.

Good News! I got VM's running under the native GlusterFS FUSE mount. During my Google searches I came across some mailing lists discussions about the problem I was having. It turns out FUSE does not support direct I/O. Apparently patches have been submitted but they have not been taken yet. In my case it doesn't matter. I'm a RHEL/CentOS whore so unless Redhat implements it, I'm SOL. In any case simply setting my VM drive images to use cache=writeback solved my problem. I tried cache=writethrough but the performance was crap. So no more need for old and busted NFSv3 mounts.

I also finished failover testing. Here are the results..

Main host running three RHEL server VM's each with 4GB ram, 2 cpu's, 20GB HD. All three VM's had been live migrated over to the backup host and back to the main again. This is to make sure everything was working and the xml files were on the backup host.

1. I had a script(io-test.sh) running in a loop on all three VM's that did the following. Create 600 2MB files and 20 30MB files then remove them.

2. While everything was humming along I pulled both power cables out of the main host to simulate a power outage or some other sudden failure.

3. Fire up virt-manager on the backup host and it was hanging at connecting to the qemu. I have a hunch that this was due to the lost connection from the main host libvirtd. Restarting libvirtd via the init script did not work. The other instance of the process was defunct and I could not kill it. Instead of wasting time I simply rebooted the backup host. After the reboot I was able to start all three VM's just fine.

4. Started the io-test.sh script on all three VM's.

5. Waited 20 minutes and plugged the main host back in and let it boot up. Checked to make sure all partitions mounted ok and that gluster mount was mounted too. I noticed that md was resyncing the raid10. Since this system has a 3.4TB raid10 that was going to take a while.

6. So, being my impatient self I only waited until it was about half way through rebuilding the RAID10. I fired up virt-manager on the main host and live migrated the three VM's over. Went without a hitch. Of course things were a little sluggish due to the RAID rebuild but it worked just fine.

I'm still in the process of getting my notes cleaned up for a how-to. Should be back by early next week with the goods.

I am trying to wrap my head around what Gluster is. Is it for building your own Isilon infrastructure on your own hardware using OSS software and using linux NFS and samba for sharing it out or is it a cluster filesystem for HPC, serving web content off of multiple servers and other uses when multiple servers might need concurrent access to the same file and NFS/CIFS won't cut it?

I am finding Gluster.org to be lacking in helping me understand this. I have a problem that this might solve, if I could just wrap my head around what it is used for.

Gluster is all of those things. One thing it cannot do is individual preference for file/directory replication. Replication is global. I recommend installing it on a few CentOS 6 VM's and setup a test cluster. It is very easy to setup and use.

Here is an early draft of my how to for using GlusterFS on KVM hosts for shared VM storage. I still need to add info about how to use virt-manager to do migrations and more details.