Right. Thing is, I let these volumes sit idle for a good twenty or thirty minutes before running that "find" command, and it never showed any self-heal activity ongoing that I could see in logs, nor were gfid's getting populated.

I mean I suppose there's nothing especially stopping me from forcing one of the bricks out, reducing to replica 1, and then running a full heal, then adding the other brick back with the directories purged and run a rebalance.

djgerm: yeah, it's kind-of pointless to do more than replica 3. This isn't a mirroring service, it's a clustered filesystem. Unless you're exceeding the read capacity of three servers, you're not getting any benefit from the additional replicas.

JoeJulian: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns, or (#2) One caveat is that the clients never learn of any other management peers. If the client cannot communicate with the mount server, that client will not learn of any volume changes.

Yes. If you don't care about that latency it should work great. I would look at quorum capabilities and decide how to best use it for your situation. With the increased possibility of network partition, it might be important to you.

guys is anyone running a CTDB setup with GlusterFS? I'm looking at the lockfile setup, and I see in the docs they recommend creating a dedicated volume for the lockfile... I'm toying with the idea of creating a file, formating it to XFS, mounting it and using it as the fs for the lockfile

hi there I'm trying to setup an hpc cluster with ~40 nodes. All nodes are connected to a san with fc, and to a infiniband network. I would like that only some dedicated storage nodes access the san and export the filesystem to the rest of the cluster via infiniband. For now I'm not sure about how many storage nodes will be needed since the range of appliaction vary a lot. The space assigned by the storage g

uys is fixed, but they can give me as many lun as I need. Does it make sense to use glusterfs for such a setup? Is it possible to combine clvmd and glusterfs such that if tomorrow I need to add a new storage node I only create re-arrange lvm volumes?

guys if a volume is started, and replicated across 2 x nodes...and one machine is shutdown down, the volume is still started.... if the 2nd node is restarted, it connects to the running volume... all good

RH ship hook scripts that help (more hinder imo) the configuration and starting of ctdb, smb, editing fstab, blah blah.... but they're really screwing up the system... they don't play nicely with systemd

Ethical2ak: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.

Yes but this will not be representative of the production. I already tested 3.7 (and 3.6.5) on staging setup, and everytime everything went fine. It's always in production with 11TB data and 1k clients that we encounter issues.

last time I tried geo-rep it was transatlantique with a poor wan link, it worked, but then took too much time to convert so we disabled it and use custom rsync scripts then. Now that I have the perfect condition to use geo-rep, well it crashed my two brick by night, make all QA testing failed ;/

JoeJulian: thanks for that, no i don't see a "hard" mount option there. On NFS, you can have soft or hard mounts as defined in your mount -o ... i'm storing VHD files and I prefer hard mount because it will not return IO errors to the guest OS, instead the VM will wait or hang until storage is available again

JoeJulian: if you sent IO errors to the guest OS, then they start thinking that they need an fsck and will mark their root are read-only... using a gluster fuse mount, I had a small network disrupt and all the VMs had to be restarted and many had to run fsck /chkdsk when they rebooted it was yucky. I never had that problem using the NFS hard mount before and I dont see a FUSE/gluster alternative :(

JoeJulian: why is XenServer so weird about using gluster over NFS? From the hypervisor i can read/write to the nfs mount at 100+ MB/sec, but within a VM with it's VHD on gluster over NFS, i seem to max out at 7MB/sec even on 10GbE

im really stuck trying to decide what to do with my platform :/ as of right now i have no clue what i should do... so many options... im curious how others would choose to set this up... i'm working with 8 servers here, 4 as virtualization nodes and 4 as storage nodes... xenserver is required on the 4 virt nodes, and I have gluster running successfully in dom0... each of the virt nodes has dual 10 GbE and 2 x 4TB SATA disks.

thanks JoeJulian , the need is to run about 100 VMs, and not have them be laggy. the 4 xenservers have enough capacity to run those VMs, now I just need to select a shared storage method so that the VMs are Agile and can live migrate between hosts

thats about it. I think i have the right hardware for the task... and I get really good performance as well until i get to the virtualization layer in Xen there where as lanning said, it's breaking up the data into very small writes

JoeJulian: i used to suffer many many cases of read-only root VMs until i found xenserver defaults is for "soft" mount. after I changed that to "hard" mount, i've never had a read-only root since then because the VM will simply freeze, forever, until storage is restored, or until it is force shutdown

JoeJulian: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.

I've got a 3 node Gluster replicate cluster - I've created a new server which is to act as a client and mount the data to a directory. It does do this, however, when I try to write to the directory it claims that it is read only. I do not have read only specified in my fstab.. what gives?

Hi JoeJulian. samba with samba-vfs-glusterfs mounting the distribution mode volume. It shows directories and files. when I create a folder and files, it does create it. but when I delete it through cifs share, it disappears but it really didn't. as if it only deleted the metadata. the folder and file is still intact.

We have customers that are in that same place. Luckily we're a datacenter so we can let them expand into our hardware using their own vcenter, and simultaneously offer them openstack so they can learn and migrate their deployments without that fear of loss.

Our VP of engineering mentioned that he saw a VSAN-esque implementation "tool" that was put in to an open beta by EMC. But the tool they were using could pool redundant storage. I'm not sure if that was DRBD or Gluster or Ceph, but that's what took me down this road. Finding a solution for shared VM storage without spending $40k on VMware VSAN.