my cluster is made up of machines each with a SINGLE hard drive. I want to use 1/2 of each disk and pool them all together and make a RAID 10 or 5 set. And then be able to access said volume RW from any node and should a node die, the filesystem keeps running.

I can't think of a way to combine (C)LVM, GFS, GNBD, and MD (software RAID) and make it work unless just one of the nodes becomes the MD master and then just exports it via NFS. Can it be done? Do commercial options exist to pull off this trick?

My backup plan is to define a shared SAN volume intended for high WRITE volume, partition it, and each node that needs to scribble manages it's own partition+filesystem. And it's up to me to make sure no 2 nodes try to own the same piece.