Fault Injection Testing. Spurious Space Depletion? Sure, Why Not?

When file systems run out of space bad things happen. We like to investigate what those “bad things” are but to do so we have to create artificially small installation directories and run CPU-intensive programs to deplete the remaining space. There is a better way on modern Linux systems.

If you should find yourself performing Linux platform fault-injection testing you might care to add spurious space free failures. The fallocate() routine immediately allocates the specified amount of file system space to an open file. It might be interesting to inject random space depletion in such areas as Oracle Clusterware (Grid Infrastructure) installation directories or application logging directories. Could a node ejection occur if all file system space immediately disappeared? What would that look like on the survivors? What happens if large swaths of space disappear and reappear? Be creative with your destructive tendencies and find out!

@goryszewskig : Thanks for stopping by! I have a kernel DLM that eats space even faster for those times I want to be nasty (fault-injection testing) but now that the fallocate() routine is prime time I promote its use for such purposes.

My lab VMs tend to have quite small filesystems so as not to waste space so I find they inadvertently fill up quite often (e.g. I’ve seen clusterware produce 500MB or more of logs without too much effort). It’s not usually pretty…

Yes you can but it is not ***immediate*** :-) That’s the evil aspect and when torture testing systems you have to be evil. So script a quick check for how much free space there is then fallocate() all but, say, 4K of it again and again in a loop in filesystems where Oracle (or any application) needs free space. It’s just a way to see whether spurious space depletion is a problem for your application to handle.

EMC Employee Disclaimer

The opinions and interests expressed on EMC employee blogs are the employees' own and do not necessarily represent EMC's positions, strategies or views. EMC makes no representation or warranties about employee blogs or the accuracy or reliability of such blogs. When you access employee blogs, even though they may contain the EMC logo and content regarding EMC products and services, employee blogs are independent of EMC and EMC does not control their content or operation. In addition, a link to a blog does not mean that EMC endorses that blog or has responsibility for its content or use.

This disclaimer was put into place on March 23, 2011.

Follow Blog via Email

Enter your email address to follow this blog and receive notifications of new posts by email.