Tuesday, May 10, 2016

Rapid Ceph Kernel Module Testing with vstart.sh

Introduction

Ceph's vstart.sh utility is very useful for deploying and testing a mock cluster directly from the Ceph source repository. It can:

Generate a cluster configuration file and authentication keys

Provision and deploy a number of OSDs

Backed by local disk, or memory using the --memstore parameter

Deploy an arbitrary number of monitor, MDS or rados-gateway nodes

All services are deployed as the running user. I.e. root access is not needed.

Once deployed, the mock cluster can be used with any of the existing Ceph client utilities, or exercised with the unit tests in the Ceph src/test directory.

When developing or testing Linux kernel changes for CephFS or RBD, it's useful to also be able to use these kernel clients against a vstart.sh deployed Ceph cluster.

Test Environment Overview - image based on content by Sage Weil

The instructions below walk through configuration and deployment of all components needed to test Linux kernel RBD and CephFS modules against a mock Ceph cluster. The procedure was performed on openSUSE Leap 42.1, but should also be applicable for other Linux distributions.

Network Setup

First off, configure a bridge interface to connect the Ceph cluster with a kernel client VM network:

Kernel VM Deployment

Build a kernel:

> cd $kernel_source_dir
> make menuconfig

$kernel_source_dir should be replaced with the actual path. Ensure CONFIG_BLK_DEV_RBD=m, CONFIG_CEPH_FS=y, CONFIG_CEPH_LIB=y, CONFIG_E1000=y and
CONFIG_IP_PNP=y are set in the kernel config. A sample can be found here.

> make
> INSTALL_MOD_PATH=./mods make modules_install

Create a link to the modules directory ./mods, so that Dracut can find them:

Conclusion

A mock Ceph cluster can be deployed from source in a matter of seconds using the vstart.sh utility.
Likewise, a kernel can be booted directly from source alongside a throwaway VM and connected to the mock Ceph cluster in a couple of minutes with Dracut and QEMU/KVM.

This environment is ideal for rapid development and integration testing of Ceph user-space and kernel components, including RBD and CephFS.