Ceph: Roll Your Own or Turnkey?

Ceph offers an open source storage alternative to those who need object, block, and file services. Initially developed at UC Santa Cruz and funded by a number of government agencies, Ceph has grown into a very interesting solution for a myriad of storage problems. Since it is open source, you have the choice of deploying it yourself or purchasing a turnkey version that simplifies installation and management. Which way should you go?

This is an age-old question whose answer says as much about the person answering it as it does about the solution. Some people like to build things themselves, and don’t mind a significant amount of tinkering to get things off the ground. Others would be more interested in turning things on and just configuring them.

The main advantage of using the open source, build your own approach is upfront cost. The open source version is “free,” as you do not pay for a license to download. You can also install it on a “free” operating systems like CentOS. The word “free” is in quotes here because there will be a significant investment in labor to install and configure a product like Ceph. While Ceph offers a significant amount of impressive technology for an open source product, the number of steps required to create a functioning Ceph server is significant. The following is a discussion of the steps taken directly from the documentation at docs.ceph.com.

One very important step that must be done before even beginning the steps listed on ceph.com is to decide what hardware to use for your Ceph nodes. Obviously a turnkey appliance would make this decision, but the administrator needs to decide what system to use, as well as how much computing power, memory size, and storage capacity it should have. You will also need to decide what type of storage controllers and disks to use inside your Ceph nodes.

Before Installing Ceph (i.e. preflight)

The first thing to do before installing Ceph is decide on and install a Linux distribution. Ceph can run on Debian/Ubuntu, RHEL/CentOS, and openSUSE. This will most likely be dictated by your environment. Once you decide on the operating system and installing it, each operating system has a mechanism for adding the Ceph repositories to the Ceph admin node. Once adding the Ceph repositories to the Ceph admin node, you’ll need to install ceph-deploy.

The Ceph admin node requires a user that has password-less sudo access to the Ceph nodes. After making sure you have NTP and an SSH server installed, you will need to create a Ceph user on each node, ensure that said user has sudo privileges, and then enable password-less SSH access from the Ceph admin node using ssh-keygen and ssh-copy-id to each Ceph node. The Ceph documentation also recommends that you add the Ceph user to the ~/.ssh/config file on the admin node so that you can SSH as these users without having to specify –username each time.

You must then perform a number of network setup tasks depending on your distribution. Some distributions are set to disable networking upon reboot, so you will need to change that. Ensure that each of the hostnames of your Ceph nodes resolves to the appropriate IP address and that you can ping them via their hostname. You then need to ensure that the appropriate port is opened up in whatever firewall your distribution uses. For example, RHEL7 uses firewalld. To configure firewalld for ceph, you must add the ceph-mon and ceph services to the public zone and ensure you make the settings permanent after reboot. You may also need to disable requiretty if it is set by default in your distribution, and then specify to disable it only for Ceph. RHEL sets SELinux to enforcing by default, and the Ceph documentation recommends disabling it during the installation process. The final preflight step is to make sure that your package manager has priority/preferences packages installed and enabled. On CentOS, you may need to install EPEL, and on RHEL, you may need to enable optional repositories.

Set Up Your Storage Cluster

Assuming you read and follow all of the preflight steps, Ceph documentation recommends creating a directory (e.g. my-cluster) for maintaining the configuration files and keys that ceph-deploy will generate for your cluster. Once you have created this directory, cd to the directory you created.

It’s interesting that at this point in the documentation, it mentions three commands that will completely purge your Ceph deployment. It’s essentially a factory reset after which you will need to completely reinstall Ceph. This suggests that many people get far down into their Ceph configuration and run into some sort of problem that can only be solved by a complete reset.

To create the cluster, use the ceph-deploy new command. It will create a Ceph configuration file, a monitor secret key ring, and a log file in the current working directory. You should then change the default number of replicas to only two so that Ceph can achieve a clean state with just two Ceph object storage daemons (OSDs). If you have more than one network interface, you should specify which interface you’re going to use in the configuration file as well. Once you have done all this you install Ceph using ceph-deploy install and specify the names of the nodes you are installing. You then add the initial monitors and gather they keys using ceph-deploy mon create-initial.

You then need to create an OSD on each node. The quick start guide uses a directory, but Ceph recommends that full installations use disks. This step uses ceph-deploy osd prepare and ceph-deploy osd activate. After this, you need to copy the configuration file to each node using the command ceph-deploy admin. If you successfully ran all these steps, you can check your cluster’s health running the command ceph health. Your cluster should return the state of active + clean. If the cluster returns the appropriate state, you will then need to create at least one monitor using ceph-deploy mon add. These dozens of setup steps are all so that you can begin storing and retrieving objects to an object store. This does not include SMB, NFS, or Swift compatibility, as those will require additional software and configuration.

What About a Turnkey System?

Comparing installing Ceph yourself to a turnkey system couldn’t be more night and day. While it’s certainly possible to configure your own Ceph cluster from scratch, it certainly is much more complicated than the standard Web server or NFS server, partly because it requires a cluster of at least five servers. Having a preconfigured system could make things significantly easier. Instead of enabling and disabling various services simply to begin running the appropriate ceph-deploy commands, you simply turn on the systems and follow the appropriate prompts.

Consider, for example, the Aquari Storage product that offers a turnkey Ceph cluster. The entire preflight checklist and creation of clusters is part of the storage installer. Instead of having to go back and forth between your operating system documentation and the Ceph documentation, you just follow a series of prompts that will install your operating system, install the SSH server, create the appropriate user, push it out to the nodes, deploy Ceph, create and name the cluster, and add the monitors. Once it’s installed, additional actions such as adding and removing nodes and OSD’s are all handled via the administrative GUI (or command line if you prefer). The Aquari Storage software also supports adding, configuring, and managing SMB, NFS, S3, and Swift support all via the GUI.

StorageSwiss Take

Those interested in experimenting with a Ceph deployment will probably not be intimidated by the myriad tasks mentioned above. They are most likely Linux system administrators, after all. Linux administrators are not historically afraid of having to get their hands dirty to get a new system up and running. But they are also not above using a commercially available product that helps make their lives easier. The decision on which is appropriate for your environment is left as an exercise to the reader, but it is important to know that a fully supported, easy to configure product does exist for Ceph.

Share this:

Like this:

Related

W. Curtis Preston (aka Mr. Backup) is an expert in backup & recovery systems; a space he has been working in since 1993. He has written three books on the subject, Backup & Recovery, Using SANs and NAS, and Unix Backup & Recovery. Mr. Preston is a writer and has spoken at hundreds of seminars and conferences around the world. Preston’s mission is to arm today’s IT managers with truly unbiased information about today’s storage industry and its products.