Announcing clr-cloud-init

We've just released a new cloud-init implementation written specifically for Clear Linux* OS.

You may already be thinking: "Why another implementation?"

Two existing projects already implement a standard cloud instance initialization process. There's a Python-based implementation that might be considered the "reference standard." And there's a go-based implementation created by the CoreOS* project. Both these projects are excellent implementations, yet we chose not to use them in our project.

As all software engineers do, we aim to do as much as we can in as little time and effort as we can. This is a bit of a paradox, so let me elaborate. We want to provide as much functionality as we can, but we don’t want to do useless work, so if we determine that some functionality is unlikely to be used, we're not likely to include it. We also want our OS to start as fast as it can, but we don't want to cut out functionality that everyone needs. These are natural conflicts that exist in the realm of OS resource management.

One of the largest resources that the OS needs is storage. We ideally want to provide as much functionality as possible, but this increases the size of our cloud image. And that increased storage has a cost. While ideally Copy On Write algorithms should make spinning up cloud instances extremely cheap, these methods aren't used in many cloud deployments, and most of the time cloud instances are still copied verbatim.

Ultimately, we strive to make the cloud images as minimal as we can. Once a cloud instance is provisioned, we have an excellent and efficient method available to expand the set of available software, in the form of swupd, which allows installation of entire functional software bundles with minimal download bandwidth consumption. This resource consumption is optional, however. So making the initial resource consumption lower is a win for every cloud instance.

In several other projects we've contributed to, Intel has striven to "race to idle." This is an excellent optimization strategy that pushes us to reduce the cost of doing basic things and eliminate things we don't need to do. Especially at boot time, this matters a lot.

Our cloud images should therefore really be as minimal as possible, feature-wise, and as small as possible. And by that we mean that if you wish to deploy a nginx, or mariadb, or postfix cloud instance and don't need Python* on your cloud instance, it shouldn't be there. We can't do that if the absolute minimal cloud image already requires a significant Python installation just to initialize that cloud instance for further use. And our base image currently doesn't depend on go-based code, either.

We have a strong preference for writing plumbing code like this in C, but not very often do we find that we should really replace the existing implementations that are already out there with a new version. In this case we weighed all the pros and cons carefully and decided that the cost of maintaining a limited feature set implementation in C was worth doing, considering all the factors. The project itself did not take that long to implement after the decision, about two person-months or so, although that's a somewhat rough guesstimate.

Enough about why! Here are the technical details on the project:

clr-cloud-init is implemented in C, based on glib, json-glib, YAML* and a bit of cURL*. It fetches OpenStack metadata and cloud-config user data, and initializes your cloud instance for you. This is akin to "customization," the part that makes the cloud instance actually useful to the end user. These configuration "blobs" are provided to your cloud instance through (usually) an HTTP service. The cloud-init binary will run at the first boot of your cloud instance, fetching these data blobs and then processing them.

These blobs provide useful data for OpenStack orchestration components and a way for a user to make the cloud instance ready for its intended use. This could be done, for example, by updating all software and installing new software. Or this could be accomplished by starting some system services, configuring user accounts and configuration files, provided access credentials, etc.

Implemented in C, the binary runs extremely fast. Ignoring any network latencies, cloud-init takes about 30ms to complete initialization of an instance in a typical test case scenario. Its storage footprint is well under 1MB ignoring dynamic libraries.

clr-cloud-init provides a modular implementation to support more cloud-config functionality. Currently supported are user and groups creation, adding SSH keys, manipulating system services, writing config files, performing package updates and installing new package bundles. We are planning to add a few more pieces of basic functionality in the coming weeks.