Introducing Continuuity Loom: Modern Cluster Management

March 20, 2014

Jonathan Gray, Founder & CEO of Cask, is an entrepreneur and software engineer with a background in open source and data. Prior to Cask, he was at Facebook working on projects like Facebook Messages. At startup Streamy, Jonathan was an early adopter of Hadoop and HBase committer.

We started off with simple scripts. But scripts had multiple issues – they required a lot of setup, there was no central management keeping track of available clusters, and the scripts had to be changed whenever we moved from one cloud to another, or one software deployment to another. Next, we started looking at different SCM (software configuration management) technologies and picked Chef. Combining scripts and Chef recipes, we’re able to get to one-button push, but it became extremely complex for two reasons – there were an increasing number of customizations needed for each new type of cluster and a significant amount of setup was needed on every developer’s box.Our flagship platform, Continuuity Reactor, utilizes several technologies within the Apache HadoopTM ecosystem, so as we started building it, our developers needed an easy way to test Reactor on real distributed systems. They needed a fast, self-service way to provision clusters, both on our own hardware and in public clouds, because testing against real clusters is such a critical and frequent component of our development cycle. Our vision was to enable all Continuuity developers to be able to quickly create a cluster with the push of a button.

When we launched Reactor, we wanted to enable developers to deploy our single-node Sandbox Reactor to a public cloud directly from the website. This forced us to think harder about designing a complete production system while still providing an easy and flexible solution for our developers. We wrote a combination of scripts and Chef recipes, designed a system that can templatize the Hadoop cluster, added capabilities to make it extendable to other services and providers, and made it very developer and ops friendly – that’s when Loom was born.

Today, Loom is used to provision Sandbox Reactors on the Continuuity Cloud and as an internal DevOps tool to test new and incremental features on an ongoing basis. We’ve been developing and using Loom for almost a year and we’re noticing the benefits: our developer and DevOps productivity has doubled, and our development and test cycles have become faster.

We built Loom for ourselves. But today, we’re open sourcing it for anyone to use because we believe that the broader developer community can benefit from it.

What is Continuuity Loom?

Continuuity Loom is cluster management software that provisions, manages, and scales clusters on public clouds and private clouds. Clusters created with Loom utilize templates of any hardware and software stack, from simple standalone LAMP-stack servers and traditional application servers like JBoss to full Apache Hadoop clusters comprised of thousands of nodes. Clusters can be deployed across many cloud providers (Rackspace, Joyent, etc.) and virtualization platforms (like OpenStack) while utilizing common SCM tools (Chef, scripts, etc.).

Loom from Your Laptop

Developers can use Loom to provision and decommission a cluster directly from their laptop. Loom creates a very simple user API that allows the developer to choose the type of cluster, how many nodes, and which cloud. Any developer can go from zero to Hadoop, from their laptop to any cloud, in minutes.

Worry-Free Provisioning

Loom simplifies the installation and configuration of any software stack, including Hadoop, and ensures that all installations are verified before a cluster is made available. Developers can create custom cluster types ranging from single VM application servers to large-scale, distributed, heterogeneous clusters.

Download Loom for Free

As a developer, building Big Data applications using Continuuity Reactor is just one part of the equation. The other part involves provisioning and managing your Hadoop cluster, and deploying your applications to the Hadoop cluster. With Loom, we’re empowering DevOps in a truly self-service way. Download and run Loom on a Mac laptop (currently experimental), and start or spin up Hadoop clusters and other software stacks across any private or public cloud.

Open Sourcing Loom

Loom is a relatively new project but highly extensible in the providers and services that can be supported. Open sourcing Loom allows others to add support for new software stacks, currently unsupported recipes/scripts, and new infrastructure/provisioning systems. Help us make Loom better by contributing here and please report any issues, bugs, or ideas.

Loom for the Enterprise

In many companies, developers must submit requests to individuals in other parts of the organization to obtain a Hadoop cluster or to run distributed software on a group of machines. This process can be incredibly slow and burdensome for both the developer and IT, decreasing developer productivity and increasing IT overhead. With Loom, the IT department can codify reference architectures and maintain a centrally controlled catalog of cluster templates that can be provisioned directly by developers. IT administrators then use a centralized dashboard to monitor and manage multiple clusters concurrently across multiple clouds.

Download Loom to start managing and provisioning your own clusters today, or contact sales if you need enterprise support for Loom.