jclouds Proposal for Apache Incubator

Abstract

jclouds is an open source cloud agnostic library that enables developers to access a variety of supported cloud providers using one API.

Proposal

jclouds provides two portable API abstractions for cloud computing and cloud storage in the form of Java or Clojure libraries. By abstracting away vendor-specific details, jclouds supplies a solution for users to avoid lock-in to a specific cloud vendor. It also provisions hooks to make it easy to take advantage of specific features from those vendors.

Background

jclouds began in March of 2009 while Adrian Cole was working on the Amazon S3 plugin for Infinispan. He discovered that existing S3 libraries would not pass the concurrent test suite needed for write behind persistence. When it became clear that a new library was needed, jclouds formed as an independent project founded by Cole under the mentorship of Manik Surtani.

The fledgling jclouds team soon encountered two portability issues: runtime portability with Google App Engine and compatibility with the similar cloud files service from Rackspace.

The legacy of Google App Engine is still present within jclouds and is expressed by the 'driver' design which doesn't require threads. Supporting Rackspace started a legacy of cloud portability which would soon extend to provisioning. As jclouds supports newer and cutting edge cloud technologies, it is in a constant state of evolution.

jclouds was founded in Shoreditch, London. The ensuing 4 months backpacking between European Java User Groups accounts for the initial population of European developers and users within the community. The initial contributors to jclouds included Adrian Cole, James Murty, Andrew Newdigate, and Andrew Phillips. Manik Surtani served as a mentor, and there were a number of other advisers as well.

Rationale

There are many providers that offer IaaS clouds, but few APIs that can bridge the gap between cloud providers. jclouds supplies an abstraction layer for both compute and storage resources that makes it easy to work with many different cloud APIs using a simple and singular API. jclouds provides this abstraction layer for the Java and Clojure communities. It is a unique niche within the cloud community.

Initial Goals

Build a more centralized community pulling together jclouds developers, jclouds integrators, and service providers who use jclouds.

Although jclouds has a fairly collaborative community, the committers wish to further bridge the gap between developers and community by encouraging more community involvement in development processes.

Break down from a monolithic release process into one that is more decentrilized.

Bring the jclouds core code base consisting of ComputeService and Blobstore into Apache.

jclouds will not move all existing provider code into Apache. The goal is to move the provider code for those clouds that have complete portability and are tried, tested, and true. This includes code for OpenStack, Amazon EC2, and Google Cloud.

Current Status

Meritocracy

jclouds has an active community of contributors who are encouraged to become full-fledged jclouds committers. A jclouds committer understands the importance of seeking community feedback, actively listening to suggestions, and fostering relationships with community members who make contributions. The entirety of jclouds is built upon the relationships that have been built with community members. This camaraderie will continue to be promoted as jclouds is introduced into the Incubator and beyond.

jclouds encourages involvement from its community members, and the issues are open and available to any developers who wish to contribute to the project. The jclouds team currently seeks help and asks for suggestions utilizing the jclouds and jclouds-dev Google groups and IRC on irc.freenode.net#jclouds. Any and all submissions for changes are reviewed and taken into consideration.

There have been over 100 individuals who have contributed code to jclouds throughout the years, and this number has continued to increase at a rate of 1 or 2 per month. It is accepted and anticipated that the amount of developer participation within the community will increase under the guidance of the Apache Software Foundation (ASF).

Core Developers

The core contributors are a diverse group comprised of both unaffiliated developers and those hailing from small to large companies. They are scattered geographically, and some are highly experienced industry as well as open source developers. Though their backgrounds may be diverse, the contributors are united in their belief in community driven software development.

More detailed information on the core developers and contributors in general can be found under the section on homogeneous developers.

Alignment

jclouds adoption is growing, and it is no longer feasible for it to remain as an isolated project. Apache is experienced in dealing with software that is very widely accepted and has a growing audience. The proposers believe that the jclouds team can benefit from the ASF's experience and its broad array of users and developers.

jclouds supports several Apache projects and options exist for integration with others. Apache ACE, Apache Karaf Cellar, Apache Camel, and Apache Whirr all utilize jclouds APIs. Apache Karaf integration with jclouds provides managed service factories and command line support for creating and administering blobstore and compute as OSGi services. Apache Maven may be used with jclouds to build new projects.

Known Risks

Orphaned products

jclouds is already being utilized at multiple companies that are actively participating in improving the code. The thriving community centered around jclouds has seen steady growth, and the project is gaining traction with developers. The risks of the code being abandoned are minimal.

Inexperience with Open Source

jclouds began as an open source project in March of 2009 and has remained so for 4 years. Additionally, many of the committers have extensive experience with other open source projects. jclouds fosters a collaborative and community-driven environment.

In the interest of openly sharing technology and attracting more community members, several of our developers also regularly attend conferences in North America and Europe to give talks about jclouds. jclouds meetups are also planned every few months for developers and community members to come together in person and discuss ideas.

Homogeneous Developers

Thanks in part to founder Adrian Cole's love of travel, jclouds has attracted committers across North America, Europe, and Asia. The committers' technical and professional backgrounds are diverse as well, with the range of experience being several months to 18+ years. The committers are employed at both smaller companies, such as Andrew Gaul from the startup Maginatics, and larger corporations, as is the case with Matt Stephenson from Google. Though there are currently a handful of contributors from the same companies, none of the initial committers are from the same company. jclouds committer Andrew Bayer is also a PMC member and committer on the Apache projects Bigtop, Flume, Sqoop, and Whirr.

The jclouds developers thrive upon the diversity of the community. The jclouds IRC channel is always active, and the developers often collaborate on fixes and changes in the code. They are always happy to answer users' questions as well.

jclouds is interested in continuing to expand and strengthen its network of developers and community members through the ASF.

Reliance on Salaried Developers

None of the developers are directly funded by jclouds. Though some of the developers are paid by their employer to contribute to jclouds, many jclouds developers contribute code and documentation on their own time and have done so for a lengthy period. Given the current stream of development requests and the committers' sense of ownership of the jclouds code, this arrangement is expected to continue with jclouds' induction into the ASF.

Relationships with Other Apache Products

jclouds and Apache Libcloud address similiar use cases. However, jclouds supplies these services for the Java and Clojure communities whereas Libcloud provides them for the Python ecosystem.

While jclouds does not directly rely upon any Apache project, it does support several Apache projects and has options to collaborate with several others. More specifically, jclouds currently supports Apache Whirr, Apache ACE, Apache Karaf, and Apache Camel, and options exist to use Apache Maven as a build tool with the jclouds API.

jclouds includes support for the Apache CloudStack API and is used as a compatibility test tool for its EC2 interface. jclouds can also be used to test Apache Deltacloud EC2 portability.

An Excessive Fascination with the Apache Brand

jclouds recognizes the fortitude of the Apache brand, but the motivation for becoming an Apache project is to strengthen and expand the jclouds community and its user base. While the jclouds community has seen steady growth over the past several years, association with the ASF is expected to expedite this pattern of growth. Development is expected to continue on jclouds under the Apache license whether or not it is supported by the ASF.

Documentation

The jclouds project documentation is publicly available at the following sites:

http://jclouds.org: installation guide, user guides, development resources, news, resources to get started