OpenStack is facing an important choice: does define a new set of API’s, one of many such efforts in cloud infrastructure, or does it build around the existing AWS API’s? So far, OpenStack has had it both ways, with some new API work and also some AWS-based effort. I’m writing to make the case for a tighter definition of mission around the de facto standard infrastructure API’s of EC2, S3 and a few other elements of AWS.

What prompted this blog was my overhearing (or, seeing an email on a list) the statement that cloud infrastructure projects like OpenStack, Eucalyptus and others should “innovate at the level of the API and infrastructure concepts”. I’m of the view that any projects which try to do so will fail and are not worth spending your or my time on. They are going to be about as successful as projects that try to reinvent HTTP to make it better/faster/cleaner/whatever. Which is to say – not successful at all, because no new protocol with the same conceptual goals will match the ecosystem that exists today around HTTP. There will of course be protocol innovation, the last word is never written, but for the web, it’s a done deal. All the proprietary and ad-hoc things that preceded HTTP have died, and good riddance. Similarly, cloud infrastructure will converge around a standard API which will be imperfect but real. Innovation is all about how that API is implemented, not which API it is.

Nobody would say the web server market lacks innovation. There are many, many different companies and communities that make and market web server solutions. And each of those is innovating in some way – focusing on a different audience, or trying a different approach. Yet that entire market is constrained by a public standard: HTTP, which evolves far more slowly than the products that implement it.

There are also a huge number of things that wrap themselves around HTTP, from cache accelerators to 3G content compressors; the standardisation of that thin layer has created a massive ecosystem and driven fantastic innovation, even as many of the core concepts that drove HTTP’s initial design have eroded or softened. For example, HTTP was relentlessly stateless, but we’ve added cookies and cacheing to address issues caused by that (at the time radical) design constraint.

Today, cloud infrastructure is looking for its HTTP. I think that standard already exists in de facto form today at AWS, with EC2, S3 and some of the credential mechanisms being essentially the core primitives of cloud infrastructure management. There is enormous room for innovation in cloud infrastructure *implementations*, even within the constraints of that minimalist API. The hackers and funders and leaders and advocates of OpenStack, and any number of other cloud infrastructure projects both open source and proprietary, would be better off figuring out how to leverage that standardisation than trying to compete with it, simply because no other API is likely to gain the sort of ecosystem we see around AWS today.

It’s true that those API’s would better be defined in a clean, independent forum analogous to the W3C than inside the boiler-room of development at any single cloud provider, but that’s a secondary issue. And over time, it can be engineered to work that way.

More importantly for the moment, those who make an authentic effort to fit into the AWS protocol standard immediately gain access to chunks of the AWS gene pool, effectively gratis. From services like RightScale to tools like ElasticFox, your cloud is going to be more familiar, more effective and more potent if it can ease the barriers to porting from AWS. No two implementations will magically Just Work, but the rough edges and gotchas can get ironed out much more easily if there is a clear standard and reference implementations. So the cost of “porting” will always be lower between clouds that have commonality by design or heritage.

For OpenStack itself, until that standard is codified, I would describe the most successful mission statement as “to be the reference public cloud provider scale implementation of cloud infrastructure compatible with AWS core API’s”. That’s going to give all the public cloud providers who want to compete with Amazon the best result: they’ll be able to compete on service terms, while convincing early adopters that the move to their offering will be relatively painless. All it takes, really, is some humility and the wisdom to recognise the right place to innovate.

There will be many implementations of those core API’s. One or other will be the Apache, the “just start here” option. But it doesn’t matter so much which one that is, frankly. I think OpenStack has the best possible chance to be that, but only if they stick to this crisp mission and don’t allow themselves to be drawn into front-end differentiation for the sake of it. Should that happen, OpenStack will be vulnerable to another open source project which credibly aims to achieve the goals outlined here. Very vulnerable. Witness the ways in which Eucalyptus is rightly pointing out its superior AWS compatibility in comparison with OpenStack.

For the public cloud providers that hope to build on OpenStack, API differentiation is poison in a juicy steak. It looks tasty, but it’s going to cost you the race prematurely. There were lots of technical reasons why alternatives to Windows were *better*, they just failed to become de facto standards. As long as Amazon doesn’t package up AWS as an on-premise solution, it’s possible to establish a de facto standard around something else, but that something else (perhaps OpenStack) needs to be AWS-compatible in some meaningful way to get enough momentum to matter. That means there’s a window of opportunity to get this right, which is not going to stay open indefinitely. Either Amazon, or another open source project, could close that window on OpenStack’s fingers. And that would be a pity, since the community around OpenStack has tons of energy and goodwill. In order to succeed, it will need to channel that energy into innovation on the implementation, not on trying to redefine an existing standard.

Of course, all this would be much easier if there were a real HTTP-like standard defining those API’s. The web had the enormous advantage of being founded by Tim Berners-Lee, in an institution like CERN, with the vision to setup the W3C. In the case of today’s cloud infrastructure, there isn’t the same dynamic or set of motivations. Amazon’s position of vagueness on the AWS API’s is tactically perfect for them right now, and I would expect them to maintain that line while knowing full well there is no real proprietary claim in a public network API, and no real advantage to be had from claiming otherwise. What’s needed is simply to start codifying existing practice as a draft standard in a credible forum of experts, with a roadmap and the prospect of support from multiple vendors. I think that would be relatively easy to arrange, if we could get Rackspace, IBM and HP to sit down and commit to doing it. We already have HP and Rackspace at the OpenStack table, so the signs are encouraging.

A good standard would:

* be pragmatic about the fact that Amazon has already made a bunch of decisions we’ll live with for ever.
* have a commitment from folk like OpenStack and Eucalyptus to aim for compliance
* include a real automated functional test suite that becomes the interop benchmark of choice over time
* be open to participation by Amazon, though that will not likely come for some time
* be well documented and well managed, like HTTP and CSS and HTML
* not be run but the ITU or ISO

I’m quite willing to contribute resources to getting such a standard off the ground. Forget big consortiums or working groups or processes or lobbying forums, what’s needed are a few savvy folk who know AWS, Eucalyptus and OpenStack, together with a very few technical writers. Let me know if you’re interested.

Now, I started out by saying that I was writing to make the case for OpenStack to be focused on a particular area. It’s a bit cheeky for me to write anything of the sort, of course, because OpenStack is a well run project that has an excellent steering group, which recently held a poll of contributors to appoint some new members, none of which was me. I’ve every confidence in the leadership of the project, despite the tremendous pressure they are under to realise the hopes of so many diverse users and companies. I’m optimistic for the potential OpenStack has to accelerate cloud technology, and in Canonical we put a considerable amount of effort into making OpenStack deployment a smooth experience for Ubuntu users and Canonical customers. Ubuntu Cloud Infrastructure depends now on OpenStack. And I have a few old friends who are also leaders in the OpenStack community, so for all those reasons I thought it worth making this perspective public.

This entry was posted
on Thursday, September 8th, 2011 at 7:35 pm and is filed under thoughts.
You can follow any responses to this entry through the
RSS 2.0 feed.
Both comments and pings are currently closed.

33 Responses to “Innovation and OpenStack: Lessons from HTTP”

Agreed, a standard would be a good and one based on AWS would have most traction. However, why being restricted to what AWS does, if there is a good reason to innovate and it stays compatible with the existing implementation? For example, WebDAV and RESTful web services have been quite successful in innovating over HTTP without breaking compatibility.

Mark, I’ve been following the debate and I have to admit that I’m quite divided on the topic. On the one hand, I can agree with all your points. As a consumer of many cloud APIs and a reviewer of countless proposed ones, I have yet to see a better API. We can all piss on aspects of the AWS APIs, but as a whole, as a functioning and evolving set they are absolutely unmatched.

Where I have reservations is that the APIs do embody a quite specific take on cloud computing, in particular those of a very large scale service provider with a specific implementation architecture. As soon as you move to private clouds and try to integrate the cloud with the existing network things start shifting and you may need somewhat different concepts. Also, not everyone wants the EBS model for disk volume storage and other technologies do benefit from slightly different API calls or semantics. Or take a look at the VPC2 functionality, does it really make sense to duplicate that in OpenStack? Or the permissions management (IAM), which is quite tricky in many ways.

What I think would be a great position is to define a “portable EC2” subset, support that in OpenStack, and add variations or extensions around that, keeping the flavor of the API. A well chosen subset could make a majority of tools just work and would reduce the burden of porting those that use functionality beyond the portable subset. I’d be very much in favor of such an approach. That would also start putting some pressure on Amazon to collaborate because they suddenly risk becoming incompatible with a widely adopted “portable EC2” API down the road.

I’m torn here. On the one hand, you make a very strong argument in the direction of pragmatism, and the points you make around picking up tooling and ecosystem right off the bat are obviously correct. I’m also excited to have you involved in just about anything … you have a clear track record of passionate support for projects you are involved with. On the other hand, I personally wouldn’t touch the AWS APIs with a 100 foot pole. You are, of course, correct that there is no proprietary claim in a public network API. However, Amazon does publish a claim of copyright over the API spec, and Jeff Bezos is the guy who patented one-click shopping. As a person who does not have the wherewithal to withstand a legal attack (even if it’s a baseless one) I think that anything associated with Amazon has to be viewed with extreme skepticism.

I’m more than a bit disappointed by all of the tool writers who flew directly to AWS-direct apis when things like libcloud and jclouds exist. In the database world, we’ve had multiple protocols for forever, but we’ve also had JDBC and DB API and DBI … and those are the things programmers use. Who would write an app in direct OCI calls? Or in direct MySQL library calls? By the same argument about pragmatic de-facto standards, it’s quite easy to argue that MySQL is the de-facto standard on the web (and by a much larger install-base majority, if we want to count it out). Does that mean we should attempt to get PostGreSQL and Oracle to ditch their protocols and adopt the MySQL one? Or, do we just count on rich protocol abstraction layers and get tools to use those? (the second is obviously the choice we make in everything except extremely high performance situations)

In any case – as I said, I’m torn. I want OpenStack to be successful in terms of user/install base – but I also want it to be successful in terms of freedom. I don’t want to be beholden to the whims of Bezos, nor do I want to grant him W3C status for the cloud just because he was first to market. The world of hosted services and cloud is already extremely dangerous ground for those of us concerned with software freedom (thanks, btw, for AGPL’ing Launchpad!) and I have grave concerns about placing important standards-related work in the hands of a single for-profit US corporation.

The question remains though, as I believe that your assessment of the current state of things is spot on – is it possible to move people away from the infected AWS apis? Or are we effectively trapped by them from here on out? I’d hate to have to open a new Bug #1 in a few years…

[…] Innovation and OpenStack: Lessons from HTTP September 8, 2011 | Posted by admin OpenStack is facing an important choice: does define a new set of API’s, one of many such efforts in cloud infrastructure, or does it build around the existing AWS API’s? So far, OpenStack has had it both ways, with some new API work and also some AWS-based effort. I’m writing to make the case for a tighter definition of mission around the de facto standard infrastructure API’s of EC2, S3 …Full Article […]

By the way HTTP has been evolving since its inception. People only use a portion of its potential. I’ve heard HTTP be called the ‘Universal API’. Perhaps that’s the right way to approach cloud storage.

Tim Berners-Lee has written quite a good piece on cloud storage combined with the social web, which is perhaps the direction things are going. Tho this was actually the original vision of the Web.

If this is really a time for pause and reflection before setting out on an important step forward, I’d like to urge the following:

Drop the notion of a URI-driven RPC-style API running over HTTP and instead adopt a media type-driven Hypermedia-style API that can run over multiple protocols (HTTP, XMPP, etc.).

To do this what is needed is some time designing a new domain-specific media type (application/openstack+xml, application/openstack+json) that contains elements/properties to express domains-pecifics (machines, instances, memory, etc.) and uses hypermedia controls (links and forms w/ arguments tied to protocol verbs) to provide context-sensitive appliation flow control appropriate to the state of the requesting user and target device.

While it will take some work at the start to agree on a media type design, some key advantages can flow from this work:
– easily extendable over time w/o resorting to versions that break existing implementations
– the flexibility to implement the existing openstack, AWS, or any other ‘proprietary’ APIs *behind* the media type and offer _true_ interoperability
– the establishment of a long-term development path that can take advantage of new protocols (SPDY, WAKA, etc.) as they appear

Best of all, this media-type driven work can go on without abandoning or slowing the already valuable work the existing OpenStack API; without invalidating any of the existing implementations.

I would say, instead of “learning from HTTP” it would be quite valuable to “learn from HTML” since, despite changes in features, clients, and servers, HTML 1.0 still works today; side-by-side w/ HTML5 more than ten years on.

You could create a Community Group at W3C. It is not tied to any membership at all. It addresses exactly your use case. It gives a platform for doing so, and you can even benefit from the Patent Policy. http://www.w3.org/community/

I’m not sure I will have the popular opinion here but I thought I’d throw in my two cents.

In my opinion, the cloud API space is already fractured as AWS, Azure (they have ‘VM roles’ now so its effectively a IAAS as well), Rackspace, GoGrid and many others all have different web APIs. I would like to suggest that API standardization should be left to another day if attempted at all when there are real incentives for cloud providers to more heavily participate in this.

I also think it’s unrealistic to say another stack would ever support the full AWS API as it is a moving target with additions every month. Instincts tell me (though maybe not correct ones) the result of efforts on standardizing on the AWS API would be that tooling developed for OpenStack could move to AWS but not the other way around. Even if this kind of effort was seen as worthwhile for the more static APIs, I think the first step would be to get Amazon on board with a set of standardized tests like you suggest — but I really don’t see why Amazon would be interested in effectively helping people get off of AWS (no one else uses AWS as the standard today so the only possible result of standardization would be an immediate loss in customers for Amazon — this assumes shortsightedness on amazons parts as such a move may result in the positive growth of cloud computing in general resulting from the benefits of standardization — common tooling etc).

I would like to suggest that the development of OpenStack should take the path of least resistance: follow the AWS API where its easiest to follow as most of the hard thought has already been done by Amazon, but where there is a strong opinion about the ‘correctness’ (usability, feasibility etc etc) of specific AWS API, OpenStack could vary it as it would like.

All this said, I do understand the significant value of portability. My solution to portability then would be some concept of adapter APIs either at the client or server side. These may emulate any other cloud provider’s API using OpenStack’s. These could be developed after OpenStack and as a result be much lower risk projects.

Yes, I agree, “perfect AWS” compatibility is a pipe dream. I think we should focus on the core: S3, Ec2, and credentials/security. All the rest is fluff, essentially, and folk would be wise not to get locked into it.

We agree entirely that there’s some risk of Amazon bluster around the API. But we probably both agree that doing so would damage AWS permanently – everyone would rush to get off, and whatever neutral forum was closest to AWS would be the immediate beneficiary. So even though there is a history of litigiousness and IP bluster at Amazon, I’d not expect them to make this mistake.

Right now, there’s no codification of the “core API subset” which can be trusted to exist on multiple clouds. If we can identify that subset, document it, have it implemented in both OpenStack and Eucalyptus with a test-suite that works against AWS, OpenStack and Eucalyptus, we will have started the process of defining HTTP-for-the-cloud. I think there’s a good chance tool writers might stick to that subset, because they would have the option of deploying and testing internally, which they don’t with AWS.

On the API front, I think the JDBC analogy is quite useful, because all of the JDBC databases are SQL (or at least set oriented) AFAIK. That means that there’s little *conceptual* variation, even if the details of implementaiton vary widely, and as a result, there’s a lot of innovation and diversity in the database market. I guess my argument here is that there’s no need for OpenStack to innovate on the concepts – it would be better to mirror the core conceptual architecture that’s been established. That would make it easier to make a real cross-cloud API.

@Mark I think the JDBC analogy it’s brilliant. Let’s OpenStack innovate without AWS compatibility constraints, and the sue something like http://incubator.apache.org/deltacloud/ to provide mobility between different cloud implementations.

I doubt deltacloud will gain any real traction, because unlike JDBC, the underlying systems don’t have the same consensus around model. With JDBC, all the back end databases are SQL, or at least set-based, so they share enough of a framework that a common API works. The problem with all the new cloud infrastructure bids is they are trying to innovate in the model itself. That means deltacloud is always going to offer the worst possible lowest common denominator, unless the underlying clouds accept that they need a common model. And if they’re going to go for a common model, why not fall in line with a common core network API too?

Serious cloud users like Thorsten above will tell you that the AWS API’s are well done. I’m sure they can be improved, and the goal of my post was to suggest how we can shift the API from being AWS specific to something steered by a professional, pragmatic and independent team.

I like your description of it as a “portable EC2 subset” though in practice I think we need some storage and credentials, so “portable AWS subset” would be more accurate. But if we can actually define that subset, get multiple open source implementations of it, and a test suite, we’ll have taken a concrete step towards putting that API on a trajectory to formal standardisation. And we’ll have drawn a nice clear line in the sand for tool vendors to say “stick in this subset and your stuff will work much more widely”, which they don’t have today.

Mark, thank you for your writings and participation, which I keep finding extremely pertinent.

Just wanted to say that your point “not be run but the ITU or ISO” sounds like a typo. Did you mean “not be run by the ITU or ISO”, and if so, can you clarify this point? I am interested in your views about standards governance.

ITU and ISO standards processes are extremely heavyweight, and have generally failed to match the agility needed for internet operations. Consider LDAP vs X.500, or the debacle of the ISO’s inability to maintain a high quality review process through all the Microsoft Office document standard discussions. I spent too many hours of my life implementing X.509 to want to go through that again :-). In my view, neither the ISO or the ITU is capable of steering a successful cloud protocol definition process. So yes, I stand by the view that the best thing that could happen is a small, semi-formal working group of people who are savvy enough to know how things are working in practice today, and independent enough to want to make a practical level playing field while avoiding the typical consortium effect of large players stalling on the creation of an “open” standard while they try to outmanoeuvre one another on the product front.

As a counter point to the argument that Mark is making about adopting Amazon’s AWS, I would encourage everyone to familiar themselves with the history of the gopher protocol (rfc1436) which was in wide use before http was very popular.

Specifically everyone should review the role that the U. of Minnesota played as the originator of gopher protocol and who had intellectual property rights to the protocol.

@mark well said, and agreed. i remember arguing about innovation from rackspace in the cost benefit + api effectiveness.. but the market is clear. additionally paas pricing costs @ scale (esp with recent appengine changes) argue pretty effectively for the aws api with iaas with opensource automation as a win for most orgs. azure is also interesting an alternative public api. re aws, the degree to which large commercial entities can implement the aws api, without legal consideration, will be in doubt till there’s a public statement on the manner. wrt to openstack, aws compatibility seems to lack a champion willing to implement.

@monty, thorsten.. aws is the sql of the cloud, sure there are many datahbase implementation specific features and eccentricities.. but they do so against a common standard of operation. the basic cloud operations have been abstracted across by several projects… but the reality seems they do so at the expansion to so many marginal implementations of the aws feature set, they require breaking the abstraction to get to a useful production feature set that goes beyond a basic virtual machine api. the occi initiative could be interesting, but its innovation beyond a simple virtualization api is at glacial speeds.

@monty.. re ip concerns.. its valid, but the reality is there is already a vc backed company with products based soley around providing an opensource implementation of the aws api for years.

With Gnome3 and all its api’s already existing. New developers were paid to develop a counter shell for it. It would have been more easier if the gnome shell would have been customized to canonical’s taste and then forced it down the gnome3 devs throat if it found popular use. Now with other “cloud” related products trying to innovate, they are being stifled and bulldozed into using common api’s which U like and think are the most optimum. Two products two different strategies. kudos.

The DMTF has just published Work In Progress versions of its Cloud Infrastructure Management Interface (CIMI) specs: http://dmtf.org/standards/cloud. Though not the first IaaS-level standard (OCCI likes to claim that), CIMI is more like AWS/EC2 in its model and approach.

Mark – Great article.
In a series of blog posts about Cloud Lock-In I relate to the different layers of the cloud i.e. I P and S. – I think that the following is relevant to the discussion here –

“As a market leader, Amazon AWS will be (already is) followed by other IaaS vendors. Those will solve the same scalability and operational issues by the same sense and logic of AWS. Basically this means an evolution of IaaS platform standards. Smart cloud integration layer will enable “plug & play” a different IaaS platform or even orchestrate several in parallel. To strengthen my point I bring as an example several cloud start-ups (solving IaaS issues such as governance, usage and security) that developed their product to solve issues for Amazon AWS consumers and seriously target support of other IaaS vendors’ platforms such as Rackspace cloud and vCloud. In regards to lock-in the public cloud is great !”

This is an interesting, important and difficult discussion. I’ve probably started to reply to this post 5 times, but always end up changing my mind halfway through.

I believe I understand the value of the EC2 API. I’ve put quite a bit of effort into its implementation in OpenStack, but there’s still lots of work to be done for sure and I think it should be a priority. EC2 has plenty of interesting features that we should be adding and that could keep us busy for a long time.

However, we have people today who want to add really interesting features to OpenStack that have no counterpart in EC2 (most notably in the networking area). While we certainly could extend our implementation of the EC2 API to expose this functionality, it’s extremely unlikely that when Amazon gets around to adding similar concepts they will do so by exposing an API identical to what we’ve built. Years from now, if OpenStack becomes as ubiquitous as we’d like, then things might be different. We might be in a position to call the shots and innovate independently of or even in cooperation with Amazon.

Right now, that’s just not the case.

I’d be happy to explore adding a special “openstack” version to the EC2 API (any EC2 API request includes a version identifier denoting which version of the API your client implements) which has these extra concepts, but the “native” OpenStack API isn’t going away.

[…] subject of much public debate on the OpenStack lists, at the conference and online (Wilamuna & Shuttleworth). So far, I have held back from the community discussion because I think both sides are being […]