The latest version of the Haizea Lease Manager (Technology Preview 1.3) was released a few days ago, so this seems like a good opportunity to talk about why Haizea exists and what it means to OpenNebula users.

As most readers of this blog know, OpenNebula allows you to manage the dynamic deployment of virtual machines (VMs) on a pool of physical resources. There are many reasons why you would want to virtualize your infrastructure (see the OpenNebula use cases at the bottom of this page), and the one I will focus on here is creating a “private cloud” (a subject that has been discussedpreviously on this blog).

Since we’re entering the perilous terrain of buzzwordiness, let me stop for a second to clarify that whenever I use the term “cloud” in this post, I specifically mean an “Infrastructure-as-a-Service (IaaS) cloud”, such as Amazon’s EC2, where computational infrastructure is provisioned on-demand as virtual machines on a large data center. Yes, I realize “cloud” can and does mean many other things (although we’re still far from agreeing on what it means) but, for now, let’s stick to the IaaS aspect of clouds.

One of the the characteristics that is frequently attributed to clouds is that of “infinite capacity”. Thus, large cloud providers like Amazon EC2, Flexiscale, and ElasticHosts have evolved towards an immediate provisioning model: when users asks for additional capacity, they get it, subject to some reasonable limitations (there may be a delay in setting up the extra VMs, providers may have limits on how much capacity one single user can request, etc.) If you assume infinite capacity, this provisioning model is pretty reasonable. There is no need to, for example, allow users to make reservations in advance: if you need resources from 2pm to 4pm, why would you need to reserve them? Just show up at 2pm and they will be there for you, because capacity is “infinite”.

Of course, there is no such thing as “infinite capacity”, but large cloud providers can at least provide the illusion of infinite capacity. However, if you have a relatively small number of resources (compared to Amazon or Google) and want to build a “private cloud” with OpenNebula on top them, you’re probably in no position to assume you’ve got infinite capacity.

But why would you want to create a “private cloud”? Isn’t outsourcing your infrastructure to large external providers (instead of keeping them in-house), thus reducing your IT expenses, one of the biggest selling points of cloud computing? Sure, but some of us will still have our own IT infrastructure to manage and, although datacenter virtualization has been around since before “clouds” became “the next big thing”, there are certain benefits to managing your infrastructure like a “private cloud”:

You can provide your in-house users with all the benefits of deploying their machines on EC2, without actually paying EC2 to do it. My intuition is that, if you already have an IT infrastructure that is mostly amortized, this will make sense financially (unlike a business with no IT infrastructure, where relying on a large cloud provider makes more sense than making a huge initial investment on new infrastructure). That said, I will be more than happy to be corrected on this point, as it is simply an intuition.

You can become a cloud provider. Having a “private cloud” doesn’t preclude the possibility of adding a public interface, using tools like Nimbus or Eucalyptus, and turning all or part of your private cloud into a public cloud that can be accessed by external users via the Internet.

All of the above. If you’re servicing in-house users, it’s almost certain that your infrastructure will be underutilized some of the time. This unused capacity could be sold to external users.

This is all nice and dandy but, as I said earlier, a private cloud can’t assume it has “infinite capacity”. Thus, relying on an immediate provisioning model just doesn’t hold water. Requests for resources are going to have to be prioritized, queued, pre-reserved, and even rejected. Tools for building private clouds will need to support more sophisticated resource scheduling than just immediate provisioning, and this is where Haizea comes in.

Haizea is a lease manager that can be used as a drop-in replacement for OpenNebula’s scheduler, providing scheduling features not found in other cloud and virtualization solutions, such as efficient support for advance reservations, queuing of best-effort requests and, coming soon, pluggable scheduling policies. While still supporting an immediate provisioning model, Haizea also allows OpenNebula users to pre-reserve resources (in anticipation of capacity peaks) or queue requests that can afford to wait a while (another feature that will be added to Haizea in the future is best-effort scheduling with deadlines, so there will be a finite bound on the waiting time). Again, if you have a datacenter of Amazonic proportions, Haizea probably makes no sense. But if you have a more modest datacenter, and want to build a private cloud on it, you will need to be more judicious about how you slice up your resources amongst users (and I suspect that most of us fall into the non-Amazonic category).

To wrap this up, I’d like to refer to a technical report that has been getting quite a bit of press lately, “Above the Clouds: A Berkeley View of Cloud Computing“. This report has been getting mixedreviews and, personally, I can’t say I agree with many of the things they say, particularly the way they dismiss private clouds right from the outset. However, I think they raised a good point in “Number 5 Obstacle [for Cloud Computing]: Performance Unpredictability”, where they stated:

The obstacle to attracting HPC is not the use of clusters; most parallel computing today is done in large clusters using the message-passing interface MPI. The problem is that many HPC applications need to ensure that all the threads of a program are running simultaneously, and today’s virtual machines and operating systems do not provide a programmer-visible way to ensure this. Thus, the opportunity to overcome this obstacle is to offer something like “gang scheduling” for Cloud Computing.

Haizea, in fact, has supported VM gang scheduling from day one. The lease abstraction used in Haizea allows users to request not just individual VMs, but groups of VMs that must be treated atomically. In other words, VM that must either all be running simultaneously or not at all (which involves gang-scheduling those VMs)

So, if you’re interested in virtual machine scheduling that goes beyond immediate provisioning, I invite you to check out Haizea. It’s still a technology preview, but it’s being actively developed and a 1.0 release shouldn’t be too far off.

A Distributed Virtual Infrastructure (VI) Manager is responsible for the efficient management of the virtual infrastructure as a whole, by providing functionality for deployment, control and monitoring of groups of interconnected Virtual Machines (VMs) across a pool of resources. An added functionality of these management tools is the dynamic scaling of the virtual infrastructure with resources from remote providers, so seamless integrating remote Cloud resources with in-house infrastructures. This novel functionality allows to add and remove capacity in order to meet peak or fluctuating service demands, so providing the foundation for interoperation between Cloud infrastructures. The distributed virtual infrastructure would run on top of a geographically distributed physical infrastructure consisting of resources from the private cloud and several external cloud providers.

Since release 1.0, OpenNebula distribution includes the plugins required to supplement local resources with Amazon EC2 resources to satisfy peak or fluctuating demands. This novel feature has been illustrated in several use cases for computing clusters and web servers. The open and flexible architecture of OpenNebula makes quite simple to create new plugins to access other cloud providers. In order to illustrate this and to provide KVM users with an utility access to remote resources, the OpenNebula team has just released the plugins required to dynamically grow the infrastructure using ElasticHosts resources. ElasticHosts is the world’s first public cloud based upon KVM, providing scalable and flexible virtual server capacity for cloud hosting. An interesting result is that a private infrastructure could dynamically grow using resources from different Cloud providers according to provisioning policies based on resource availability, performance, costs, availability…

The release of these new plugins represents a new step towards an open-source framework for cloud infrastructure federation, which is one of the main goasl of the Reservoir project, European research initiative in virtualized infrastructures and cloud computing.

The OpenNebula Team is releasing a new plug-in to interface the ElasticHosts cloud provider, so it can be used to dynamically increase capacity of your virtualized infrastructure to meet fluctuating or peak demands. This can happen when the local fabric runs out of capacity to spawn a new virtual machine, therefore it may be interesting to add capacity using cloud providers.

ElasticHosts offers KVM based virtualized hosts in a cloud like fashion, i.e., à la Amazon EC2, using a very neat RESTful API. Uploading images (drives, in ElasticHosts speak) previously configured with the service that needs to meet a increased demand would allow the cloudbursting described above through OpenNebula.

Information on how to download and install the ElasticHosts plug-in can be found in the OpenNebula Trac.

So, do you want to transform your rigid and compartmented infrastructure, into a flexible and agile platform where you can dynamically deploy new services and adjust their capacity?. If the answer is yes, you want to build what is nowadays called a private cloud.

In this mini-howto, you will learn to setup such private cloud in 5 steps with Ubuntu and OpenNebula. We assume that your infrastructure follows a classical cluster-like architecture, with a front-end (cluster01, in the howto) and a set of worker nodes (cluster02 and cluster03).

First you’ll need to add the following PPA if you’re running Ubuntu 8.04 LTS (Hardy) or Ubuntu 8.10 (Intrepid), you should be fine if you are running Ubuntu 9.04 (Jaunty Jackalope):

The OpenNebula team is happy to announce the availability of OpenNebula 1.2, the second stable release of the project. This is an important milestone for the project and marks that most of the components of OpenNebula are now in place.

What is OpenNebula?

The OpenNebula virtual infrastructure engine provides efficient, dynamic and scalable management of groups of interconnected VMs within datacenters involving a large amount of virtual and physical servers. OpenNebula supports Xen and KVM platforms and can interface with remote cloud sites, being the only tool able to access on-demand to Amazon EC2 to dynamically scale the local infrastructure based on actual usage. OpenNebula also exhibits an open and flexible architecture which allows the definition of new algorithms for virtual machine placement, and its integration with any virtualization platform, infrastructure cloud offering and third-party component in the cloud ecosystem, such as cloud-like remote interfaces, virtual image managers, and service managers. OpenNebula is one of the components being enhanced in the context of the European Union’s Reservoir Project, which aims to develop the open source technology to enable deployment and management of complex IT services across different administrative domains.

New Features and Highlights in OpenNebula 1.2

OpenNebula 1.2 presents important improvements in the following areas:

Image Management. OpenNebula 1.2 features a general mechanism to transfer and clone VM images. The new Transfer Manager is a modular component that embrace the driver-based design of OpenNebula, so it can be easily extended and integrated with third-party developments and virtually any cluster storage architecture. The new TM allows you to re-use VM images as you can mark them as clonable, and also you can save space as swap images are now created on-the-fly by OpenNebula.

Networking. With OpenNebula 1.0 it is difficult to track the MAC/IPs in use by the running VMs, and its association with physical networks. This mechanism does not scale when dealing with tens of VMs. The new Virtual Network Manager module allows you to define virtual networks and it leases IP-MAC pairs to VMs, so you do not have to keep track of the addresses in use. In this way it is pretty much like an embedded DHCP server. Additionally, the leases are built in such a way that you can easily obtain the IP form the MAC when booting the VM. OpenNebula 1.0 networking is still supported, in case you do not want to use the new functionality.

Robustness and scalability, OpenNebula 1.2 has been tested in the management of hundreds of running VMs to ensure that the code meets production level requirements.

The OpenNebula team would like to thank everyone that sent comments, reported bugs and provide patches. It definitely helped to get a better OpenNebula 1.2. Specially, we would like to thank the great labour done by Soren Hansen, reflected in submitted patches that helped to get OpenNebula closer to work with Ubuntu and Debian.

Nowadays it is difficult to find a place where virtualization benefits are not praised. One of my favorite ones is consolidation, that has finish with the “one application one server” paradigm. However, when you start placing your services on different hosts, you will find quickly that the new model does not scale per se. Soon (ten VMs is enough), you’ll be trying to answer questions like: Where did I put our web server? Is MAC address already in use? Is this the updated image for the ubuntu server? and the alike.

That was our situation last year; although we organized ourselves with a wiki and a database to store all the information, it became evident that we need a management layer on top of the hypervisors. We started the development of OpenNebula to implement our ideas about the distributed management of Virtual Machines.

Today OpenNebula is a reality, and has reach a production quality status (at least we are using it in production). I’ll briefly show you how we setup our infrastructure with OpenNebula, KVM and Xen.

The Physical Cluster

Our infrastructure is a 8-blade cluster, each one with 8GB and two quad-core CPUs (8 cores/blade). These blades are connected with a private LAN an to the Internet, so they can host public services. The output of the onehost list command is as follows – it’s been colorized for dramatic purposes ;) :

Originally the blades were configured as a classical computing cluster.

The Virtual Infrastructure

Now, thanks to OpenNebula the cluster can be easily configured to host multiple services. For example we are using the 8 blades for:

Two grid clusters (4 nodes each) for our grid projects. Needless to say, that we can add/remove nodes to these clusters as we need them

A Grid broker based on the Gridway metacheuler

A web server, serving this page actually

A coupe of teaching clusters for our master students, so they can break anything

A cluster devoted to the RESERVOIR project

And my favorite, a cluster were we develop OpenNebula. The nodes of this cluster are Xen capable, and they run as KVM virtual machines. So yes, we are running VMs in Xen nodes which are KVM VMs.

This is the output of the onevm list command:

One of the most useful features of the new version of OpenNebula is the networking support. You can define several networks, and OpenNebula will lease you MAC addresses from the network you want. So you do not have to worry about colliding addresses. The MAC addresses are assigned using a simple pattern so you can derive the IP from the MAC (see the OpenNebula Networking guide for more info).

We are using a virtual network for each virtual cluster, output from onevnet list and onevnet show commands:

Thanks to the hypervisors and OpenNebula, we can shape our 8 blades to be any combination of services, and we can resize these services to suite our needs. As can be seen above, you can easily find where is the webserver running, so you can get a console in case you need one, or migrate the VM to other blade…

Although the management is now so easy with OpenNebula, setting up everything (installing the physical blades, the VMs, configure the network…) is a difficult task. Kudos to Javi and Tino for getting the system running!.

Last month I’ve been invited to give a couple of talks about Cloud computing in the wonderful C3RS (Cisco Cloud Computing Research Symposium) and in a Spanish e-science meeting. (The slides are available online, if you want to check). Although the audiences were quite heterogeneous, there is a recurrent question among the participants of these events: How can I set my private cloud?. Let me briefly summarize the motivation of the people asking this:

Lease compute capacity from the local infrastructure. These people acknowledge the benefits of virtualizing their own infrastructure as a whole. However, they are not interested, in selling this capacity over the internet, or at least is not a priority for them. This is, they do not want to become a EC2 competitor, so they do not need to expose to the world a cloud interface.

Capacity in the cloud. They do not want to be the new EC2 but they want to use EC2. The ability of moving some services, or part of the capacity of a service, to an external provider is very attractive to them.

Open Source. Current cloud solutions are proprietary and closed, they need an open source solution to play with. Also, they are using some virtualization technologies that would like to see integrated in the final solution.

I say to these people, take a look to OpenNebula. OpenNebula is a distributed virtual machine manager that allows you to virtualize your infrastructure. It also features an integral management of your virtual services, including networking and image management. Additionally, it is shipped with EC2 plug-ins that allow you to simultaneously deploy virtual machines in your local infrastructure and in Amazon EC2.

OpenNebula is modular-by-design to allow its integration with any other tool, like the Haziea lease manager, or Nimbus that gives you a EC2 compatible interface in case you need one. It is a healthy open source software being improved in several projects like RESERVOIR, and it has a growing community.

MIT Technology Review has just published an interesting article entitled “Openning the Cloud” about open-source technological components to build a cloud-like infrastructure. The article focuses on the IaaS (Infrastructure as a Service) paradigm, describing the components required to develop a solution to provide virtualized resources as a service. The article briefly describes the following technologies: OpenNebula, Globus Nimbus, and Eucalyptus.

In the OpenNenula project, we strongly believe that a complete Cloud solution requires the integration of several of the available components, with each component focused on a niche. The open architecture and interfaces of the OpenNevula VM Manager allow its integration with third-party tools, such as capacity managers, cloud interfaces, service adapters, VM image managers…; so supporting a complete solution for the deployment of flexible and efficient virtual infrastructures. We maintain an Ecosystem web page with information about third-party tools to extend the functionality provided by OpenNebula.