With Ubuntu 12.04.2, the kernel team introduced the idea of the "hardware enablement kernel" (HWE), originally intended to support new hardware for bare metal server and desktop. In fact, the documentation indicates that HWE images are not suitable for Virtual or Cloud Computing environments. The thought was that cloud and virtual environments provide stable hardware and that the newer kernel features would not be needed.

Time has proven this assumption painfully wrong. Take for example the need for drivers in virtual environments. Several of the Cloud providers that we have engaged with have requested the use of the HWE kernel by default. On GCE, the HWE kernels provide support for their NVME disks or multiqueue NIC support. Azure has benefited from having an updated HyperV driver stack resulting in better performance. When we engaged with VMware Air, the 12.04 kernel lacked the necessary drivers.

Perhaps more germane to our Cloud users is that containers are using kernel features. 12.04 users need to use the HWE kernel in order to make use of Docker. The new Ubuntu Fan project will be enabled for 14.04 via the HWE-V kernel for Ubuntu 14.04.3. If you use Ubuntu as your container host, you will likely consider using an HWE kernel.

And with that there has been a steady chorus of people requesting that we provide HWE image builds for AWS. The problem has never been the base builds; building the base bits is fairly easy. The hard part is that by adding base builds, each daily and release build goes form 96 images for AWS to 288 (needless to say that is quite a problem). Over the last few weeks -- largely in my spare time -- I've been working out what it would take to deliver HWE images for AWS.

I am happy to announce that as of today, we are now building HWE-U (3.16) and HWE-V (3.19) Ubuntu 14.04 images for AWS. To be clear, we are not making any behavioral changes to the standard Ubuntu 14.04 images. Unless users opt into using an HWE image on AWS they will continue to get the 3.13 kernel. However, for those who want newer kernels, they now have the choice.

For the time being, only amd64 and i386 builds are being published.. Over the next few weeks, we expect the HWE images to reach full feature parity including release promotions, and indexing. And I fully expect that the HWE-V version of 14.04 will include our recent Fan project once the SRU's complete.

[UPDATE] The Image ID's have been updated with the latest builds which now include Docker 1.6.2, the latest LXD and of course the Ubuntu Fan driver.

This week, Dustin Kirkland announced the Ubuntu Fan Project. To steal from the description, "The Fan is not a software-defined network, and relies on neither distributed databases nor consensus protocols. Rather, routes are calculated deterministically and traffic carries no additional overhead beyond routine IP tunneling. Canonical engineers have already demonstrated The Fan operating at 5Gpbs between two Docker containers on separate hosts."

My team at Canonical is responsible for the production of these images. Once the official SRU's land, I anticipate that we will publish an official stream over at cloud-images.ubuntu.com. But until then, check back here for images and updates. As always, if you have feedback, please hop into #server on FreeNode or send email.

GCE Images

Images for GCE have been published to the "ubuntu-os-cloud-devel" project.

I am pleased to announce initial Vagrant images [1, 2]. These images are bit-for-bit the same as the KVM images, but have a Cloud-init configuration that allows Snappy to work within the Vagrant workflow.

Note: due to the way that Snappy works, shared file systems within Vagrant is not possible at this time. We are working on getting the shared file system support enabled, but it will take us a little bit to get going.

If you want to use Vagrant packaged in the Ubuntu archives, in a terminal run::

Well, the next Ubuntu 12.04 Cloud Images on Microsoft Azure will start using the HWE kernel. We have been working with Microsoft to validate using the 3.13 kernel on 12.04 and are pleased with the results and the stability. We spent a lot of time thinking about and testing this change, and in conference with the Ubuntu Kernel, Foundations and Cloud Image teams, feel this change will give the best experience on Microsoft Azure.

By default, the HWE kernel is used on official images for Ubuntu 12.04 on VMware Air, Google Compute Engine, and now Microsoft Azure.

Any 12.04 Image published to Azure with a serial later than 20140225 will default to the new HWE kernel.

One of the perennial problems in the Cloud is knowing what is the most current image and where to find it. Some Clouds provide a nice GUI console, an API, or some combination. But what has been missing is a "dashboard" showing Ubuntu across multiple Clouds.

In that light, I am please to announce that we have a new beta Cloud Image Finder. This page shows where official Ubuntu images are available. As with all betas, we have some kinks to work out, like gathering up links for our Cloud Partners (so clicking an Image ID launches an image). I envision that in the future this locator page will be the default landing page for our Cloud Image Page..

The need for this page became painfully apparent yesterday as I was working through the fallout of the Ghost Vulnerability (aka CVE 2015-0235). The Cloud Image team had spent a good amount of time pushing our images to AWS, Azure, GCE, Joyent and then notifying our partners like Brightbox, DreamCompute, CloudSigma and VMware of new builds. I realized that we needed a single place for our users to just look and see where the builds are available. And so I hacked up the EC2 Locator page to display other clouds.

Please note: this new page only shows stable releases. We push a lot of images and did not want to confuse things by showing betas, alphas, dailies or the development builds. Rather, this page will only show images that have been put through the complete QA process and are ready for production work loads.

This new locator page is backed by Simple Streams, which is our machine-formatted data service. Simple Streams provides a way of locating images in uniform way across the cloud. Essentially our new Locator Page is just a viewer of the Simple Stream Data.

A few years ago when our fine friends on the kernel team introduced the idea of the "hardware enablement" (HWE) kernel, those of us in the Cloud world looked at it as curiosity. We thought that by in large, the HWE kernel would not be needed or wanted for Virtual Cloud instances.

And we were wrong.

So wrong in fact, that the HWE kernel has found its way into the Vagrant Cloud Images, VMware's vCHS, and Google's Compute engine as the default kernel for the Certified Images. The main reason for these requests is that virtual hardware moves at a fairly quick pace. Unlike traditional hardware, Virtual Hardware can be fixed and patched at the speed that software can be deployed.

The feedback in regards to Azure has been the same: users and Microsoft has asked for the HWE kernel consistently. Microsoft has validated that the HWE kernel (3.16) running Ubuntu 14.04 on Windows Azures passes their validation testing. In our testing, we have validated that the 3.16 kernel works quite well in Azure.

Therefore, starting with the latest Windows Azure image [1], all the Ubuntu 14.04 images will track the latest hardware enablement kernel. That means that all the goodness in Ubuntu 14.10's kernel will be the default for 14.04 users launching our official images on Windows Azure.

If you want to install the LTS kernel on your existing instance(s), simply run:

We are pleased to announce that Ubuntu 12.04 LTS, 14.04 LTS, and 14.10 are now in beta on Google Compute Engine [1, 2, 3].

These images support both the traditional user-data as well the Google Compute Engine startup scripts. We have included the Google Cloud SDK, pre-installed as well. Users coming from other Clouds can expect to have the same great experience as on other clouds, while enjoying the features of Google Compute Engine.

From an engineering perspective, a lot of us are excited to see this launch. While we don't expect too many rough edges, it is a beta, so feedback is welcome. Please file bugs or join us in #ubuntu-server on Freenode to report any issues (ping me, utlemming, rcj or Odd_Bloke).

Finally, I wanted to thank those that have helped on this project. Launching a cloud is not an easy engineering task. You have have build infrastructure to support the new cloud, create tooling to build and publish, write QA stacks, and do packaging work. All of this spans multiple teams and disciplines. The support from Google and Canonical's Foundations and Kernel teams have been instrumental in this launch, as well the engineers on the Certified Public Cloud team.

Getting the Google Cloud SDK:

As part of the launch, Canonical and Google have been working together on packaging a version of the Google Cloud SDK. At this time, we are unable to bring it into the main archives. However, you can find it in our partner archive.

Cloud Images and Bash Vulnerabilities

The Ubuntu Cloud Image team has been monitoring the bash vulnerabilities. Due to the scope, impact and high profile nature of these vulnerabilties, we have published new images. New cloud images to address the lastest bash USN-2364-1 [1, 8, 9] are being released with a build serials of 20140927. These images include code to address all prior CVEs, including CVE-2014-6271 [6] and CVE-2014-7169 [7], and supersede images published in the past week which addressed those CVEs.

Addressing the full scope of the Bash vulnerability has been an iterative process. The security team has worked with the upstream bash community to address multiple aspects of the bash issue. As these fixes have become available, the Cloud Image team has published daily[2]. New released images[3] have been made available at the request of the Ubuntu Security team.

Canonical has been in contact with our public Cloud Partners to make these new builds available as soon as possible.

Cloud image update timeline

Daily image builds are automatically triggered when new package versions become available in the public archives. New releases for Cloud Images are triggered automatically when a new kernel becomes available. The Cloud Image team will manually trigger new released images when either requested by the Ubuntu Security team or when a significant defect requires.

Please note: Securing Ubuntu cloud images requires that security updates be applied regularly [5], using the latest available cloud image is not sufficient in itself. Cloud Images are built only after updated packages are made available in the public archives. Since it takes time to build the images, test/QA and finally promote the images, there is time (sometimes considerable) between public availablity of the package and updated Cloud Images. Users should consider this timing in their update strategy.

For years, the Ubuntu Cloud Images have been built on a timer (i.e. cronjob or Jenkins). Every week, you can reasonably expect that stable and LTS releases to be built twice a week while our development build is build once a day. Each of these builds is given a serial in the form of YYYYMMDD.

While time-based building has proven to be reliable, different build serials may be functionally the same, just put together at a different point in time. Many of the builds that we do for stable and LTS releases are pointless.

When the whole heartbleed fiasco hit, it put the Cloud Image team into over-drive, since it required manually triggering builds the LTS releases. When we manually trigger builds, it takes roughly 12-16 hours to build, QA, test and release new Cloud Images. Sure, most of this is automated, but the process had to be manually started by a human. This got me thinking: there has to be a better way.

What if we build the Cloud Images when the package set changes?

With that, I changed the Ubuntu 14.10 (Utopic Unicorn) build process from time-based to archive trigger-based. Now, instead of building every day at 00:30 UTC, the build starts when the archive has been updated and the packages in the prior cloud image build is older than the archive version. In the last three days, there were eight builds for Utopic. For a development version of Ubuntu, this just means that developers don't have to wait 24 hours for the latest package changes to land in a Cloud Image.

Over the next few weeks, I will be moving the 10.04 LTS, 12.04 LTS and 14.04 LTS build processes from time to archive trigger-based. While this might result less frequent daily builds, the main advantage is that the daily builds will contain the latest package sets. And if you are trying to respond to the latest CVE, or waiting on a bug fix to land, it likely means that you'll have a fresh daily that you can use the following day.

Many of our Cloud Image users have inquired about the availability of updated Ubuntu Cloud Images in response to the Heartbleed OpenSSL Vulnerability [1]. Ubuntu released update Ubuntu packages for OpenSSL 08 April 2014 [2]. Due to the exceptional circumstances and severity of the Heartbleed OpenSSL bug, Canonical has released new 12.04.4 LTS, 12.10 and 13.10 Cloud Images at [3].

Canonical is working with Amazon to get the Quickstart and the AWS Marketplace links updated. In the meantime, you can find new AMI ID's at [3] and [4]. Also, the snapshot's for Amazon have the volume-create permission granted on the latest images.

If you are running an affected version of OpenSSL on 12.04 LTS, 12.10 or 13.10, you are strongly encouraged to update. For new instances, it is recommended to either use an image with a serial newer than 20140408, or update your OpenSSL package immediately upon launch. Finally, if you need documentation on enabling unattended upgrades, please see [10].

Ubuntu Server 11.04 has proven to be a venerable server platform, it has nonetheless, reached its end of life on October 28, 2012. Whether or not you are blissfully unaware, end-of-life means that updates and security patches have been discontinued.

As part of the EOL'ing of a release, the mirrors are retired over time. Last week the mirrors for 11.04 were retired from archive.ubuntu.com, which in turn propagated through to the S3 EC2 mirrors. Any person using Ubuntu 11.04 and the S3 mirrors or archive.ubuntu.com will be unable to install software.

Over the last week, the Cloud Image team has fielded several questions from distraught users caused by the continued use of 11.04. We strongly suggest that those running Ubuntu Server 11.04 and the recently expired 11.10 and 8.04 LTS upgrade to a supported release to prevent any disruptions to their infrastructure. The current supported LTS is 12.04 with 13.04 being the latest stable release. Ubuntu 10.04 LTS is supported until April of 2015.

Those who continue to run expired Ubuntu releases may experience issues and may be required to mitigate the movement of mirrors from the S3 and main archive servers to old-releases.ubuntu.com/ubuntu [2]. While Ubuntu 8.04 LTS and 11.10 are currently available at the main archive and S3 locations, they will be removed anytime, per policy [3]

The Symptom

For those who are still running 11.04 and are running the 11.04 Cloud Image, you mostly likely have encountered or will encounter some ugly error messages when you try to access the S3 archives for EC2 images in the form of 404 and 403 errors. For example:

Over the course of working with Microsoft on Windows Azure, we have had the goal of bringing the same experience on Azure as our users have on EC2. As part of our QA process, we publish daily images (http://cloud-images.ubuntu.com) for EC2 and OpenStack users.

Today, I am pleased to announce that Windows Azure Cloud Image dailies are now being published for Ubuntu Server 12.04 LTS, 12.10, 13.04 and the current development version 13.10. Due to the way that Windows Azure image publication works, these images will appear with in a three or four hours of the EC2 images and will be published to all Windows Azure regions.

However, the daily images will not be available in the Windows Azure Gallery; these images are published to API users. In the coming weeks, we'll throw up some pages to help our API users find the current images, but for now, you can use the API Query tools to find the images.

As you can see, daily builds are clearly marked as "DAILY_BUILD" and include both the code name and the version number. Canonical provided images are all prefixed with our publisher GUUID of "b39f27a8b8c64d52b05eac6a62ebad85__"

While we make every effort at maintaining quality, daily images are not officially supported and may have issues as they are not rigorously QA'ed. As part of our release process, we take a daily, put it through QA and then promote the image. If you see any problems with any of the daily builds, please head over to Launchpad.net and file us a bug.

Finally, our daily image publishing will be restricted to the last five images for any one series. Like on EC2, all versions of Ubuntu Server released Cloud Images will remain indefinitely, with the exception of the pre-Windows Azure GA images (i.e. images with a serial of less than 20130414).
Read more

We are pleased to announce that Canonical has stood up official mirrors in HP Cloud's AZ-1, 2, and 3 regions.

If you are using Ubuntu Server 12.10 Cloud Images, there is no action to take; 12.10 images are by default configured to use the new mirror address.

For Ubuntu 12.04 instances, the default Ubuntu image does not automatically use the in-HP Cloud mirrors. We are currently working with HP to publish a new image that defaults to the local mirrors. If you would like to switch to the new in-HP mirrors, simply run:

Note: *.clouds.archive.ubuntu.com is configured using split-horizon DNS. This means that the DNS answer to queries is based on the askering IP address; only queries originatingwithin HP Cloud are answered with the HP Cloud mirror addresses.If your DNS resolver[s] is not based in HP Cloud, then you will be unable to benefit from these new mirrors.

Shortly after introducing the Vagrant images, a number of users provided very valuable feedback. The general gist was "sure this is a nice, but useless." For Cloud Images, we definitely take our user feedback to heart.

The 12.10 and 13.04 images now include Chef, Puppet and Juju clients. Also, the 13.04 images work now that the annoying Virtualbox installation error has been fixed. Users report that Chef and Chef Solo provisioning work with out any problems.

For 12.04, providing Chef support is somewhat more difficult as there are no official Ubuntu provided versions of Chef. Policy restricts us from providing third-party software on any image hosted on ubuntu.com. However, 12.04 does include Puppet and Juju.

The inclusion of Juju was added at the request of Juju charmers. In a future blog post, I'll illustrate how to use Vagrant for Juju charm development.

Finally, a common query that I get is about this particular error message:

[default] No guest additions were detected on the base box for this VM! Guest

additions are required for forwarded ports, shared folders, host only

networking, and more. If SSH fails on this machine, please install

the guest additions and repackage the box to continue.

This is not an error message; everything may continue to work properly,

in which case you may ignore this message.

I came up a nice long explanation as to the root cause (tl;dr: the Vagrant Cloud Images are _never_ booted and therefore the agent doesn't report to VirtualBox its information). And then the engineer in me started to think that this might be a trivial fix. Anyhow, in the next few days, this ugly error message will disappear for our daily builds (for Raring the message is gone as of today).

In conclusion, I wanted to say thank you to all the people who have dropped me an email for feature requests, rants and feedback. As always please feel to drop me a line, and I'll take a look at making these better.

Earlier we announced[1] that Canonical had worked this cycle to enable more frequent releases to the Ubuntu Cloud Images stable and long term releases. As of today, we are pleased to announce that Ubuntu Server 10.04 LTS, 11.10, 12.04 LTS and 12.10 are now fully enabled to follow the kernel SRU schedule with automated update releases. This means that within 24 hours of most SRU kernel releases, a new Ubuntu Cloud Image will be published.

Please note: with this change, the release notes have been moved the http://cloud-images.ubuntu.com/releases website. You can find them under <SUITE>/release/unpacked/release-notes.txt. Effective today, all emails announcing these new updates are discontinued.

However, at this time, 12.04 LTS and 12.10 Cloud Images are not yet being promoted automatically to Windows Azure. We expect that as Windows Azure moves closer to General Availability (i.e. moves out of preview status) that automatic promotion will be enabled.

Traditionally, updates for the stable release and long term stable release Cloud Images have been on an ad-hoc basis; reasons for releasing new images were generally restricted to security, critical bugs, and stale images. This ad-hoc update cycle meant that updated images were only released every three months or so, and for older releases, as often as six months.

While quality has always been a concern and top priority, during this cycle, Canonical has worked to vastly improve the QA infrastructure to support our Cloud Images. For example, when a new kernel is released, the daily build for that image is now put through the complete QA process. This change in process has allowed us to identify and automatically evaluate whether or not an image is a good candidate for update release.

As such, we are pleased to announce in the next few weeks, we will be turning on automated updates for Ubuntu Server 10.04 LTS, 11.10, 12.04 LTS, and 12.10. This means that approximately every three to four weeks, a new, freshened image will be released. The release cadence will follow the kernel SRU process.

The first updated image to be released under this process was 10.04 LTS[1].

There are a variety of ways to find the released Cloud Images. The two easiest ways are to go the AMI Finder[2] or use http://cloud-images.ubuntu.com/releases/<SUITE>/release. For example, http://cloud-images.ubuntu.com/releases/lucid/release would bring you to the last AMI's for Ubuntu Server 10.04 LTS.

Due to this change, we will discontinuing the email notifications of updated images to the various email lists for updated images. At UDS-R in Copenhagen[3], we discussed email notifications and the decision was reached to discontinue them. Replacing email notification is the RSS feed[4] and release notes (example from 10.04 LTS)[5].

As Cloud Image suites are migrated to automated releases, we will follow up on this announcement.

Finally, for 12.04 LTS and later, this change will introduce lock-step update releases with Windows Azure. As Windows Azure moves towards GA, we have been working to have the same releases for the Ubuntu Server Cloud Images on both EC2 and Windows Azure.

As always, your feedback is most appreciated. Please feel free to follow on either this post or to email concerns direct to me.

We are pleased to announce the availability of beta Vagrant Cloud Images. These images have been customized to work with the Vagrant development environment, and are based on the Ubuntu Cloud Images. As such, these are vanilla images. They do, however, have the Virtualboxguest additions found in the Universe archive (required for Vagrant integration). For those who use Vagrant, your feedback is essential. Please feel free to send feedback via the ubuntu-cloud@lists.ubuntu.com mailing list.The images are approximately 256M in size, and are configured for 512MB of RAM. They use a custom cloud-init user-data to drive the first boot. And of course, they have the vagrant user with vagrant insecure SSH key pre-installed. During the beta period, we will not be promoting any of the Vagrant boxes with the regular releases of the Ubuntu Cloud Images and will only publish the daily image builds; after the beta period these images will be promoted with the releases.To kick the tires on the Vagrant images, take a look at http://cloud-images.ubuntu.com/vagrant. I will be working with the fine folks at Vagrant to get the official Ubuntu Vagrant images listed at http://www.vagrantbox.es/If you are interested in learning about the Vagrant development environment, head on over to Vagrant for more information.

For sometime people have been asking me "when will Cloud Images sport a Twitter account?" Well, wait no longer, because the Ubuntu Cloud Image Builder now has a Twitter Account.

The Cloud Image process will now Tweet when a new image is build and published -- dailies, new release updates and new versions being releases. For right we're only Tweeting EC2 information, but once Windows Azure goes GA, we'll start Tweeting that too.

So in the meantime, you can follow our faithful Cloud Builder as it tweets merrily its build progress at @UbuCloudImages. But I'll have to warn you, the Cloud Builder won't response to tweets, so we're not snubbing you if there is no response.

If Twitter isn't your cup of tea, or coffee, or <insert liquid refreshment here>, how about RSS. A while back in November, we introduced RSS feeds for the Ubuntu Cloud Images.

There is a feed for both released images and dailies. The feeds are really simple: they show all the builds that are available. This is a great way to track new releases of the Ubuntu Cloud Images if you don't want want to follow Twitter, hate checking your email or you don't care much for reading our announcement emails.