Posts tagged with 'canonical'

This is a series of posts on reasons to choose Ubuntu for your public or private cloud work & play.

We run an extensive program to identify issues and features that make a difference to cloud users. One result of that program is that we pioneered dynamic image customisation and wrote cloud-init. I’ll tell the story of cloud-init as an illustration of the focus the Ubuntu team has on making your devops experience fantastic on any given cloud.

Ever struggled to find the “right” image to use on your favourite cloud? Ever wondered how you can tell if an image is safe to use, what keyloggers or other nasties might be installed? We set out to solve that problem a few years ago and the resulting code, cloud-init, is one of the more visible pieces Canonical designed and built, and very widely adopted.

Traditionally, people used image snapshots to build a portfolio of useful base images. You’d start with a bare OS, add some software and configuration, then snapshot the filesystem. You could use those snapshots to power up fresh images any time you need more machines “like this one”. And that process works pretty amazingly well. There are hundreds of thousands, perhaps millions, of such image snapshots scattered around the clouds today. It’s fantastic. Images for every possible occasion! It’s a disaster. Images with every possible type of problem.

The core issue is that an image is a giant binary blob that is virtually impossible to audit. Since it’s a snapshot of an image that was running, and to which anything might have been done, you will need to look in every nook and cranny to see if there is a potential problem. Can you afford to verify that every binary is unmodified? That every configuration file and every startup script is safe? No, you can’t. And for that reason, that whole catalogue of potential is a catalogue of potential risk. If you wanted to gather useful data sneakily, all you’d have to do is put up an image that advertises itself as being good for a particular purpose and convince people to run it.

There are other issues, even if you create the images yourself. Each image slowly gets out of date with regard to security updates. When you fire it up, you need to apply all the updates since the image was created, if you want a secure machine. Eventually, you’ll want to re-snapshot for a more up-to-date image. That requires administration overhead and coordination, most people don’t do it.

That’s why we created cloud-init. When your virtual machine boots, cloud-init is run very early. It looks out for some information you send to the cloud along with the instruction to start a new machine, and it customises your machine at boot time. When you combine cloud-init with the regular fresh Ubuntu images we publish (roughly every two weeks for regular updates, and whenever a security update is published), you have a very clean and elegant way to get fresh images that do whatever you want. You design your image as a script which customises the vanilla, base image. And then you use cloud-init to run that script against a pristine, known-good standard image of Ubuntu. Et voila! You now have purpose-designed images of your own on demand, always built on a fresh, secure, trusted base image.

Auditing your cloud infrastructure is now straightforward, because you have the DNA of that image in your script. This is devops thinking, turning repetitive manual processes (hacking and snapshotting) into code that can be shared and audited and improved. Your infrastructure DNA should live in a version control system that requires signed commits, so you know everything that has been done to get you where you are today. And all of that is enabled by cloud-init. And if you want to go one level deeper, check out Juju, which provides you with off-the-shelf scripts to customise and optimise that base image for hundreds of common workloads.

As we move from “tens” to “hundreds” to “thousands” of nodes in a typical data centre we need new tools and practices. This hyperscale story – of hyper-dense racks with wimpy nodes – is the big shift in the physical world which matches the equally big shift to cloud computing in the virtualised world. Ubuntu’s popularity in the cloud comes in part from being leaner, faster, more agile. And MAAS – Metal as a Service – is bringing that agility back to the physical world for hyperscale deployments.

Servers used to aspire to being expensive. Powerful. Big. We gave them names like “Hercules” or “Atlas”. The bigger your business, or the bigger your data problem, the bigger the servers you bought. It was all about being beefy – with brands designed to impress, like POWER and Itanium.

Things are changing.

Today, server capacity can be bought as a commodity, based on the total cost of compute: the cost per teraflop, factoring in space, time, electricity. We can get more power by adding more nodes to our clusters, rather than buying beefier nodes. We can increase reliability by doubling up, so services keep running when individual nodes fail. Much as RAID changed the storage game, this scale-out philosophy, pioneered by Google, is changing the server landscape.

In this hyperscale era, each individual node is cheap, wimpy and, by historical standards for critical computing, unreliable. But together, they’re unstoppable. The horsepower now resides in the cluster, not the node. Likewise, the reliability of the infrastructure now depends on redundancy, rather than heroic performances from specific machines. There is, as they say, safety in numbers.

We don’t even give hyperscale nodes proper names any more – ask “node-0025904ce794”. Of course, you can still go big with the cluster name. I’m considering “Mark’s Magnificent Mountain of Metal” – significantly more impressive than “Mark’s Noisy Collection of Fans in the Garage”, which is what Claire will probably call it. And that’s not the knicker-throwing kind of fan, either.

The catch to this massive multiplication in node density, however, is in the cost of provisioning. Hyperscale won’t work economically if every server has to be provisioned, configured and managed as if it were a Hercules or an Atlas. To reap the benefits, we need leaner provisioning processes. We need deployment tools to match the scale of the new physical reality.

That’s where Metal as a Service (MAAS) comes in. MAAS makes it easy to set up the hardware on which to deploy any service that needs to scale up and down dynamically – a cloud being just one example. It lets you provision your servers dynamically, just like cloud instances – only in this case, they’re whole physical nodes. “Add another node to the Hadoop cluster, and make sure it has at least 16GB RAM” is as easy as asking for it.

With a simple web interface, you can add, commission, update and recycle your servers at will. As your needs change, you can respond rapidly, by adding new nodes and dynamically re-deploying them between services. When the time comes, nodes can be retired for use outside the MAAS.

As we enter an era in which ATOM is as important in the data centre as XEON, an operating system like Ubuntu makes even more sense. Its freedom from licensing restrictions, together with the labour saving power of tools like MAAS, make it cost-effective, finally, to deploy and manage hundreds of nodes at a time

Here’s another way to look at it: Ubuntu is bringing cloud semantics to the bare metal world. What a great foundation for your IAAS.

By 14.04 LTS Ubuntu will power tablets, phones, TVs and smart screens from the car to the office kitchen, and it will connect those devices cleanly and seamlessly to the desktop, the server and the cloud.

Unity, the desktop interface in today’s Ubuntu 11.10, was designed with this specific vision in mind. While the interface for each form factor is shaped appropriately, Unity’s core elements are arranged in exactly the way we need to create coherence across all of those devices. This was the origin of the name Unity – a single core interface framework, that scales across all screens, and supports all toolkits.

Canonical and the Ubuntu community have established Ubuntu’s place in desktop, server and cloud deployments. We have also invested in the design and engineering of Unity, motivated by the belief that desktop interfaces would merge with mobile, touch interfaces into a seamless personal computing platform in the future. Today we are inviting the whole Ubuntu community – both commercial and personal – to shape that possibility and design that future; a world where Ubuntu runs on mobile phones, tablets, televisions and traditional PC’s, creating a world where content is instantly available on all devices, in a form that is delightful to use.

A constantly changing world

The way we access the Internet, connect to our friends, listen to music, watch films and go about our daily lives is rapidly evolving. We now use a diverse set of devices with an array of operating systems, which have a range of connectivity. Few people are exclusively loyal to a single technology provider.

Consider this quote from Paul Maritz of VMWare:

“Three years ago over 95 percent of the devices connected to the Internet were personal computers. Three years from now that number will probably be less than 20 percent. More than 80 percent of the devices connected to the Internet will not be Windows-based personal computers.” Paul Maritz, 29 August 2011 VM World Keynote.

Make no mistake – just as the world is changing for manufacturers so is it changing for Linux distributions. Today, 70% of people in Egypt access the Internet solely via the phone. Even in the US that figure is a startling 25%.

Ubuntu is well positioned

Ubuntu will thrive in this new reality.

Our established collaboration with the silicon vendors that are driving this converging market are critical. Intel, ARM and AMD will make the chip-sets that will power this future and Ubuntu works with all of them on all technologies.

Our engagement with the PC market will help bring the results of this work to a huge audience – partnerships with the likes of Dell, HP, Asus, Lenovo, Acer, IBM, Vodafone and more are a gateway to users who want continuous, connected, cross-device computing.

We are determined to bring more free software to more people around the world, and building that future hand in hand with device manufacturers is the best way to do it. There is no winner in place yet. This opportunity remains wide open, but only to products that deliver excellent experiences for users, across a full range of device categories.

The investment we have already made in the interface accommodates the touch scenarios required in some form factors and, with a little love and attention, will work equally well in mouse, keyboard or stylus-driven environments. Ubuntu will not be restricted to small screen or large screen environments but encompasses both and all the form factors in between. We will see our work on the Ubuntu platform land in a variety of formats current and yet to be invented. It is without doubt the most exciting phase in the history of Ubuntu.

Ubuntu One and the software centre

Ubuntu’s personal cloud and app centre services are appropriate for all these environments. They deliver the required storage, syncing and sharing capabilities that are not just a convenience but a requirement as we move to a universe where content is increasingly shared but the devices that access them become more diverse. Ubuntu One’s support for other OSes show the ability of Ubuntu to play nice with others, recognising that the divergence is strength. It allows users to choose the devices they prefer but still delivering the benefits of Ubuntu-centred strategy.

The next steps

We are describing this at UDS to energize the entire Ubuntu ecosystem around this challenge. Canonical will provide the heavy lifting needed to put us in the ball park, but there are opportunities for participation, contribution and engagement by all elements of the broader Ubuntu community, both corporate and individual.

Our developers, our partners’ developers and the broader open source development community share this opportunity. There is a great deal to discuss, and an array of strands we need to pull together at UDS. But the direction is clear and the prize is great – to bring more free software to more people in more delightful ways than ever before.

There are a couple of scenarios that we’re focused on at the moment, where we can offer standardised engagements:

Telco’s building out cloud infrastructures for public cloud services. These are aiming for specific markets based on geography or network topology – they have existing customers and existing networks and a competitive advantage in handling outsourced infrastructure for companies that are well connected to them, as well as a jurisdictional advantage over the global public cloud providers.

Cloud infrastructure prototypes at a division or department level. These are mostly folk who want the elasticity and dynamic provisioning of AWS in a private environment, often to work on products that will go public on Rackspace or AWS in due course, or to demonstrate and evaluate the benefits of this sort of architecture internally.

Cloud-style legacy deployments. These are folk building out HPC-type clusters running dedicated workloads that are horizontally scaled but not elastic. Big Hadoop deployments, or Condor deployments, fall into this category.

Cloud has become something of a unifying theme in many of our enterprise and server-oriented conversations in the past six months. While not everyone is necessarily ready to shift their workloads to a dynamic substrate like Ubuntu Cloud Infrastructure (powered by OpenStack) it seems that most large-scale IT deployments are embracing cloud-style design and service architectures, even when they are deploying on the metal. So we’ve put some work into tools which can be used in both cloud and large-scale-metal environments, for provisioning and coordination.

With 12.04 LTS on the horizon, OpenStack exploding into the wider consciousness of cloud-savvy admins, and projects like Ceph and CloudFoundry growing in stature and capability, it’s proving to be a very dynamic time for IT managers and architects. Much as the early days of the web presented a great deal of hype and complexity and options, only to settle down into a few key standard practices and platforms, cloud infrastructure today presents a wealth of options and a paucity of clarity; from NoSQL choices, through IAAS choices, through PAAS choices. Over the next couple of months I’ll outline how we think the cloud stack will shape up. Our goal is to make that “clean, crisp, obvious” deployment Just Work, bringing simplicity to the cloud much as we strive to bring it on the desktop.

For the moment, though, it’s necessary to roll up sleeves and get hands a little dirty, so the team I mentioned previously has been busy bringing some distilled wisdom to customers embarking on their cloud adventures in a hurry. Most of these engagements started out as custom consulting and contract efforts, but there are now sufficient patterns that the team has identified a set of common practices and templates that help to accelerate the build-out for those typical scenarios, and packaged those up as a range of standard cloud building offerings.

Prompted in part by the critique of Canonical’s code contributions to the kernel and core GNOME infrastructure, I’ve been pondering whether or not I feel good about what I do every day, and how I do it. It’s important for me to feel that what I do is of service to others and makes the world a better place for it having been done. And in my case, that it’s a contribution commensurate with the good fortune I’ve had in life.

Two notes defined for me what I feel I contribute, in this last month. One was a thank-you from New Zealand, from someone who is watching Ubuntu 10.04 make a real difference in their family’s life. For them it seems like a small miracle of human generosity that this entire, integrated, working environment exists and is cared for by thousands of people. The other was a support contract for tens of thousands of desktops running Ubuntu 10.04 in a company. Between those two, we have the twin pillars of the Ubuntu Project and Canonical: to bring all the extraordinary generosity of the free software community to the world at large, as a gift, free of charge, unencumbered and uncrippled, and to do so sustainably.

The first story, from New Zealand, is about someone who is teaching their children to use computers from a young age, and who has observed how much more they get done with Ubuntu than with Windows, and how much more affordable it is to bring computing to all the kids in their community with Ubuntu. For them, the fact that Ubuntu brings them this whole world of free software in one neat package is extraordinary, a breakthrough, and something for which they are very grateful.

It’s a story that I hope to see replicated a hundred million times. And it’s a story which brings credit and satisfaction not just to me, and not just to the people who make Ubuntu the focus of their love and energy, but to all of those who participate in free software at large. Ubuntu doesn’t deserve all the credit, it’s part of a big and complex ecosystem, but without it that delivery of free software just wouldn’t have the same reach and values.

We all understand that the body of free software needs many organs, many cells, each of which has their own priorities and interests. The body can only exist thanks to all of them. We are one small part of the whole, it’s a privilege for us to take up the responsibilities that we do as a distribution. We have the responsibility of choosing a starting point for those who will begin their free software journey with Ubuntu, and we work hard to make sure that all of those pieces fit well together.

Ubuntu, and the possibilities it creates, could not have come about without the extraordinary Linux community, which wouldn’t exist without the GNU community, and couldn’t have risen to prominence without the efforts of companies like IBM and Red Hat. And it would be a very different story if it weren’t for the Mozilla folks and Netscape before them, and GNOME and KDE, and Debian, and Google and everyone else who have exercised that stack in so many different ways, making it better along the way. There are tens of thousands of people who are not in any way shape or form associated with Ubuntu, who make this story real. Many of them have been working at it for more than a decade – it takes a long time to make an overnight success while Ubuntu has only been on the scene six years. So Ubuntu cannot be credited solely for the delight of its users.

Nevertheless, the Ubuntu Project does bring something unique, special and important to free software: a total commitment to everyday users and use cases, the idea that free software should be “for everyone” both economically and in ease of use, and a willingness to chase down the problems that stand between here and there. I feel that commitment is a gift back to the people who built every one of those packages. If we can bring free software to ten times the audience, we have amplified the value of your generosity by a factor of ten, we have made every hour spent fixing an issue or making something amazing, ten times as valuable. I’m very proud to be spending the time and energy on Ubuntu that I do. Yes, I could do many other things, but I can’t think of another course which would have the same impact on the world.

I recognize that not everybody will feel the same way. Bringing their work to ten times the audience without contributing features might just feel like leeching, or increasing the flow of bug reports 10x. I suppose you could say that no matter how generous we are to downstream users, if upstream is only measuring code, then any generosity other than code won’t be registered. I don’t really know what to do about that – I didn’t found Ubuntu as a vehicle for getting lots of code written, that didn’t seem to me to be what the world needed. It needed a vehicle for getting it out there, that cares about delivering the code we already have in a state of high quality and reliability. Most of the pieces of the desktop were in place – and code was flowing in – it just wasn’t being delivered in a way that would take it beyond the server, or to the general public.

The second email I can’t quote from, but it was essentially a contract for services from Canonical to help a company move more than 20,000 desktops from Windows to Ubuntu. There have been several engagements recently of a similar scale, the pace is accelerating as confidence in Ubuntu grows. While Linux has long proven itself a fine desktop for the inspired and self-motivated developer, there is a gap between that and the needs of large-scale organisations. There isn’t another company that I’m aware of which is definitively committed to the free software desktop, and so I’m very proud that Canonical is playing that role in the free software ecosystem. It would be sad for me if all the effort the free software community puts into desktop applications didn’t have a conduit to those users.

There’s nothing proprietary or secret that goes into the desktops that Canonical supports inside large organisations. The true wonder for me is that the story from New Zealand, and the corporate story, both involve exactly the same code. That to me is the true promise of free software; when I have participated in open source projects myself, I’ve always been delighted that my work might serve my needs but then also be of use to as many other people as possible.

Ubuntu is a small part of that huge ecosystem, but I feel proud that we have stepped up to tackle these challenges.

Canonical takes a different approach to the other companies that work in Linux, not as an implicit criticism of the others, but simply because that’s the set of values we hold. Open source is strengthened by the fact that there are so many different companies pursuing so many different, important goals.

In recent weeks it’s been suggested that Canonical’s efforts are self-directed and not of benefit to the broader open source community. That’s a stinging criticism because most of us feel completely the opposite, we’re motivated to do as much as we can to further the cause of free software to the benefit both of end-users and the community that makes it, and we’re convinced that building Ubuntu and working for Canonical are the best ways to achieve that end. It’s prompted a lot of discussion and consideration for each of us and for Canonical as a whole. And this post is a product of that consideration: a statement for myself of what I feel I contribute, and why I feel proud of the effort I put in every day.

What do we do for free software? And what do I do myself?

For a start, we deliver it. We reduce the friction and inertia that prevent people trying free software and deciding for themselves if they like it enough to immerse themselves in it. Hundreds of today’s free software developers, translators, designers, advocates got the opportunity to be part of our movement because it was easy for them to dip their toe in the water. And that’s not easy work. Consider the effort over many years to produce a simple installer for Linux like http://www.techdrivein.com/2010/08/massive-changes-coming-to-ubuntu-1010.html which is the culmination of huge amounts of work from many groups, but which simply would not have happened without Canonical and Ubuntu.

There are thousands of people who are content to build free software for themselves, and that’s no crime. But the willingness to shape it into something that others will find, explore and delight in needs to be celebrated too. And that’s a value which is celebrated very highly in the Ubuntu community: if you read planet.ubuntu.com you’ll see a celebration of *people using free software*. As a community we are deeply satisfied to see people *using* it to solve problems in their lives. That’s more satisfying to us than stories about how we made it faster or added a feature. Of course we do bits of both, but this is a community that measures impact in the world rather than impact on the code. They are very generous with their time and expertise, with that as the reward. I’m proud of the fact that Ubuntu attracts people who are generous in their contributions: they feel their contributions are worth more if they are remixed by others, not less. So we celebrate Kubuntu and Xubuntu and Puppy and Linux Mint. They don’t ride on our coattails, they stand on our shoulders, just as we stand on the shoulders of giants. And that’s a good thing. Our work is more meaningful and more valuable because their work reaches users that ours alone could not.

What else?

We fix it, too. Consider the https://wiki.ubuntu.com/PaperCut Papercuts project, born of the recognition that all the incredible technology and effort that goes into making something as complex as the Linux kernel is somehow diminished if the average user gets an incomprehensible result when they do something that should Just Work. Hundreds of Papercuts have been fixed, across many different applications, benefiting not just Ubuntu but also every other distribution that ships those applications. If you think that’s easy, consider the effort involved to triage and consider each of thousands of suggestions, coordinating a fix and the sharing of it. The tireless efforts of a large team have made an enormous difference. Consider this: saving millions of users one hour a week is a treasury of energy saved to do better things with free software. While the Canonical Design team played a leading role in setting up the Papercuts project, the real stars are people like http://www.omgubuntu.co.uk/2010/06/maverick-papercut-hunting-season-opens.html Vish and Sense who rally the broader papercuts team to make a difference. Every fix makes a difference, on the desktop http://ubuntuserver.wordpress.com/2010/01/20/ubuntu-server-papercuts-project/ and the server.

At a more personal level, a key thing I put energy into is leadership, governance and community structure. When we started Ubuntu, I spent a lot of time looking at different communities that existed at the time, and how they managed the inevitable tensions and differences of opinion that arise when you have lots of sharp people collaborating. We conceived the idea of a code of conduct that would ensure that our passions for the technology or the work never overwhelmed the primary goal of bringing diverse people together to collaborate on a common platform. I’m delighted that the idea has spread to other projects: we don’t want to hoard ideas or designs or concepts, that would be contrary to our very purpose.

We setup a simple structure: a technical board and a community council. That approach is now common in many other projects too. As Ubuntu has grown, so that governance has evolved, there are now multiple leadership teams for groups like Kubuntu and the Forums and IRC, who provide counsel and guidance for teams of LoCo’s and moderators and ops and developers, who in turn strive for technical perfection and social agility as part of an enormous global community. That’s amazing. When people start participating in Ubuntu they are usually motivated as much by the desire to be part of a wonderful community as they are to fix a specific problem or ease a specific burden. Over time, some of those folks find that they have a gift for helping others be more productive, resolving differences of opinion, doing the work to organise a group so that much more can be achieved than any one individual could hope to do. Ubuntu’s governance structures create opportunities for those folks to shine: they provide the backbone and structure which makes this community able to scale and stay productive and happy.

A project like Ubuntu needs constant care in order to defend its values. When you are tiny and you put up a flag saying “this is what we care about” you tend to attract only people who care about those things. When the project grows into something potent and visible, though, you tend to attract EVERYONE, because people want to be where the action is. And then the values can easily be watered down. So I continue to put energy into working with the Ubuntu community council, and the Canonical community team, both of which are profoundly insightful and hard-working which makes that part of my work a real pleasure. The Ubuntu community council take their responsibility as custodian of the projects community values very seriously indeed. The CC is largely composed of people who are not affiliated with Canonical, but who nevertheless believe that the Ubuntu project is important to free software as a whole. And the awesome Jono Bacon, the delightful Daniel Holbach, and unflappable Jorge Castro are professionals who understand how to make communities productive and happy places to work.

Something as big as the Ubuntu community cannot be to the credit of me or any other individual, but I’m proud of the role I’ve played, and motivated to continue to play a role as needed.

In more recent years I’ve come to focus more on championing the role of design in free software. I believe that open source produces the best quality software over time, but I think we need a lot more cogent conversations about the experiences we want to create for our users, whether it’s on the desktop, the netbook or the server. So I’ve put a lot of my leadership energy into encouraging various communities – both Ubuntu and upstream – to be welcoming of those who see software through the eyes of the new user rather than the experienced hacker. This is a sea change in the values of open source, and is not something I can hope to achieve alone, but I’m nevertheless proud to be a champion of that approach and glad that it’s steadily becoming accepted.

There were designers working in free software before we made this push. I hope they feel that Canonical’s emphasis on the design-lead approach has made their lives easier, and the community at large more appreciative of their efforts and receptive to their ideas. But still, if you *really* care about design in free software, the Canonical design team is the place to be.

I do some design work myself, and have participated most heavily in the detailed design of Unity, the interface for Ubuntu Netbook Edition 10.10. That’s an evolution of the older UNR interface; most importantly, it’s a statement that Linux desktops don’t need to be stuck in the 90′s, we can and will attempt to build new and efficient ways of working with computers. I’ve been delighted with the speed at which some of Unity’s facilities have been adopted by hundreds of projects, their goal is to make using Linux easier and classier for everyone, so that pace of adoption is a measure of the speed at which we are reducing the friction for new users discovering a better way to use their PC’s.

Design without implementation would leave us open to accusations of wanting others to do our work for us, so I’m proud also to lead a wonderful team that is doing the implementation of some of those key components. Things like dbusmenu have proven useful for bringing consistency to the interfaces of both GNOME and KDE applications running under Unity, and I very much hope they are adopted by other projects that need exactly the facilities they provide. I’d credit that engineering team with their focus on quality and testability and their desire to provide developers with clean API’s and good guidance on how best to use them. If you’ve used the full set of Indicators in 10.10 then you know how this quiet, persistent work that has engaged many different projects has transformed the panel into something crisp and efficient. Utouch is coming up for its first release, and will continue to evolve, so that Ubuntu and GNOME and KDE can have an easy road to multi-touch gesture interface goodness.

Beyond my own personal time, I also support various projects through funding. Putting money into free software needs to meet a key test: could that money achieve a better outcome for more people if it were directed elsewhere? There are lots of ways to help people: $100,000 can put a lot of people through school, clothed and fed. So I really need to be confident that the money is having a real, measurable impact on people’s lives. The thank you notes I get every week for Ubuntu help sustain that confidence. More importantly, my own observation of the catalytic effect that Ubuntu has had on the broader open source ecosystem, in terms of new developers attracted, new platforms created, new businesses launched and new participants acknowledged, make me certain that the funding I provide is having a meaningful consequence.

When Ubuntu was conceived, the Linux ecosystem was in a sense fully formed. We had a kernel. We had GNOME and KDE. We had X and libc and GCC and all the other familiar tools. Sure they had bugs and they had shortcomings and they had roadmaps to address them. But there was something missing: sometimes it got articulated as “marketing”, sometimes as “end-user focus”. I remember thinking “that’s what I could bring”. So Ubuntu, and Canonical, have quite explicitly NOT put effort into things which are obviously working quite well, instead, we’ve tried to focus on new ideas and new tools and new components. I see that as an invigorating contribution to the broader open source ecosystem, and I hear from many people that they perceive it the same way. Those who say “but Canonical doesn’t do X” may be right, but that misses all the things we do, which weren’t on the map beforehand. Of course, there’s little that we do exclusively, and little that we do that others couldn’t if they made that their mission, but I think the passion of the Ubuntu community, and the enthusiasm of its users, reflects the fact that there is something definitively new and distinctive about the project. That’s something to celebrate, something to be proud of, and something to motivate us to continue.

Free software is bigger than any one project. It’s bigger than the Linux kernel, it’s bigger than GNU, it’s bigger than GNOME and KDE, it’s bigger than Ubuntu and Fedora and Debian. Each of those projects plays a role, but it is the whole which is really changing the world. So when we start to argue with one another from the perspective of any one slice of free software, we run the risk of missing the bigger picture. That’s a bit like an auto-immune disease, where the body starts to attack itself. By definition, someone else who is working hard all day long to bring free software to a wider audience is on the same side as me, compared to 99% of the rest of the world, if I want to think in terms of sides. I admire and respect everyone who puts energy into advancing the cause of free software, even if occasionally I might differ on the detail of how it can be done.

A few months ago we took on the challenge of building a version of Ubuntu for the dual-boot, instant-on market. We wanted to be surfing the web in under 10 seconds, and give people a fantastic web experience. We also wanted it to be possible to upgrade from that limited usage model to a full desktop.

The fruit of that R&D is both a new desktop experience codebase, called Unity, and a range of Light versions of Ubuntu, both netbook and desktop, that are optimised for dual-boot scenarios.

The dual-boot, web-focused use case is sufficiently different from general-purpose desktop usage to warrant a fresh look at the way the desktop is configured. We spent quite a bit of time analyzing screenshots of a couple of hundred different desktop configurations from the current Ubuntu and Kubuntu user base, to see what people used most. We also identified the things that are NOT needed in lightweight dual-boot instant-on offerings. That provided us both with a list of things to focus on and make rich, and a list of things we could leave out.

Instant-on products are generally used in a stateless fashion. These are “get me to the web asap” environments, with no need of heavy local file management. If there is content there, it would be best to think of it as “cloud like” and synchronize it with the local Windows environment, with cloud services and other devices. They are also not environments where people would naturally expect to use a wide range of applications: the web is the key, and there may be a few complementary capabilities like media playback, messaging, games, and the ability to connect to local devices like printers and cameras and pluggable media.

We also learned something interesting from users. It’s not about how fast you appear to boot. It’s about how fast you actually deliver a working web browser and Internet connection. It’s about how fast you have a running system that is responsive to the needs of the user.

Unity: a lightweight netbook interface

There are several driving forces behind the result.

The desktop screenshots we studied showed that people typically have between 3 and 10 launchers on their panels, for rapid access to key applications. We want to preserve that sense of having a few favorite applications that are instantly accessible. Rather than making it equally easy to access any installed application, we assume that almost everybody will run one of a few apps, and they need to switch between those apps and any others which might be running, very easily.

We focused on maximising screen real estate for content. In particular, we focused on maximising the available vertical pixels for web browsing. Netbooks have screens which are wide, but shallow. Notebooks in general are moving to wide screen formats. So vertical space is more precious than horizontal space.

We also want to embrace touch as a first class input. We want people to be able to launch and switch between applications using touch, so the launcher must be finger friendly.

Those constraints and values lead us to a new shape for the desktop, which we will adopt in Ubuntu’s Netbook Edition for 10.10 and beyond.

First, we want to move the bottom panel to the left of the screen, and devote that to launching and switching between applications. That frees up vertical space for web content, at the cost of horizontal space, which is cheaper in a widescreen world. In Ubuntu today the bottom panel also presents the Trash and Show Desktop options, neither of which is relevant in a stateless instant-on environment.

Second, we’ll expand that left-hand launcher panel so that it is touch-friendly. With relatively few applications required for instant-on environments, we can afford to be more generous with the icon size there. The Unity launcher will show what’s running, and support fast switching and drag-and-drop between applications.

Third, we will make the top panel smarter. We’ve already talked about adopting a single global menu, which would be rendered by the panel in this case. If we can also manage to fit the window title and controls into that panel, we will have achieved very significant space saving for the case where someone is focused on a single application at a time, and especially for a web browser.

We end up with a configuration like this:

Mockup of Unity Launcher and Panel with maximised application

The launcher and panel that we developed in response to this challenge are components of Unity. They are now in a state where they can be tested widely, and where we can use that testing to shape their evolution going forward. A development milestone of Unity is available today in a PPA, with development branches on Launchpad, and I’d very much like to get feedback from people trying it out on a netbook, or even a laptop with a wide screen. Unity is aimed at full screen applications and, as I described above, doesn’t really support traditional file management. But it’s worth a spin, and it’s very easy to try out if you have Ubuntu 10.04 LTS installed already.

Ubuntu Light

Instant-on, dual boot installations are a new frontier for us. Over the past two years we have made great leaps forward as a first class option for PC OEM’s, who today ship millions of PC’s around the world with Ubuntu pre-installed. But traditionally, it’s been an “either/or” proposition – either Windows in markets that prefer it, or Ubuntu in markets that don’t. The dual-boot opportunity gives us the chance to put a free software foot forward even in markets where people use Windows as a matter of course.

And it looks beautiful:

Ubuntu Light, showing the Unity launcher and panel

In those cases, Ubuntu Netbook Light, or Ubuntu Desktop Light, will give OEM’s the ability to differentiate themselves with fast-booting Linux offerings that are familiar to Ubuntu users and easy to use for new users, safe for web browsing in unprotected environments like airports and hotels, focused on doing that job very well, but upgradeable with a huge list of applications, on demand. The Light versions will also benefit from the huge amount of work done on every Ubuntu release to keep it maintained – instant-on environments need just as much protection as everyday desktops, and Ubuntu has a deep commitment to getting that right.

The Ubuntu Light range is available to OEM’s today. Each image will be hand-crafted to boot fastest on that specific hardware, the application load reduced to the minimum, and it comes with tools for Windows which assist in the management of the dual-boot experience. Initially, the focus is on the Netbook Light version based on Unity, but in future we expect to do a Light version of the desktop, too.

Given the requirement to customise the Light versions for specific hardware, there won’t be a general-purpose downloadable image of Ubuntu Light on ubuntu.com.

Evolving Unity for Ubuntu Netbook Edition 10.10

Unity exists today, and is great for the minimalist, stateless configurations that suit a dual-boot environment. But in order embrace it for our Netbook UI, we’ll need to design some new capabilities, and implement them during this cycle.

Those design conversations are taking place this week at UDS, just outside Brussels in Belgium. If you can’t be there in person, and are interested in the design challenges Unity presents for the netbook form factor, check out the conference schedule and participate in the discussion virtually.

The two primary pieces we need to put in place are:

Support for many more applications, and adding / removing applications. Instant-on environments are locked down, while netbook environments should support anybody’s applications, not just those favored in the Launcher.

Support for file management, necessary for an environment that will be the primary working space for the user rather than an occasional web-focused stopover.

We have an initial starting point for the design, called the Dash, which presents files and applications as an overlay. The inspiration for the Dash comes from consoles and devices, which use full-screen, media-rich presentation. We want the Dash to feel device-like, and use the capabilities of modern hardware.

The Unity Dash, showing the Applications Place

The instant-on requirements and constraints proved very useful in shaping our thinking, but the canvas is still blank for the more general, netbook use case. Unity gives us the chance to do something profoundly new and more useful, taking advantage of ideas that have emerged in computing from the console to the handheld.

Relationship to Gnome Shell

Unity and Gnome Shell are complementary for the Gnome Project. While Gnome Shell presents an expansive view of how people work in complex environments with multiple simultaneous activities, Unity is designed to address the other end of the spectrum, where people are focused on doing one thing at any given time.

Unity does embrace the key technologies of Gnome 3: Mutter, for window management, and Zeitgeist will be an anchor component of our file management approach. The interface itself is built in Clutter.

The design seed of Unity was in place before Gnome Shell, and we decided to build on that for the instant-on work rather than adopt Gnome Shell because most of the devices we expect to ship Ubuntu Light on are netbooks. In any event, Unity represents the next step for the Ubuntu Netbook UI, optimised for small screens.

The Ubuntu Netbook interface is popular with Gnome users and we’re fortunate to be working inside an open ecosystem that encourages that level of diversity. As a result, Gnome has offerings for mobile, netbook and desktop form factors. Gnome is in the lucky position of having multiple vendors participating and solving different challenges independently. That makes Gnome stronger.

Relationship to FreeDesktop and KDE

Unity complies with freedesktop.org standards, and is helping to shape them, too. We would like KDE applications to feel welcome on a Unity-based netbook. We’re using the Ayatana indicators in the panel, so KDE applications which use AppIndicators will Just Work. And to the extent that those applications take advantage of the Messaging Menu, Sound Indicator and Me Menu, they will be fully integrated into the Unity environment. We often get asked by OEM’s how they can integrate KDE applications into their custom builds of Ubuntu, and the common frameworks of freedesktop.org greatly facilitate doing so in a smooth fashion.

Looking forward to the Maverick Meerkat

It will be an intense cycle, if we want to get all of these pieces in line. But we think it’s achievable: the new launcher, the new panel, the new implementation of the global menu and an array of indicators. Things have accelerated greatly during Lucid so if we continue at this pace, it should all come together. Here’s to a great summer of code.

One of the driving mantras for us is “less is more”. I want us to “clean up, simplify, streamline, focus” the user experience work that we lead. The idea is to recognize the cost of every bit of chrome, every gradient or animation or line or detail or option or gconf setting. It turns out that all of those extras add some value, but they also add clutter. There’s a real cost to them – in attention, in space, in code, in QA. So we’re looking for things to strip out, as much (or more) as things to put in.

I’m not sure we’ll go as far as Microsoft has with their new Windows Phone 7 UI (links to .PPTX), which uses a design language called Metro. It’s radically pared back, and very cool work. It will be interesting to see if they’ve gone too far, or if users take to the more abstract feel of it.

It’s not hard to get people enthusiastic about the idea that less is more. However, it’s quite hard to get people to agree on which bits can be less. It turns out that one person’s clutter is another person’s most useful and valued feature.

Less, it turns out, is still less.

So, for example, consider tooltips on the panel. In bug #527458, there’s some discussion about a decision I made to deprecate tooltips on panel indicators. For quite a lot of people, that’s a little less too far.

On that particular decision, we’ll have to let time tell. For the moment, the decision stands. I’m the first to admit fallibility but I also know that it would be impossible to get consensus around a change like that. If those tooltips are, on balance, really just clutter, then unless someone is willing to take a decision that will be unpopular, they will be clutter forever. And it’s easier for me to make a decision like that in Ubuntu than for virtually anybody else. I apologise in advance for the mistakes that I will certainly make, and which others on the design team may make too, but I think it’s important to defend our willingness to pare things back and let the core, essential goodness shine through. We have to balance innovation and change with clarification and focus. We can’t *stop* innovating and changing, and we have to be willing to remove things that someone will miss.

The bug is a good place to continue the discussion about that particular issue. But I thought it would be useful to issue a call to arms, and invite suggestions from people on the Ayatana list as to what elements of the existing Ubuntu desktop can be trimmed back, on balance making the whole better.

There’s a growing awareness and excitement about the importance of design in free software. A few years ago, folks laughed out loud when it was suggested that design is a good thing for the free software community to build expertise in. And it’s been slow going, admittedly. It’s hard to bring clarity in a crowd. Or mob. We’ve been doing our part to lead that at Canonical and in the Ubuntu community, both through internal work and through public forums. If you’re interested in design and Free Software, then Ubuntu and Ayatana and related forums are great places to participate. And your participation is welcome!

Jono Bacon, Alan Pope, and many others have written, yesterday we published a new visual story and style for Ubuntu. The core design work was lead by Marcus Haslam, Otto Greenslade and Dominic Edmunds, who are the three visual artists leading our efforts in the Canonical Design team. Once we had the base ideas in place we invited some anchor members of the Ubuntu Art community to a design sprint, to test that the concept had the legs to work with the full range of forums, websites, derivatives and other pieces of this huge and wonderful project. And apparently, it does!

Here are some additional thoughts.

Embracing both Ubuntu and Canonical

One of the real challenges for us has been to find a branding and design strategy which spans the spectrum of audiences, forums and dialogues that we cover. With Ubuntu, it’s my specific dream to find a constructive blend of commercial and community interests, not only for Canonical but for other companies. That has made our design and branding work difficult – the distinctive look of Ubuntu lent itself well to pure community messaging, but it was hard to do a brochure for Canonical data center services for Ubuntu on servers. We have not only Ubuntu, but also Kubuntu and an important range of derivatives that all have a role in our ecosystem.

So we spent a lot of time trying to distill the requirements down into a set of three dimensions:

We found a set of ideas which each represent those spectrums, and which work together.

For example, we identified a palette which includes both a fresh, lively Orange, and a rich, mature Aubergine, which work together. The use of Aubergine indicates Commercial involvement of one form or another, while Orange is a signal of community engagement. The Forums will use the Orange elements more strongly, and a formal product brochure, with descriptions of supporting services, would use more of the Aubergine.

On the consumer/enterprise spectrum, we took inspiration from the aerospace industry, and identified a texture of closely spaced dots. When you see more of that, it means we’re signalling that the story is more about the enterprise, less of that, and it’s more about the consumer. Of course, there are cross-overs, for example when we are talking about the corporate desktop, where we’ll use that closely space dot texture as a boundary area, or separator. We also identified shades of Aubergine that are more consumer, or more enterprise – the darker shades mapping to a stronger emphasis on enterprise work.

And on the end-user / engineer spectrum, we took inspiration from graph paper and engineering blue prints. When you see widely spaced patterns of dots, or outline images and figures, that’s signalling that the content is more engineering-oriented than end-user oriented.

And finally, we found a number of themes which enhanced and echoed those ideas. We use a warm gray supporting colour to give shape to pages and documents, and we built on the dots and circles to create a whole style for figures, illustrations and pictograms.

The beauty of this is that we can now publish content that spans the full range, and we generally know when we start the design process what sorts of visual cues we want to be signalling. Instead of having these different mental domains fight with one another, we can now convey quite subtle collaboration between community and corporate, or work which is aimed at engineers and developers from enterprises as opposed to developers working with consumers. Time will tell how it shapes up, but for now I’m celebrating the milestone and the efforts of the team that pulled it together. There’s something there for everyone who wants to participate in the great hubbub of Ubuntuness that is our shared experience of free software.

So, for example, here’s a conference banner. The strong use of Aubergine suggests that it’s more corporate messaging (Canonical is heavily involved). Orange is used here more as a highlight. The Aubergine is darker, and there’s quite a lot of the fine dot pattern. Below the image is a set of scales showing where on those spectra this work is pitched.

As another example, here’s a brochure with an emphasis on end-users who are thinking about adopting Ubuntu’s cloud infrastructure. Again, the fine dot patterns suggests a more enterprise focus, as does the use of the dark aubergine. You can see the circle metaphor used in the quote callout.

And here’s a similar brochure, but with a more developer or engineering oriented focus: note the use of the graph-paper theme with wide spaced dots, and outline shapes.

Finally, here’s an example of a brochure and CD cover for Ubuntu:

As you can see the idea is to signal a mix of both community and Canonical involvement in the message, addressing consumer audiences with a mix of developers and end-users.

A new Ubuntu font

We have commissioned a new font to be developed both for the logo’s of Ubuntu and Canonical, and for use in the interface. The font will be called Ubuntu, and will be a modern humanist font that is optimised for screen legibility. It will be published under an open font license, and considered part of the trade dress of Ubuntu, which will limit its relevance for software interfaces outside of Ubuntu but leave it free for use across the web and in printed documents.

It will take a few months for the font to be finalised, initial elements will be final in the next week which will be sufficient for the logo and other bits and pieces, but I expect to see that font widely used in 10.10. The work has been commissioned from world-renowned fontographers Dalton Maag, who have expressed excitement at the opportunity to publish an open font and also a font that they know will be used daily by millions of people.

Initial coverage will be Western, Arabic, Hebrew and Cyrillic character sets, but over time we may be able to extend that to being a full Unicode font, with great kerning and hinting for print and screen usage globally. We are considering an internship program, to support aspiring fontographers from all corners of the world to visit London and work with Dalton Maag to extend the font to their own regional glyph set.

The critical test of the font is screen efficiency and legibility, and its character and personality are secondary to its fitness for that purpose. Nevertheless, our hope is that the font has a look that is elegant and expresses the full set of values for both Canonical and Ubuntu: adroitness, accountability, precision, reliability, freedom and collaboration. We’ll publish more as soon as we have it.

A good start

It’s been an exciting process, but I have the sense that we are just getting started. The language will get richer, we will find new things that we want to communicate, and new treatments and visual themes that resonate well with these starting points. We’ll find new ways to integrate this on the web, and on the desktop (look out for the two new themes, Radiance and Ambiance). I hope we’ll see the language being used to good effect across everything we do, both commercial and community oriented. There’s a range of expression here that should be useful to artists across the spectrum. Let me know how it works for you.

When you present yourself on the web, you have 15 seconds to make an impression, so aspiring champions of the web 2.0 industry have converged on a good recipe for success:

Make your site visually appealing,

Do something different and do it very, very well,

Call users to action and give them an immediate, rewarding experience.

We need the same urgency, immediacy and elegance as part of the free software desktop experience, and that’s is an area where Canonical will, I hope, make a significant contribution. We are hiring designers, user experience champions and interaction design visionaries and challenging them to lead not only Canonical’s distinctive projects but also to participate in GNOME, KDE and other upstream efforts to improve FLOSS usability.

Fortunately, we won’t be working in a vacuum. This is an idea that is already being widely explored. It’s great to see that communities like GNOME and KDE have embraced user experience as a powerful driver of evolution in their platforms. Partly because of the web-2.0 phenomenon and the iPhone, there’s a widely held desire to see FLOSS leap forward in usability and design. We want to participate and help drive that forward.

There’s also recognition for the scale of the challenge that faces us. When I laid out the goal of “delivering a user experience that can compete with Apple in two years” at OSCON, I had many questions afterwards about how on earth we could achieve that. “Everyone scratches their own itch, how can you possibly make the UI consistent?” was a common theme. And it’s true – the free software desktop is often patchy and inconsistent. But I see the lack of consistency as both a weakness (GNOME, OpenOffice and Firefox all have different UI toolkits, and it’s very difficult to make them seamless) and as a strength – people are free to innovate, and the results are world-leading. Our challenge is to get the best of both of those worlds.

I don’t have answers to all of those questions. I do, however, have a deep belief in the power of the free software process to solve seemingly intractable problems, especially in the long tail. If we articulate a comprehensive design ethic, a next-generation HIG, we can harness the wisdom of crowds to find corner cases and inconsistencies across a much broader portfolio of applications than one person or company could do alone. That’s why it’s so important to me that Canonical’s design and user experience team also participate in upstream projects across the board.

In Ubuntu we have in general considered upstream to be “our ROCK”, by which we mean that we want upstream to be happy with the way we express their ideas and their work. More than happy – we want upstream to be delighted! We focus most of our effort on integration. Our competitors turn that into “Canonical doesn’t contribute” but it’s more accurate to say we measure our contribution in the effectiveness with which we get the latest stable work of upstream, with security maintenance, to the widest possible audience for testing and love. To my mind, that’s a huge contribution.

Increasingly, though, Canonical is in a position to drive real change in the software that is part of Ubuntu. If we just showed up with pictures and prototypes and asked people to shape their projects differently, I can’t imagine that being well received! So we are also hiring a team who will work on X, OpenGL, Gtk, Qt, GNOME and KDE, with a view to doing some of the heavy lifting required to turn those desktop experience ideas into reality. Those teams will publish their Bzr branches in Launchpad and of course submit their work upstream, and participate in upstream sprints and events. Some of the folks we have hired into those positions are familiar contributors in the FLOSS world, others will be developers with relevant technical expertise from other industries.

One strong meme we want to preserve is the idea that Ubuntu, the platform team, is still primarily focused on integration and distribution. We will keep that team and the upstream work distinct to minimise the conflict of interest inherent in choosing the patches and the changes and the applications that actually ship each six months as part of an Ubuntu release.

Of course, there’s a risk to participation, because you can’t easily participate without expressing opinions, visions, desires, goals, and those can clash with other participants. It’s hard to drive change, even when people agree that change is needed. I hope we can find ways to explore and experiment with new ideas without blocking on consensus across diverse and distributed teams. We have to play to our strengths, which include the ability to diverge for experimental purposes to see what really works before we commit everyone to a course of action. It will be a challenge, but I think it’s achievable.

All of this has me tapdancing to work in the mornings, because we’re sketching out really interesting ideas for user interaction in Launchpad and in the desktop. The team has come together very nicely, and I’m thoroughly enjoying the processes, brainstorming and prototyping. I can’t wait to see those ideas landing in production!