Given 10 quad-core servers in a cloud running 40 virtual machines. 20 virtual machines are simultaneously discarded. What power savings would you expect from PowerNap?

This is actually a rather complex combinatorics problem. I’ll restate it as such:

Given 10 buckets, each of which can hold 0 – 4 items, how many ways can 20 items be distributed among those 10 buckets? Furthermore, what is the statistical distribution of empty buckets?

It’s been 10 years since my university statistics classes, and my brain melted trying to derive the permutation formula. But a big thanks to my good friend Matt Dobson for solving that! If you’re interested, you can view his combinatorics formula here.

It took me about an hour to hack a Python script that could empirically solve this problem by brute force, generating a comprehensive list of all of the permutations. You can now download and run /usr/bin/powernap_calculator provided by the powernap package in Karmic.

In a cloud with [10] hosts, which can handle [4] guests-per-host, currently running [20] guests,you may expect the following:

[ 5.2%] likely that [0/10] of your hosts would powernap, for a [0%] power savings[ 27.5%] likely that [1/10] of your hosts would powernap, for a [10%] power savings[ 42.5%] likely that [2/10] of your hosts would powernap, for a [20%] power savings[ 21.8%] likely that [3/10] of your hosts would powernap, for a [30%] power savings[ 2.9%] likely that [4/10] of your hosts would powernap, for a [40%] power savings[ <1%] likely that [5/10] of your hosts would powernap, for a [50%] power savings

The overall expected value is [19.0%] power savings.

real 0m46.919suser 0m46.227ssys 0m0.276s

So at this snapshot in time, when your cloud suddenly dropped to 50% utilization, the expected value is a 19% power savings. Note that expected value is a very specific statistics term. Wikipedia says:

Right, so we might expect about a 19% power savings for this random moment in time, when we simultaneously reduce our utilization by 50%.

However, if we observe the cloud over time, and with perhaps more realistic usage patterns, the distribution should be much better than random. VMs will come and go at more staggered intervals than simultaneous destruction of half the instances.

And you will have another tremendous factor working in your favor — Eucalyptus’ greedyschedulingalgorithm. This ensures that each time a new VM is launched, it will land on a system that’s already running. This is known as an annealing system — one that’s constantly correcting itself — since under-utilized systems will automatically powernap, and new VMs will fill in the gaps on running hardware. Pretty cool, I think.

So I’m curious…

What does (or would) your cloud look like?

How many -h|–hosts, -p|–guests-per-host, and -g|–guests do you usually have?

What does powernap_calculator say about your parameters?

Post your results!

Note that the powernap_calculator algorithm is exponential, O((p+1)h), so large values of (p, h) will take a very long time to compute! I’m totally open to code review of powernap_calculator and any performance enhancements ;-)

In my cloud, I have 8 dual-core hosts. I typically limit my usage to 2 guests-per-host. And I rarely run more than 4 VMs at a time.

$ powernap_calculator -h 8 -p 2 -g 4In a cloud with [8] hosts, which can handle [2] guests-per-host, currently running [4] guests,you may expect the following power savings:[ 26.3%] likely that [4/8] of your hosts would powernap, for a [50%] power savings[ 63.2%] likely that [5/8] of your hosts would powernap, for a [62%] power savings[ 10.5%] likely that [6/8] of your hosts would powernap, for a [75%] power savingsThe overall expected value is [60.5%] power savings.

My servers run at about 80 Watts fully loaded. My electricity is about $0.10 per kilowatt-hour. So a year’s worth of electricity with all 8 servers running all of the time is 8 * .08 KW * 24 hr/day * 365 days/year * $0.10/KW-hr = $560/year.

I like the prospects of saving approximately 60.5% of that, or $339/year!

Since each system can hold 0, 1, or 2 virtual machines, we’re going to use a base-3 number (which is p+1). And we’re going to generate all possible 4-digit base-3 numbers (which is (p+1)h). In our case, that’s 34 or 81 scenarios to consider. For each of those scenarios, we convert the decimal integer to a list of each of the digits 0-2, and sum the list, throwing out any “invalid scenarios”, where the sum does not add up to the target number of guests, 3, g. Our valid scenarios is actually much smaller, the following 16:

We also enabled cpu frequency scaling on the Ubuntu Server, defaulting to the on-demand governor. This ensures that Ubuntu Servers run at minimum cpu frequency and power consumption when under-utilized.

pwrkap is an open source project from the IBM Linux Technology Center which provides an energy use monitor and power capping enforcement tools (just what the Slashdot article is discussing!). We partnered with the LTC, working with Darrick Wong, to deliver this technology as a package in Ubuntu Universe.

powerman is another new package in Jaunty. Here, we worked with Arnaud Quette of Eaton to provide in Ubuntu a tool for manipulating Power Distribution Units (PDUs) from a central location–useful for remote operation in data centers and cluster computing.

Finally, we embarked on a tremendous effort to make the Ubuntu Server a better host and client in virtual and cloud computing environments. Cloud, grid, and utility computing, in a generic sense, provide far more scalable resources at the data center level. And being able to move computing efforts between your data center and someone else’s (such as Amazon) also provides some interesting options on the power savings front.

PowerNap is a new free software project from Canonical that acts as sort of a “screen saver” for servers. Ubuntu Servers running PowerNap can be configured to take a specific action (like suspending, or hibernating, or powering off) when deemed inactive (no keyboard mouse activity, and none of some list of processes running).

Eucalyptus has been enhanced to leverage PowerNap and PowerWake, to maintain a far more energy efficient cloud. Eucalyptus uses PowerNap to suspend, hibernate, or power-off nodes that are not currently running any virtual machines. New VM requests are served from the available capacity of running systems. Eucalyptus will PowerWake sleeping systems only if load requires. With PowerNap, data centers will finally realize the energy savings promised by cloud computing.

We are working on packaging Condor for Karmic. Condor is a system that “scavenges” otherwise unused computing cycles, leveraging them for a higher purpose. Think “protein folding” or “SETI@home”, except in your data center, for your grid-capable applications. As such, Condor is less about saving power, but more about increasing utilization and efficiency or your computing resources. You could perhaps choose to PowerNap your under utilized hardware and save energy, or instead Condor your systems and task them to other work.

We have also synchronized the mpich2 package from Debian, thanks to some excellent work from a few Debian developers. mpich2 is an extremely important library for high performance, grid computing applications. Whereas Condor is intended for general purpose grid computing, mpich2 is used by developers and users of very specific applications. Like Condor, mpich2 is also about using available computing resources as efficiently as possible.

So what’s next? I certainly hope to continue working on energy efficiency in the Ubuntu Server. I have a few ideas about what we could do in 10.04.

10.04 (future)

Low Power Architectures

I have blogged a couple of times now (here, and here), about running the Ubuntu Server on Dell Mini’s. These systems have Intel Atom processors, and run the lpia architecture. I would like to see us work more on this, and perhaps partner with the vendors on an Ubuntu Server product for these architectures.

Beyond that, ARM is a fascinating architecture, and will have a tremendous effect on the way we think about computing power. ARM based servers, with solid-state disks will soon run on fractions of a watt of power. Some people are excited about laptops that might have 24 hours of battery. I’m excited about Servers that have a 24 hour built-in battery backup, consume 1% of the power of their predecessors, and can fit in nooks and crannies in every room of your house.

July 13, 2009

A savvy Koala knows that the best way to conserve energy is to goto sleep, and these days even servers can suspend and resume, so imagineif we could make it possible to build a cloud computing facility thatdrops its energy use virtually to zero by napping in the midday heat,and waking up when there's work to be done. No need to drink at theenergy fountain when there's nothing going on. If we get all of thisright, our Koala will help take the edge off the bear market.

I have just uploaded PowerNap to Karmic, and we are well on our way to integrating the technology into the 9.10’s Ubuntu Enterprise Cloud.

Actually, I spent last week in sunny Santa Barbara, California working with Dan Nurmi, of Eucalyptus Systems. We shot some amateur digital videos of Ubuntu Karmic Servers, PowerNap/PowerWake, Eucalyptus, and a Watt meter in action. I’ll get those posted soon!

I’ll go into much deeper technical detail on the design and implementation of PowerNap over the next few weeks in subsequent posts, but I’ll give an overview here…

How Does It Work?

PowerNap operates sort of like a screen saver for servers. Besides watching the console and terminals for keyboard activity, it also watches the system’s process table for activity.

An administrator defines a list of regular expressions describing some critical MONITORED_PROCESSES that should be running. When powernapd notices that all of the MONITORED_PROCESSES have been absent from the process table for some configurable ABSENT_SECONDS, powernapd emits a warning to all users of the system that it will run powernap, unless canceled within the next GRACE_SECONDS.

Thus, if no instance of kvm runs for 5 minutes, then the system will emit a warning, and powernap after a 1 minute grace period.

PowerNap Now!

Alternatively, a system administrator can force the system to powernap immediately by either running /usr/sbin/powernap directly, or sending powernapd the “now” signal with: service powernap now. In fact, this is what the Eucalyptus Node Controller does, such that it can maintain state, and directly control its managed nodes.

What constitutes a powernap?

So, powernap will first check if /etc/powernap/action is executable, and if so, it will run that file. This will allow you, as an administrator, to run any arbitrary script or program of your design when powernapd determines that your server has become idle. Your script could send an email, for example.

echo "Inefficient server, wasting energy" | mail Al_Gore@example.com

But in the default case, powernap will check if your server supports suspend-to-ram, and if so, it will pm-suspend your system. Otherwise, it will suspend-to-disk, or power the system off, depending on the sleep states supported by your hardware.

Slick, huh? :-D

Beyond the Cloud

While PowerNap is bespoke for the Ubuntu Enterprise Cloud, I have implemented it in manner that I hope is generically useful.

My hardware supports S3 suspend-to-ram, so this is great! If 4 minutes go by, where I’m not running any of my media players (mplayer, vlc, xine, mythfrontend.real, xmms), I’m given a 1 minute grace period, and then my system suspends. I have configured wake-on-usb and wake-on-lan in the BIOS, so I can resume the system in a couple of seconds either by tapping a key or sending a WoL magic packet.

But in the mean time, I’ve reduced the power consumption of 4 systems by 90%, for most of every day while I’m not directly using MythTV!

What about Waking Systems?

Which brings me to PowerNap‘s kid brother…PowerWake. /usr/bin/powerwake is another Python script. This script is designed to be a smarter, remote waking utility. Currently, it supports wake-on-lan, but it will eventually support other mechanisms, such as IPMI, and perhaps NUT.

With respect to wake-on-lan, it’s “smarter” than some other wake-on-lan utilities because it uses a hierarchy of cache files, configuration files, and the current arp table, such that you can wake a system by MAC address, or IP address, or hostname. I find this far more convenient than trying to remember or look up MAC addresses. powerwake respects static configuration in /etc/ethers and maintains a dynamically learning cache of known MAC addresses in /var/cache/powerwake/ethers.

Interesting?

I’m eager to hear what other uses you might have for PowerNap and/or PowerWake for your data centers, basements, and living rooms!

July 10, 2009

So the backport of KVM-84 to Hardy and Intrepid has been in the works since March, and we’re now rounding 3rd base.

I’ve produced a couple of release candidates and fixed a few remaining issues. Thank you to everyone who has tested these packages and provided feedback.

The final step before releasing the backport is to ensure that these latest changes get uploaded to jaunty-updates, such that the package is in sync among Hardy, Intrepid, and Jaunty.

One more call for testing…

So I’ve been working hard on this, and I’m at a point where I require assistance from the community. I get emails on a weekly basis from people asking for advice on getting involved in Ubuntu. Here’s your shot ;-)

There is a package in jaunty-proposed that needs to be pushed to jaunty-updates before the Hardy and Intrepid backports can take place. In order to promote the package to jaunty-updates, I need users to verify that the new package fixes the four bugs that I think it fixes, and does not cause regressions.

Please, if you have a system running Jaunty + KVM, give the -proposed package a shot, and provide feedback in the following 4 bugs: