Article Index

The Beowulf mailing list provides detailed discussions about issues
concerning Linux HPC clusters.
In this article I review some postings to the
Beowulf list on
power usage (always a good topic) from two discussion threads
on the beowulf list. I'll also review some discussions about
24-port GigE switches from 2004. Plus a bonus update to the age old question Where can I get
a cheap GigE switch?

Max FLOPS/WATTS

Producing the most performance for a given power level is becoming an
increasingly important issue. Given that Intel has currently stopped
development of more powerful CPU cores because of the huge power
consumption (the new Prescott dissipates about 103W of power), this
question has taken on new meaning. On February 16 of 2004, Camm
Maguire wanted to find the maximum flops (floating point operations
per second) to watts ratio for a cluster. Joshua Baker-LePain was
the first to reply. He thought the Dell Optiplex SX270 provided a
good ratio. These boxes consist of an Intel P4 with Hyperthreading
up to 3.2 GHz with on-board GigE, a laptop type hard drive and a
simple 150 Watt power supply.

{mosgoogle right}

At this point, the discussion broke into two very interesting threads.
The first thread began with a reply from Jim Lux, the "space" cluster
expert. Jim pointed out that this very topic is very near and dear
to his heart because he's designing power constrained systems for
space based computations. Jim's first reply began by asking some
very good questions. Was Camm interested in the power consumption
at the wall plug per node? Or is it the power consumption for the
CPU only? Should memory be included? How about networking power
consumption? Jim pointed out that the networking part of the power
equation is very interesting because you could have a number of low
power nodes with a high-speed interconnect that itself uses a fair
amount of power. In this case Jim thought it might be better to
use a few high power nodes to save on the power for the networking.
Jim also pointed out that you could forgo the networking switch and
connect the nodes in a torus or cube fashion but this will cost
some in CPU performance since the nodes have to route the data.

Andrew Carter responded to Jim's posting with a quick comment. He
mentioned that something like the Via Eden/Nehemiah running at
1 GHz uses about 7W of power. He mentioned that the Mars Rovers are
using about 5W of power to transmit back to Earth. Jim Lux responded
that the power number was pretty accurate but it required a nice
cyrogenically cooled receiver on Earth and hundreds of kilowatts
to transmit from Earth to Mars. If makes you wonder that if we
colonize planets and other bodies in space could we make an
interplanetary Grid.

The second thread started with some postings from William Dieter who
pointed out a cluster design tool at
Aggregate.org
that allows you to design based
on power. It allows you to input cluster parameters such as memory,
hard disk space, etc. and weight the resulting designs based on power
consumption. The designs are based on a set of rules for individual
parts. Bogdan Costescu responded that the rules could be somewhat
simplistic in that reducing the power on the NIC, for example, would
drive you to a NIC that increases the load on the CPU. William
responded that the rules aren't perfect and there is still some
development to be done (volunteers anyone?).

This type of discussion, power vs. performance, is going to become
more common as processor power consumption increases dramatically.

Power Consumption for Opterons

The previous discussion made some points about power consumption.
Another discussion centering on the Opteron also developed as well.
As with many narrow topics, the discussion became much more general
providing some interesting ideas and interesting comments.

On March 9, 2004, Trent Piepho asked about power consumption for
dual Opterons measured with a "kill-a-watt" type device (you can
plug the device into a wall outlet and then plug your node into
the device to measure power usage, etc.). There were a number of
replies to this posting with some very good numbers. Bill Broadley
posted some numbers for a Sun V20z Sunfire (dual 2.2 Ghz Opterons)
which had dual SCSI drives and 4 Gigs of memory. Idling the box
was measured using 237-249 watts. When running a code called
pstream
which is related to the Stream benchmark,
Bill measured 260-277 watts for one instance of pstream and
265-280 watts for two instances of pstream. The unsinkable Robert
Brown posted some numbers for a dual Opteron machine he was
testing (dual 242 Opteron) under a load average of 2. He measured
150 watts using his
kill-a-watt
device. He thought this was quite
good and remarked that it was better than a dual Athlon that was
just idling. Andrew Wang thought that both sets of results were
very good and mentioned that an idling Itanium2 used about 120
watts just for the CPU alone. Mark Hahn also posted that his dual
Opteron (dual 240's) used about 250 watts maximum while running two
copies of Stream and one copy of Bonnie++. Robert Brown then posted
that he tested his dual Opteron again and with a load factor of
3 he measured about 182 watts.

There was some discussion about Roberts numbers because they were
much lower than either Bill's or Mark's. Jim Lux jumped in to say
that the number sounded reasonable (Jim was taking a break from
building his space based Death Star cluster). The 180 watts converts
to about 140 watts DC which allows for about 100 watts total for
the CPUs, 10 watts or so for the fans, and 30 watts for the board
logic and RAM.

Trent Piepho also posted some numbers for various Intel
Pentium III and Xeon machines under various types of loads. The
dual 2.4 GHz Xeon box was a pretty hefty box with a total of 16
SATA drives, a single IDE drive, 1 GB of memory, CD-ROM, and 6
high-speed case fans (measured at 4.4 watts each). When running
RAID and Bonnie++, the box was using about 534 watts (Ouch).