I bought it because I was curious about folding performance of the Magny-Cours chips and how it compared with the current Xeon line-up of chips.

Outside of Bit-Tech.Net which compared dual Xeon x5650s and dual Opteron 6174, there wasn't really much out there, and this comparison was done using BigAdv A2 units, which were de-commissioned by this time.

Anyway, I found the performance was very close to a well-overclocked Core I7 980x running at 4.1 Ghz or higher, except on BigAdv 2684 units, where the dual Opteron could out-fold even a 4.4Ghz Core I7 980x.

Generally I was getting between 55 - 60K PPD on 2685, 2686, and 2692 units and around 42K PPD on 2684 BigAdv units. Not bad, but I wanted more!

Power usage at the time was around 303 - 309 watts, which really was in line with a Core I7 980x system running a modern day graphics adapter. The Asus KGPE-D16 board comes with a ultra-low end graphics adapter, but it also comes with an IPMI/KVM adapter allowing console access without keyboard and mouse, which works great. It even allows BIOS access and the ability to force power off/on the server. Very nice.

In the meantime I also decided to get an SR-2 with twin Xeon x5650s. I then remembered why I hated overclocking Core I7 based stuff and it was even worse with dual Xeon chips. I got the SR-2 to 3.9Ghz and it went between 88K and 147K PPD on BigAdv units. Impressive no doubt, but going 4.0Ghz and higher actually resulted in a loss of folding speed so I left it at that.

A while after that, after corresponding with a fellow KGPE-D16 and Opteron 6168 duallie owning NCIX folder who was looking to get rid of some Opteron 6168 "Black Edition" chips, we worked out a deal for me to get this second set of Opteron 6168s. He wanted to go SR-2. I had already done so, and I had been interested in those chips since seeing them listed on EBay a few months earlier.

Fortuitous luck one might say.

After playing around with multiplier overclocking I settled on a 2.6Ghz overclock and got between 60K PPD and 92K PPD with BigAdv units.

Not bad either, and I could go even higher to 2.8, maybe 2.9Ghz, but at that point I changed my mind.

QUAD box!!!!!!!!!!!!!!!!!!!

I already had my original Opteron 6168 retail chips, safely tucked away back in their boxes, and I now had two more 6168s. So why not? The big cost had already been absorbed and either I could re-sell my original 6168s at a loss, or keep them and see what I could do.

If worse came to worst I would sell them and keep the "Black Editions".

So I needed to get 8 more 1GB DIMMS, which luckily enough had dropped in price like a rock to nearly half of what I had bought them at just a few months earlier. I originally bought them for about $40.00 per DIMM back in August, but now in November they were $23 - $24.00 a DIMM!

The Opterons are unique because to see full performance you need quad-channel DDR3 RAM. Each chip "package" holds two 6-core CPUs, each with a dual-channel RAM IMC, so four channels in total.

This monstrous SWTX board is wider and nearly as deep as the SR-2 which is one of the biggest consumer motherboards (as many of us know) and hard to find a case for. It's 13" deep but over 16" wide!

So I special ordered it in November and last Friday the big monster board came in. Huge, heavy and as rigid as they come, but a thing of beauty.

Here's a shot with the CPUs installed:

So I hooked it up to the Enermax Revolution 85+ 1050W, stuck in the CPUs (as pictured) and it was off to the races. Right now I'd figured I would run 'er naked and figure out the case situation once the excitement wore off and my head was screwed back on

I had bought some SuperMicro SNK-P0043P and mounted a couple of CM BladeMaster 92mm 2500 RPM PWM fans on them and things were kept cool, and I bought two more TT ITBU CL-P0314 heat sinks which were Socket F 1207 HSF units, but using the 4.1" spacing I required to screw them into the G34 compatible screwholes (also pictured).

I then decided to stick the 16 1 GB DIMMs in and hook up all THREE EPS 12V power connectors. I needed to get a dual molex to EPS 12V connector as the board has THREE whereas the Enermax 1050W Revolution85+ has two stock.

The result!

I recently ordered two Noctua NH-U9DO A3 G34 compatible CPU coolers from Team 37412's sponsor to change out the two smaller coolers. Initially they were pretty buzzy but with the AMD fans on them they've quieted down.

So far at 100% folding load (6900 BigAdv) units the hottest CPUs are running in the low 53 - 54 celsius range and the larger SuperMicro SNK-P0043P equipped CPUs are only at 39 celsius.

I then installed Windows 2008 R2 x64 from my Technet sub using the IPMI by mapping the Windows ISO from my main PC as a "virtual DVD". This took about 40 minutes from start to finish over a gigabit connection, and I then proceeded to install and run Folding@home with the BigAdv switch.

Power Usage:

540 - 545 watts for the whole shebang at 100% load. This is around the same as my SR-2 @ 3.9Ghz with dual x5650s and lowly ATI Radeon 3670 GPU (no GPU folding).

Folding Performance:

With SMP A3 unit 6024 I was able to get a time-per-frame (TPF) of between 57 and 58 seconds. This results in around 100K PPD.

With BigAdv unit P6900 this box is ripping through it at a TPF of 10:51 and a PPD of 172K PPD!

Both times are almost exactly HALF of what I was getting with the dual, stock-clocked 1.9Ghz Opterons so if the question is asked, YES, folding@home SMP units seem to scale almost linearly in terms of doubling cores from 24 to 48.

If this trend holds up, I should be getting 2684 units to fold in under a day. The twin 6168s with 24 1.9Ghz cores can do a TPF of 27:50 (average) with 2684s and if I halve that, I will get a TPF of around 14 minutes. A TPF of 14:24 (average) on a 2684 will get one done in a 24 hour period.

The Opterons seem to really outstrip the Xeons/Core I7s at 2684s. Other units seem to be close. I have a benchmark that can simulate a 2686 which I will try later and post more numbers for that.

Even my SR-2 at 3.9Ghz and 24 threads can only do a TPF of 16:50 on 2684 units, meaning an average of 28 hours per unit.

If this baby can pull 14 minute TPFs on 2684, it will result in a PPD of over 115K !

So why did I do this?

1) I was really curious how folding@home scales.
2) I wanted a multi-CPU box "for the ages"! I plan to keep this box for a number of years and I really wanted it to kick some rear-end.
3) General hardware curiosity. I was able to get those two extra "Black Edition" Opteron 6168s for such a great price that I wanted to put them to good use.

As the build progresses with a case, and as more folding@home unit types get crunched, I will update this thread.

Feel free to ask any question you have! I will check back regularly on a daily basis to answer anything I can.

Thanks for reading! Stay Hard Canucks!

PS. This box, plus my SR-2 and associated other components will allow us to have close to 500,000 PPD extra for next year's chimp challenge.

Go Canada

__________________Xeven: How about 10^8.450980400142567e-001 -as a possible replacement for "10e"

Thanks for posting here. This is amazing... To bad theses opterons are so expensives vs the xeon 5639 or othe Intel of the same type.

New vs. New the Opterons are nearly $300.00 cheaper each vs. Xeon x5650s, and the second set of "Black Edition" processors cost me about the same as the 5639 units you guys got from Ebay.

I got my SR-2 from Newegg earlier this year for $599.99 plus $20.00 shipping and the Xeons cost me about $900.00 each before taxes. I had 6 2GB DIMMs already, and the Lian Li PC-P80 case was transferred from the original KGPE-D16 dual 6168 setup to the SR-2.

1 050 $ Shipped for two CPU
600 $ SR-2
200 $ = Ram 12 Gigs ( I should have gone for 6 gigs in retrospect)
60 $ WDD 640G black (I already bought it for prior Build - backup)
I added some GTX480's but this is for my 3 LCD gaming, so cost should not matter here.
Lets put a 50 $ value on some basic GPU
250 $ PSU, I got the AX 1200W, but a lower graded one Like any 850 W would have been enough.

total : 2 200 $ Taxe in, shipping in.

Your quad socket cool factor is no match, but I think the Sr-2 is better for now, because of overclocking. If the opteron do go down in price, then the equation differ. you also got a great deal on the ''black edition'' Opteron

I did also promise myself no more upgrade before LGA2011 and/or Buldozer. No more GPU buys before the next generation. All this should take me to Q3 2011. Hope I can supress the upgrade bug till then. Anyway, I dont see any single CPU solution beeting a Sr-2 before that time. (I hope !)