Home labs – a scalable vSphere whitebox

Having recently upgraded my home lab’s storage I decided it was also time to upgrade my aging hosts which date back to 2007. They’ve done well to survive and still be useful(ish) five years later but they’re maxed out at 8GB RAM and it’s becoming increasingly difficult to do anything with that. I briefly considered adding SSDs as host cache but that doesn’t address some of their other shortcomings such as no support for Fault Tolerance, VMDirectPath or any type of KVM functionality.

More power!

The problem for me was that these solutions all maxed out at 16 or 32GB RAM per host, a limitation of the single socket Xeon’s architecture. That’s a lot of memory for a home lab server today but to ensure that this server can last five years I really wanted more scalability. I wasn’t too fussed about noise as I use my cellar for my lab, and power consumption was a secondary concern. The server features of the Supermicro boards appeal to me (and many Supermicro motherboards are compatible with vSphere) so I browsed their range looking for the one that best met my requirements. My final parts list ended up as;

The total cost comes to around £1150. I’m branding mine the vHydra after the mythical multi-headed dragon!Note: In the US this is significantly cheaper, coming in at $1450, or about £900.

For the money I get a powerful server that can replace all three of my current 8GB hosts and more than match their performance while consuming less power and space, plus compatibility with all VMware features including FT (and full EPT for nested ESXi). With two onboard storage controllers, support for SAS drives, and VMDirectPath I can run a Nexenta VM to supplement my shared storage plus I gain remote KVM and the option of future memory upgrades (up to a max of 256GB RAM!). If you want to save some cash you can always buy a single CPU (and fewer DIMMs) and add the second later if your requirements increase.NOTE: FT is supported on the Xeon E5-26xx CPUs but at the time of writing VMware’s SiteSurvey tool hasn’t been updated so will erroneously report it’s not supported. Given that the web client is the way forward it’s quite possible SiteSurvey won’t be updated in its current VI client format.

It might sound crazy expensive but it’s less than an extra level MacBookPro with anti-glare screen which makes it sound more reasonable (to me at least). If it helps my career over four years it seems like a good investment. Compared to a single Baby Dragon II it’s more expensive and larger (ATX vs micro-ATX) but if you’re considering two Baby Dragons it’s a closer call – it costs slightly less, has less compute (no hyperthreading and a lower core MHz) but more memory scalability (although you’ll need RDIMMs), SAS support, and a slightly smaller footprint. With two Baby Dragons however you have more flexibility as you have two physical instances of ESXi – no nesting and a ‘proper’ cluster is quite compelling. Two servers is also better for experimenting with other technologies (hypervisors which may not work nested) etc but I went with my ‘vHydra’ because I had a case, power supply, and some RDIMM’s which I could reuse (which the Baby Dragon wouldn’t accept) – your circumstances may vary.

Tip: It’s worth bearing in mind the 32GB limit on the free version of ESXi – unless you’re a vExpert or they reinstate the VMTN subscription you’ll be stuck with 60 day eval editions if you go above 32GB (or buying a licence!). For a full discussion of scale up vs scale out I’d recommend the VCAP-DCD Design Workshop. Having considered scale up vs out for my lab I think I’ve got enough thoughts for a future blogpost…

Baby dragon II

The vHydra

Cost

£635

£1150

Compute

4 cores @ 3.2GHz
(plus hyperthreading)

8 cores @ 1.8GHz
(no hyperthreading)

Memory

16GB
(Max 32GB)

64GB
(Max 256GB using RDIMMs
Max 64GB using UDIMMs)

Storage

2 SATA3
4 SATA2

2 SATA3
4 SATA2

Network

One 82574L
One 82579LM (custom driver)

Two 82574L

Motherboard

Micro-ATX
9.6" x 9.6"

ATX
12" x 10"

Case size

279mm x 262mm x 373mm
(W x D x H)

210mm x 381mm x 490mm
(W x D x H)

Noise

Power

80W

190W (see below for update)

Motherboard and CPU choices

The X9DRL-3F motherboard is relatively new having been launched in March this year. Being dual socket it requires Xeon E5-26xx series processors (tech details here, reviewed here, with some performance figures via El Reg) which are also new having been launched in May this year. The Xeon E5 CPU range, like the Xeon 5xxx range previously, is designed for high end server and workstations and crucially for me supports much higher memory density;

The cheapest is the E5-2603 (£157 at Lambdatek) with a clock speed of 1.8GHz and four cores, although no hyperthreading. The max memory speed is 1066Mhz whereas the motherboard will support 1333MHz.

Next up is the E5-2609 (£230 at Lambdatek) with a clock speed of 2.4GHz, four cores, and again no hyperthreading and a max memory speed of 1066MHz.

Another option is the E5-2620 (£305 at Lambdatek) with a clock speed of 2.0GHz and six cores plus hyperthreading, giving 12 threads for VMs plus it supports 1333MHz memory.

The Xeon E5-26xx range is comprehensive so if you need more compute power you can easily scale up if you have the funds.

Tip: You’ll need to order a heatsink separately as it doesn’t come with the CPU even if you buy the retail version. Details of the Intel heatsinks are in the above review link and the Supermicro ones can be found in this PDF (I went with the SNK-P0048AP4 which is fine for a standard ATX case). Be careful if you buy a different motherboard/CPU combination as some Supermicro boards use an uncommon ‘narrow ILM’ socket fixing. This board uses the more common square ILM. The two CPU sockets are quite close together so some of the aftermarkets heatsinks might not fit.

The motherboard has 8 DIMM slots but you need both CPUs to use all eight slots – with a single CPU the most you can use is four. It’s flexible as it’ll accept both buffered/registered and unbuffered RAM, although you’ll need to use registered memory if you want to go above 64GB.

Tip: In the UK most resellers don’t hold stock of the Supermicro motherboards and there’s a two week wait while they’re ordered from the US. Frustrating!

Power supply choices

Being a dual socket board you’ll need a bit more power compared to a desktop. You need a PSU that’s compliant with ATX 2.02 and SSI standards (often referred to as EPS12V) which means it’ll have the usual 24 pin connector and at least one 8 pin (or a 4+4pin) CPU connector. This motherboard needs a PSU with twoeight pin power connectors which is less common – I had two EPS12V certified PSUs and neither had enough. I went for the 600W Silverstone Strider Plus as it’s their lowest spec PSU which offers dual EPS12V connectors and it gets goodreviews. If you go for a different PSU make sure it’s not too deep – one of my PSUs was 180mm (vs 160mm for the Silverstone) and it wouldn’t fit the Lian-Li case without hitting the first CPUs RAM modules. I used Thermaltake’s PSU calculator to size my PSU correctly.

UPDATE – 26th Nov 2012 – I’ve been monitoring power usage and my vHydra idles (with a few VMs that do nearly nothing) at around 85W. With electricity prices around 12.8p/kWh that’s approx 25p/day, or £1.75/wk, or £91/yr for this one server so factor that into your lab costs.

Hi Edward,
It’s a great home lab. I would like have an home lab like your, but is some expensive for my needs. So my home lab is an laptop HP dm4 Intel core i3 with 12GB RAM.
I’ve installed StarWind iSCSI emulator and using VMware workstation 9. I’ve created two virtual ESXi in the workstation.
All run fine!, so I’m happy.

Hey Ed, its been a while. How is this build treating you these days? This article has me thinking a bit differently on how I would build out my next lab. Now thinking about a single box loaded with storage inside and carved out virtually.

Hey Sean, good to hear from you. Unfortunately I’ve been out of the office on paternity leave for the bulk of the year so haven’t had time for any work, even in my lab at home. This means my vHydra has been sitting powered off since mid Jan – a huge waste of potential! I’m hoping to power it back up soon and get stuck back in to learning (vCloud.next and Puppet are high on my list) but kids sure do soak up spare time. I was indeed running nested ESXi previously though.

Hey Ed, congrats on the new addition to your family. I hear you there, kids will definetely take up your time. Thanks for the info on the build. I am very interested in going the all-in-one route and just carving it all out virtually.

Sadly no updates Nick. I did power it on last weekend as I needed to get some data off an old VM, but I’ve not had time to invest in setting up new stuff. I’m itching to do so and there’s plenty to play with (VSAN, vFRC, Puppet integration, PernixData, the list goes on and on) but I need to quit my job to have enough time!

A couple of people have posted recently about a newer Supermicro motherboard which comes with 10GB NICs among a few other improvements – maybe that would be a better choice? Have a look at Eric Bussink’s blog (http://www.bussink.ch/?p=1450).

Site Administration

Disclaimer

These rants and raves are solely my opinion and do not reflect the opinions of my employers.
Any of my code, configuration references, or suggestions should be researched and verified in a lab environment before attempting in a production environment.
Agreement to use any of my code or recommendations removes me from any liability as such....and I shamelessly stole this disclaimer from Jase McCarty's site!