HomeLab Server Guide

I configured a purpose-built server for hosting virtual machines in a home virtualization lab. This page explains the build and some things I learned in the process.

A competent IT professional needs a way to safely experiment and prototype. That requires some appropriate hardware to use for a virtual-machine hosting platform.

Knowing this, I recently did a lot of tedious research and plenty of intensive contemplation about HomeLab servers and what a proper virtual-machine environment for hands-on experimentation requires. The goal was build a good server that has as many desirable features as possible but doesn't cost any more than necessary.

Happily, the diligent effort paid off. For a very reasonable price you can build an efficient and capable HomeLab Server that will provide years of dependable VM-hosting service.

Traits of a Server

This Page's Outline

Here are a few attributes of a well-built HomeLab server. This section will explain why workstation-grade hardware is far superior to conventional PC hardware when you need a server but don't want a noisy and cumbersome rackmount system in your house.

Software Compatibility

Choose components with a strong preference toward compatibility. VMWare ESXi, FreeBSD, Linux, Windows, and probably even Mac OS will run well on this machine.

High Availability

Workstations and servers are more durable than PCs. They're designed so they can operate under a continuous day-and-night workload. Durability is especially important for depreciated (bargain-priced) off-lease computers that are out of warranty.

Expandability

More expansion slots, more memory slots, and dual-CPU capability all add value to your server. High maximum memory capacity is especially important.

ECC RAM

ECC (Error-Correction Code) memory is beneficial because it provides data integrity. Enterprise servers have ECC memory because data integrity is more important for servers than for PCs. If a single memory bit erroneously flips in your office-suite document the consequences are not likely to be critical. On the other hand, if a bit flips when the computer is writing a file to your encrypted storage volume the consequences could be more significant.

CPU Features

More cores and more on-CPU cache memory add value to your server. Built-in Features like AES New Instructions are also beneficial. i-Series processors have desktop-oriented features whereas Xeon CPUs are built specifically for workstations and servers.

Energy Efficiency

Components with this often-overlooked quality enable your machine run cooler, quieter, and possibly more reliably as a result. It will also run longer on UPS battery power.

Without a doubt your server should be made from a workstation rather than a PC. There are reasons why new workstations cost so much more than new PCs.

Precision T5500 Workstation

The machine I chose is a robust and scalable Dell Precision T500 Workstation. It's a super-duty compact-and-quiet desktop workstation for professional technical and scientific applications. A machine like this opens up a vast number of possibilities for running VMs.

I first discovered the T5500 because I was searching for a used machine with at least 8GB of ECC RAM to use as a FreeNAS or NAS4Free fileserver. This machine can do that and a whole lot more (to say the least!).

Additional drives can be adapted into 5.24" drive bays and the 3.5" bay.

Expansion Features

Dual-CPU Capable (requires second-CPU Riser)

A 64-bit PCI-X (PCI Extended) slot

Plenty of PCI-E (PCI Express) slots

A PCI slot. How quaint.

Efficient (85% or 88%) 875W power supply

T5500 Lowlights:

SATA II 3.0Gb/s host controller (current is 6.0, better for SSDs)

Broadcom 5754 Gigabit Ethernet (Would have preferred Intel)

A 650 volt-amp UPS is more that adequate to provide always-on power.

CPU Upgrade

The T5500's CPU socket is a "Socket LGA 1366". It accepts 55XX-series ("Gainestown") and 56XX-series ("Westmere-EP") efficient-performance CPUs. Early T5500's came with 55XX CPUs and later models came with 56XX CPUs.

An optional second-CPU socket takes the form of a riser assembly, which also adds three additional RAM slots. The riser assembly is three parts, the riser itself (F623F), a heatsink, (W715F) and a fan (F306F).

56XX-series CPUs are more desirable. They're more energy-efficient, they have reduced latency in virtualization, and they provide AES New Instructions (AES-NI) that dramatically improve throughput for disk encryption and other cryptographic operations.

Inexpensive Xeon L5630 and L5640 CPUs are available that minimize power consumption for cooler, quieter, and less-expensive operation. X5650 and X5660 processors use more power but have more grrr.

RAM Upgrade

The main motivation to get this machine was because it supports ECC (Error-Correction Code) RAM -- and plenty of it. ECC RAM is strongly recommended for best results with the ZFS filesystem, especially if encryption is enabled.

The best way to expand the capabilities of a virtual-machine server is usually by adding RAM. You can install up to 72GB of memory in a T5500 according to Dell -- more according to the Internet.

Fortunately, even though high-grade ECC RAM was pricey when T5500s were new, high-quality name-brand sticks of compatible memory are available at very reasonable prices now. In fact, because of supply and demand, used server/workstation RAM is less expensive than used PC RAM.

Recommended name-brands: Hynix, Samsung, Micron

If you need lots of RAM, 24GB kits are available as 3x8GB kits. These higher-capacity 8GB modules cost more per GB. The premium is 70%, still not bad at under $4/GB. Filling all the slots (three 3x8 kits) in a dual-CPU system gives you the full rated capacity of 72GB of RAM.

Combining different RAM (module capacities, voltages, single-rank with dual-rank, etc.) in the same system can be tricky. The system will perform differently based on the combination of modules you use.

FWIW, I chose to remove the RAM that came with mine and install three low-voltage 8GB modules, opting for slightly better speed and better expandability. A less-expensive option (typically by about $30) would have been to install six 4GB modules. Installing three low-voltage modules rather than six standard-voltage modules also saves power and runs cooler.

The T5500 accepts "PC3L-10600R" (1.35V Registered ECC) or "PC3-10600R" (1.5V Registered ECC) modules. For best results, install double-ranked (2R) modules in sets of three, where the voltage of all installed modules matches -- either all 1.5V or all 1.35V modules.

Consult the Precision T5500 Service Manual for installing RAM sticks in the optimum manner.

Hard Drives

The second-best way to expand the capabilities of a virtual-machine server is usually by adding storage space.

The Precision T5500 has two internal 3.5-inch hard drive bays. It's possible to add more in the 5.24" drive bays. If necessary, external (eSATA) drives can be added as well.

New hard drives specifically designed for network attached storage (NAS) are recommended.

ProTip: Bad drives, when they occur, tend to leave a factory in batches. For best results, either buy drives from a known-good batch (hard to do) or get drives from different batches (easy to do).

Network Adapter

Brand

Part #

Intel

EXPI9402PT

HP

NC360T

Dell

X3959

IBM

39Y6128

Dual-Port Intel Pro/1000 PT NICs

Recent-model Intel Pro/1000 PT server network adapters are a safe bet. They're commonly re-branded and sold by server manufacturers under various model names.

Single- dual- and quad-port models are widely available. Intel's part numbers are EXPI9400PT, EXPI9402PT, and EXPI9404PT respectively. At current prices, expect to pay roughly $8 per port.

Graphics

Most T5500 machines have nVidia Quadro video cards installed. Graphics capability can easily be improved by swapping in a more-modern (5 years newer) and more energy-efficient GPU card.

For example, if your T5500 has a Quadro FX 580 card, common benchmark results predict that a well-chosen new card will provide 37% faster 2D performance while consuming half as much power at less than 19W TDP rather than 40W. DirectX support goes from 10 to 11 and OpenGL support jumps from v3.3 to v4.5. Bonus features are: 3D performance is 2.5X as fast and it has twice as much RAM.

SATA Adapter

SATA III 6.0Gb/s capability can be added in a PCI-E slot. For spinning drives this won't make much, if any, difference. For fast SSDs you're wise to go with SATA III.

Optimum HomeLab Build

This is an annotated list of the build I ultimately chose. I started with a $357 base configuration (plus hard drives), with a couple of upgrades soon to follow.

Price Summary

These prices current as of late October, 2015.

Initial Build

Component

Price

Dell T5500

$175

CPU Upgrade

28

Upgrade to 24GB

78

100GB SLC SSD

70

Quad-port NIC

30

Total

$381

Storage

For my build I already have three 1TB desktop hard drives I can use. Here are minimum prices for drives. New, (not refurbished) name-brand drives are recommended.

Lowest-price NAS drives (pairs):

Capacity

2X

3X

1TB

$74 / 1TB

$111 / 2TB

2TB

110 / 2TB

165 / 4TB

3TB

130 / 3TB

195 / 6TB

4TB

300 / 4TB

450 / 8TB

5TB

340 / 5TB

510 / 10TB

Next Upgrades

Near-future upgrade components (prices may vary, obviously):

Phase One - Add CPU and another 24GB RAM:

2nd-CPU Riser Assy.

$99

Upgrade to 48GB

78

Total

$177

Phase Two - Always-on Power, Upgrade GPU:

600VA UPS

50

Upgrade GPU

50

Upgrade to 72GB

78

Total

$178

A storage upgrade will likely intervene in there somewhere. We'll see how it goes with storage.

If more CPU power is necessary, a CPU Upgrade from dual-L56230 to dual-L5640 (currently $138 for a pair) goes from 8 to 12 cores and each core is a bit more powerful. Overall benchmark performance increases by 44%.

So let's say we install three new Hitachi 2GB drives for 4TB of available storage and do all of the upgrades.

4 drives?

Links

These are just links (meaning they're not "affiliate" links).

Features that stand out:

4 Cores (8 vCPUs @ only 5W TDP per vCPU)

2nd (matching) CPU in hand

18GB ECC RAM

Support for three 3.5" drives instead of two

4-display Support

T5500 Workstation

The machine appears to have come from the factory with three spinning hard drives installed. The third hard drive is in the lower 5.25" drive bay, below the optical drive and above the 3.5" bay. The SSD, in 3.5" a mounting kit should be able to fit into the 3.5" bay with only an additional SATA data cable because a SATA power cable is provided.

CPU Upgrade

Two Low-power 56XX Quad-core CPUs with hyperthreading and AES New Instructions:

Enterprise SSD

This enterprise/industrial-grade SLC drive is a beast. List price was nearly $1,000 when it was new in 2010. It's designed for high IOPs and harsh environments, neither of which will happen in my HomeLab server. It sips 2.2W under load in a server and .125W at idle.

It will be used as the boot drive, for swap space, for a filesystem journal (ZIL, or "ZFS Intent Log") and for a high-performance filesystem cache (L2ARC, or "Level 2 Adjustable Replacement Cache"). Planned allocation for those are: 60GB, 12GB, 8GB and 15GB respectively. 5GB will be unused. The ZIL improves performance and data-integrity. The L2ARC improves disk I/O performance.

This would theoretically speed up the SSD. Cards like these seem to use only two lanes of the PCI-Express bus. Best I can tell "PCIe 2.0 x2" limits throughput to below SATA III 6.0Gbps, but still it's faster than SATA III at 3.0 Gbps. Marvell 88SE9235 chipset appears to be a good choice. YMMV.

This card uses a Fresco Logic FL1100EX chipset, which I believe is a good one. As with hte SATA card, YMMV because it's hard to determine which chipsets are compatible with various operating systems and hypervisors.

Q & A

What's the very least expensive build you'd recommend?

An adequate single-CPU T5500 with a decent CPU (55XX will work, 56XX preferred), 12GB of RAM, two cheap 1TB drives from eBay ($74) and a 600VA UPS ($50) comes at around $300.

Do I really need a UPS?

Computers are sensitive electronic devices that tend not to tolerate blackouts, surges, and brownouts. If data integrity is important for your server a UPS is cheap insurance. At minimum you should install a good surge protector, which costs roughly half as much as the UPS does.