The replacement boards came in from Shuttle, and they’ll be going back into the chassis. But as you may have seen at the end of the last post, I discovered the Intel Next Unit of Computing server line. The NUC line current includes three models.

All three have three USB 2.0 ports outside (one front, two rear), as well as two USB headers inside, conceivably useful for a USB flash drive or flash reader. They also have an mSATA-capable Mini-PCIe slot as well as a short mini-PCIe slot suitable for a WiFi/Bluetooth card. And there are two DDR3 SODIMM slots, supporting a reported 16GB of RAM (the processor supports 32GB, but the board/kit do not mention this). They all include VT-x with EPT.

I don’t see the Celeron being that useful for virtualization labs, but these are rather multifunctional for a little 4″ square computer. Imagine putting a broadband modem (3G/4G/Wimax) inside for reasonably potent portable kiosk purposes (VESA mount kit included). A card reader and a DVD burner for saving and sharing (and even editing) photos. Intel’s WiDi wireless display technology is supported as well, if you have a suitable receiver. Or use it with a portable projector for presentations on the go (no more fiddling with display adapters for presentations at your meetings!).

But we’re talking about a VMware lab here.

And let me get this out of the way… this was one of the coolest features of the NUC.

That’s correct, the box has its own sound effects.

Let’s get this party started…

Those of you of my era and inclinations may remember when KITT’s brain was removed and placed in a non-vehicle form factor on the original Knight Rider tv series. When I got ready to RMA my Shuttle motherboards, I was thinking about this sort of effect for a couple of VMs on the in-service ESXi server that was about to have its board sent to southern California. And that’s what I had to do. I couldn’t quite miniaturize the server Orac-style, but that thought had crossed my mind as well.

I backed up the VMs from the original system using vSphere Client, so that I could re-deploy them later to the NUC. Someday I’ll get Veeam or something better going to actively back up and replicate my VMs, but for the limited persistent use of my cluster (cacti and mediawiki VMs) this was sufficient.

One gotcha: Fixing the NUC network…

I originally tried reusing the 4GB usb drive my existing server was booting from, but it didn’t recognize the Ethernet interface. I installed a fresh 5.0u2 on a new flash drive, and still no luck. I found a post at tekhead.org that detailed slipstreaming the new driver into ESXi’s install ISO. I did so, installed again, and was up and running.

I did have to create a new datastore on the mSATA card — my original server had used a small Vertex 2 SSD from OCZ, which obviously wouldn’t work here. But I was able to upload my backed up OVF files and bring up the VMs very quickly.

And one warning I’ll bring up is that the unit does get a bit warm, and if you use a metal USB flash drive, it will get hot to the touch. My original ESXi lab box used a plastic-shelled USB drive, and I’m inclined to go back to that.

What’s next, Robert?

My next step is going to be bringing central storage back. There is a new HP MicroServer N54L on the market, but my N40L should be sufficient for now–put the 16GB upgrade in and load it up with drives. As those of you who saw my lab post last year know, it was running FreeNAS 8, but I’m thinking about cutting over to NexentaStor Community Edition.

I’ve taken the original Shuttle box and replaced a mid-tower PC with it for my primary desktop. I will probably set the other one up with a Linux of some sort.

And in a week or so I’ll grab a second NUC and build it out as a second cluster machine for the ESXi lab. All five of them are slated to go into my new EXPEDIT shelving thingie in the home office, and I’ll bring you the latest on these adventures as soon as they happen.

About a year ago I bought a homebuilt Intel Core i7 (1st generation) desktop from a friend to run VMware ESXi on. He had gone to the trouble of assembling the system with a beautiful Gigabyte motherboard, and getting 4.1 to run on it, and I got a good deal on the system with 6GB of RAM and a 2TB hard drive.

I upgraded to 12GB, then to 24GB, but never put it into use.

Two months ago, I started it up and ran some computationally intensive software on it and discovered it was munching 320W. And it’s a mid-tower size case. Somewhat unwieldy for an apartment with a few other computers already running, and a significant other who doesn’t appreciate a living room that resembles a small colo.

It gets… smaller…

About that time, I think it was Kendrick Coleman who mentioned a new Shuttle barebones XPC system, the SH67H3, that in typical XPC form factor supported a second generation i7 processor and 32GB of RAM. Four slots of DDR3. Problem was threefold.

1) Shuttle on the VMware HCL? Unlikely.

1a) Onboard LAN and SATA controllers supported? Almost as unlikely.

2) 8GB DIMMs were expensive. And how could I in clear conscience run a system capable of 32GB with just 16GB of RAM?

3) Have you seen my holiday credit card bill?

So I was willing to risk 1a, live with 1 (as I’m not buying support or expecting it), and wait out 2 until memory prices came down.

Once 3 was resolved, I emptied my wallet into the cash register at Central Computers and bought the SH67H3 barebones XPC, and an i7-2600s (low power) processor. I had a pair of 2GB DDR3 DIMMs to use until I could upgrade, so I went about installing. I hung a SATA DVD drive off the system and installed ESXi 5 to the flash drive, and all went well.

Well, not quite.

Turned out one of the two DIMMs was bad, keeping the Shuttle from taking off, so to speak. Brief monitor sync and then it went out of sync, no beeps, no signs. I tried one DIMM, it worked; tried the other, it didn’t. Swapping the DIMM slots didn’t help. So I booted with one DIMM, 2GB, the minimum to run the ESXi installer.

No dice.

Turns out system reserved memory and/or shared video RAM managed to pull me under 2GB, and the installer quit on me.

So I realized I had 6 4GB DIMMs in the old VMware box, and I pulled two to get the Shuttle system going. Bueno. Just short of 10GB and it installed pretty well. The Shuttle disk and network were supported under ESXI 5.0.0 without any additional effort.

It got… better…

By the time this happened, I found some 8GB DDR3 DIMMs on Amazon from Komputerbay. These were not on the Shuttle compatibility list, but they were less than half the price, so I took a fortuitous risk. I’ve bought memory from them before, for the last ESXi server I built (at my last job), so I was willing to try out a pair. The memory was $58/stick, and I paid $10 for expedited shipping (twice, as I bought two pairs separately just in case). They worked fine, survived memtest86+, and made me happy.

I added a 4GB Onyx flash drive from Maxell, a very low profile drive that hides on the back of the system, to install the hypervisor onto. (Picture shows it in an extension USB pod, to show how much it sticks out. It actually fits in a regular soda bottle cap.)

For disk storage, I put a four-drive SATA enclosure in the 5.25″ half-height bay, and occupied the two SATA3 and the two SATA2 ports on the motherboard. The first bay got a 50GB SATA2 SSD I had on hand, for the initial datastore, and the second has a 500GB 7200RPM SATA disk.

I’m almost embarrassed to admit that the first VM I built on this system was Windows 7 Professional, but it was. And it worked pretty well.

Then the little one spilled a handful of change behind an electric plug and blew up the circuit breaker, while I was away from home… so it’s been on hold for a little while.

What’s in the box?

I bought the following new:

Shuttle SH67H3 barebones ($240 at Amazon)

Intel Core i7-2600S processor, retail box ($300)

4x Komputerbay 8GB DDR3 RAM ($53 per stick, $212 total)

Four-drive 2.5″ SATA cage ($71)

Intel PCIe x1 Gigabit Ethernet adapter ($40)

The following came from stock.

4GB Maxell Onyx flash drive ($9)

50GB OCZ Vertex 2 SSD ($126, much more when I bought it)

500GB 7200RPM SATA drive ($120 today, much less when I bought it)

So to build the whole mess today, I’d pay about $1,118 plus tax and sometimes shipping.

What’s next, Rob?

Well, I’m going to be a bit limited by 4 2.5″ drive bays, although I will probably put some more drives in there. I have some 32GB SSDs that are gathering dust, and a couple of 500GB disks, so we’ll see how that goes. The Patriot Pyro SSDs are coming down in price (after rebate) at the local Fry’s store, so maybe I’ll make use of the SATA3 channels.

But for now, my next step is going to be a home NAS (that I threatened to do a while back) starting from an HP N40L Microserver. The Microserver, and its 8GB of DDR3 ECC RAM, came in last month. FreeNAS 8 is currently running on this system, with an internal USB flash drive, although I’m tempted by OpenFiler’s ability to serve as a fibre channel target.

I will probably put the 8GB of RAM back into the mid-tower VMware box and use it as a second node, put some multiport cards into both ESXi servers, and power up a Summit 400-48T switch for the backbone of my virtualization network. I’m still watching for absurdly affordable PCIe 10GB Ethernet cards (since my Summit 400 has two 10GBE ports), but all I have for now is PCI-X, and only one of the three involved machines has even PCI.

I also now have a second location for lab equipment, as you may see in my write-up of the new store I’m starting. So the old desktop, and probably a Fibre Channel-enabled OpenFiler on a small SAN, will go over there. I can replicate across a 20ms latency link once, and have a pretty valid test environment for anything I’m likely to do.

Random thoughts

The LAN (RTL 8111E Gigabit Ethernet), and SATA onboard on the SH67H3 were supported out of the box, no oem.tgz tweaking needed. I had an Apricorn PCIe SSD/SATA3 controller that I plugged in with the SSD, but it wasn’t recognized by ESXi, so I went forward with the drive bay options.

I haven’t tried the SATA RAID on this system. I wouldn’t expect it to be supported, and I’d be inclined to use FreeNAS or OpenFiler or NexentaStor Community Edition to handle resilience, rather than the onboard RAID. If I get a chance, I’ll configure a pair of disks under the onboard RAID just to see how it works, or if it works. But it’s not a long-term prospect for this project.

Other people doing similar stuff

My friend Chris Wahl just put together his home whitebox systems. He went a bit more serverward, and he’s going with a pre-constructed NAS from Synology (which was tempting for me).

Kendrick Coleman wrote about his “Green Machines” project for his lab, and has built out a bit more (and put a bit more detail into his shipping list).

Simon Gallagher of vinf.net fame is well known in Europe for his vTARDIS projects, virtualizing virtualization inside virtual machines. Or as Nyssa said in Castrovalva, it’s when procedures fold back on themselves. I was reading about this, and doing a little bit of it on a quad core desktop at my last job, so I think he gets credit for my thinking about this scale of virtualzation in the first place.

Share this:

Like this:

Welcome to RSTS11…

I’m a 17 year veteran of Silicon Valley/Bay Area system administration, now retired from doing Real Work(tm). I’ve done networks, storage, IT, operations, caffeine procurement, and just about anything else that plugs in or acts like it. I’ve worked in 149-person and 149,000-person companies.

Today I work for Cisco designing solutions and telling stories around big data and analytics. See the links above for disclosures and caveats to my coverage here.

My thoughts here are my own, and should not be taken to represent any company or entity other than me.