Read on…

In Production…..

I have introduced these machines to the production environment, they run great… I had to add a 10Gb SFP+ Ethernet card to the XServe, and now I can NetBoot all 160 machines at the same time.. It works so well that all of the machines are back up and running within 45 seconds…. That’s fast for a NetBoot.. Here are some final pics of the cabinet.

Time Goes By………..

Well, here we are, 2 years have passed and these 2 racks (yup there are 2 of them now) have been running non-stop at ~97% utilization. I have had 10 hosts fail, but these failures have all been related to SSD or RAM, and were quickly fixed. Last year, these racks endured an 40 mile move to our new datacenter and continue to run fine. Over the past year I did have to replace the front fans on the first rack, but the second rack is running without issue.

Not a silly question at all.. These machines are one small part of a farm of ~2000 machines consisting of Windows, Mac and linux hosts running on some very powerful systems. These systems run 24/7 in our Build and Test environment constantly building and testing the software we produce.

This is very impressive. I’m just curious as to why there are so few pictures included.
And the cooling difference… that’s really impressive.
I’m far from apple, all of this is basically because apple doesn’t allow to run their OS on other HW?

for as much as apple appears care about their consumer products, they sure seem to not care about enterprise datacenter scalablity…. Don’t get me wrong, you daisy chaining a bunch of mac mini’s is an innovative on-off solution, and you did a great job. Unforuntatly not mainstream solution, which is what is needed if they want to be as dominant in the datacenter as they are with consumer electronics.

The mini’s will work fine alone, but in our environment we require that each time a machine is rebooted it returns with a fresh copy of our internal OS X build. This requirement reduces the occurrences of sporadic errors in out Build and Test environment. The XServe is there to provide NetBoot services to the 160 Clients.

Hmmm… With those requirements, why not use Apple Hardware running Hyper-V with RemoteFX enabled in order to virtualize GPU resources running virtualized OSX instances on top? That way, OSX gets virtualized while still retaining GPU access for OpenGL as you require.

The only thing though is that I do not know if HyperV would support running OSX/Server but maybe you guys could pull it off.

I really appreciate you taking the time to offer help, I looked into virtualization, but our testing environment has the power to max out CPU and RAM on each machine so it is in our best interest to have actual physical hardware under each copy of the OS.

I have experimented with automotive fans for cooling in racks as well, but was never able to get more than 6months out of one before they seized up. Being brush fans im concerned you may have a similar problem; unless you found brushless models? I would also make sure the fans have some sort of inline fuse between power supply. I burnt through two power supplies before I wised up to that. They sure move some mean air. Neat setup!

hi steve, this sounds like a very interesting project. it caught my eye on the endgadet website.. We are a Dubai based manufacturing company that specializes in design & manufacturing of custom made server racks. id like to get in touch and send you some more details. im sure we can work something out that meets your requirements. Please get back to me if your interested.

I think this is really cool and inventive but the price must be between $80k – $100k, is there no mainstream solution that’s comparable? We have an IBM BladeCenter with HX5 blades that have Dual 8-core processors, hyperthreaded that’s 32 Logical processors, with 256GB of ram and VMware vSphere they run approx 30 vms each for Dev and Test environments. New technology is backwards compatible as it emerges and we’ve rotated the blades out to newer powerful units every year.

What would hold me back from your solution is the availability of replacement units and maintenance contracts. Apple seems to change their Mac minis every 6 months can you simple replace with the newest model or do you have to keep replacement stock onhand?

I applaud your company for having faith in you and taking the risk to develop this project.

I used the Small Tree 10Gb Ethernet card with SFP+ The server is also equipped with 3 Intel 320 Series 300GB SSD’s in a triple mirror and 36GB of RAM. I can reboot all 160 Mac Minis at the exact same time and they all come back up in well under a minute. The XServe isn’t even breathing hard, CPU is under 20% and the SSD’s aren’t even accessed once the NetBoot image is in RAM, and the machine only uses 4 of the 10Gb Ethernet link. I’m going to add the Small tree card to the 3 other NetBooted racks I have.

It would be awesome to get everything mechanical related thats left, (eg. electric fans), out of the picture and you would have probably made the first timeless machines of all times. Keep up the awesome work. Hope to have mine soon!

Really cool design and idea, my only question/comment is how will you deal with building either more or replacing your current xserve server? Since even in such a impressive and complex system you still need a xserve, which I sadly regret Apple retiring. Besides that though, super cool. I honestly love this idea, and think it’s cool your company was willing to drop a dime on 160 mac minis, my issue is less with how you made it possible more so how Apple gave you only one way to do it. But kudos on coming up with a solution, I only wish Apple would work more with companies to deliver business hardware on a workable scale.

What’s your process for getting new systems from their out-of-the-box config (where they want to boot from the factory installed OS on the internal disk) to NetBooting? Presumably, you’re not hooking up a keyboard to each and holding down N. :)

Unfortunately, we are doing just that, only instead of the “n” key we hold the “option” key and choose our custom installer, once our custom image is on the machine, we choose our “Build and Test” NetBoot image, and off they go! Once they are in the Build and test Pool, we have custom scripting that allows us to change the image they NetBoot between test cycles, That scripting is built around the “bless” command. I hope to have some time to write up the entire process on this blog, and I may include some of my notes from the last time I spoke at the Splunk conference.

Hi Steve, you have inspired me to build a few of these racks for internal purposes once the 2014 mini refresh is out in a few months. Q? Source of the racks and plastic heat redirectors to order them and any special instructions I should tell them? Also did you put a power strip and gigabit switch in each rack so 2 connections out the back or insufficient room so 8 connections out the back?