Oracle Blog

A thin thinker down under

Tuesday Sep 23, 2008

After receiving the mandatory "Waiver for Internal Funding for Equipment" (WIFE) approval, I received a few days ago my new toy, a super-duper Sun Ultra 24 workstation, possibly the last PC I may ever buy (but then again, wasn't 640K enough for all needs?) It's a welcome addition to finally replace my 6 and a half year old Frankenstein PC (AMD Athlon XP1800+) which served faithfully over the years.

Careful thought went into this purchase:
- it had to be as powerful as possible without breaking the budget
- it had to be super quiet, as it will be on most of the time in a public part of my house
- it had to to have room for heaps of memory and disk
- it needed to be capable of running pretty much any O/S, but above all, VWare ESX, and as it turns out, just when I was about to place the order, VMware saved me some money by making ESXi free.

In the end, I chose the Ultra 24 because:
- it's VERY well put together
- it met all the requirements above
- it was competitive with anything out there in the market, and better than most in terms of memory capacity (8GB)
- and well, staff discount... So I added the obligatory 22" screen (it is \*fantastic\*) and a Sun Ray 2 to accompany my already existing batallion of Sun Rays.

Anyway, this blog entry is about the steps to get ESXi to run on it, so here we go.

START SAFE: LOAD THE LATEST FIRMWARE FOR THE ULTRA 24
For completeness, I updated the system to the latest firmware, downloaded from here:
www.sun.com/desktop/workstation/ultra24/downloads.jsp

FIRST CHALLENGE: SATA DISKS ARE NOT SUPPORTED BY ESXi (or are they?)
I tried SATA once on an Ultra 20 and got quite far: I even built a VM, only to find that the VI client would not see it even though it existed on the VMFS volume. Go figure. One day, when I get a spare drive, I'll try SATA on its own. For now, read on...

As with any other "workstation" in the market, the Ultra 24 can house SATA and SAS disks, but that doesn't mean that the internal controller can handle SAS disks, so buying these disks was not an option (and I couldn't afford them anyway), so to get there for now, I bought 4 x 250GB SATA disks to go in. With every major vendor box, you find that although the disks themselves are a commodity, the brackets that hold them in aren't, so I went for the easy way out, cheap for now with all the brackets in place for the future, so that an upgrade to bigger disks can be had anytime.

The next thing to solve was getting ESXi to be happy with these drives. As it turns out, I had earlier experimented successfully with a Sun branded LSI card that's fully supported by VMware ESX (Sun part # SG-XPCIE4SAS3-Z), which in more generic terms is a "4-Port SAS PCI-E HBA LSI3041E with B3 silicon" - although I'm yet to understand (or care) what "B3 silicon" means.... We used these cards to run a Sun VDI bootcamp training course on a raft of Sun Ultra 20 systems with 4GB RAM.

So, I plugged in the card into one of the PCI-E slots of the Ultra 24, and lo and behold, the next reboot the system saw the card and the LSI adapter framework which would later allow me to configure RAID options with my disks.

SECOND CHALLENGE: HOW TO BETTER RAID THE DISKS FOR MY MULTIPLE USES?
So now I have four disks and all sorts of possible configurations... But at the end of the day, for now my needs are defined: I want to run mostly ESXi as safely as possible, so that I can leave a number of VMs running to host some of my lab stuff with confidence (Sun Ray server on Solaris/CentOS/SLES,SGD, Windows VMs, LDAP server, etc) but allow some room to run Windows natively for other very important things like, well, Team Fortress 2, and when the time is ripe, play with Sun xVM.

So the choice was given to me when I realised that the LSI RAID card would let me create a mirror across 3 disks... Cool - 3 disks as a mirrored set of 375GB and an independant disk to multiboot from other operating systems, Windows XP and Solaris xVM at the very least. And yes, I tested the mirror - I pulled out a drive, and the LSI framework reported it was working in degraded mode until I plugged the drive back in (after a shutdown of course). This type of mirroring may or may not be as fast as a standard two way (think of the implications and the design of such a mirror) but that doesn't bother me too much just yet.

THIRD CHALLENGE: THE INSTALLATION ITSELF (NO PROBS HERE... BUT)
After this all worked, I simply loaded the ESXi 3.5 u2 CD on the DVD tray, booted and installed happily, UNTIL... the initial ESXi screen came back displaying that the IP address was 0.0.0.0.

FOURTH CHALLENGE: THE NETWORK
0.0.0.0... Huh? So after a bit of digging I found out that VMware doesn't particularly like the onboard Gigabit Ethernet NIC, which although being an onboard Intel Gigabit, the particular chipset (82566DM-2) is not in the support list "yet" (see http://www.grid.org/blog/cameron/updating-vmware-esxi-disk-dump-file among others) and so, since I wanted instant satisfaction by now and had found the potential workaround too hard to implement, I decided that my time would be better spent by investing $47 in a stock standard, ESX supported, Intel 1000/pro Gigabit card, which worked on the next reboot.

FIFTH CHALLENGE: BOOT EASILY FROM ONE DRIVE/LUN OR THE OTHER
As it turns out, the U24 does not let you boot from different Hard Disks/LUNs. You can only specify ONE preferred hard disk in the BIOS, and the boot menu shows only this one, along with whatever other options your system may have (CD, ETH, USB). After a bit of toing and froing, I found that booting into the BIOS everytime I wanted to change boot disks was VERY painful, whenever the time came for Team Fortress 2. The only option presented to me... USB boot!

SIXTH CHALLENGE: BOOT ESXi FROM A USB DRIVE
What do you know, there's an internal USB port within the Ultra 24. How useful's that? I had always wanted to give this a try, and now the time was ripe. Googling for instructions I found the more official ones at http://communities.vmware.com/message/825686, but there are plenty of repeat intructions out there. Of course, for a Unix/Mac guy, winimage is not a requirement and a simple "dd" command did the trick quite nicely, although you do seem to need a 2GB drive or \*smaller\*. So, now, I've become a USB boot fan, because that shows up in the boot menu! As a side note, booting ESXi off USB is not particularly fast, but it does make the setup a lot more portable and convenient. At the end of the day, you can easily import the VMs that are already residing on the VMFS to any ESX bootable environment, so I can choose to boot from disk or USB and still have my VMs.

SEVENTH CHALLENGE: LIMITATIONS OF ESXi
Almost finished here. I just wanted to document the downside of ESXi in my particular setup:
1) No access to the CDROM. You can't map a CD to a VM, which is a bit disappointing. Very little documentation on this but it seems to have to do with the fact that the CDROM on the Ultra 24 is IDE...
2) No USB mapping. That's a sad one, but I knew in advance, so I'm not too fussed.

By now, it's a happy ending - I have built a few VMs, got my Sun VDI server now running within, my XFX 9800GT plays TF2 like a charm and the machine has been rock solid. Good to go til 2014 I hope!!!