The Challenges

We have Apple XServe 3.1 servers here. They are equipped with a JBOD, Promise VTrek E610F. It connected via Fibre Channel. We have configured them for RAID6 28 TB.

The best part of the challenge is that XServe 3.1 use EFI. Some people would suggested us to use rEFIt and chainload it to GRUB2. We could use “bless” command from OSX.

The problem is, we have stripped the original 150 GB harddrive. The reason was because it was only had one drive without RAID. Data integrity was in question so we have to use only the JBOD.

Another problem is because PC BIOS couldn’t handle more than 2 TB. We just have to use GPT. Besides, GPT is the part of EFI.

So, I took the challenge to not use the OSX but to install directly from GNU/Linux installer from EFI to a GPT partition.

The Preparation

I got this challenge from my friend because he gave up on installing the XServe. From his previous attempt, I got 64 Bit Mac (AMD64) alternate install CD downloaded from Ubuntu. I don’t know where he got the ISO. Please download here. Not bad, I can use it to debootstrap the partitions.

To start, boot the CD and choose Expert Install. The Expert Install mode would let us select few things manually. Here’s the things that I’ve selected:

If you don’t want to have remote install, you could disable network-console. In my case, I don’t like to be in the server room and to move XServe with its JBOD is daunting. I would like to configure everything from my nice warm room office.

After configuring the network jump to the run shell option. Or, after we have configure SSH credential, we could SSH to the machine with username “installer”. And choose to start a shell. Either way, we would be presented with a little console program: BusyBox. It isn’t feature-rich like BASH, but it serves right with compact size.

On The Shell

First, we partition our drive. We are using parted instead of fdisk because the later doesn’t support GPT partition type. The Apple’s EFI specification needs a FAT32 partition to load the EFI binary. That partition should be at the range of the first 2 TB disk. It doesn’t have to big, just few megs actually would do. So, here’s my partition:

20 MB FAT32 partition at the beginning with BOOT flag on.

20 GB partition for our GNU/Linux partition.

We have lots of RAMs, so we disable swap partition. But, yours might differ. From my research, the semantic web/4th paradigm world, it seems people prefer XFS because it is optimized for big data. So, from the beginning I choose XFS. Again, yours might differ. So, here’s the two commands:

I have difficulty to format the FAT32 partition, I think the Installer CD is dying. So, I decided to format it later. Now, let’s install it via bootstrap it and copy the files that were configured earlier from the installer:

1

2

3

# debootstrap squeeze /mnt http://kambing.ui.ac.id/debian/

# cp /etc/{fstab,hostname,hosts,resolv.conf} /mnt/etc

# cp /etc/network/interfaces /mnt/etc/network/interfaces

BusyBox can’t do the array like the above. You have to parse it one by one (copy fstab, then hostname, and so on). I just put that way for convenience. Next, we mount the important filesystems and chroot to our new system. I explicitly use BASH instead of DASH, the default shell in Ubuntu.

1

2

3

4

# mount /dev /mnt/dev -o bind

# mount none /mnt/proc -t proc

# mount none /mnt/sys -t sysfs

# chroot /mnt /bin/bash

On Chrooted Shell: The Basic Packages

First thing to do is to add other repositories to “/etc/apt/sources.list”.

Time is crucial for server. Install NTP server to have our server clock right. Don’t forget to edit the editor to get our nearest NTP server.

1

2

3

# apt-get install ntp

# $EDITOR /etc/ntp.conf

# /etc/init.d/ntp restart

We can check the avaibility of the NTP servers.

1

# ntpq -p

Last thing, let’s install our Linux kernel.

1

# apt-get install linux-image-2.6-amd64 firmware-linux-nonfree

The next one is the most challenging one: making our server boot from EFI on GPT partition.

Building GRUB2 from Unstable

I have many failed attempts using the default grub-efi 1.98 from Squeeze. As I realize, the Natty installer is using 1.99 version of GRUB2. I decided to go to that route.

GRUB2 1.99 is available at unstable and must get backported to Squeeze. We must compile it ourselves. So, here’s how to.

Get all the requirements for GRUB2 in Squeeze. There is another optional package we need: libfuse-dev for building grub-mount. It can be skipped, but it’s not that bad to add few bytes on that functionality. We need also autogen because the GRUB2 source is taken from Bazaar.

1

2

# apt-get build-dep grub2

# apt-get install autogen libfuse-dev

Next, add Debian source repository for unstable/SID. Don’t add the binary, it will upgrade your system to unstable/SID.

We could build the package manually, but I prefer the Debian way. Fortunately, the showstopper of doing the Debian way is only the GCC requirement. Unstable/SID is using gcc-4.5 while Squeeze using gcc-4.4. The solution is simple, find all the occurrences and change all of them.

There are three files that affected. You could use sed/awk, or using your favorite editor to change all of the gcc-4.5 to gcc-4.4. I’m using ViM with regex. Of course, it is advised that we change the version also. I’m not doing it because we are not trying to publish it. This build has bug, yet sufficient for our need.

Okay, the next thing is to build all of the packages.

1

# dpkg-buildpackage -b -uc -d

All of the Debian packages are built on the top of our source directory. Let’s install all of the EFI versions. But wait, before you could install, the package needs to install efibootmgr, an application for setting the default drive to boot on UEFI (Universal EFI) compatible board. Unfortunately, XServe is using Apple’s own EFI implementation, so the application is useless. We must use bless, an application on OSX for that purpose. Oh, btw, install the ucf and os-probes.

1

2

3

4

# apt-get install efibootmgr ucf os-probes

# cd .. && dpkg -i grub-common_1.99-11_amd64.deb \

grub2-common_1.99-11_amd64.debgrub-efi-amd64*\

grub-efi_1.99-11_amd64.deb

Now, we are ready to get into the show.

Installing GRUB2 EFI

Before we take further, I want to say something. I am gratefully to the developer of the grub-install for making an undocumented parameter –debug. That parameter saves me and made me to see what’s going on.

Okay, now we need to prepare our partition for EFI. It should be FAT32 and mounted at “/boot/efi”. Let’s create one.

1

2

# mkfs.vfat /dev/sda1

# mkdir -p /boot/efi && mount /dev/sda1 /boot/efi

Next, we run the GRUB installer.

1

# grub-install --bootloader-id=BOOT --removable --debug

The parameter “–bootloader-id=BOOT –removable” is what I found out after I analyze “/usr/sbin/grub-install” script. Basically, I want GRUB2 installs its EFI file at “(hd0,gpt1)/EFI/BOOT/BOOTX64.EFI” (in its FAT32 partition). Some UEFI board may have other vendor such as Ubuntu, GRUB, etc. But, XServe doesn’t understand other vendor, yet. We could only use the default prefix where XServe’s EFI would search for EFI binary.

Don’t restart yet, because next is the most important one.

Fix GRUB2 EFI

You know why I used the “–debug”? It was because I kept on going to the GRUB Rescue mode with error message telling that GRUB could not found the proper disk. In the GRUB Rescue, I’ve found out that the prefix was wrong. The prefix was pointing to ” ((null),gpt2)/boot/grub”. Wew, that was wrong.

After I’ve found the problem, it seems that the “/usr/sbin/grub-install” didn’t generate the right prefix. From the debug, I’ve found the culprit:

One of system administrator at Universitas Indonesia. His current interests are in smart card technology, pervasive computing, and free/open source software. He is an evangelist of free/open source movement and using FOSS actively.

how did you get the XServe booting the Ubuntu CD? I have installed rEFIt and tried to boot with hold down service-ID button and hold down option (alt) key but it wont start.

Martin

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

Hi, Martin, I only press hold the [Option] button when the screen still gray. I still press it and the bootloader popped out one after another. From my experience, it displayed the HDD icon first then the CD icon. After that, I released [Option] button. I choose the CD icon with my mouse and it booted.

The CD have a file: “/EFI/BOOT/BOOTX64.EFI”. XServe firmware can take that and boot it. I’m doing the whole installation without rEFIt.

Man i follow your post, but i cant boot the CD, so there is any way to do that, even the fedora for MAC with EFI support, maybe the Xserve dont boot with CD, so there is some config file or something in it that i can change.

btw the ubuntu iso doesnt have efi folder so that boot with option key cant be true…

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

I’m sorry, I think you are right. My coworker didn’t download it from there. I have removed it.

Btw, I didn’t use rEFIt or anything. I was using pure EFI boot manager in XServe 3.1. I have to remove the internal disks and was only using JBOD. So, I have to wait few minutes or so while holding the “Option” key until the blank gray boot screen showing bootable disk icons one by one.

I hope these help you. Thank you for your correction.

http://aosidesigns.comze.com Isaac Alcocer

Well, sorry for late reply…first thank you i was making my own prerelease 12.04 ubuntu xD lol… i will try with that iso, but its taking sooo long so maybe in a few days come back with some news, im gonna uninstall rEFIt to doit like ur post =D

cheers

btw nice redesign

http://staff.blog.ui.ac.id/jp/ Jan Peter Alexander Rajagukguk

Thanks. Btw, if you have to make a prerelease 12.04, you might want to copy the efi directory in (not really) my ISO. It seems it would boot into grub.

Haha… take care there. I hope you could share the result. I wish to give it a try but all the machines have gone production. No machine left to test…

http://aosidesigns.comze.com Isaac Alcocer

Well i follow ur post and found a finger error =D in:

# apt-get install efibootmgr ucf os-probes

well all really good =D and some updates… the grub its now using gcc-4.6, the only 4.5 text its in just some commets xD… but i couldnt make it work, i install the grub and all but i could make it run, get stuck on Welcome to GTUB! and nothing happends… to bad, well i will try to see whats wrong… ur post helped me very much =D and thank you

AFAIK, it could, but it isn’t the latest Ubuntu. I haven’t check it thoroughly. I only use it for debootstrapping.

http://aosidesigns.comze.com Isaac Alcocer

Isaac Alcocer:
Well i follow ur post and found a finger error =D in:
# apt-get install efibootmgr ucf os-probes
well all really good =D and some updates… the grub its now using gcc-4.6, the only 4.5 text its in just some commets xD… but i couldnt make it work, i install the grub and all but i could make it run, get stuck on Welcome to GTUB! and nothing happends… to bad, well i will try to see whats wrong… ur post helped me very much =D and thank you

aaa sorry should be “os-prober”

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

Thanks for all of the corrections, I slipped a very serious command. Btw, if the GRUB is already loaded, I suspect it is about a GRUB driver that not being loaded or conflicted.

http://aosidesins.comze.com Isaac Alcocer

Well after read all ur post again and again i found that you could make a format to FAT32, i forgot to do it so xD i think this could be, now i out of work so im just reporting some solutions or idea that can solve the problem.

I have difficulty to format the FAT32 partition, I think the Installer CD is dying. So, I decided to format it later. Now, let’s install it via bootstrap it and copy the files that were configured earlier from the installer

and i still got the MacOS so i will give a full format with parted or some MacCD to do all again and make sure to format for FAT32

http://staff.blog.ui.ac.id/jp/ Jan Peter Alexander Rajagukguk

Ah, yeah, I think that should get highlighted…

I have hardware problem, that is my laptop is too old and the CD drive is broken. So, I have to use whatever available.

Well im back with some news =D, im still stuck on grub, i never done it before, so this is exiting… but this time i change some things, first i format all HDD to FAT32, then did the parted part, then i install Ubuntu Lucid well bootstrapped, here first research for some grub info and looks like there is some .deb call it grub-efi-amd64 but this packet install grub on /boot/gurb, not on efi so i install it manually same packet: as #grub-install –root-directory=/boot/efi, but now i can see the Welcome to GRUB! texto, so i dunno if u can help me with the grub part cause i dunno a thing about that xD, im a next next install guy xD in that area…

btw yes i can install any ubuntu or linux distro just need the right drivers =D. the boot part its what cause me some troubles….

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

Wow, congrats! When GRUB in the “Welcome to GRUB!” means it have completed the first stage. Before entering the OS selector, it tries to load bunch of drivers. That’s why I think it is the best if you use the newest GRUB by compiling or backporting it yourself. Btw, words from my friend tells that Linux 3.3 have the ability to runs directly from EFI without the need of bootloader (GRUB). I haven’t tested it yet, though.

parted's file system manipulation code is not as robust as what you'll find in

dedicated,file-system-specific packages like e2fsprogs.We recommend

you useparted only tomanipulate partition tables,whenever possible.

Support forperforming most operations on most types of file systems

will be removed inan upcoming release.

Warning:The existing file system will be destroyed andall data on the<br/>

partition will be lost.Doyou want tocontinue?

Yes/No?yes

yes

Partition number?1

1

File system type?[ext2]?fat32

fat32

Error:Partition too big/small forafat32 file system.

looks like parted doesn’t allow format some fat32 partitions lower than 100MB :S… but its rare cause in ur commands u say that is fat32 and there is no error log or message, how u format that partition?… using chroot and gparted?

I have reformatted you comment so that I can read your valuable information. I don’t know a wordpress plugin to have better formatting on comments. So, I manually add pre html tag to your code.

Well, anyway…

It looks like I have a faulty writings. Thanks pointing out about the FAT32 of being too small. I have checked my server and you are right, I used 200 MB not 20 MB. I suggested a 20 MB because the EFI partition only took 136KB of free space. That also could be done via one of these commands:

The later wasn’t tested by me and I’ve just found it on the Internet now. But, the former is what I have used in the past. It’s just that I have repeated the process and typo still persists. Thanks for pointing out my mistake.

Oh, btw, the EFI blob seems like a chainloader that would loading the real GRUB into memory. That’s why I don’t suspect anything. Because if you have come to the “Welcome to GRUB” screen, it means you have completed loading GRUB. And the reason for not continue would be because GRUB failed to load its second stage drivers.

Isaac Alcocer

Sorry, but there is something wrong with ur comments post style and tags =D maybe its the stylesheet or some script, i dunno …

well i success to install linux-image-2.6-amd64 , but i didnt install extras, to be honest i dunno wut that is XD… but this was early this day and again am out of work, and i leave it installing linux image… if everything goes well in a few hours post wut i did =D cheers

http://staff.blog.ui.ac.id/jp/ Jan Peter Alexander Rajagukguk

Yeah, I don’t know either about wordpress stuff, hahahaha…

Well, if anything works, I think I would like to set this post on a page. Thanks for your enormous help.

http://aosidesins.comze.com Isaac Alcocer

Well it boot 2 times on linux, then a grub (Grub2-1.99-18) error appears,
elf header smaller than expected u.u i think i can’t use that grub, but a will try to reinstall the grub and reconfigure and should be working

Isaac Alcocer

hi again, this is a request, i dunno if u can upload the grub2-efi cause i try again and looks like these versions has the same issue with the header :S so i think u got the right ones and luck xD..

please =D if that works i will send all info about it =D and all the issues i got and i will be always thank you for this awesome post =D

http://staff.blog.ui.ac.id/jp/ Jan Peter Alexander Rajagukguk

I have put the debs in:

1

http://kambing.ui.ac.id/jp/GRUB-EFI-DEBS/

In case something missing, check the source and other debs in:

1

http://kambing.ui.ac.id/jp/GRUB-EFI-DEBS.BUILD/

Good luck.

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

Oh, btw, look out that this version have quirks like what I wrote above. The most important is you have to manually runs every necessary grub commands instead of invoking it by using “grub-install” utility.

Isaac Alcocer

Great i just download them all =D, i will start from 0 so i’ll be back in a few hours, THANKS ALOT REALLY YOU HELPED ME ALOT =D

http://aosidesins.comze.com Isaac Alcocer

Hi … again… well this time i got some news about my problem:

I make the install of the .debs that you post(i have some error but can easy repair that with apt-get -f install)

I try this 2 this week the first time everything ok until reboot, and appear unknown filesystem error in the grub screen, after that i format ALL with AppleCD, and restart again. (looks like this was cause i format grub partition with fat32 no just vfat (16){i read should work with fat32 but looks like in apple doesn’t xD}

Everything ok reboot 2 times, red screen looks awesome with 2 boot options Debian and recovery, then the 3 time EFI header appear :S, i was tired so i try to lunch the SO by myself and did this:

1

2

3

4

5

6

set prefix=(hd0,gpt2)/boot/grub

set root=(hd0,gpt2)

insmod linux

linux(hd0,gpt2)/vmlinuz root=/dev/sdb2 ro

initrd(hd0,gpt2)/initrd.img

boot

but i could, ELF header error appears in this part :

1

insmod linux

looks like this error is on that .mod so XD i dunno what it is, cause i don’t know a sh*t about grub drivers and manual installation. so I’m uploading some pics that u can see and tell me what’s wrong, but thanks a lot, i will try different ways to fix this =D i reaally thank you for all your help =D.. good luck and cheers

I feel that you are already in the GRUB shell and everything have been read well. Maybe, just a hunch, the linux module were already loaded.

Btw, I don’t know what’s wrong with this one. I hope you could find the problem and find what’s wrong. If I can recap, here’s what I do:

I booted with the CD which I have provided.

I formatted the harddrive into two partition: EFI and root using parted. Tried 200MB for EFI instead of 20MB like suggested in my writting.

I reformatted the EFI with mkfs.vfat.

Installed the system with debootstrap and then hooked the /boot/efi with first partition

Installed the full system and the GRUB I provided.

Ran the grub-install and the workaround (grub-mkimage)

Rebooted

I think that was what I’ve done so far. I hope this helps because that’s all that I have done and I hope you find something from this information. Good luck and may you tame the magical beast.

Isaac Alcocer

i got some news fisrt like i said i wil test the ubuntu 12.04 but i have desktop64bits, on boot and holding Alt, looks like works (booteable usb) but gurb never show menu.list, so i will try today with server distro, and new news xD looks like ubuntu 12.04 use a stable version of Grub2 for UEFI systems (grub2/grub-efi-amd64-bin_1.99-21ubuntu3_amd64.deb) http://releases.ubuntu.com/12.04/ubuntu-12.04-server-amd64.list check it out =D, i found a teacher and he told me could it be a filesystem, i need to check the MBR cilinders and repair it. if it doesnt works then i will do a refit boot :S wish me luck

http://aosidesins.comze.com Isaac Alcocer

BTW i hate ur wordpress comment section xD it does wut it likes……..!!! and i dont know if u can change code tag to another one more legible… aaa and erease this coment =D should be good, a btw it should be couldn’t on the line insmod

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

Yeah, I too hate the comment section. But, I don’t have a knowledge in PHP, so I don’t know how to improve this comment section. Hahaha…

Btw, I use the PRE html tag directly for coding section. Hahaha…

http://aosidesins.comze.com Isaac Alcocer

Ok i’m DONE WHIT THIS….

sorry man i could do you way, the bad news its that with http://www.ubuntu.com/download/server/thank-you?distro=server&release=lts&bits=64 you can boot in CD mode, install Ubuntu 12.04, and it does works no problems so far just some issuse with some driver nvidia nothing than cant be repair, =D so i got a linux fully working, btw if you wanna debian just backport this grub files =D and use it =D and works, i found my problem, its that i got 119.8GB no 121.2GB on my SSD so the sectors get corrupt and that happend =D i found it when i try to repair the HDD =S so bad, so i give it a try with PRECISE PANGOLIN, and “Voila”, this works for me =D since i really want KVM on UBUNTU so its great, THANKS ALOT i Lear so much from you, even how GRUB works xD.

THANKYOU and GOOD LUCK

There is an error on grub if you install Ubuntu 12.04, first you cant use a BOOTEABLE USB, this is a bug on Ubuntu, you need a CD/DVD, second in GRUB after first boot(make sure you install SSH) log in via ssh and do this:
$sudo$EDITOR/etc/default/grub
inside you just need to edit GRUB_CMDLINE_LINUX_DEFAULT=”” and replace it with GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash nomodeset” or your alternative GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash nomodeset gfxpayload=keep” thats all =D

CHEERSS PPL =D

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

I alsofeel thankful for your sharing. Your suggestion about using Ubuntu 12.04 might be the best solution. It’s very neat and easy.

Good luck also and may the source be with you.

João Pereira

Can you please share the full steps needed to install Ubuntu 12.04 server on a xserve? It is just plug the cd, select it on boot and install (for those that dont want to keep osx server, only Ubuntu installed)?

Isaac Alcocer

well looks like u got some problems. well look, i just download a fresh version of Ubuntu 12.04 server 64 bits amd , burn it and boot itselfs then just some issues that u can repair =D no big deal, really easy… =D i use KVM but i have some troubles with connection betwen qemu://system and a remote virt-manager =D so i dunno wut problems u have. thats all i did just burn a disk and repair the problems =D… the video driver problems.

João Pereira

Can you please share the full stepts to get Ubuntu 12.04 server installed on a xserve 3.1? I just want to have Ubuntu, not dual boot with osx. It is just needed to burn the image, plug it, select on boot and install as done on a PC without anything else?

http://jpmrblood.blogspot.com Jan Peter Alexander Rajagukguk

I’m sorry I can’t help you with that. I haven’t try it. It was the commenter who suggested that.

Nope. That’s just the case of a certain model of MacBook in 12.04. I’m doing fine with my setup. The only problem is the Apple RAID card couldn’t detected. But, I’m using older (Squeeze) kernel and using workarounds for installing. Don’t know if 12.04 have already taken care the issue since it seems people said it was easy using it.

Anyway, the post you mention using rEFIt as bootloader. My method is using EFI (the BIOS replacement) directly. My reason was that I didn’t know how to use Mac, so I chose this route.

http://profiles.google.com/aosi87 Isaac Alcocer

Hi again, yes im back, i was researching some old articles of mine, and this one apears =D, tnx google, BTW i still hate ur comment section does what it likes…

I just saw the update on your post, that says too much about you

And YES it works like a charm =D… i got a fully functional Ubuntu LTS 12.04 on it, using KVM and a lots of VMs, like 35 just in that server, works really well, no problems at all, none, not even fan, power, drivers (i use command line so, i guess NVidia should work on desktop animations), even crtitical updates WORKS, its a **** great piece of hardware, and tnx to you i can have the best of it with DEBIAN and UBUNTU =D.. THANK YOU SO MUCH =D…

c ya, and the source be with you too =D.

http://staff.blog.ui.ac.id/jp/ jpmrblood

I’m glad to help. And yeah, I couldn’t helped it. I’m a System engineer not a website designer, so sorry about the suck commenting system. Hahaha…. ;-P