I just discovered a real nasty thing with grub, raid and RedHat 7.3. It goes like this:

I just spent the last 2 days building a server. I have RedHat 7.3 as the os, 2 ide drives running software raid 1, and these are using ext3 as the filesystem.

7.3 has been out a while and a lot of the packages are out of date, so I do a update all RPMs to bring the server up to spec. This includes updating the kernel to 2.4.18-5. It shouldn't be a problem.

I check /boot, and it's all been updated to the correct kernel. I check grub, and these files are pointing to vmlinuz 2.4.18-5. I'm happy and I reboot.

Grub loads. It prompts with "Red Hat Linux (2.4.18-3)". I think "What the...?" I try booting it to see what happens, expecting obscure error codes, kernel panics, and the end of civilisation as we know it.

It starts booting, then it loads vmlinuz-2.4.18-3, alogn with the rest of the OS. Of course all the device drivers fail because the symbols are wrong. I'm amazed, astounded and astonished. I search for signs of 2.4.18-3. It's not there. Totally absent. Not a 2.4.18-3 anywhere on the server. Totally eradicated. I check two different ways just to be sure. I am totally confused, confounded and chastened. The server just semi-successfully booted a ghost; I had thought I felt a chill enter the room. I wrote that off to the AC coming on, maybe it hadn't. I think "Buh?"

I search the internet for answers - it knows everything - this has to have happened before. This will be fixed in two minutes. No problem.

I find lots of questions relating to this. No answers. Nothing. Not a clue. Not a hint.

This is a problem.

I wonder if maybe civilisation did end when I booted a kernel that didn't exist and could only have been a ghost. I idly wonder if Dr. Egon Spengler would have any ideas. He seems like the type that would know about a ghost kernel.

I assume grub is storing the config file elsewhere. It doesn't explain how it's loading the previous kernel though; it can't be storing that much data in the MBR. I try running grub-install. It won't run because it says "/dev/md0 does not have any BIOS drive". Of course not. I try forcing it to do something. Anything. Because I have everything on a raid device it refuses. I try everything short of trying to find a sharp stick and poking it, but it just won't work. By now I'm looking at losing 2 days of a server install because I have no way to get all my config files off the machine (no devices are working, remember?). By now, I'm beginning to drool and idly wondering if it will stain my shirt.

I decide that the system is pretty irreprable, and elect to try desperate measures.

Suddenly I see the old boot directory. vmlinuz-2.4.18-3 is there, along with the old grub files that it won't change. I wipe the drool off my chin, and type the following:

mkdir old
mv * old
cp -r /boot/* .
ls

All new files are there, I double check permissions and links. I type: sync; init 6

Grub appears on the screen. It happily prompts me to boot "Red Hat Linux (2.4.18-5)". The new kernel boots and all is right with the world again. I've exorcised the ghost of 2.4.18-3 and I can go back to breaking MySQL again.

Man, I've been having some wierd problems with Borland C++.
I can't get librarys to link, io.h won't #include (well it
includes, it just won't recognise anything that io.h should
provide...) It's
being a royal pain. I wish I knew what was up with it.

So now I'm unemployed again. My contract ended on Monday.
This came as a surprise to all involved, as everyone else
thought it was at least Friday. Contract Services notified
security to revoke my pass. They didn't notify me, my firm,
or my manager about this, so nobody else knew. This came as
quite a surprise to me on Tuesday morning when I turned up
for work and was refused entry into the building. I had to
get my manager to sign me in as a
visitor so I could at least clear out my desk, and finish up
stuff.

Quite reasonably all parties involoved were more than a
little annoyed about the way my contract ended, especially
me. I'd been after my manager for weeks to try and get a
definite finish date, but he couldn't get any information
about it.

Whats even more stupid is that although they need people in
the department, they can't hire anyone. They can hire
contractors, but only for a year (because legal is scared of
the permatemp problem Microsoft had). So they let us go,
then hire more contractors.

My firm told me that they had a position available at
Microsoft in Redmond. There were a few problems with this:

I don't like Microsoft, and I have ethical problems
with working for them

Redmond is on the other side of Seattle, and it's about a 3
hour commute given Seattle traffic

Microsoft don't pay contractors that well.

So here I am updating my resume, and I've decided to go back
to Ireland for 3 weeks and visit family and friends.

So, now I'm the last contractor in our department. It's a
bad situation too. The second last contractor, Scott, is a
Novell
guy. During the last round of layoffs they fired a Novell
engineer, leaving just Scott and Clinton. Clinton is
currently back in the army on reserve duty. Theres no idea
when he will be back. This leaves us with zero Novell
people. Actually half a person. One of the 64 bit linux
guys will be taking over Novell duties, but he will be also
have to do 64 bit stuff...

No word on my job, but I've only got a couple more weeks
before I'll find out... Most likely I'll be looking for a
new job soon...