HOWTO Replace RedHat with Gentoo, Remotely over SSH

Author: Jordan Ritter <jpr5@darkridge.com>

Introduction

What is this?

This is a HOWTO that describes how to take a stock RedHat9 system and
convert it to Gentoo, remotely over ssh and while it is
running. These instructions have been tested with RedHat9,
but very likely apply to everything from RedHat9 down to RedHat 7.2,
and include everything from advice and gotchas around how to get
another distribution installed over RedHat while it is running, to
strategies for cleaning up the system of unused files once the
transformation is complete.

Why do that?

While everyone has their own tastes in Linux distributions and
installed software, there are frequently times when one doesn't have a
real choice, and only the most popular distributions prevail. For
example, remote ISPs that offer leasable colocated systems will
provide only a short list of the most common Linux distros for
pre-installation, and while it's almost a sure thing that RedHat is on
that list, there is no guarantee that your favorite one is.

I had a remotely colocated server installed with RedHat9 that I had no
physical access to. I love Gentoo, though, and so through much trial
and tribulation (and about 80 vmware checkpoints) I figured out how to
fulfull the Gentoo dream without needing to access the physical
console.

Does it really work?

Yes, it really works! It probably seems a bit scary, but I have tried
this process now a dozen times and the only problems I've ever had
were the result of typo mistakes that can easily be avoided with more
careful attention. No more RedHat9 on my systems -- all are 100% pure
Gentoo now, baby!

Wait... I have questions about Gentoo..

This HOWTO assumes a basic knowledge of Gentoo. Some time is taken to
explain the finer points and possible pitfalls and gotchas, but to
successfully install Gentoo Linux really requires ready access to the
Gentoo Installation Documentation, in addition to this document.
Print it out in advance or have it handy before doing this.

OK, I'm ready. I can sue if this doesn't work, right?

Nope.

THESE INSTRUCTIONS ARE PROVIDED AS-IS AND WITHOUT WARRANTY. ALL
USERS ACCEPT FULL RESPONSIBILITY AND LIABILITY IN THE EVENT OF SYSTEM
FAILURE, DATA LOSS, OR ANY OTHER FAILURE OR LOSS RESULTING FROM THESE
INSTRUCTIONS.

Phew, ok! Here we go!

The General Strategy

In summary, the transformation process is actually fairly simple.
Since Gentoo, once unpacked, will run its own binaries from its own
libraries, all we're really doing is unpacking Gentoo on top of RedHat
without wiping out our remote access mechanism (sshd), then removing
the old RedHat files.

The entire process can be distilled down into the following simple
steps:

Preparation -- Make copies of important files and configuration
items, Download Gentoo tarballs, Get sshd running without PAM, and Tag
old files for later removal;

Step 1: Preparation

There are several things that need to be prepared before actually
starting the transformation process. Create a directory on a
partition you have plenty of space on and don't intend to touch (like
``/tmp/gentoo''), and use that to store temporary files and such, as
explained below.

Again, please have handy in some form the Gentoo Installation
Documentation, as this HOWTO won't take the time to explain the
various Gentoo-specific concepts in detail.

All instructions assume the user is logged in as ``root''.

Compile a kernel with relevant driver support compiled-in.

Just to be on the safe side, you should compile a fully-static kernel.
In other words, we want a kernel that will run the system that has no
externel module dependencies. When we start removing libraries and
old files and such, it is quite possible we might accidentially rm
something that those exceedingly tedious RedHat kernels rely on. Just
compile your own and make sure your system boots and runs from it
before you even start this process. I suggest sticking with 2.4.x for
now so as not to complicate things.

NOTE: I suggest not enabling devfs support just yet.
Gentoo can still boot without it, and it's better to have a kernel
that can boot both systems RedHat + Gentoo than one that may only be
able to boot under Gentoo (dunno how well RedHat supports devfs
out-of-the-box).

Also, just to be safe, copy the kernel (again, after you've tested it)
to the safe directory where you're keeping copies of things.

Turn off PAM in OpenSSH's sshd_config file.

Just like with the kernel, we have to be careful about the differences
between the PAM setup in RedHat and the PAM setup in Gentoo. You
don't need PAM in order for the thing to function, so
regardless of whether you like it or not just disable it so as to
ensure you'll be able to login again.

Best way to do this is:

Modify ``/etc/ssh/sshd_config'', set ``UsePAM'' to ``no'', save the file.

Kill sshd with ``kill PID'', where you got the PID from the
previous step. This will only kill the listening sshd, and will
not affect any already-connected sessions.

Without logging out, run ``/usr/sbin/sshd''.

Separate from your current logged-in session, try to re-ssh back into the box.

If you get back in, you're fine on this step.

Most folks will have no problem with this. If you do experience
problems, consult ``/var/log/messages'', and in the worst-case
scenario run sshd in the foreground with debugging turned on. That
should give you enough detail about what's going on to figure out how
to get things running fine.

Download the latest snapshots/stage3 tarball.

The only thing you need is the latest stage3 tarball. Make sure to
decide ahead of time what platform architecture you are using (x86,
pentium3, pentium4, etc), as that's an obvious benefit of Gentoo
that you'll want to take advantage of. For the purposes of this
HOWTO, we assume the safe ``x86'' architecture.

You can find a full list of Gentoo mirrors at
http://www.gentoo.org/main/en/mirrors.xml. As of this
writing, the most recent stage3 tarball is:

Additionally, take a quick snapshot of what modules are loaded, for
later inspection, with ``/sbin/lsmod > /tmp/gentoo/modules.txt''.

Mark all current files with a special date.

Assuming the author isn't on crack, how would one know what files to
delete after the Gentoo installation overlays the RedHat one? Easy!
Set the timestamp on all files to a special date before proceeding. I
used my birthday!

First of all, we need to be careful about touching all files, because
the kernel will detect accesses to ``/dev'' and try to autoload
modules for things that you may not have present or installed, which
will make things take a lot longer. So first we temporarily
disable modprobe by moving it out of the way, and then move it back
when we're done. Also, some filenames and directories have spaces in
them, so we need to be careful of that also.

Step 2: Installation

OK! So now it's time to take the dive. Yes, it's a little scary, but
follow these instructions and you should be fine.

First, the most effective way to conduct this installation is with two
(2) shells, both at or su'd to the root user. We'll call the first
shell the "setup" shell and the second shell the "repair" shell.
Let's assume that you used "/tmp/gentoo" as the safe directory to
store copies of downloaded files and important files and configuration
items.

From the "setup" shell: Unpack the stage3 tarball onto the root partition.

cd / && tar jxvf /tmp/gentoo/stage3-*.tar.bz2

Synchronize Portage.

emerge sync

While the portage synchronization is happening, continue on by
completing the following operations from the "repair" shell:

Re-set the root password. This is because the stage3 tarball
replaced your password and shadow file.

Add your user account to the wheel group in ``/etc/group'', so you
can ``su''.

IMPORTANT: At this point, test that you can ssh back into
the box. You may have to restart the listening sshd server, but
that's unlikely.

Re-add any other necessary account info, referencing
``/tmp/gentoo/passwd,shadow,group'' files. If any UIDs or GIDs change
between the old files vs. the new, make sure to run chown across the
filesystem to adjust them.

And that's it! Now you're ready to reboot and come back up on
Gentoo. At this point the best thing to do is go over your
configuration and make sure you have the important things covered
before you reboot:

That the bootloader is properly configured to point to your
preferred, working kernel.

That your kernel supports the necessary drivers to properly boot
the network interface.

That the Gentoo network configuration looks correct and doesn't
have any typos.

That your fstab is correct and the right file systems will be
mounted at boot time.

That you can login to the system remotely, and that you can su to
root.

That the remote access mechanism, openssh, is configured properly
to load at boot time. This can be via ``/etc/conf.d/local.start'' or
the ``rc-update'' mechanism.

When you're ready, reboot your system: ``shutdown -r now''.

Step 3: Clean-up

The remaining tasks revolve around cleaning up the old RedHat files
that Gentoo is not going to use (to get that disk space back), and a
few OS security hardening items that can't hurt.

If you're like me, you install the minimum amount of software
components necessary to run the system. If however you like
installing lots of other things, then keep in mind that you need to
consider the spirit of the following instructions rather than the
literal directions, as you may have other components you'll want to
isolate or protect.

Protect certain files and directories.

Before we install any more software, we need to take advantage of the
timestamp magic from the Preparation Step and nuke all the old files.
But before we do that, there are some directories and files we want to
make sure are protected no matter what. The following is a command
line that touches the timestamps of various files, and it should be
good enough if you haven't done anything beyond the scope of these
instructions.

Recall that we ``touch''ed the entire file system back in the
Preparation Step. This was to make it easy to find them after the
installation -- the Gentoo steps would overwrite whatever files were
in common with RedHat, thereby updating their timestamps, while the
files that were untouched would retain the old timestamp. Now we'll
run a ``find'' looking for directories and files older than a certain
number of days, and then ``rm'' them.

NOTE: the ``9125'' number below references the number of days
since my birthday, which again was the date I chose in the Preparation
Step. If you chose a different date, you need to adjust this number
accordingly.

Doing this will screw up the installation of groff (man) and emacs.
I'm not sure how, but the solution is simple and easy enough so
instead of drilling down to figure this out, I just provide you with
the solution:

emerge groff
emerge emacs
emerge vi

See? Simple.

Optional: Remove old RedHat devices and special files.

In the interest of being complete, we also want to consider wiping out
any and all "special" files (devices, pipes, sockets, etc) from RedHat
that Gentoo doesn't use. Again, remember that the timestamps will be
recent if the Gentoo Installation overwrote any of the files, so you
should be safe.

NOTE: I did not do this step. I don't know if it works, but
there won't be much disk space to recover from this step and it just
wasn't important to me in light of the fact that the Gentoo system
prefers to use DevFS and will take over the ``/dev'' directory.

Once you're off of the RedHat kernel, you should ``rmdir /initrd'' as
well.

IMPORTANT: don't nuke the initrd directory, even if it's empty,
if you're still on the RedHat kernel -- the RedHat kernel boot will
fail and hang when it goes to mount the initrd image if the mount
point is missing.

Secure the system.

As a general practice, I don't like my users doing much on a system
beyond what I have specifically provided for them. To that end, I
favor limiting the permissions of standard system components, because
you never know what's going to become a security risk.