Used the VMware "Open existing virtual machine" to load the suspended image. I don't run it yet.

Copy the md5 file to floppy. On a trusted Linux system I decompressed it and pulled out just the file names to another file on the floppy.
mount /dev/fd0 /mnt
cp /mnt/*.gz .
gzip -d linux-suspended-md5s.gz
cut -c35-99 host79-2003-08-06.md5 >/mnt/files

Copied the netstat, ls, find and ps commands to the floppy as well, since I may not be able to trust the versions on the compromised system.
cp /bin/netstat /mnt
cp /bin/ls /mnt
cp /bin/ps /mnt
cp /usr/bin/find /mnt
umount /mnt
(I later discovered that find is not statically linked and would not run on the Red Hat 7.2 system. I will remember this next time.)

Now I resume the suspended image within VMware. This is what I see:

The first signs that something is amiss is the line "eth0: Promiscuous mode enabled."
This indicates that a sniffer program has probably been run.

I see another sign that something is afoul.
The .bash_history file has been changed to a symlink to /dev/null.
This will cause bash to stop saving the command history to a file and
stop us from getting the very useful information that it would provide.
This symlink was created at Aug 10 15:30.
The sslstop directory was created 24 minutes later.
This appears to be a program to change some config options for apache.

Tell VMware to connect the floppy drive.

Mount the floppy disk prepared earlier:
mount /dev/fd0 /mnt

Run the netstat command to look at active connections:
/mnt/netstat -anp

There are several suspicious items in this output, which I will discuss in the answer to question #3 below.

To see which files have been modified, I need to generate a new set of checksums:
for i in `cat /mnt/files`
do
md5sum $i
done >/mnt/newsums.md5

Unmount the floppy:
umount/mnt

On the trusted the Linux system, I perform a diff of the checksum files to see which files have changed.
mount /dev/fd0 /mnt
diff host79-2003-08-06.md5 /mnt/newsums.md5 >sums.diff
Checksum diff output
Suspicious items are marked in red.

Many files will have changed during the normal operation of the system. I will ignore these.
The changed files that do concern me are:

/usr/bin/top

/bin/netstat

/bin/ls

/bin/ps

/sbin/ifconfig

These are files that are frequently trojaned by a rootkit.
(Now I am glad that I have trusted versions on the floppy.)
I will discuss these further in the answer to question 8 below.

This process has shown me which files have been modified, but it does not show me which files have been added.
To get this information I need a list of files in a format to compare to the list I created above.
I create this list and remove the files from the /proc, /dev, and /mnt directories:
mount /dev/fd0 /mnt
find / | grep -v ^/proc/ | grep -v ^/dev/ | grep -v ^/mnt/ >files2

To see which files have been added:
diff /mnt/files files2 >files.diff
Files diff output
Suspicious items are marked in red.

Again, many files will have been added during the normal operation of the system.
I look for suspicious additions and find several.
The one that gets my attention first is /lib/.x, so I take a closer look at that:
/mnt/ls -alR /lib/.x

Notice that several of the files are owned by the apache user.
This is unusual and leads me to believe that they were placed there by an apache exploit.
I investigate these files using the commands file, less, and strings to discover the following.
Output from these commands

The file cl is the log cleaner "Die Putze" which means that we will get little useful information from the system logs
(In fact, after looking in the /var/log directory, I find that the file messages
has been symlinked to /dev/null and the apache log directory has been deleted.)

The file inst is a script to create the file sk.

The file sk is a rootkit called sucKIT.
There is no longer any doubt that this system has been compromised.

The directory s contains files for what appears to be an SSH server in the file named xopen.
Two copies of this program have been run and are listening on TCP port 3128 and UDP port 3049.
This is discussed further in the answers below.

The file lsn is a compressed executable, but I believe it is a password sniffer.
A Google search for "lsn sniffer" provides ample evidence to support this belief.

I continue looking through the list of added files.
The directory /etc/opt/psybnc has been added.
Lets have a look at it:
/mnt/ls -alR /etc/opt/psybnc
psybnc directory listing

This is the psyBNC program, which is used to hold open an IRC connection and to
proxy IRC in order to hide the user's IP address.
There are three instances of this program currently running, and one has an active connection.
This is discussed in the answers below.

The only language files included with this version are English, Italian, and German.
The intruder probably speaks one or more of these languages.

Now we know that the intruder likes to use IRC, so lets search memory for any clues we might find.
The file /proc/kcore is mapped to the kernel core memory.
strings /proc/kcore | less

I use the search function of less to look for MSG, JOIN, and other common IRC strings.
I find that the intruder has connected to fairfax.va.us.undernet.org and joined the channels
#RedCode, #Ef~#Wurst, and #AiaBuni (which has a topic of "a bunch of hacked comps and the hackers")

Two snippets of conversation were found. One looks like the admin of this system talking to the intruder:

PRIVMSG decedatu :&LT;NeThUs`&GT; If you want a shell all you needed to was ask. If you don't mind please
PRIVMSG decedatu :&LT;NeThUs`&GT; let me know what you changed the root password to and which exploit you
PRIVMSG decedatu :&LT;NeThUs`&GT; used. I don't keep that box very secure as you now know. If you wish to
PRIVMSG decedatu :&LT;NeThUs`&GT; have a shell let me know those two pieces of information and I will
PRIVMSG decedatu :&LT;NeThUs`&GT; hook
PRIVMSG decedatu :&LT;NeThUs`&GT; it up for you.
PRIVMSG decedatu :&LT;NeThUs`&GT; -Mike

Since many users of IRC talk in "chat shorthand" style, it can sometimes be difficult to identify the actual language.
However, I do not believe that this is German or Italian.
I am guessing that it is Romanian. To test this theory, I do Google searches for spunetzi, vretzi, and totzi.
The results for all three searches were overwhelmingly Romanian.
This guess is further backed up by the discovery (from the search of kcore, see above) of several systems involved in this incident,
which are Romanian. Using the whois command I find the following.

The time stamp is about two hours before the /lib/.x files were created, so I will assume that
this is the intruder poking around on the web server.
The IP address 213.154.118.219 (extreme-service-11.is.pcnet.ro) is another system in Romania.
This is consistent with the IP addresses discovered earlier.

Question #2
Explain the impact that your actions had on the running system.

The actions taken to analyze the system will have a small impact on the system.
Various log files will be modified and some swap space and disk space will be overwritten.
Since this could destroy some evidence of the compromise, best practices recommend
that minimal investigation be performed on a live system.
The normal practice would be to create an image of the disk using dd or some other
imaging tool and perform the analysis on that image.
Since this is a virtual machine and we already have a suspended image, I will ignore this step.
In fact, since I can easily restore the system back to the state it was in when suspended,
I am free to analyze the live system with little concern about the impact of my actions
(aside from isolating it from the network.)
I can always return to the original suspended image for a more thorough forensic examination.

Question #3
List the PID(s) of the process(es) that had a suspect port(s) open (i.e. non Red Hat 7.2 default ports).

The trusted copy of the netstat command (see above) showed several suspicious ports open.
I used the /proc directory to get the full path for the suspicious processes.
The /proc directory shows many details for each active process in a directory with the PID as the name.
For example, the file /proc/15119/exe will be a symbolic link to the actual executable file.
See this example

What I found:

Process /etc/opt/psybnc/initd (PID=15119) has ports 65336 and 65436 open. These are non-standard ports.

Process /lib/.x/s/xopen (PID=25241) has port 3128 open. This port is normally used by a web proxy such as squid.

Another instance of process /lib/.x/s/xopen (PID=25239) has UDP port 3049 open. This port is normally used by CFS.

Question #4
Were there any active network connections? If so, what address(es) was the other end and what service(s) was it for?

The trusted netstat command (see output above) shows an active connection on port 65336 from a system with
IP address 213.154.118.200 (sanido-08.is.pcnet.ro), which is an ISP in Romania.
The process used for this connection is psybnc, which is a tool for holding open IRC
connections and for proxying IRC connections to hide your IP address.
We can deduce from this that the service is most likely IRC or IRC DCC protocol.

Question #5
How many instances of an SSH server were installed and at what times?

The original (/usr/sbin/sshd) has a file date of Sep 6, 2001.
This file was probably installed when the system was created.

The file /lib/.x/s/xopen was installed by the intruder at Aug 10, 2003 15:32.

Question #6
Which instances of the SSH servers from question 5 were run?

By examining the /proc directory I discovered:

The sshd program (PID=699) was run at Aug 10, 2003 20:31.

The xopen program (PID=25239) was run at Aug 10, 2003 20:31

The xopen program (PID=25241) was run at Aug 10, 2003 20:31

The xopen program (PID=25247) was run at Aug 10, 2003 20:31

Question #7
Did any of the SSH servers identified in question 5 appear to have been modified to collect unique information?
If so, was any information collected?

Once again, I look at the /proc directory to get some further information on a suspicious process.
(PID=25247 See the output)
This process has the same files open as the other instances of the xopen program.
Strangely, though, its exe link does not show up. I think this may be caused by the program
overlaying itself with the lsn program, which is compressed.
It has one additional file open: /lib/.x/s/mfs, which I found earlier to contain password sniffer output.
Therefore, I conclude that this program has been modified to capture passwords.
The file shows several FTP connections and one Telnet connection, however, no passwords were recorded.
This was most likely caused by none being entered.

Question #8
Which system executables (if any) were trojaned and what configuration files did they use?

The diff of the checksums showed that several files were replaced with trojan versions.
Examining these files with the strings command hinted at which configuration files they used.

The commands /usr/bin/top and /bin/ps both use /dev/ttyop for configuration.
This file tells them which process to hide.

The command /bin/netstat uses the file /dev/ttyoa, which tells it which network connections to hide.

The command /bin/ls uses the file /dev/ttyof, which tells it which files to hide.

The command /sbin/ifconfig also appears to have been replaced.
This is typically done to hide that the network interface is in promiscuous mode.

This behavior is consistent with the operation of several common root kits.

Question #9
How and from where was the system likely compromised?

The running apache version is 1.3.20, which has a well-known vulnerability in the SSLv2 handshake.
This is the vulnerability exploited by the "Slapper" worm and may be the exploit used to compromise this system.
I believe that this or some other apache exploit was used to compromise this system.
The files created in the /lib/.x directory which are owned by the apache user lead me to this conclusion.

The search for apache log file remnants showed which system was accessing the web server shortly
before the compromise occurred.
This leads me to believe that the compromise was most likely executed from the same address:
213.154.118.219 (extreme-service-11.is.pcnet.ro).

Bonus Question:
What nationality do you believe the attacker(s) to be, and why?

Romanian. The IP address seen in the apache log remnant was Romanian.
The active network connection was from Romania.
Several other Romanian IP addresses were encountered in this investigation.
But, most convincingly, the IRC chat conversation discovered appeared to be in Romanian.