The Challenge:In March, 2001
a Solaris system was compromised. A collection of tools, utilities and
files were uploaded onto the system by the blackhat. One of the files was
encrypted. For this challenge, we have changed the name of the encrypted
file to "somefile". You can download this file as
somefile.zip,
MD5 Checksum=eb7ed869ffcfe72d4b48caf57e648910, or
somefile.tgz,
MD5 Checksum=f7964d9860cbf8135ef64bcf5b96facb.

The first step
is to download the file and confirm the MD5 Checksum, this validates the
integrity of the file. Then the analysis can begin. wget(1)
is a Unix utility that retrieves files from the World Wide Web.

1. Identify the encryption algorithim used to encrypt the file.
2. How did you determine the encryption method?

We will go ahead and answer questions one and two together. We begin by analyzing
somefile
with the file(1) command, which determines that this file is a binary data
file. So it cannot be analyzed using standard text based utilities,
such as vi(1) or Notepad.exe. Instead, we have to use some viewer
that takes the binary data and converts it to hexadecimal information.
In this case, we select the handy utility xxd(1). This utility makes
a hexadecimal dump of binary data, or does the reverse (its an extremely
powerful utility). You can also use the hexdump(1) utility. In this case,
we run xxd against our binary somefile for file analysis.

Above is the hexadecimal
representation of the binary file. Reviewing the file shows a pattern,
there are no ASCII values, everything is above decimal value 127.
This is very odd, as almost any binary file should have some ASCII characters.
Most likely there has been some type of character substitution or modification,
indicating there is a pattern and we can potentially break the code.
If this had been a strong encryption scheme, the characters would have
been truly random.

When we analyzed this file, we had the advantage of knowing this was an
ASCII configuration file for the universial rootkit. That means we knew
the approximate format of the config file. The first character was most
likely a '[' (0x5B), and the last was most likely a newline '\n'(0x0A).
David Dittrich initially thought it was ROT13 or some Ceasar substitution
cipher. Working off the two characters above, that proves impossible.
Next on the 'easy to do' list is XOR.

3. Decrypt the file, be sure to explain how you decrypted the file.

The C code below contains the solution for decrypting the file. The
code contains the line "printf("Mask: %02x\n", '['^0xA4);". This
is because the first character in the encoded file is hex 0xA4, which we
believed to be the '[' character. Due to the properites of XOR, if we XOR
0xA4 with '[', we will get the mask used, which was 0xFF. Now, the rest
of the C code goes through and XORs every encoded character with 0xFF,
resulting in the original file. To learn more
about XOR, binary and Hexadecimal values we highly recommend this
review
of Assembler.

4. Once decrypted, explain the purpose/function of the file and why it was encrypted.

This is a configuration
file for a rootkit. This file determines what process, services,
and files will be hidden from the system administrator. The computer
is now configured to lie to the administrator. The purpose of encrypting
the file is to make it more difficult for the rootkit to be detected, analyzed,
and recovered from.

Bonus Question:
This encryption method and file are part of a security toolkit.
Can you identify this toolkit?

This configuration file belongs to
the Universal Rootkit,
designed by K2. Side note. David Dittrich, rain forest puppy, and
Marty Roesch worked on this together while at CanSecWest. Once they
had completed the decode and full analysis of the rootkit, K2 coincidently showed
up at the conference (and was immediately pummeled :)