ESET CONFidence 2010 Crackme - WriteUp

ESET proposed a crackme during the CONFidence conference. Challenge
started on November, 29th and lasted two days. The goal was to find a
valid username/serial combination. Challenge was won by Dmitry Sklyarov,
from ElcomSoft. This article will present a solution for the crackme,
and the steps needed to write a key generator for it. Even if the
crackme size is only 11.5kB, many tricks are used to make the
verification routine tricky enough to resist a few hours.

An easy start

The crackme is a Windows application, and asks for a name and a serial
number. The serial must be composed of two parts, separated by a hyphen.
The first part is 8 chars long, the second one is 20 chars.

First, the name is hashed with MD5. The resulting hash is converted to a
string of 16 lower chars, using a basic encoding function:

for(i=15;i>=0;i--){out[i]='a'+(hash[i]%26);}

The resulting string is converted to UNICODE and hashed with the
RtlHashUnicodeString function. This function is exported by
ntdll.dll. It returns a 32 bits hash using the x65599 (?) hashing
algorithm. Input string is processed case insensitive. A pseudo-code for
this function would be:

Reversing CRC32

The first part of the serial, composed of 8 chars, is hashed using
CRC32. The computation is done with the RtlComputeCrc32 function.
The initial value of the CRC32 is fixed to 'ESET' (0x45534554). The
resulting hash must be equal to the x65599 hash previously computed.

Obtaining a string with a given CRC32 is easy, if you can fix 4 bytes in
it. For more information about CRC32 reversing, check this paper. The
additional difficulties here are the custom start value and the fact
that this string must be written using a custom set of 32 chars:
"2WQKHTL3VJBYG6PZCM9AXF0UED5RS7N8".

How to handle such case? I chose to generate a string of 4 random chars
taken in the charset, and to fix its CRC adding 4 bytes to it, so that
the CRC is equal to the x65599 hash. Then, if the 4 added bytes are in
the charset, the problem is solved. Else, I try with another string.

It is a small bruteforce: considering the possibility of each char to be
in the charset is 32/256, the probability that the 4 added chars are in
the charset is 1/2^12, which is really acceptable.

A strange decimal conversion

The second part of the serial is 20 chars long. It is converted to a 64
bit integer n in the sub_4012DC function. This function is a bit
tricky. Basically, it does a decimal string to integer conversion: each
time a byte is read, n is multiplied by 10, the serial byte is
converted to an integer between 0 and 9, and added to n.

How are the chars converted to a decimal value? This is the difficulty
of this part. Input chars are checked like this:

0x32 is subtracted to each char. Resulting value must be different from
0. Then, two loops are executed. They iterate reading the 5 lower bits
of each char. Each loop exits when the read bit is different from the
previous one. Once the two loops have been executed, the 5 bits have to
be read. That means that to be valid, a char must be composed of

a sequence of 1 followed by a sequence of 0,

or a sequence of 0 followed by a sequence of 1.

Only 10 5-bit values validate this condition: 11111, 01111, 00111,
00011, 00001, 10000, 11000, 11100, 11110. Adding 0x32 to each of these
values, we obtain the 10 bytes charset: 'Q', 'A', '9', '5', '3', '2',
'B', 'J', 'N' and 'P'.The value between 0 and 9 is contained, after the
two loops, in the ebx register. It means that it is equal to the number
of iterations the two loops have been executed while reading the byte.

To sum up, we know that sub_4012DC converts a string encoded in a 10
bytes custom charset into a 64 bit integer. In order to write a key
generator, the inverse function has to be coded. It takes a number as
input, and generates the corresponding encoded string. Here is the code:

ElGamal signature

The last part of the crackme is about a ElGamal signature verification.
ElGamal signatures are composed of two numbers (r, s).

Retrieving message and signature

The first part of the serial, whose generation is explained above, is
hashed with MD5. A 16 byte digest is obtained. It is used to generate a
64 bit integer. This integer is the r in the ElGamal signature. The
function that converts the digest to a 64 bit integer is the following
one:

Basically it casts the digest into an array of unsigned integers ll,
and computes `` & 0x7fffffffffffffff``.

s is the second part of the serial after its conversion to a number
using explained above.

The user name is hashed with MD5. The resulting hash is converted to a
64 bit integer with sub_4012C2. Let's call this number h.

Finding group parameters

ElGamal security relies on the discrete logarithm problem. To make it
short, it consists in finding x such as g^x = y mod p, given g, y and
p.g, y and p are public parameters.

The signature is checked in sub_40121A. This function calls two big
integers functions, powmod64 (sub_4011B0) which is a basic
square-and-multiply modular exponentiation, and mulmod64
(sub_401178). Before being able to generate valid signatures, public
parameters have to be identified.

The group generator is easy to spot: it is present in the signature
verification function.

A last problem

The code just above generates r and s values that sign the
message h. The problem is different here: remind that r comes
from the MD5 of the first part of the serial. It cannot be chosen
randomly, and must be considered as a fixed value...

r = pow(g, k, p) is fixed. What is rather annoying is that k,
normally chosen at random, is necessary to compute s. How to
retrieve k from r? Solving another discrete logarithm problem is
necessary...

r depends on the hash of the first part of the serial. This part
depends on the hash of the user name. Hence, for each user name, a
discrete logarithm has to be solved to generate a valid signature.

Finally, once k has been found, one has to check that it is co-prime
with p-1, else k has no inverse mod p-1, and s cannot be computed.

Once all theses steps have been understood, it is time to code a key
generator. Source code is attached. The more interesting parts in this
source are the CRC32 reversing function, and the runtime DLP solver,
which uses Pohlig-Hellman and Pollard's Rho algorithm to compute rather
efficiently the logarithms, even if it is written in Python.

I thank the ESET team for this nice crackme. Hiding such tricks in a so
small piece of code is tremendous =)