The famous zombie researcher “Donn Beach” almost created an immunization
against the dipsomanie virus. This severe disease leads to the inability to
defend against Zombies, later causes a complete loss of memory and finally
turns you into one of them. Inexplicably Donn forgot where he put the
license key for his centrifuge. Provide him a new one and humanity will owe
you a debt of gratitude for fighting one of the most wicked illnesses
today.
https://ctf.fluxfingers.net/challenges/donn_beach.exe
ctf.fluxfingers.net tcp/2055
credits: 500 +3 (1st), +2 (2nd), +1 (3rd)

This Win32 executable starts by asking a name, hashing it and comparing it to a
constant, then asks a key, does several computations on it using a VM
obfuscated with SSE3 instructions, and compares the result of these
computations to four integer constants.

We can safely patch the name hashing and make sure that the name hash value is
the constant we want - using hardware breakpoints, we can see that the name
itself isn't used later, but the name hash is. The key is composed of three 32
bits integers, read from stdin like this (in hex): AAAAAAAA-BBBBBBBB-CCCCCCCC.

Before running code in the VM, the executable initializes the VM state with all
the inputs to the algorithm:

Name hash

First part of the key (key1)

Second part of the key (key2)

Third part of the key (key3)

Pointer to the current instruction

Pointer to a constant 256 bytes array

Stack pointer (points to freshly allocated memory)

After running the VM code, it unpacks these values from the state and stores
them back to stack variables. They are then compared to the constant values.

Looking inside the VM code a bit closer for 1 hour, and stepping into it with a
debugger, we can notice several interesting things:

The bytecode is interlaced with the VM code in the binary. The x86 code
regularly contains long multi-byte NOPs, in which the VM code is placed. The
VM simply ignores any instruction it does not know and skips to the next
byte, so it will only execute the instructions from inside the NOPs.

The VM state is contained in MMX registers mm0 to mm3, scrambled. Bytes
of each of the VM 8 32 bits registers are shuffled to fill these 4 64 bits
registers.

The instruction pointer always goes forward, and there does not seem to be
anything that increments it with a non constant increment. This means the VM
does not support jumps of any sort, so the logic inside of the VM is very
reduced.

The 8 VM registers initially contain the following values:

Reg 0: Constant 256 bytes array ptr

Reg 1: Name hash

Reg 2: key1

Reg 3: key2

Reg 4: key3

Reg 5: 0

Reg 6: Stack pointer

Reg 7: EIP

Something also makes our life a lot easier: inside the instruction handlers, to
read a register value, the code does not inline the SSE instructions to
unshuffle and unpack the register. Instead, it gets a function pointer from a
table which contains 8 register read functions (one for each register), and
calls that function to get the register value in mm4. The same can be
observed for register writes. This allows us to very easily notice the
instructions reading and writing to registers.

Using all of these infos, I started to statically reverse engineer all the
instruction handlers present in the binary. After one additional hour of work
and a lot of laughs after I was rickrolled by an instruction handler, a
disassembler was ready:

Now that we have the algorithm, we still need to generate a key that will
result in valid values in the end. As I was lazy and it was getting late in the
evening, I implemented the algorithm with Z3Py and asked it to solve the
problem for me. Unfortunately I failed several times to implement the
algorithm, and the iteration time was quite long because Z3 needed 20-30
minutes to get me a key matching my description of the problem, so we only got
the answer in the morning.

After running for 20 minutes, this gave me the following valid key:
e5304760-47b7c45f-f59a8f29.

Later, a friend tried to find a better way to solve this problem, and noticed
that it was reductible to a 32 bits bruteforce. Using this method, we found the
previous key, but also a second valid key: b6b09bf0-f23daa06-ac4ee747.