HackingWeek 2015 - Reverse 4

19 May 2015

How we managed to solve a Keygenme with Python and Z3 a high-performance theorem prover.

Introduction

This Reverse Engineering challenge was probably the most interesting one I ever solved. It took me a few days to complete it but it was totally worth it because I learned about symbolic execution and how to use Z3 The Theorem Prover.

I will write more articles about Z3 in a near future but until a later time I will explain how to solve this challenge.

Context

The goal of this Reverse Engineering challenge is to find the password for which the given ELF 64-bit binary will print "Success !".To make the solving a bit easier we are given a few hints :

The password must be alphanumeric [a-zA-Z0-9].

The md5sum of the expected password is 3efafa3e161a756e6e1711dbceaf9d68.

Static Analysis

We can say that the binary is quite small, it is less than 4 Kb, we won't have too much code to analyse.

The interesting part of the code is located between 0x04006A0 and 0x04008F9.We can find a hashing routine from 0x04006A0 to 0x040085D which we will name calculate and a routine to check if a hash is valid from 0x0400863 to 0x04008F9 which we will name verify.

Calculate

The routine calculate iterates for each bytes of the password.

The value of the hash is stored in RCX.

After 0x040070DRCX is always equals to 0x555555.

Verify

The routine verify takes RCX as only argument.

If the hash is valid the routine verify returns 1, otherwise it returns 0.

Finding the valid hash

So the first step is to find for which hash the routine verify will return 1.When the routine calculate jumps to verify.Because RCX is a 64 bit register, trying all values of RCX to find a valid hash would take too much time, we need to do something smarter.

Finding the valid hash with Z3

Z3 is a high-performance theorem prover being developed at Microsoft Research. It is able to solve or simplify complex equations. We are going to use Z3 Python binding to solve several steps of this challenge.

The first step was to convert the Assembly code to Python, since the code is not too large we can do it in a few minutes by hand.The second step is to make the script comply with Z3. Sometimes you have to specify between Arithmetic and Logic operations.

calculate output

The Python script will print a large amount of valid passwords but it might take days before it find the password we are looking for.We need to develop something faster than Z3 but with the same rules and less voodoo magic.

Finding the expected password

Z3 the theorem prover will help us to create rules, instead of using is_alphanum we will try all alphanumeric char on each RDX values.This will give us all possible characters for the password.

Get combinations

Using Z3 we are able to find all possible characters for a password of length 10.

Bruteforcing

I implemented my own script which will try all the combinations with the rules defined by Z3.During the competition I split the tasks on 8 threads to reduce the time of the attack because there are 49,015,612,800 possibilities and my laptop is rather slow.