Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It's 100% free, no registration required.

I am trying to find an encryption algorithm for a game client, but I need to find the actual register/stack in the client before I can decrypt/encrypt the packets correctly (on my emulated server side). The problem is that I have no idea how to find the function.

In OllyDbg, is there a way I can trace during my debugging in order to find the function right after user input? The user input is a string converted to a char array, which is then encrypted in the executable. In this case, it would mean sending a packet to the game client, and then the game client decrypts the packet.

What do you mean by "find the function right after user input"? On which side is the input - client-side or server-side? After that, do you mean the function that directly follows the input, a function that encrypts the input before sending the packet to the server, or a function that decrypts the packet received by the client? Please clarify in your question itself (i.e. not in a comment). Also, please give a bit more explanation on how it all works.
–
JMcAfreakAug 2 '13 at 15:32

Do you have access to both server and client side binaries?
–
PSSAug 8 '13 at 12:15

4 Answers
4

There is clearly not enough of information to get you the exact answer. However, I will try my best to demonstrate logic behind what happens. That in a way might give you an idea how to go about locating the encryption function. I will assume you are using Windows, since it has not being stated otherwise.

The main logic of ordinary (and when I say ordinary, I mean there is no deliberate attempt to obfuscate or conceal normal execution flow) server/client TCP/IP network application boils down to the following:

It is obvious, Send and Receive on both client and server side is where all of the exchange happening. Usually, data will get encrypted by one side (client or server). Shortly thereafter, it will get transmitted to the other side. Other side will receive the data and decrypt it:

data -> encrypt -> send <----> receive -> decrypt -> data

In order to find a function that performs encryption, you need to locate send() function, determine location of the buffer being sent, go back and see what function sets(creates/modifies) that particular buffer location.

OllyDbg comes with some features that particularly useful in this situation. Firstly, we need to find all send() and receive() functions. We will search for intermodular calls. In disassembly window right click somewhere to bring up the following menu:

Select Search For -> All Intermodular Calls. It will bring up the window with all of the calls found. We specifically are looking for WS2_32.recv:

Set breakpoint on the function by pressing F2. Run the binary. Whenever WS2_32.recv breakpoint hits you can examine the stack window and determine location of the buffer:

We can see, that our buffer is at location 0x001CFBB0 with the size of the buffer at 0x80, which is 128 bytes. Let's go to to the buffer location by pressing Ctrl + G and typing location. We get our buffer:

Now, you need to step through and see whatever happens to that buffer, and data. I would assume one of the next function will try to decrypt it.

Once you narrow down to specifics you should next concentrate on narrowing down the message processing loop. Is it a classic while (TRUE) loop with switch cases for each message (Windows Message Pump) etc etc

Once the scope is narrowed down to a manageable size, several options exist to trace them.
For example with:

Windbg you can use ( wt watch and trace when you are on a function prologue ) it will execute the function and provide you a list of subfunctions

with ollydbg you can use run trace feature or hit trace feature. Several plugins exist that enhance this feature

for ida look for Trace Replayer documentation

I hope you are on windows. If you are on another OS then GDB also has certain tracing features available.