Calculating an Integer Overflow

I was playing an exploit game yesterday, and had to compute an exact value for an integer overflow, which made me think (when I’ve run into this before, I’ve just had to get ‘close enough’). In the binary, it compares some user input to the integer 9, which it must be “less than”

The vulnerability is possible (at least in part) to the shl, which can be used to wrap the integer and bypass the jle check. It’s fairly obvious there is an integer overflow here, and in fact, calculating n to be an exact value is also not difficult. So in my case I wanted n in the memcpy call to equal exactly 80.

The very first thing I did was to look at this http://en.wikipedia.org/wiki/Two’s_complement, which I remember having to do in school. It’s not complicated, but once you start throwing algebra in… anyway, so instead of using math I just wrote a wrapper program on the same machine.

Then just plop this in a calculator. Remember to divide by 4 to undo the multiply

>>> (-2147483647*2 + 78)/4.0
-1073741804.0

I entered this in the appropriate place, and set a breakpoint on the call to memcpy.

(gdb) x/d $esp+8
0xbffff2b8: 80

Success, we’ve managed to set n to 80. This one took more time to write out than to solve, but hey, maybe it will be useful for someone. Plus I needed a filler today… I have some cool stuff I’m working on, but it won’t be ready until at least next post, or maybe the post after :)