Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.

I am currently looking at a function inside a Win32 executable's main module. After allocating memory on the stack (sub esp) and saving some registers on the stack, the value of esp is XORed with a global variable.

mov eax, esp
xor eax, DWORD PTR ds:[<some address>]
push eax

I wonder whether this is some kind of stack protection technique?

Edit: I wrote the question from memory. The actual instruction sequence is:

2 Answers
2

Yes, this is an implementation of what's often called "Stack Canaries", a method of stack Buffer Overflow Protection. That example you're describing is specifically the method used by Visual Studio, enabled by default since Visual Studio 2005, implemented since Visual Studio 2003. It is also called GS protection due to the fact Visual Studio provided the flags, /GS to enable and /GS- to disable, that protection and override the default behavior.

What is stack buffer overflow protection?

There are multiple stack buffer overflow protection techniques implemented by different compilers and 3rd party protection tools, but they all revolve around the same fundamental idea: When a stack buffer overflow is exploited, the attacker often overwrites the return address located on the stack to redirect code execution to a controlled address. Stack canaries work by prefixing the ret instruction with some sort of validation that the stack, and specifically the return address, was not altered by an attacker prior to the ret instruction being executed (which will result in a pop from the stack and placing the popped value in the instruction pointer).

How does the provided example protect the stack from such exploits?

To answer that, we'll need to provide the full implementation details of which you just provided the first half. Most stack overflow canary protections usually include inserting function prolog and epilog, and you only provided the former.

This prolog starts with allocating the stack variable's space just as any "regular" function would. It then continues to fetch a value randomly generated at the process start, and stored in a specific memory address,
into register EAX and then xor-ing it with the current stack pointer. The resulting value is then stored on the stack.

Then, just before the function's ret instruction is executed, which under normal buffer overflow attack circumstances will fetch the overwritten return address and will execute attacker controlled code, a validation that the original stack cookie value was kept is made, triggering a safe failure in case the value differs from the expected stack cookie.

The logic in most of the cookie/canary based protections is the following:

On function entrance store a specific value, deterministic but unpredictable from the attacker's perspective. Preferably, one that's dependant on the function's execution conditions (such as the current ESP). That value should be placed between the function's return address and any buffer-overflow potential buffer or variable.

On function's exit, just before using the potentially overwritten return address, validate the stored cookie is same as expected, fail if it was somehow changed.

As classic buffer overflow attacks are rewriting the entire stack in a sequential manner (meaning that to overwrite the return address from stack variable buf, all data located on the stack between the return address and buf might also be overwritten), any overwrite of the return address must also modify the stack canary. As long as the attacker cannot predict the canary before triggering the function call, the attack will fail due to the canary validation performed just before ret is executed.

Was that the end of stack based buffer overflows?

No, for several reasons the battle over stack overflow exploitation and protection carried on (and is still ongoing, in certain circumstances):

Soon after the first canary protections implemented in Visual Studio, attacks against SEH exception structures (which are allocated on the stack to handle exceptions) began, and provided several anti-SEH buffer overflow protections (such as SafeSEH, which underwent several versions until it was fully reliable in protecting against such attacks including the later SEHOP).

Additionally, information leakage bugs were used to predict (and increase prediction chances) of canary values, which enabled bypassing the canary check and made stack based buffer overflows possible.

Slightly similar to #2 but specific to canary protections, in certain cases an attacker could exploit the flow of the process's execution to slowly extract the canary values byte by byte, thus reducing a 2**32 (4294967296 possibilities) brute force to only 256*4 (1024 possibilities) brute force, making many attacks a lot more plausible.

Buffer overflow bugs allowing non-linear overwrites were also being used, to "skip" overwriting the stack canary (or most of it) to either completely avoid the need to predict the canary value, or reduce the modification range to a lower, 1 byte range. Such common examples are looks that overwrite only on certain conditions or those whom overwrite only 1 byte of a dword. These would also make the return address modification limited but were still useful (and occasionally more so, in certain cases where ASLR was also used).

So, in contrast to Igor's answer, the return address is not immediately protected by xoring it with the cookie, but by checking whether the canary has been overwritten (assuming, as you stated, that a stack overflow will overwrite it)?
– InkassoHeinziSep 2 '17 at 18:19

In the code snippet you provided (which is more similar to the one I provided) the randomly generated cookie is XORed with the the ESP register (which points to the current location of the stack). In the code snippet Igor quoted, the cookie is XORed with where the return address is placed (on the stack). Both approaches work, and I can see slight advantages to either. However I think the main point to take from this answer is the overall reasoning and logic behind the stack cookie, and not the specific implementation used. The change might come from different protection versions.
– NirIzrSep 2 '17 at 18:27

The prolog contains an instruction that fetches a copy of the cookie,
followed by an instruction that does a logical xor of the cookie and
the return address, and then finally an instruction that stores the
cookie on the stack directly below the return address. From this point
forward, the function will execute as it does normally. When a function returns, the last thing to execute is the function’s epilog, which is the opposite of the prolog.

When compiled with /GS, the security checks are also placed in the
epilog:

The stack's copy of the cookie is retrieved and then follows with the
XOR instruction with the return address. The ECX register should
contain a value that matches the original cookie stored in the
__security_cookie variable. The stack space is then reclaimed, and then, instead of executing the RET instruction, the JMP instruction to
the __security_check_cookie routine is executed.