Some background information must be covered first, before we can start talking about the attack outright.

The return address is a memory location that will have control transferred to it when the retopcode is executed. This is used to implement functions; when a program uses call foo to launch a function foo, the computer will push the address of the next instruction into the stack, and perform other diagnostic operations before control is transferred to foo. When foo returns via ret, the return address is popped off the stack and used to replace the instruction pointer, thereby relinquishing control to the instruction immediately after call foo, as it should.

When a function is called, it will typically do a number of things to set up the computer state so the function can... erm... function. One of these things would be to allocate some of the space on the stack for local variables. If, for example, I have a local character buffer of size 100 bytes, 100 bytes will be allocated off the top of the stack for this variable.

Thus, when the actual function code begins, the stack will look somewhat like this:

The way a stack buffer overflow works is that it uses the overflow to overwrite the return address. When a static-sized buffer has data larger than its size copied into it, the extra data is written into the memory space following it. This implies that if a sufficiently big chunk of data is copied into the buffer, it will cause the return address to be replaced.

Since the return address is meant to point to the code which foo should secede control to once it finishes, replacing the return address would allow an attacker to cause the computer to execute code from just about anywhere in the program's allocated memory space.

What most exploit programs do is replace the return address with a pointer to the beginning of the buffer. The data stored at the beginning of this buffer is actually some machine code which can perform various operations using the priviledge offered by the program being exploited. This is typically something that spawns a shell, aka shellcode.

While this seems to be quite simple, in reality it is much trickier. An exploit writer must take into consideration the size of buffer space available, as well as if any characters will be filtered out by the function before data is written into the buffer. Also, because the location of the buffer in memory is dynamic and can change with each separate run of the program, overwriting the return address with the memory location of the buffer requires various workarounds and to some extent guessing.