3 Answers
3

I saw what you did there, asking about memory allocation & security then mentioning pointers ;)

In my opinion, the best way to start off learning about memory (in general) is to write code. I'm not sure there is any other way of approaching this. Not just any code, find a language which forces memory management on the programmer, no managed environments such as Java or the .NET Framework. These are great languages for RAD, but don't provide the lower level aspects that would be required. I recommend C or C++. Learn about pointers, dangling references, dynamic memory allocation & deallocation, etc.

Assembly is a fantastic language if you want to learn the ins and outs of how software runs, but it also is hella-difficult to understand. You don't necessarily need to be able to write assembly code, though. Just pickup a few documents on the language and I guarantee you'll increase your understanding of memory substantially.

Sometimes the only way to truly understand how something works, is to take it apart. Pick up a book on operating system architecture such as Operating Systems - Design and Implementation or a reverse engineering book Reversing: Secrets of Reverse Engineering. Take some stuff apart, become familiar with a debugger and analyze applications running in memory. Start toying around with breakpoints and see what your debugger tells you about whats going on.

In conclusion, the best way in my opinion to learn about memory is to be responsible for managing it. Start off slow, learn C++ and work towards harder stuff.

Buffer Overflows might be what you are trying to refer to. Once you figure out exactly what it is you want to learn, you may want to build your own testing environment to observe how it works. DVWA is a great place to start with that.

The memory allocator may influence security in the following way: the choice of memory allocator may affect the ease or difficulty of exploiting buffer overrun and other memory-safety-related vulnerabilities.

For instance, randomizing the location at which objects are allocated can disrupt exploits that rely upon these objects to be at predictable locations -- thus defeating the attacker. Thus, a well-designed memory allocator can play a role in implementing ASLR (address-space layout randomization). In general, a memory allocator can potentially help defend against some "heap exploits".

See, e.g., the following research papers for some work on building hardened memory allocators that make it harder to exploit these kinds of vulnerabilities:

Kernel Pool Exploitation on Windows 7, Tarjei Mandt, Blackhat 2011. The Windows 7 kernel's memory allocator was designed to try to stop exploitation of certain kinds of memory-safety vulnerabilities. This paper finds that the protection is imperfect and describes how an attacker could defeat the protection under certain conditions.

Here's some more information about how deployed operating systems are implementing these ideas: