All programmers write executable code for computers, but what sets system programmers apart from application programmers is the purpose of the software they write. Application programming produces software that makes a computer's hardware generate something for the user, be it a spreadsheet or the graphics for a game. System programming produces software that accesses and controls the inner workings of a computer's hardware and operating system.

Application programming generally involves issuing system commands to utilize the basic functions of a computer's hardware and operating system, such as storing a particular piece of data in the computer's physical memory or a file on the hard drive. These kinds of programs are not concerned with details of how the hard drive or physical memory are working. Conversely, system programmers concern themselves with the details of how an operating system and hardware components work. This allows them to build the software that defragments hard drives and checks the integrity of a computer's physical memory.

In addition to being able to build such tools, system programmers are typically experts in the core functioning of operating systems. All programmers are familiar with system calls, thread management, and input/output handling, but system programming requires the software engineer to be able to manipulate these operating system mechanisms. This allows a system programmer to perform specialized installations and automate system maintenance tasks.

Knowledge of an operating system's core is also necessary in order to maximize an application's performance on a particular hardware configuration. For example, very busy online retailers need their websites and transaction processing systems to run as efficiently, and reliably, as possible. Using his or her knowledge about the internal mechanics of operating systems and hardware components, such as how to make a particular operating system optimize its thread handling or which algorithms run fastest on which hardware components, a system programmer can help fine tune an application's performance.

This detailed access to the inner workings of hardware and operating system components requires system programming to be done in a language that allows this kind of low level hardware access. Languages such as Java®, Python™, or Ruby on Rails® are what programmers refer to as high level languages. This means that they make application programming easier by not making the programmer handle the fine details of hardware management. System programming requires exactly this kind of access, so system programmers use a low level language such as C or C++.