CPU Affinity

Bind specific processes to specific processors with a new system call.

The ability in Linux to bind one or more
processes to one or more processors, called CPU affinity, is a
long-requested feature. The idea is to say “always run this
process on processor one” or “run these processes on all
processors but processor zero”. The scheduler then obeys the
order, and the process runs only on the allowed processors.

Other operating systems, such as Windows NT, have long
provided a system call to set the CPU affinity for a process.
Consequently, demand for such a system call in Linux has been high.
Finally, the 2.5 kernel introduced a set of system calls for
setting and retrieving the CPU affinity of a process.

In this article, I look at the reasons for introducing a CPU
affinity interface to Linux. I then cover how to use the interface
in your programs. If you are not a programmer or if you have an
existing program you are unable to modify, I cover a simple utility
for changing the affinity of a given process using its PID.
Finally, we look at the actual implementation of the system
call.

Soft vs. Hard CPU Affinity

There are two types of CPU affinity. The first, soft
affinity, also called natural affinity, is the tendency of a
scheduler to try to keep processes on the same CPU as long as
possible. It is merely an attempt; if it is ever infeasible, the
processes certainly will migrate to another processor. The new O(1)
scheduler in 2.5 exhibits excellent natural affinity. On the
opposite end, however, is the 2.4 scheduler, which has poor CPU
affinity. This behavior results in the ping-pong effect. The
scheduler bounces processes between multiple processors each time
they are scheduled and rescheduled. Table 1 is an example of poor
natural affinity; Table 2 shows what good natural affinity looks
like.

Hard affinity, on the other hand, is what a CPU affinity
system call provides. It is a requirement, and processes must
adhere to a specified hard affinity. If a processor is bound to CPU
zero, for example, then it can run only on CPU zero.

Why One Needs CPU Affinity

Before we cover the new system calls, let's discuss why
anyone would need such a feature. The first benefit of CPU affinity
is optimizing cache performance. I said the O(1) scheduler tries
hard to keep tasks on the same processor, and it does. But in some
performance-critical situations—perhaps a large database or a
highly threaded Java server—it makes sense to enforce the affinity
as a hard requirement. Multiprocessing computers go through a lot
of trouble to keep the processor caches valid. Data can be kept in
only one processor's cache at a time. Otherwise, the processor's
cache may grow out of sync, leading to the question, who has the
data that is the most up-to-date copy of the main memory?
Consequently, whenever a processor adds a line of data to its local
cache, all the other processors in the system also caching it must
invalidate that data. This invalidation is costly and unpleasant.
But the real problem comes into play when processes bounce between
processors: they constantly cause cache invalidations, and the data
they want is never in the cache when they need it. Thus, cache miss
rates grow very large. CPU affinity protects against this and
improves cache performance.

A second benefit of CPU affinity is a corollary to the first.
If multiple threads are accessing the same data, it might make
sense to bind them all to the same processor. Doing so guarantees
that the threads do not contend over data and cause cache misses.
This does diminish the performance gained from multithreading on
SMP. If the threads are inherently serialized, however, the
improved cache hit rate may be worth it.

The third and final benefit is found in real-time or
otherwise time-sensitive applications. In this approach, all the
system processes are bound to a subset of the processors on the
system. The specialized application then is bound to the remaining
processors. Commonly, in a dual-processor system, the specialized
application is bound to one processor, and all other processes are
bound to the other. This ensures that the specialized application
receives the full attention of the processor.

Getting the New System Calls

The system calls are new, so they are not available yet in
all systems. You need at least kernel 2.5.8-pre3 and glibc 2.3.1;
glibc 2.3.0 supports the system calls, but it has a bug. The system
calls are not yet in 2.4, but patches are available at
www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity.

Many distribution kernels also support the new system calls.
In particular, Red Hat 9 is shipping with both kernel and glibc
support for the new calls. Real-time solutions, such as MontaVista
Linux, also fully support the new interface.

Comments

Comment viewing options

Thank you...
I have worked in the past on commercial flight simulators where everything relies on real time processing architectures (essentially 80's workstation multi-processor /w shared memory bus tech). I have always wondered what it would be like to leverage real-time processing capabilities on a PC without the OS getting in the way.

Not only did you give the best description of the application of affinity I've seen so far, you answered my 'how do you limit the os to one processor' question.

I can think of so many applications where this would be useful. Ex, performance testing that, for the first time, can be accurately measure without the margin of error typically included by os task switching on the same core.

I really hope that an affinity tool will eventually be added to the vanilla kernel. It would be hugely beneficial to have a tool like yours (with added capability to view affinity) that could easily manage processor affinity.

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.