For the most part, multiprocessor operating systems are just regular operating systems. They handle system calls, do memory management, provide a file system, and manage I/O devices. Nevertheless, there are some areas in which they have unique features. These include process synchronization, resource management, and scheduling. This chapter excerpt takes a brief look at multiprocessor hardware and then moves on to these operating systems issues.

From the author of

From the author of

A shared-memory multiprocessor (or just multiprocessor henceforth) is
a computer system in which two or more CPUs share full access to a common RAM. A
program running on any of the CPUs sees a normal (usually paged) virtual address
space. The only unusual property this system has is that the CPU can write some
value into a memory word and then read the word back and get a different value
(because another CPU has changed it). When organized correctly, this property
forms the basis of interprocessor communication: one CPU writes some data into
memory and another one reads the data out.

Below we will first take a brief look at
multiprocessor hardware and then move on to the unique issues facing multiprocessor operating systems.

8.1.1 Multiprocessor Hardware

Although all multiprocessors have the property that every CPU can address all
of memory, some multiprocessors have the additional property that every memory
word can be read as fast as every other memory word. These machines are called
UMA (Uniform Memory Access) multiprocessors. In contrast,
NUMA (Nonuniform Memory Access) multiprocessors do not have this
property. Why this difference exists will become clear later. We will first
examine UMA multiprocessors and then move on to NUMA multiprocessors.

UMA Bus-Based SMP Architectures

The simplest multiprocessors are based on a single bus, as illustrated in
Fig. 8-1(a). Two or more CPUs and one or more memory modules all use the same
bus for communication. When a CPU wants to read a memory word, it first checks
to see if the bus is busy. If the bus is idle, the CPU puts the address of the
word it wants on the bus, asserts a few control signals, and waits until the
memory puts the desired word on the bus.

If the bus is busy when a CPU wants to read or write memory, the CPU just
waits until the bus becomes idle. Herein lies the problem with this design. With
two or three CPUs, contention for the bus will be manageable; with 32 or 64 it
will be unbearable. The system will be totally limited by the bandwidth of the
bus, and most of the CPUs will be idle most of the time.

The solution to this problem is to add a cache to each CPU, as depicted in
Fig. 8-1(b). The cache can be inside the CPU chip, next to the CPU chip, on the
processor board, or some combination of all three. Since many reads can now be
satisfied out of the local cache, there will be much less bus traffic, and the
system can support more CPUs. In general, caching is not done on an individual
word basis but on the basis of 32- or 64-byte blocks. When a word is referenced,
its entire block is fetched into the cache of the CPU touching it.

Each cache block is marked as being either read-only (in which case it can be
present in multiple caches at the same time), or as read-write (in which case it
may not be present in any other caches). If a CPU attempts to write a word that
is in one or more remote caches, the bus hardware detects the write and puts a
signal on the bus informing all other caches of the write. If other caches have
a ''clean'' copy, that is, an exact copy of what is in
memory, they can just discard their copies and let the writer fetch the cache
block from memory before modifying it. If some other cache has a
''dirty'' (i.e., modified) copy, it must either write it
back to memory before the write can proceed or transfer it directly to the
writer over the bus. Many cache transfer protocols exist.

Yet another possibility is the design of Fig. 8-1(c), in which each CPU has
not only a cache, but also a local, private memory which it accesses over a
dedicated (private) bus. To use this configuration optimally, the compiler
should place all the program text, strings, constants and other read-only data,
stacks, and local variables in the private memories. The shared memory is then
only used for writable shared variables. In most cases, this careful placement
will greatly reduce bus traffic, but it does require active cooperation from the
compiler.

UMA Multiprocessors Using Crossbar Switches

Even with the best caching, the use of a single bus limits the size of a UMA
multiprocessor to about 16 or 32 CPUs. To go beyond that, a different kind of
interconnection network is needed. The simplest circuit for connecting n
CPUs to k memories is the crossbar switch, shown in Fig. 8-2.
Crossbar switches have been used for decades within telephone switching
exchanges to connect a group of incoming lines to a set of outgoing lines in an
arbitrary way.

At each intersection of a horizontal (incoming) and vertical (outgoing) line
is a crosspoint. A crosspoint is a small switch that can be electrically
opened or closed, depending on whether the horizontal and vertical lines are to
be connected or not. In Fig. 8-2(a) we see three crosspoints closed
simultaneously, allowing connections between the (CPU, memory) pairs (001, 000),
(101, 101), and (110, 010) at the same time. Many other combinations are also
possible. In fact, the number of combinations is equal to the number of
different ways eight rooks can be safely placed on a chess board.

One of the nicest properties of the crossbar switch is that it is a
nonblocking network, meaning that no CPU is ever denied the connection it
needs because some crosspoint or line is already occupied (assuming the memory
module itself is available). Furthermore, no advance planning is needed. Even if
seven arbitrary connections are already set up, it is always possible to connect
the remaining CPU to the remaining memory.

One of the worst properties of the crossbar switch is the fact that the
number of crosspoints grows as n2. With 1000 CPUs and
1000 memory modules we need a million crosspoints. Such a large crossbar switch
is not feasible. Nevertheless, for medium-sized systems, a crossbar design is
workable.

UMA Multiprocessors Using Multistage Switching Networks

A completely different multiprocessor design is based on the humble 2 X 2
switch shown in Fig. 8-3(a). This switch has two inputs and two outputs.
Messages arriving on either input line can be switched to either output line.
For our purposes, messages will contain up to four parts, as shown in Fig.
8-3(b). The Module field tells which memory to use. The Address
specifies an address within a module. The Opcode gives the operation,
such as READ or WRITE. Finally, the optional Value field may contain an
operand, such as a 32-bit word to be written on a WRITE. The switch inspects the
Module field and uses it to determine if the message should be sent on
X or on Y.

Our 2 X 2 switches can be arranged in many ways to build larger multistage
switching networks (Adams et al., 1987; Bhuyan et al., 1989; and Kumar and
Reddy, 1987). One possibility is the no-frills, economy class omega
network, illustrated in Fig. 8-4. Here we have connected eight CPUs to eight
memories using 12 switches. More generally, for n CPUs and n
memories we would need log2n stages, with n/2
switches per stage, for a total of (n/2)log2n
switches, which is a lot better than n 2
crosspoints, especially for large values of n.

The wiring pattern of the omega network is often called the perfect
shuffle, since the mixing of the signals at each stage resembles a deck of
cards being cut in half and then mixed card-for-card. To see how the omega
network works, suppose that CPU 011 wants to read a word from memory module 110.
The CPU sends a READ message to switch 1D containing 110 in the Module
field. The switch takes the first (i.e., leftmost) bit of 110 and uses it
for routing. A 0 routes to the upper output and a 1 routes to the lower one.
Since this bit is a 1, the message is routed via the lower output to 2D.

All the second-stage switches, including 2D, use the second bit for routing.
This, too, is a 1, so the message is now forwarded via the lower output to 3D.
Here the third bit is tested and found to be a 0. Consequently, the message goes
out on the upper output and arrives at memory 110, as desired. The path followed
by this message is marked in Fig. 8-4 by the letter a.

As the message moves through the switching network, the bits at the left-hand
end of the module number are no longer needed. They can be put to good use by
recording the incoming line number there, so the reply can find its way back.
For path a, the incoming lines are 0 (upper input to 1D), 1 (lower input
to 2D), and 1 (lower input to 3D), respectively. The reply is routed back using
011, only reading it from right to left this time.

At the same time all this is going on, CPU 001 wants to write a word to
memory module 001. An analogous process happens here, with the message routed
via the upper, upper, and lower outputs, respectively, marked by the letter
b. When it arrives, its Module field reads 001, representing the
path it took. Since these two requests do not use any of the same switches,
lines, or memory modules, they can proceed in parallel.

Now consider what would happen if CPU 000 simultaneously wanted to access
memory module 000. Its request would come into conflict with CPU 001's
request at switch 3A. One of them would have to wait. Unlike the crossbar
switch, the omega network is a blocking network. Not every set of
requests can be processed simultaneously. Conflicts can occur over the use of a
wire or a switch, as well as between requests to memory and replies
from memory.

It is clearly desirable to spread the memory references uniformly across the
modules. One common technique is to use the low-order bits as the module number.
Consider, for example, a byte-oriented address space for a computer that mostly
accesses 32-bit words. The 2 low-order bits will usually be 00, but the next 3
bits will be uniformly distributed. By using these 3 bits as the module number,
consecutively addressed words will be in consecutive modules. A memory system in
which consecutive words are in different modules is said to be
interleaved. Interleaved memories maximize parallelism because most
memory references are to consecutive addresses. It is also possible to design
switching networks that are nonblocking and which offer multiple paths from each
CPU to each memory module, to spread the traffic better.

Single-bus UMA multiprocessors are generally limited to no more than a few
dozen CPUs and crossbar or switched multiprocessors need a lot of (expensive)
hardware and are not that much bigger. To get to more than 100 CPUs, something
has to give. Usually, what gives is the idea that all memory modules have the
same access time. This concession leads to the idea of NUMA multiprocessors, as
mentioned above. Like their UMA cousins, they provide a single address space
across all the CPUs, but unlike the UMA machines, access to local memory modules
is faster than access to remote ones. Thus all UMA programs will run without
change on NUMA machines, but the performance will be worse than on a UMA machine
at the same clock speed.

NUMA machines have three key characteristics that all of them possess and
which together distinguish them from other multiprocessors:

There is a single address space visible to all CPUs.

Access to remote memory is via LOAD and STORE instructions.

Access to remote memory is slower than access to local memory.

When the access time to remote memory is not hidden (because there is no
caching), the system is called NC-NUMA. When coherent caches are present,
the system is called CC-NUMA (Cache-Coherent NUMA).

The most popular approach for building large CC-NUMA multiprocessors
currently is the directory-based multiprocessor. The idea is to maintain
a database telling where each cache line is and what its status is. When a cache
line is referenced, the database is queried to find out where it is and whether
it is clean or dirty (modified). Since this database must be queried on every
instruction that references memory, it must be kept in extremely-fast
special-purpose hardware that can respond in a fraction of a bus cycle.

To make the idea of a directory-based multiprocessor somewhat more concrete,
let us consider as a simple (hypothetical) example, a 256-node system, each node
consisting of one CPU and 16 MB of RAM connected to the CPU via a local bus. The
total memory is 232 bytes, divided up into 226 cache lines
of 64 bytes each. The memory is statically allocated among the nodes, with
016M in node 0, 16M32M in node 1, and so on. The nodes are connected
by an interconnection network, as shown in Fig. 8-5(a). Each node also holds the
directory entries for the 218 64-byte cache lines comprising its
224 byte memory. For the moment, we will assume that a line can be
held in at most one cache.

To see how the directory works, let us trace a LOAD instruction from CPU 20
that references a cached line. First the CPU issuing the instruction presents it
to its MMU, which translates it to a physical address, say, 0x24000108. The MMU
splits this address into the three parts shown in Fig. 8-5(b). In decimal, the
three parts are node 36, line 4, and offset 8. The MMU sees that the memory word
referenced is from node 36, not node 20, so it sends a request message through
the interconnection network to the line's home node, 36, asking whether its
line 4 is cached, and if so, where.

When the request arrives at node 36 over the interconnection network, it is
routed to the directory hardware. The hardware indexes into its table of
218 entries, one for each of its cache lines and extracts entry 4.
From Fig. 8-5(c) we see that the line is not cached, so the hardware fetches
line 4 from the local RAM, sends it back to node 20, and updates directory entry
4 to indicate that the line is now cached at node 20.

Now let us consider a second request, this time asking about node 36's
line 2. From Fig. 8-5(c) we see that this line is cached at node 82. At this
point the hardware could update directory entry 2 to say that the line is now at
node 20 and then send a message to node 82 instructing it to pass the line to
node 20 and invalidate its cache. Note that even a so-called
''shared-memory multiprocessor'' has a lot of message
passing going on under the hood.

As a quick aside, let us calculate how much memory is being taken up by the
directories. Each node has 16 MB of RAM and 218 9-bit entries to keep
track of that RAM. Thus the directory overhead is about 9 X 218 bits
divided by 16 MB or about 1.76 percent, which is generally acceptable (although
it has to be high-speed memory, which increases its cost). Even with 32-byte
cache lines the overhead would only be 4 percent. With 128-byte cache lines, it
would be under 1 percent.

An obvious limitation of this design is that a line can be cached at only one
node. To allow lines to be cached at multiple nodes, we would need some way of
locating all of them, for example, to invalidate or update them on a write.
Various options are possible to allow caching at several nodes at the same time,
but a discussion of these is beyond the scope of this book.