Using GOP to Achieve MPI_Allreduce Functionality

In this example, we look at the gop function and the functions that build on it: gplus and gcat. These seemingly simple functions turn out to be very powerful tools in parallel programming.

The gop function allows us to perform any associative binary operation on a variable that is defined on all the labs. This allows us not only to sum a variable across all the labs, but also to find its minimum and maximum across the labs, concatenate them, and perform many other useful operations.

When doing parallel programming, we often run into the situation of having a variable defined on all the labs, and we want to perform an operation on the variable as it exists on all the labs. For example, if we enter an spmd statement and define

on all the labs, we might want to calculate the sum of the values of x across the labs. This is exactly what the gplus operation does, it sums the x across the labs and duplicates the result on all labs:

spmd
s = gplus(x);
end

The variables assigned to inside an spmd statement are represented on the client as Composite. We can bring the resulting values from the labs to the client by indexing into the Composite much like that of cell arrays:

s{1} % Display the value of s on lab 1. All labs store the same value.

ans =
78

Also, gop, gplus, and gcat allow us to specify a single lab to which the function output should be returned, and they return an empty vector on the other labs.

spmd
s = gplus(x, 1);
end
s{1}

ans =
78

This example shows how to perform a host of operations similar to addition across all the labs. In MPI, these are known as collective operations, such as MPI_SUM, MPI_PROD, MPI_MIN, MPI_MAX, etc.

Create the Input Data for Our Examples

The data we use for all our examples is very simple: a 1-by-2 variant array that is only slightly more complicated than the x we defined in the beginning:

Now that we have initialized our vector x to different values on the labs, we can ask questions such as what is the element-by-element sum of the values of x across the labs? What about the product, the minimum, and the maximum? As to be expected from our introduction,

spmd
s = gplus(x);
end
s{1}

ans =
90 102

returns the element-by-element addition of the values of x. However, gplus is only a special case of the gop operation, short for Global OPeration. The gop function allows us to perform any associative operation across the labs on the elements of a variant array. The most basic example of an associative operation is addition; it is associative because addition is independent of the grouping which is used:

(a + b) + c = a + (b + c)

In MATLAB®, addition can be denoted by the @plus function handle, so we can also write gplus(x) as

spmd
s = gop(@plus, x);
end
s{1}

ans =
90 102

We can concatenate the vector x across the labs by using the gcat function, and we can choose the dimension to concatenate along.

To conclude our tour of the associative operations that are built into MATLAB, we look at the bitwise AND, OR, and XOR operations. These are represented by the @bitand, @bitor, and @bitxor function handles.