Collective communications transmit data among all processes
in a group specified by an intracommunicator object.
One function, the barrier,
serves to synchronize processes without passing data.
MPI provides the following collective communication functions.
collective communication functions.

o Barrier synchronization across all group members (Section 4.4)
o Global communication functions, which are illustrated in Figure 4.1.
They include.
-Broadcast from one member to all members of a group (Section 4.5)
-Gather data from all group members to one member(Section 4.6)
-Scatter data from one member to all members of a group (Section 4.7)
-A variation on Gather where all members of the group receive the result
(Section 4.8) this is shown as "allgather" in Figure 4.1.
-Scatter/Gather data from all members to all members of a group
(also called complete exchange or all-to-all) (Section 4.9). This
is shown as "alltoall" in Figure4.1
o Global reduction operations such as sum,max,min,or user-defined
functions.
This includes
-Reduction where the result is returned to all group members and
a variation where the result is returned to one member (Section4.10)
-A combined reduction and sctter operation(Section4.10.5)
-Scan across all members of a group (also called prefix)(Section4.11)

Figure: Collective move functions illustrated
for a group of six processes. In each case, each row of boxes
represents data locations in one process. Thus, in the broadcast,
initially just the first process contains the item Ao, but after the
broadcast all processes contain it.

Figure gives a pictorial representation of the
global communication functions. All these functions (broadcast
excepted) come in two variants: the simple variant, where all
communicated items are messages of the same size, and the ``vector''
variant, where each item can be of a different size. In addition, in
the simple variant, multiple items originating from the same process
or received at the same process, are contiguous in memory; the vector
variant allows to pick the distinct items from non-contiguous
locations.
collective, vector variants

Some of these functions, such as broadcast or gather, have a single
origin or a single receiving process. Such a process is called the
root.
root
Global communication functions basically comes in
three patterns:

Each process communicates with each process (itself included):
allgather and alltoall.

The syntax and semantics of the MPI collective functions was
designed to be consistent with point-to-point communications.
collective, compatibility with point-to-point
However, to keep the number of functions and their argument lists
to a reasonable level of complexity, the MPI committee made
collective functions more
restrictive than the point-to-point functions, in several ways. One
collective, restrictions
restriction is that, in contrast to point-to-point communication,
the amount of data
sent must exactly match the amount of data specified by the receiver.

A major simplification is that collective functions come in blocking
versions only. Though a standing joke at committee meetings concerned
the ``non-blocking barrier,'' such functions can be quite useful and may be included in a future version of MPI.
collective, and blocking semanticsblocking

Collective functions do not use a tag argument. Thus, within
each intragroup communication domain, collective calls are matched
strictly according to the order of execution.
tagcollective, and message tag

A final simplification of collective functions concerns modes. Collective
collective, and modesmodes
functions come in only one mode, and this mode may be regarded as
analogous to the standard mode of point-to-point. Specifically, the
semantics are as follows. A collective function (on a given process)
can return as soon as its participation in the overall communication
is complete. As usual, the completion indicates that the caller is now
free to access and modify locations in the communication buffer(s).
It does not indicate that other processes have completed, or even started,
the operation. Thus, a collective communication may, or may not, have
the effect of synchronizing all calling processes. The barrier, of course,
is the exception to this statement.

This choice of semantics was made so as to allow a variety of implementations.

The user of MPI must keep these issues in mind. For example, even
though a particular implementation of MPI may provide a broadcast with the
side-effect of synchronization (the standard allows this), the standard
does not require this, and hence, any program that relies on the
synchronization will be non-portable. On the other hand, a correct and
portable program must allow a collective function to be synchronizing.
Though one should not rely on synchronization side-effects, one must
program so as to allow for it.
portabilitycorrectness
collective, and portabilitycollective, and correctness

Though these issues and statements may seem unusually obscure, they are
merely a consequence of the desire of MPI to:

allow efficient implementations on a variety of architectures; and,

be clear about exactly what is, and what is not, guaranteed by the standard.