<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\setlength{\marginparwidth}{0.5in}
\setlength{\marginparsep}{0.3in}
\newcommand{\snir}{\marginpar[$\bf \top$]{
\mbox{\hspace{0.3in} $\bf \top$}}}
\newcommand{\rins}{\marginpar[$\bf \bot$]{
\mbox{\hspace{0.3in} $\bf \bot$}}}

\chapter{Point-to-Point Communication}
\label{sec:pt2pt}
\label{chap:pt2pt}
%\footnotetext[1]{Version of April 10, 1995}

\section{Introduction}
\label{sec:pt2pt-intro}

Sending and receiving of messages by processes is the basic \MPI/
communication mechanism.
The basic point-to-point communication operations are {\bf send} and {\bf
receive}.   Their use is illustrated in the example below.

\begin{verbatim}
#include "mpi.h"
main( argc, argv )
int argc;
char **argv;
{
    char message[20];
    int myrank;
    MPI_Status status;
    MPI_Init( &amp;argc, &amp;argv );
    MPI_Comm_rank( MPI_COMM_WORLD, &amp;myrank );
    if (myrank == 0)    /* code for process zero */
    {
        strcpy(message,"Hello, there");
        MPI_Send(message, strlen(message), MPI_CHAR, 1, 99, MPI_COMM_WORLD);
    }
    else                /* code for process one */
    {
        MPI_Recv(message, 20, MPI_CHAR, 0, 99, MPI_COMM_WORLD, &amp;status);
        printf("received :%s:\n", message);
    }
    MPI_Finalize();
}
\end{verbatim}

In this example, process zero ({\sf myrank = 0}) sends a message to process one
using the
{\bf send} operation \mpifunc{MPI\_SEND}. The
operation specifies a {\bf send buffer} in the sender memory from which the
message data is taken.  In the example above, the send buffer consists of the
storage containing the variable {\sf message} in the memory of process zero.
The location, size and type of the send buffer are specified by the first three
parameters of the send operation.  The message sent will contain the 13
characters of this variable.
In addition, the send operation associates an {\bf envelope} with the
message.  This envelope specifies the message destination and contains
distinguishing information that can be used by the {\bf receive} operation to
select a particular message.  The last three parameters of the send operation
specify the envelope for the message sent.

Process one ({\sf myrank = 1}) receives this message with the
{\bf receive} operation \mpifunc{MPI\_RECV}.
The message to be received is selected according to the value of its
envelope, and the message data is stored into the {\bf receive
buffer}.  In the example above, the receive buffer consists of the storage
containing the string {\sf message} in the memory of process one.
The first three parameters of the receive operation specify the location, size
and type of the receive buffer.  The next three
parameters are used for selecting the incoming message.   The last parameter is
used to return information on the message just received.

The next sections describe the blocking send and receive
operations.  We discuss send, receive, blocking communication semantics,
type matching requirements, type conversion in
heterogeneous environments, and more general communication modes.
Nonblocking communication is addressed next, followed by
channel-like constructs and send-receive operations.  We then consider general
datatypes that allow one to transfer efficiently
heterogeneous and noncontiguous data.
We conclude with the description of calls for explicit packing and unpacking of
messages.

\section{Blocking Send and Receive Operations}
\label{sec:pt2pt-basicsendrecv}
\subsection{Blocking send}
\label{subsec:pt2pt-basicsend}

The syntax of the blocking send operation is given below.

\begin{funcdef}{MPI\_SEND(buf, count, datatype, dest, tag, comm)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (nonnegative integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\end{funcdef}

\mpibind{MPI\_Send(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm)}

\mpifbind{MPI\_SEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, IERROR}

The blocking semantics of this call are described in Sec. \ref{sec:pt2pt-modes}.

\subsection{Message data}

\label{subsec:pt2pt-messagedata}


The send buffer specified by the \func{MPI\_SEND} operation consists of
\mpiarg{count} successive entries of the type indicated by
\mpiarg{datatype}, starting
with the entry at address \mpiarg{buf}.   Note that we specify the
message length
in terms of number of {\em elements}, not number of {\em bytes}.  The former is
machine independent and closer to the application level.

The data part of the message consists of a sequence of \mpiarg{count}
values, each of the type indicated by \mpiarg{datatype}.
\mpiarg{count} may be zero,
in which case the data part of the message is empty.
The basic
datatypes that can be specified for message data values correspond to
the basic datatypes of the host language.
Possible values of this argument for Fortran and the
corresponding Fortran types are listed below.

\vspace{5ex}

\begin{center}
\begin{tabular}{|l|l|}
\hline
MPI datatype &amp; Fortran datatype \\
\hline
\type{MPI\_INTEGER} &amp; \ftype{INTEGER} \\
\type{MPI\_REAL} &amp; \ftype{REAL} \\
\type{MPI\_DOUBLE\_PRECISION} &amp; \ftype{DOUBLE PRECISION} \\
\type{MPI\_COMPLEX} &amp; \ftype{COMPLEX} \\
%\type{MPI\_DOUBLE\_COMPLEX} &amp; \ftype{DOUBLE COMPLEX} \\
\type{MPI\_LOGICAL} &amp; \ftype{LOGICAL} \\
\type{MPI\_CHARACTER} &amp; \ftype{CHARACTER(1)} \\
\type{MPI\_BYTE} &amp; \\
\type{MPI\_PACKED} &amp; \\
\hline
\end{tabular}
\end{center}
\vspace{5ex}
Possible values for this argument for C and the corresponding C
types are listed below.
\vspace{5ex}
\begin{center}
\begin{tabular}{|l|l|}
\hline
MPI datatype &amp; C datatype \\
\hline
\type{MPI\_CHAR} &amp; \ctype{signed char} \\
\type{MPI\_SHORT} &amp; \ctype{signed short int} \\
\type{MPI\_INT} &amp; \ctype{signed int} \\
\type{MPI\_LONG} &amp; \ctype{signed long int} \\
\type{MPI\_UNSIGNED\_CHAR} &amp; \ctype{unsigned char} \\
\type{MPI\_UNSIGNED\_SHORT} &amp; \ctype{unsigned short int} \\
\type{MPI\_UNSIGNED} &amp; \ctype{unsigned int} \\
\type{MPI\_UNSIGNED\_LONG} &amp; \ctype{unsigned long int} \\
\type{MPI\_FLOAT} &amp; \ctype{float} \\
\type{MPI\_DOUBLE} &amp; \ctype{double} \\
\type{MPI\_LONG\_DOUBLE} &amp; \ctype{long double} \\
\type{MPI\_BYTE} &amp; \\
\type{MPI\_PACKED} &amp; \\
\hline
\end{tabular}
\end{center}

\vspace{5ex}

The datatypes \type{MPI\_BYTE} and \type{MPI\_PACKED} do not correspond to a
Fortran or C datatype.  A value of type \type{MPI\_BYTE} consists of a byte
(8 binary digits).  A
byte is uninterpreted and is different from a character.
Different machines may have
different representations for characters, or may use more than one
byte to represent characters.  On the other hand, a byte has the same
binary value on all machines.
The use of the type \type{MPI\_PACKED} is explained in
Section~\ref{sec:pt2pt-packing}.

\MPI/ requires support of the datatypes listed above, which match the basic
datatypes of Fortran 77 and ANSI C.
Additional \MPI/ datatypes should be provided if the host language has
additional data types:
\type{MPI\_LONG\_LONG\_INT}, for (64 bit) C integers declared to be of type
{\sf longlong int};
\type{MPI\_DOUBLE\_COMPLEX} for double precision complex in
Fortran declared to be of type \type{DOUBLE PRECISION};
\type{MPI\_REAL2},
\type{MPI\_REAL4} and \type{MPI\_REAL8} for Fortran reals, declared to be of
type \type{REAL*2}, \type{REAL*4} and \type{REAL*8}, respectively;
\type{MPI\_INTEGER1} \type{MPI\_INTEGER2} and
\type{MPI\_INTEGER4} for Fortran integers, declared to be of type
\type{INTEGER*1}, \type{INTEGER*2} and \type{INTEGER*4}, respectively; etc.

\begin{rationale}
One goal of the design is to allow for \MPI/ to be implemented as a library,
with no need for additional preprocessing or compilation.
Thus, one cannot assume that a communication call has information on the
datatype of variables in the communication buffer; this information must be
supplied by an explicit argument.
The need for such datatype information will become clear in
Section~\ref{subsec:pt2pt-conversion}.
\end{rationale}


\subsection{Message envelope}
\label{subsec:pt2pt-envelope}

In addition to the data part, messages carry information that can be used to
distinguish messages and selectively receive them.  This information consists
of a fixed number of fields, which we collectively call
the {\bf message envelope}.   These fields are
\begin{center}
source \\
destination \\
tag \\
communicator
\end{center}

The message source is implicitly determined by the identity of the
message sender.  The other fields are specified by arguments in the send
operation.

The message destination is specified by the \mpiarg{dest}
argument.

The integer-valued message tag is specified by the \mpiarg{tag} argument.
This integer can be used by the program to distinguish different types of
messages.
The range of valid tag values is {\sf 0,...,UB}, where the value of {\sf UB} is
implementation dependent. It can be found by querying the
value of the attribute \const{MPI\_TAG\_UB}, as
described in Chapter~\ref{chap:environment}.  \MPI/ requires that {\sf UB} be no
less than 32767.

The \mpiarg{comm} argument specifies the {\bf communicator} that is used for
the send operation.
Communicators are explained in Chapter~\ref{chap:context}; below is a brief
summary of their usage.

A communicator specifies the
communication context for a communication operation.
Each communication context provides a separate ``communication
universe:'' messages are always received within the context they were
sent, and messages sent in different contexts do not interfere.

The communicator also specifies the set of processes that share this
communication context.   This {\bf process group}
is ordered and processes are identified by their
rank within this group.  Thus, the range of valid values for \mpiarg{dest} is
{\sf 0, ... , n-1}, where {\sf n} is the number of
processes in the group.  (If the communicator is an inter-communicator, then
destinations are identified by their rank in the remote group.  See
Chapter~\ref{chap:context}.)

A predefined communicator \const{MPI\_COMM\_WORLD} is provided by \MPI/. It
allows communication with all processes that are accessible after \MPI/
initialization and processes are identified by their rank in the
group of \const{MPI\_COMM\_WORLD}.

\begin{users}
Users that are comfortable with the notion of a flat name space for
processes, and a single communication context, as offered by most
existing communication libraries,  need only use the predefined
variable \const{MPI\_COMM\_WORLD} as the \mpiarg{comm} argument.
This will allow communication with all the processes available at
initialization time.

Users may define new communicators, as explained in Chapter~\ref{chap:context}.
Communicators provide an important encapsulation mechanism for libraries and
modules.  They allow modules to have their own disjoint communication universe
and their own process numbering scheme.
\end{users}

\begin{implementors}
The message envelope
would normally be encoded by a fixed-length message header.
However, the actual encoding
is implementation dependent. Some of the information (e.g.,
source or destination) may be implicit,  and need not be
explicitly carried by messages. Also, processes may be identified by
relative ranks, or absolute ids, etc.
\end{implementors}

\subsection{Blocking receive}
\label{subsec:pt2pt-basicreceive}


The syntax of the blocking receive operation is given below.

\begin{funcdef}{MPI\_RECV (buf, count, datatype, source, tag, comm, status)}
\funcarg{\OUT}{buf}{initial address of receive buffer (choice)}
\funcarg{\IN}{count}{number of elements in receive buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each receive buffer element (handle)}
\funcarg{\IN}{source}{rank of source (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Recv(void*~buf, int~count, MPI\_Datatype~datatype, int~source, int~tag, MPI\_Comm~comm, MPI\_Status~*status)}

\mpifbind{MPI\_RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS(MPI\_STATUS\_SIZE), IERROR}

The blocking semantics of this call are described in Sec. \ref{sec:pt2pt-modes}.

The receive buffer consists of
the storage containing \mpiarg{count} consecutive elements of the type
specified by \mpiarg{datatype}, starting at address \mpiarg{buf}.
The length of the received message must be less than or equal to the length of
the receive buffer. An overflow error occurs if all incoming data does
not fit, without truncation, into the receive buffer.

If a
message that is shorter than the receive buffer arrives, then only those
locations corresponding to the (shorter) message are modified.

\begin{users}
The \mpifunc{MPI\_PROBE} function described in
Section~\ref{sec:pt2pt-probe} can be used to receive messages of
unknown length.
\end{users}

\begin{implementors}
Even though no specific behavior is mandated by \MPI/ for erroneous programs, the
recommended handling of overflow situations is to return in
\mpiarg{status} information about the
source and tag of the incoming message.  The receive operation will return an
error code.  A quality implementation will also ensure that no memory
that is outside the receive buffer will ever be overwritten.

In the case of a message shorter than the receive buffer, \MPI/ is
quite strict in that it allows no modification of the other locations.
A more lenient statement would allow for some
optimizations but this is not allowed.  The implementation must be ready to
end a copy into the receiver memory exactly at the end of the receive
buffer, even if it is an odd address.
\end{implementors}


The selection of a message by a receive operation is governed by
the value of the message envelope.
A message can be received by a receive operation
if its envelope matches the \mpiarg{source}, \mpiarg{tag} and
\mpiarg{comm} values specified by the
receive operation.  The receiver may specify a wildcard
\const{MPI\_ANY\_SOURCE}
value for \mpiarg{source}, and/or
a wildcard \const{ MPI\_ANY\_TAG} value for \mpiarg{tag},
indicating that any source and/or tag are acceptable.   It cannot specify a
wildcard value for \mpiarg{comm}.
Thus, a message can be received by a receive
operation only if it is addressed
to the receiving process, has a matching communicator, has
matching source unless
source=\const{ MPI\_ANY\_SOURCE}
in the pattern, and has a matching tag unless tag=\const{ MPI\_ANY\_TAG} in
the pattern.

The message tag is specified by the \mpiarg{tag} argument of the
receive operation.
The argument \mpiarg{source}, if different from \const{MPI\_ANY\_SOURCE},
is specified as a rank within the process group associated
with that same communicator (remote process group, for intercommunicators).
Thus, the range of valid values for the
\mpiarg{source} argument is
{\sf \{0,...,n-1\}}$\cup${\sf \{MPI\_ANY\_SOURCE\}}, where
{\sf n} is the number of processes in this group.

Note the asymmetry between send and receive operations:  A receive
operation may
accept messages from an arbitrary sender, on the other hand, a send operation
must specify a unique receiver.  This matches a ``push'' communication
mechanism, where data transfer is effected by the sender (rather than a
``pull'' mechanism, where data transfer is effected by the receiver).

Source = destination is allowed, that is, a process can send a
message to itself.
(However, it is unsafe to do so with the blocking send and receive
operations described above, since this may lead to deadlock. See
Sec. \ref{sec:pt2pt-semantics}.)

\begin{implementors}
Message context and other communicator information
can be implemented as an additional tag field.  It differs
from the regular message tag in that wild card matching is not allowed on this
field, and that value setting for this field is controlled by communicator
manipulation functions.
\end{implementors}

\subsection{Return status}
\label{subsec:pt2pt-status}

The source or tag of a received message may not be known if wildcard
values were used in the receive operation.
The information is returned by the \mpiarg{status}
argument of \mpifunc{MPI\_RECV}.  The type of \mpiarg{status} is
\MPI/-defined.
Status variables need to be explicitly
allocated by the user, that is, they are not system objects.

In C, \mpiarg{status} is a structure that contains two fields named
\const{MPI\_SOURCE} and \const{MPI\_TAG}, and the structure may contain
additional fields.  Thus, \mpiarg{status.MPI\_SOURCE} and
\mpiarg{status.MPI\_TAG} contain the source and tag, respectively, of
the received message.

In Fortran, \mpiarg{status} is an array of \ftype{INTEGER}s of size
\const{MPI\_STATUS\_SIZE}.  The two constants \const{MPI\_SOURCE}
and \const{MPI\_TAG} are the indices of the entries that store the
source and tag fields.  Thus \mpiarg{status(MPI\_SOURCE)} and
\mpiarg{status(MPI\_TAG)} contain, respectively, the source and the
tag of the received message.

The status argument also returns information on the length of the message
received.  However, this information is not directly available as a field
of the status variable and a call to \mpifunc{MPI\_GET\_COUNT} is required
to ``decode'' this information.


\begin{funcdef}{MPI\_GET\_COUNT(status, datatype, count)}
\funcarg{\IN}{status}{return status of receive operation (Status)}
\funcarg{\IN}{datatype}{datatype of each receive buffer element (handle)}
\funcarg{\OUT}{count}{number of received elements (integer)}
\end{funcdef}

\mpibind{MPI\_Get\_count(MPI\_Status~status, MPI\_Datatype~datatype, int~*count)}

\mpifbind{MPI\_GET\_COUNT(STATUS, DATATYPE, COUNT, IERROR)\fargs INTEGER  STATUS(MPI\_STATUS\_SIZE), DATATYPE, COUNT, IERROR}

Returns the number of elements received.  (Again, we count {\em elements},
not {\em bytes}.)   The \mpiarg{datatype} argument should match the argument
provided by the receive call that set the \mpiarg{status} variable.
(We shall later see, in Section~\ref{subsec:pt2pt-datatypeuse}, that
\mpifunc{MPI\_GET\_COUNT} may return, in certain situations, the value
\const{MPI\_UNDEFINED}.)

\begin{rationale}
Some message passing libraries use \type{\INOUT} \mpiarg{count},
\mpiarg{tag} and \mpiarg{source} arguments, thus using them both to specify
the selection criteria for incoming messages and return the actual envelope
values of the received message.
The use of a separate status argument prevents errors that are often attached
with \type{\INOUT} argument (e.g., using the
\const{MPI\_ANY\_TAG} constant as
the tag in a send).
Some libraries use calls that refer implicitly to the ``last message
received.''  This is not thread safe.

The \mpiarg{datatype} argument is passed to \func{MPI\_GET\_COUNT} so as to
improve performance. A message might be received without counting the number
of elements it contains, and the count value is often not needed.
Also, this allows the same function to be used after a call to
\func{MPI\_PROBE}.
\end{rationale}

All send and receive operations use the \mpiarg{buf}, \mpiarg{count},\mpiarg{
datatype}, \mpiarg{source}, \mpiarg{dest}, \mpiarg{tag},
\mpiarg{comm} and \mpiarg{status} arguments in the same way as the blocking
\func{MPI\_SEND} and
\func{MPI\_RECV} operations described in this section.

\section{Data type matching and data conversion}
\subsection{Type matching rules}
\label{subsec:pt2pt-typematch}

One can think of message transfer as consisting of the following three phases.
\begin{enumerate}
\item
Data is pulled out of the send buffer and a message is assembled.
\item
A message is transferred from sender to receiver.
\item
Data is pulled from the incoming message and disassembled into the receive
buffer.
\end{enumerate}

Type matching has to be observed at each of these three phases:  The type
of each variable in the sender buffer has to match
the type specified for that entry by the send operation;
the type specified by the send operation has to match the type specified by
the receive operation; and
the type of each
variable in the receive buffer has to match the type specified for
that entry by the receive operation.
A program that fails to observe these
three rules is erroneous.

To define type matching more precisely, we need to deal with two issues:
matching of types of the host language with types specified in
communication operations;
and matching of types at sender and receiver.

The types of a send and receive match (phase two) if
both operations use identical names. That is, \type{MPI\_INTEGER}
matches \type{MPI\_INTEGER}, \type{MPI\_REAL} matches \type{MPI\_REAL},
and so on.
There is one exception to this rule, discussed in
Sec. \ref{sec:pt2pt-packing}, the type \type{MPI\_PACKED} can match
any other type.

The type of a variable in a host program matches the type specified in the
communication operation
if the datatype name used by that operation corresponds
to the basic type of the host program variable.  For example, an entry with type
name \type{MPI\_INTEGER} matches a Fortran variable of type \ftype{INTEGER}.
A table giving this correspondence for Fortran and C appears in
Sec. \ref{subsec:pt2pt-messagedata}.
There are two exceptions to this last rule: an entry with type name
\type{MPI\_BYTE} or \type{MPI\_PACKED} can be used to match
any byte of storage (on a byte-addressable machine),
irrespective of the datatype of the variable that contains this byte.
The type \type{MPI\_PACKED} is used to send data that has been
explicitly packed, or receive data that will be explicitly unpacked,
see Section~\ref{sec:pt2pt-packing}.
The type \type{MPI\_BYTE} allows one to transfer the binary value of a byte in
memory unchanged.

To summarize, the type matching rules fall into the three categories below.
\begin{itemize}
\item
Communication of typed values (e.g., with datatype different from \type{
MPI\_BYTE}), where the datatypes of the corresponding entries in the sender
program, in the send call,
in the receive call and in the receiver program
must all match.
\item
Communication of untyped values (e.g., of datatype \type{MPI\_BYTE}), where
both sender and receiver use the
datatype \type{MPI\_BYTE}.  In this case, there are no
requirements on the types of the corresponding entries in the sender and the
receiver programs, nor is it required that they be the same.
\item
Communication involving packed data, where \type{MPI\_PACKED} is used.
\end{itemize}


The following examples illustrate the first two cases.
\label{page:pt2pt-3examples}

\begin{example} {\rm
\label{pt2pt-exA}
Sender and receiver specify matching types.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(1), 15, MPI_REAL, 0, tag, comm, status, ierr)
END IF
\end{verbatim}

This code is correct if both {\sf a} and {\sf b} are real arrays of
size $\ge 10$. (In Fortran, it might be correct to use this code
even if {\sf a} or {\sf b} have size $&lt; 10$: e.g., when {\sf a(1)} can
be equivalenced to an array with ten reals.)
} \end{example}

\begin{example} {\rm
\label{pt2pt-exB}
Sender and receiver do not specify matching types.

\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a(1), 10, MPI_REAL, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(1), 40, MPI_BYTE, 0, tag, comm, status, ierr)
END IF
\end{verbatim}

This code is erroneous, since sender and receiver do not provide
matching datatype arguments.
} \end{example}

\begin{example} {\rm
\label{pt2pt-exC}
Sender and receiver specify communication of untyped values.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a(1), 40, MPI_BYTE, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(1), 60, MPI_BYTE, 0, tag, comm, status, ierr)
END IF
\end{verbatim}

This code is correct, irrespective of the type and size of {\sf a} and
{\sf b} (unless this results in an out of bound memory access).
} \end{example}

\begin{users}
If a buffer of type \const{MPI\_BYTE} is passed as an argument to
\mpifunc{MPI\_SEND}, then \MPI/ will send the data stored at contiguous
locations, starting from the address
indicated by the \mpiarg{buf} argument.  This may have unexpected results when
the data layout is not as a casual user would expect it to be.
For example, some Fortran compilers implement variables of type
\const{CHARACTER}
as a structure that contains the character length and a pointer to
the actual string.  In such an environment, sending and receiving a Fortran
\const{CHARACTER} variable using the \const{MPI\_BYTE} type will not
have the anticipated result of transferring the character string.
For this reason, the user is advised to use typed communications
whenever possible.
\end{users}

\subsubsection{Type \const{MPI\_CHARACTER}}

The
type \type{MPI\_CHARACTER} matches one character of a Fortran variable of
type \ftype{CHARACTER}, rather then the entire character string stored in the
variable.  Fortran variables of type \const{CHARACTER}
or substrings are transferred as if they were arrays of characters.
This is illustrated in the example below.

\begin{example} {\rm
\label{pt2pt-exD}

Transfer of Fortran \const{CHARACTER}s.

\begin{verbatim}
CHARACTER*10 a
CHARACTER*10 b

CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_SEND(a, 5, MPI_CHARACTER, 1, tag, comm, ierr)
ELSE
    CALL MPI_RECV(b(6:10), 5, MPI_CHARACTER, 0, tag, comm, status, ierr)
END IF
\end{verbatim}

The last five characters of string \const{b} at process 1 are replaced by the
first five characters of string \const{a} at process 0.
} \end{example}

\begin{rationale}
The alternative choice would be for \type{MPI\_CHARACTER} to
match a character of arbitrary length.  This runs into problems.

A Fortran character variable is a constant length string, with no special
termination symbol.  There is no fixed convention on how to represent
characters, and how to store their length.
Some compilers pass a character argument to
a routine as a pair of arguments, one holding the address of the string and the
other holding the length of string.   Consider the case of an \MPI/
communication call that is passed a communication buffer
with type defined by a derived datatype (Section~\ref{sec:pt2pt-datatype}).
If this communicator buffer contains variables of type \ftype{CHARACTER} then
the information on their length will not be passed to the \MPI/ routine.

This problem forces us to provide explicit information on character
length with the \MPI/ call.  One could add a length parameter to the
type \type{MPI\_CHARACTER}, but this does not add much convenience and the same
functionality can be achieved by defining a suitable derived datatype.
\end{rationale}

\begin{implementors}
Some compilers pass Fortran \const{CHARACTER} arguments as a structure with a
length and a pointer to the actual string.  In such an environment, the \MPI/
call needs to dereference the pointer in order to reach the string.
\end{implementors}


\subsection{Data conversion}
\label{subsec:pt2pt-conversion}

One of the goals of \MPI/ is to support parallel computations across
heterogeneous environments.  Communication in a heterogeneous
environment may require data conversions.
We use the following terminology.
\begin{description}
\item[type conversion] changes the datatype of a value, e.g., by rounding a
\ftype{REAL} to an \ftype{INTEGER}.
\item[representation conversion] changes the binary representation of a value,
e.g., from Hex floating point to IEEE floating point.
\end{description}

The type matching rules imply that \MPI/ communication never entails type
conversion.  On the other hand, \MPI/ requires that a representation
conversion be
performed when a typed value is transferred across environments that use
different representations for the datatype of this value.
\MPI/ does not specify rules for representation conversion.  Such conversion is
expected to preserve integer, logical or character values, and to
convert a floating point value to the nearest value that can be
represented on the target system.

Overflow and underflow exceptions may occur during floating point conversions.
Conversion of integers or characters may also lead to exceptions when a value
that can be represented in one system cannot be represented in the other
system.  An exception
occurring during representation conversion results in a failure of the
communication. An error occurs either in the send operation, or the receive
operation, or both.

If a value sent in a message is untyped (i.e., of type \type{MPI\_BYTE}),
then the binary representation of the byte stored at the receiver is
identical to the binary representation of the byte loaded at the sender.  This
holds true, whether sender and receiver run in the same or in distinct
environments. No representation conversion is required.
(Note that representation conversion may
occur when values of type \type{MPI\_CHARACTER} or
\type{MPI\_CHAR} are transferred, for example, from an
EBCDIC encoding to an ASCII encoding.)

No conversion need occur when an \MPI/ program executes in
a homogeneous system, where all processes run in the same environment.


Consider the three examples, \ref{pt2pt-exA}--\ref{pt2pt-exC}.
The first program is correct, assuming that {\sf a} and {\sf b} are
\ftype{REAL} arrays of size $\ge 10$.
If the sender and receiver execute in different environments,
then the ten real values that are fetched from the send buffer will
be converted to the representation for reals on the receiver site
before they are stored in the receive buffer.  While the
number of real elements fetched from the send buffer equal the
number of real elements stored in the receive buffer, the number of
bytes stored need not equal the number of bytes loaded. For example, the
sender may use a four byte representation and the receiver an eight
byte representation for reals.

The second program is erroneous, and its behavior is undefined.

The third program is correct.  The exact same
sequence of forty bytes that were loaded from the send buffer will be
stored in the receive buffer, even
if sender and receiver run in a different environment.  The message
sent has exactly the same length (in bytes) and the same binary
representation as the message received.  If {\sf a} and {\sf b} are
of different types,
or if they are of the same type but different data representations are used,
then the bits stored in the receive buffer may encode values that are
different from the values they encoded in the send buffer.


Data representation conversion also applies to the envelope of a message:
source, destination and tag are all integers that may need to be converted.

\begin{implementors}
The current definition does not require messages to carry data
type information.  Both sender and receiver provide complete data
type information.
In a heterogeneous environment, one can either use
a machine independent encoding
such as XDR, or have the receiver convert from the sender representation to its
own, or even have the sender do the conversion.

Additional type information might be added to messages
in order to allow the system to detect mismatches between
datatype at sender and receiver.
This might be particularly useful in a slower but safer debug mode.
\end{implementors}

\MPI/
does not require support for inter-language
communication. The behavior of a program is undefined if messages are sent
by a C process and received by a Fortran process, or vice-versa.

\begin{rationale}
\MPI/ does not handle inter-language communication because there are no
agreed standards for the correspondence between C types and
Fortran types.  Therefore, \MPI/ programs that mix languages would not port.
\end{rationale}

\begin{implementors}
\MPI/ implementors may want to support inter-language communication by
allowing Fortran programs to use
``C MPI types,'' such as \const{MPI\_INT, MPI\_CHAR},
etc., and allowing C programs to use Fortran types.
\end{implementors}


\section{Communication Modes}
\label{sec:pt2pt-modes}
The send call described in Section~\ref{subsec:pt2pt-basicsend}
is {\bf blocking}:
it does not return until the message data
and envelope have been safely stored away so that the sender is
free to access and overwrite the send buffer.
The message might be copied directly into the matching receive buffer,
or it might be copied into a temporary system buffer.

Message buffering
decouples the send and receive operations.  A blocking send can
complete as soon
as the message was buffered, even if no matching receive has been executed
by the receiver.   On the other hand, message buffering can be expensive,
as it entails additional memory-to-memory copying, and it requires the
allocation of memory for buffering.  \MPI/ offers the choice of
several communication modes that allow one to control the choice of the
communication protocol.

The send call described in Section~\ref{subsec:pt2pt-basicsend}
used the {\bf standard} communication mode.  In this mode,
it is up to \MPI/ to decide whether outgoing
messages will be buffered.  \MPI/ may
buffer outgoing messages.  In such a case, the send call may complete
before a matching receive is invoked.  On the other hand, buffer space may be
unavailable, or \MPI/ may choose not to buffer
outgoing messages, for performance reasons. In this case,
the send call will not complete until a matching receive has been posted, and
the data has been moved to the receiver.

Thus, a send in standard mode can be started whether or not a
matching receive has been posted.  It may complete before a matching receive
is posted.  The
standard mode send is {\bf non-local}:  successful completion of the send
operation may depend on the occurrence of a matching receive.

\begin{rationale}
The reluctance of \MPI/ to mandate whether standard sends are buffering
or not stems from the desire to achieve portable programs.  Since any
system will run out of buffer resources as message sizes are increased,
and some implementations may want to provide little buffering, \MPI/
takes the position that correct (and therefore, portable) programs
do not rely on system buffering in standard mode.  Buffering
may improve the performance
of a correct program, but it doesn't affect the result of the program.
If the user wishes to guarantee a certain amount of buffering, the
user-provided buffer system of Sec. \ref{sec:pt2pt-buffer} should be used,
along with the buffered-mode send.
\end{rationale}

% Need a diagram, a chart, of all this...
% A Q and A on it might be good also...
% This will sound very confusing and strange to the uninitiated....ARM


There are three additional communication modes.

A {\bf buffered} mode send operation can be started whether or not a
matching receive has been posted.
It may complete before a matching receive is posted.  However, unlike
the standard send, this operation is {\bf local}, and its
completion does not depend on the occurrence of a matching receive.  Thus, if a
send is executed and no matching receive is posted, then \MPI/ must buffer the
outgoing message, so as to allow the send call to complete.   An error will
occur if there is insufficient buffer space.   The amount of available buffer
space is controlled by the user --- see Section~\ref{sec:pt2pt-buffer}.
Buffer allocation by the user may be required for the buffered mode to be
effective.

A send that uses the {\bf synchronous} mode can be started whether or
not a matching receive was posted.  However, the send will complete
successfully only if a matching receive is posted, and the
receive operation has started to receive the message sent by the
synchronous send.
Thus, the completion of a synchronous send not only indicates that the
send buffer can be reused, but also indicates that the receiver has
reached a certain point in its execution, namely that it has started
executing the matching receive.  If both sends and receives are
blocking operations then the use of the synchronous mode provides
synchronous communication semantics: a communication does not complete
at either end before both processes rendezvous at the
communication.  A send executed in this mode is {\bf non-local}.

A send that uses the {\bf ready} communication mode
may be started {\em only} if the matching receive is already posted.
Otherwise, the operation is erroneous and its outcome is undefined.
On some systems, this allows the removal of a hand-shake
operation that is otherwise required and results in improved
performance.
The completion of the send operation does not depend on the
status of a matching receive, and merely indicates that the send
buffer can be reused.   A send operation that uses the ready mode has
the same semantics as a standard send operation, or a synchronous send
operation; it is merely that the sender provides additional
information to the system (namely that a matching receive is already
posted), that can save some overhead.  In a correct program, therefore, a
ready send could be replaced by a standard send with no effect on the
behavior of the program other than performance.

Three additional send functions are provided for the three additional
communication
modes.  The communication mode is indicated by a one letter prefix:
{\sf B} for buffered,
{\sf S} for synchronous, and
{\sf R} for ready.



\begin{funcdef}{MPI\_BSEND (buf, count, datatype, dest, tag, comm)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\end{funcdef}

\mpibind{MPI\_Bsend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm)}

\mpifbind{MPI\_BSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, IERROR}

Send in buffered mode.

\begin{funcdef}{MPI\_SSEND (buf, count, datatype, dest, tag, comm)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\end{funcdef}

\mpibind{MPI\_Ssend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm)}

\mpifbind{MPI\_SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, IERROR}

Send in synchronous mode.

\begin{funcdef}{MPI\_RSEND (buf, count, datatype, dest, tag, comm)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\end{funcdef}

\mpibind{MPI\_Rsend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm)}

\mpifbind{MPI\_RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, IERROR}

Send in ready mode.

There is only one receive operation, which can match any of the send modes.
The receive operation described in the last section is {\bf blocking}:
it returns only after the receive buffer contains the newly received
message.   A receive can complete before the matching send
has completed (of course, it can complete only after the matching send
has started).

In a multi-threaded implementation of \MPI/, the system may de-schedule a
thread that is blocked on a send or receive
operation, and schedule another thread for execution in the same
address space.
In such a case it is the user's responsibility not to access or modify a
communication
buffer until the communication completes.
Otherwise, the outcome of the computation is undefined.

\begin{rationale}
We prohibit read accesses to a send buffer while it is being used, even though
the send operation is not supposed to alter the content of this buffer.  This
may seem more stringent than necessary, but the additional restriction causes
little loss of functionality and allows better performance on some systems ---
consider the case where data transfer is done by a DMA engine that is not
cache-coherent with the main processor.
\end{rationale}

\begin{implementors}
Since a synchronous send cannot complete before a matching receive is posted,
one will not normally buffer messages sent by such an operation.

It is recommended to choose buffering over blocking the sender, whenever
possible, for standard sends.  The programmer can signal his or her preference
for blocking the sender until a matching receive occurs by using the
synchronous send mode.

A possible communication protocol for the various communication modes
is outlined below.

{\sf ready send}:   The message is sent as soon as possible.

{\sf synchronous send:}
The sender sends a request-to-send message.
The receiver stores this request.
When a matching receive is posted, the receiver sends back a permission-to-send
message, and the sender now sends the message.

{\sf standard send:}
First protocol may be used for short messages, and second protocol for
long messages.

{\sf buffered send:}
The sender copies the message into a buffer and then sends it with a
nonblocking send (using the same protocol as for standard send).

Additional control messages might be needed for flow control and error
recovery.  Of course, there are many other possible protocols.

Ready send can be implemented as a standard send. In this case
there will be no performance advantage (or disadvantage) for the use of
ready send.

A standard send can be implemented as a synchronous send.
In such a case, no data buffering is needed.  However, many (most?)
users expect some buffering.

In a multi-threaded environment, the execution of a blocking communication
should block only the executing thread, allowing the thread scheduler to
de-schedule this thread and schedule another thread for execution.
\end{implementors}


\section{Semantics of point-to-point communication}
\label{sec:pt2pt-semantics}

A valid \MPI/ implementation guarantees certain general properties of
point-to-point communication, which are described in this section.

\paragraph*{Order}

Messages are {\em non-overtaking}:
If a sender sends two messages in succession to the same destination, and
both match the same receive, then this operation cannot receive the
second message if the first one is still pending.
If a receiver posts two receives in succession, and both match the same
message,
then the second receive operation cannot be satisfied by this message, if the
first one is still pending.
This requirement facilitates matching of sends to receives.
It guarantees that message-passing code is deterministic, if processes are
single-threaded and the wildcard \const{MPI\_ANY\_SOURCE} is not used in receives.
(Some of the calls described later, such as \mpifunc{MPI\_CANCEL} or
\mpifunc{MPI\_WAITANY}, are additional sources of nondeterminism.)

If a process has a single thread of execution, then any two communications
executed by this process are ordered.  On the other hand, if the process is
multi-threaded, then the semantics of thread execution may not define
a relative order between two send operations executed by two
distinct threads.  The operations are logically concurrent, even if one
physically precedes the other. In such a case, the two messages sent can be
received in any order.  Similarly, if two receive operations that are logically
concurrent receive two successively sent messages, then the two messages can
match the two receives in either order.

\begin{example} {\rm
\label{pt2pt-exE}
An example of non-overtaking messages.

\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_BSEND(buf1, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_BSEND(buf2, count, MPI_REAL, 1, tag, comm, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(buf1, count, MPI_REAL, 0, MPI_ANY_TAG, comm, status, ierr)
    CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag, comm, status, ierr)
END IF
\end{verbatim}
The message sent by the first send must be received by the first receive, and
the message sent by the second send must be received by the second receive.
} \end{example}


\paragraph*{Progress}

If a pair of
matching send and receives have been initiated on two processes, then at
least one of these two operations will complete, independently of
other actions in the system:  the send operation will
complete, unless the receive is satisfied by another message, and
completes; the
receive operation will complete, unless the message sent is consumed by another
matching receive that was posted at the same destination process.

\begin{example} {\rm
\label{pt2pt-exF}
An example of two, intertwined matching pairs.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_BSEND(buf1, count, MPI_REAL, 1, tag1, comm, ierr)
    CALL MPI_SSEND(buf2, count, MPI_REAL, 1, tag2, comm, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(buf1, count, MPI_REAL, 0, tag2, comm, status, ierr)
    CALL MPI_RECV(buf2, count, MPI_REAL, 0, tag1, comm, status, ierr)
END IF
\end{verbatim}
Both processes invoke their first communication call.
Since the first send of process zero uses the buffered mode, it must complete,
irrespective of the state of process one.  Since no matching receive is
posted, the message will be copied into buffer space.  (If insufficient
buffer space is available, then the program will fail.)
The second send is then invoked.
At that point, a matching pair of send and receive
operation is enabled, and both operations must complete.  Process one next
invokes its second receive call, which will be satisfied by the buffered
message. Note that process one received the messages in the reverse order they
were sent.
} \end{example}

\paragraph*{Fairness}

\MPI/ makes no guarantee of {\em fairness} in the handling of
communication.  Suppose that a send is posted.  Then it is possible
that the destination process repeatedly posts a receive that matches this
send, yet the message is never received, because it is each time overtaken by
another message, sent from another source.  Similarly, suppose that a
receive was posted by a multi-threaded process.  Then it is possible that
messages that
match this receive are repeatedly received, yet the receive is never satisfied,
because it is overtaken by other receives posted at this node (by
other executing threads).  It is the programmer's responsibility to prevent
starvation in such situations.


\paragraph*{Resource limitations}

Any pending communication operation consumes system resources that are
limited.
Errors may occur when lack of resources prevent the execution of an \MPI/ call.
A quality implementation will use a (small) fixed amount of resources for
each pending send in the ready or synchronous mode and for each
pending receive.  However, buffer
space may be consumed to store  messages sent in standard mode, and must
be consumed to store
messages sent in buffered mode, when no matching receive is available.
The amount of space available for buffering will be much smaller than program
data memory on many systems.  Then, it will be easy to write programs that
overrun available buffer space.

\MPI/ allows the user to provide buffer memory for messages sent in the buffered
mode.  Furthermore, \MPI/ specifies a detailed operational
model for the use of this
buffer.  An \MPI/ implementation is required
to do no worse than implied by this model.  This allows users to avoid buffer
overflows when they use buffered sends.   Buffer allocation and use is
described
in Section~\ref{sec:pt2pt-buffer}.

A buffered send operation that cannot complete because of a lack of buffer space
is erroneous.  When such a situation is detected, an error is signalled that may
cause the program to terminate abnormally.
On the other hand, a standard send
operation that cannot complete because
of lack of buffer space will merely block, waiting for buffer space to become
available or for a matching receive to be posted.  This behavior is preferable
in many situations.  Consider a situation where a producer repeatedly produces
new values and sends them to a consumer.  Assume that the producer produces
new values faster than the consumer can consume them.  If buffered sends are
used, then a buffer overflow will result.  Additional synchronization has to
be added to the program so as to prevent this from occurring.
If standard sends are used, then the producer will be automatically throttled,
as its send operations will block when buffer space is unavailable.

In some situations, a lack of buffer space leads to deadlock situations.
This is illustrated by the examples below.

\begin{example} {\rm
\label{pt2pt-exG}
An exchange of messages.

\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
END IF
\end{verbatim}
This program will succeed even if no buffer space for data
is available.  The standard send operation can be replaced, in this example,
with a synchronous send.
} \end{example}

\begin{example} {\rm
\label{pt2pt-exH}
An attempt to exchange messages.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
END IF
\end{verbatim}
The receive operation of the first process must complete before its send, and
can complete only if the matching send
of the second processor is executed. The receive operation of the second
process must complete before its send and
can complete only if the matching send of the first process is executed.
This program will always deadlock.  The same holds for any other send mode.
} \end{example}

\begin{example} {\rm
\label{pt2pt-exI}
An exchange that relies on buffering.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (rank.EQ.0) THEN
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 1, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 1, tag, comm, status, ierr)
ELSE    ! rank.EQ.1
    CALL MPI_SEND(sendbuf, count, MPI_REAL, 0, tag, comm, ierr)
    CALL MPI_RECV(recvbuf, count, MPI_REAL, 0, tag, comm, status, ierr)
END IF
\end{verbatim}
The message sent by each process has to be copied out before the send operation
returns and the receive operation starts.  For the program to complete, it is
necessary that at least one of the two messages sent be buffered.
Thus, this program can
succeed only if
the communication system can buffer at least \mpiarg{count} words of data.
} \end{example}

\begin{users}
When standard send operations are used, then a deadlock situation may occur
where both processes are blocked because buffer space is not
available.
The same will certainly happen, if the synchronous mode is used.
If the buffered mode is used, and not enough buffer space is available,
then the
program will not complete either.  However, rather than a deadlock situation,
we shall have a buffer overflow error.

A program is ``safe'' if no message buffering is required for
the program to complete.  One can replace all sends in such program with
synchronous sends, and the program will still run correctly.   This
conservative programming style provides the best portability, since program
completion does not depend on the amount of buffer space available or in the
communication protocol used.

Many programmers prefer to have more leeway and be able to use the ``unsafe''
programming
style shown in example \ref{pt2pt-exI}.  In such cases,
the use of standard sends is likely to
provide the best compromise between performance and robustness:  quality
implementations will provide sufficient buffering so that ``common practice''
programs will not deadlock.  The buffered send mode can be used for
programs that
require more buffering, or in situations where the programmer wants more
control.
This mode might also be used for debugging purposes, as buffer overflow
conditions are easier to diagnose than deadlock conditions.

Nonblocking message-passing operations, as described in
Section~\ref{sec:pt2pt-nonblock}, can be used to avoid the need for buffering
outgoing messages.  This prevents deadlocks due to lack of buffer space, and
improves performance, by allowing overlap of computation and communication,
and avoiding the overheads of allocating buffers and copying messages into
buffers.
\end{users}

\section{Buffer allocation and usage}
\label{sec:pt2pt-buffer}

A user may specify a buffer to be used for buffering messages sent in buffered
mode.   Buffering is done by the sender.

\begin{funcdef}{MPI\_BUFFER\_ATTACH( buffer, size)}
\funcarg{\IN}{buffer}{initial buffer address (choice)}
\funcarg{\IN}{size}{buffer size, in bytes (integer)}
\end{funcdef}

\mpibind{MPI\_Buffer\_attach( void*~buffer, int~size)}

\mpifbind{MPI\_BUFFER\_ATTACH( BUFFER, SIZE, IERROR)\fargs &lt;type&gt; BUFFER(*) \\ INTEGER  SIZE, IERROR}

Provides to \MPI/ a buffer in the user's memory to be used for buffering outgoing
messages.  The buffer is used only by messages sent in buffered mode.
Only one buffer can be attached to a process at a time.

\begin{funcdef}{MPI\_BUFFER\_DETACH( buffer, size)}
\funcarg{\OUT}{buffer}{initial buffer address (choice)}
\funcarg{\OUT}{size}{buffer size, in bytes (integer)}
\end{funcdef}

\mpibind{MPI\_Buffer\_detach( void**~buffer, int*~size)}

\mpifbind{MPI\_BUFFER\_DETACH( BUFFER, SIZE, IERROR)\fargs &lt;type&gt; BUFFER(*) \\ INTEGER  SIZE, IERROR}

Detach the buffer currently associated with \MPI/.  This operation will block
until all messages currently in the buffer have been transmitted.
Upon return of
this function, the user may reuse or deallocate the space taken by the buffer.

The statements made in this section describe the behavior of \MPI/ for
buffered-mode sends.
When no buffer is currently associated, \MPI/ behaves as if a
zero-sized buffer is associated with the process.

\MPI/ must provide as much buffering for outgoing messages {\em as if\/}
outgoing message
data were buffered by the sending process, in the specified buffer space,
using a circular, contiguous-space allocation policy.
We outline below a model implementation that defines this policy.
\MPI/ may provide more buffering, and may use a better buffer allocation
algorithm
than described below.  On the other hand, \MPI/ may signal an error whenever the
simple buffering allocator described below would run out of space.  In
particular, if no buffer is explicitly associated with the process, then any
buffered send may cause an error.


\MPI/ does not provide mechanisms for querying or controlling buffering done by
standard mode sends.   It is expected that vendors will provide such
information
for their implementations.

\begin{rationale}
There is a wide spectrum of possible implementations of buffered communication:
buffering can be done at sender, at receiver, or both; buffers can be dedicated
to one sender-receiver pair, or be shared by
all communications; buffering can be done in real or
in virtual memory; it can use dedicated memory, or memory shared by other
processes; buffer space may be allocated statically or be changed dynamically;
etc.   It does not seem feasible to provide a portable mechanism for querying
or controlling buffering that would be compatible with all these choices, yet
provide meaningful information.
\end{rationale}


\subsection{Model implementation of buffered mode}

The model implementation uses the packing and unpacking functions described in
Section~\ref{sec:pt2pt-packing} and the nonblocking communication functions
described in Section~\ref{sec:pt2pt-nonblock}.

We assume that a circular queue
of pending message entries (PME) is maintained.  Each
entry contains a communication request handle
that identifies a pending nonblocking
send, a pointer to the next entry and the packed message data.  The
entries are stored in successive locations in the buffer.  Free space is
available between the queue tail and the queue head.

A buffered send call results in the execution of the following code.
\begin{itemize}
\item
Traverse sequentially the PME queue from head towards the tail,
deleting all entries for
communications that have completed, up to the first entry with an uncompleted
request; update queue head to point to that entry.
\item
Compute the number, {\sf n}, of bytes needed to store entry for new message
\snir
An upper bound on {\sf n} can be computed
as follows: A call to the function \mpifunc{MPI\_PACK\_SIZE(count, datatype,
comm, size)}, with the {\tt count, datatype} and {\tt comm} arguments
used in the
\mpifunc{MPI\_BSEND} call, returns an upper bound on the amount of
space needed to buffer the message data (see Section~\ref{sec:pt2pt-packing}).
The MPI constant \const{MPI\_BSEND\_OVERHEAD} provides an upper bound on the
additional space consumed by the entry (e.g., for pointers or envelope
information).
\rins
\item
Find the next contiguous empty space of {\sf n} bytes in
buffer (space following queue tail, or space at start of buffer if
queue tail is
too close to end of buffer).  If space not found then raise buffer
overflow error.
\item
Append to end of PME queue in contiguous space the new entry
that contains request
handle, next pointer and packed message data; \mpifunc{MPI\_PACK} is used to
pack data.
\item
Post nonblocking send (standard mode) for packed data.
\item
Return
\end{itemize}

\section{Nonblocking communication}
\label{sec:pt2pt-nonblock}

One can improve performance on many systems by overlapping
communication and computation.  This is especially true on systems
where communication can be executed autonomously by an intelligent
communication controller.  Light-weight threads are one mechanism for
achieving such overlap.  An alternative mechanism that often leads to
better performance is to use {\bf nonblocking communication}.  A
nonblocking {\bf send start} call initiates the send operation, but does not
complete it.  The send start call will return
before the message was copied out of the send buffer.
A separate {\bf send complete}
call is needed to complete the communication, i.e., to verify that the
data has been copied out of the send buffer.  With
suitable hardware, the transfer of data out of the sender memory
may proceed concurrently with computations done at the sender after
the send was initiated and before it completed.
Similarly, a nonblocking {\bf receive start call} initiates the receive
operation, but does not complete it.  The call will return before
a message is stored into the receive buffer.  A separate {\bf receive
complete} call
is needed to complete the receive operation and verify that the data has
been received into the receive buffer.
With suitable hardware, the transfer of data into the receiver memory
may proceed concurrently with computations done after the receive was
initiated and before it completed.  The use of nonblocking receives may also
avoid system buffering and memory-to-memory copying, as information is provided
early on the location of the receive buffer.

Nonblocking send start calls can use the same four modes as blocking sends:
{\sf standard}, {\sf buffered}, {\sf synchronous} and {\sf ready}.  These carry
the same meaning.
Sends of all modes, {\sf ready} excepted, can be started whether a matching
receive has been posted or not; a nonblocking {\sf ready}
send can be started only if
a matching receive is posted.   In all cases, the send start call
is local: it returns immediately, irrespective of the
status of other processes.
If the call causes some system resource to be exhausted, then it will
fail and return an error code.  Quality
implementations of \MPI/ should ensure that this happens only
in ``pathological'' cases.  That is, an \MPI/ implementation
should be able to support a
large number of pending nonblocking operations.

The send-complete call returns when data has been copied out of the
send buffer.
It may carry additional meaning, depending on the send mode.

If the send mode is {\sf synchronous}, then the
send can complete only if a matching receive has started.  That
is, a receive has
been posted, and has been matched with the send.  In this case,
the send-complete call is non-local.  Note that a synchronous,
nonblocking send may complete, if matched by a nonblocking receive, before
the receive complete call occurs.  (It can complete as soon as the sender
``knows'' the transfer will complete, but before the receiver ``knows'' the
transfer will complete.)

If the send mode is {\sf buffered} then the
message must be buffered if there is no pending receive.  In this case,
the send-complete
call is local, and must succeed irrespective of the status of a matching
receive.

If the send mode is {\sf standard} then the send-complete call may
return before a
matching receive occurred, if the message is buffered.  On the other hand, the
send-complete may not complete until a matching
receive occurred, and the message was
copied into the receive buffer.

Nonblocking sends can be matched with blocking receives, and
vice-versa.

\begin{users}
The completion of a send operation may be delayed, for standard mode, and must
be delayed, for synchronous mode, until a matching receive is posted.
The use of nonblocking sends in these two cases allows the sender to proceed
ahead of the receiver, so that the computation is more tolerant
of fluctuations in the speeds of the two processes.

Nonblocking sends in the buffered and ready modes have a more
limited impact.  A nonblocking send will return as soon as possible, whereas a blocking
send will return after the data has been copied out of the sender memory.
The use of nonblocking sends is advantageous in these cases only if
data copying
can be concurrent with computation.

The message-passing model implies that communication is initiated by
the sender.
The communication will generally have lower overhead if a receive is
already posted when the sender initiates the communication  (data can be moved
directly to the receive buffer, and there is no need to queue a pending send
request).  However,
a receive operation can complete only after the matching send has occurred.
The use of nonblocking receives allows one to achieve lower communication overheads
without blocking the receiver while it waits for the send.
\end{users}

\subsection{Communication Objects}
\label{subsec:pt2pt-commobject}

Nonblocking communications use opaque {\sf request} objects to
identify communication operations and match the operation that
initiates the communication with the operation that terminates it.
These are system objects that are accessed via a handle.
A request object identifies various properties of a communication
operation, such as the send mode, the communication
buffer  that is associated with it, its
context, the tag and destination arguments to be used for a send, or the tag
and source arguments to be used for a receive.  In addition, this object stores
information about the status of the pending communication operation.

\subsection{Communication initiation}
\label{subsec:pt2pt-commstart}

We use the same naming conventions as for blocking communication: a
prefix of {\sf B}, {\sf S}, or {\sf R} is used for {\sf buffered}, {\sf
synchronous} or {\sf ready} mode.  In addition a prefix of {\sf I} (for {\sf
immediate}) indicates that the call is nonblocking.

\begin{funcdef}{MPI\_ISEND(buf, count, datatype, dest, tag, comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Isend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_ISEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Start a standard mode, nonblocking send.



\begin{funcdef}{MPI\_IBSEND(buf, count, datatype, dest, tag, comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Ibsend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_IBSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Start a buffered mode, nonblocking send.




\begin{funcdef}{MPI\_ISSEND(buf, count, datatype, dest, tag, comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Issend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_ISSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Start a synchronous mode, nonblocking send.



\begin{funcdef}{MPI\_IRSEND(buf, count, datatype, dest, tag, comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements in send buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each send buffer element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Irsend(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_IRSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Start a ready mode nonblocking send.



\begin{funcdef}{MPI\_IRECV (buf, count, datatype, source, tag, comm, request)}
\funcarg{\OUT}{buf}{initial address of receive buffer (choice)}
\funcarg{\IN}{count}{number of elements in receive buffer (integer)}
\funcarg{\IN}{datatype}{datatype of each receive buffer element (handle)}
\funcarg{\IN}{source}{rank of source (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Irecv(void*~buf, int~count, MPI\_Datatype~datatype, int~source, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_IRECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER  COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR}

Start a nonblocking receive.

These calls allocate a communication
request object and associate it with the request handle (the argument
\mpiarg{request}).
The request can be used later to
query the status of the communication or wait for its completion.

A nonblocking send call indicates that the
system may start copying data out of the send buffer.
The sender should not access any part of the send
buffer after a nonblocking send operation is called, until the send completes.

A nonblocking receive call indicates that the system may start
writing data into the receive buffer.  The receiver should not access
any part of the
receive buffer after a nonblocking receive operation is called,
until the receive completes.

\subsection{Communication Completion}
\label{subsec:pt2pt-commend}

The functions \func{MPI\_WAIT} and \func{MPI\_TEST} are used to complete a
nonblocking communication.  The completion of a send operation indicates that
the sender is now free
to update the locations in the send buffer
(the send operation itself leaves the content
of the send buffer unchanged). It does not indicate that the
message has been received,
rather, it may have been buffered by the communication
subsystem.  However, if a {\sf synchronous}
mode send was used, the completion of the
send operation indicates that a matching receive was initiated, and that the
message will eventually be received by this matching receive.

The completion of a receive operation indicates that the receive buffer
contains the received message, the receiver is now free to access it,
and that the status object is set.  It does
not indicate that the matching send operation has completed (but indicates, of
course, that the send was initiated).

We shall use the following terminology:
A {\bf null} handle is a handle with
value \linebreak \const{MPI\_REQUEST\_NULL}.
A persistent
request and the handle to it are {\bf inactive}
if the request is not associated with any ongoing
communication (see Section~\ref{sec:pt2pt-persistent}).
A handle is {\bf active} if it is neither null nor inactive.
\snir
A {\bf null} status is a status which is set to
return \mpiarg{tag = MPI\_ANY\_TAG},
\mpiarg{source = MPI\_ANY\_SOURCE}, and is also internally configured
so that calls to
\mpifunc{MPI\_GET\_COUNT} and \mpifunc{MPI\_GET\_ELEMENTS} %mansplit
return \mpiarg{count = 0}.
We set a status variable to null when the value returned by it is not
significant. Status is set in this
way so as to prevent errors due to accesses of stale information.
\rins

\begin{funcdef}{MPI\_WAIT(request, status)}
\funcarg{\INOUT}{request}{request (handle)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Wait(MPI\_Request~*request, MPI\_Status~*status)}

\mpifbind{MPI\_WAIT(REQUEST, STATUS, IERROR)\fargs INTEGER  REQUEST, STATUS(MPI\_STATUS\_SIZE), IERROR}



A call to \func{MPI\_WAIT} returns when the operation
identified by \mpiarg{request} is complete.  If the communication object
associated with this request was created by a nonblocking send or
receive call, then the object is deallocated by the call to \func{MPI\_WAIT}
and the request handle is set to \const{MPI\_REQUEST\_NULL}.
\mpifunc{MPI\_WAIT} is a non-local operation.

The call returns, in \mpiarg{status}, information on
the completed operation.  The content of the status object for a receive
operation can be accessed as
described in section~\ref{subsec:pt2pt-status}.
The status object for a send operation may be queried by
a call to \mpifunc{MPI\_TEST\_CANCELLED}
(see Section~\ref{sec:pt2pt-probe}).

One is allowed to call \mpifunc{MPI\_WAIT} with a null or inactive
\mpiarg{request} argument.
In this case the operation returns immediately with a null \mpiarg{status}.

\begin{users}
Successful return of \func{MPI\_WAIT} after a \func{MPI\_IBSEND} implies
that the user send buffer can be reused --- i.e., data has been sent
out or copied into a
buffer attached with \func{MPI\_BUFFER\_ATTACH}.   Note that, at this point,
we can no longer cancel the send (see Sec. \ref{sec:pt2pt-probe}).
If a matching receive is never
posted, then the buffer cannot be freed.  This runs somewhat
counter to the stated goal of \func{MPI\_CANCEL} (always being able to free
program space that was committed to the communication subsystem).
\end{users}

\begin{implementors}
In a multi-threaded environment, a call to
\func{ MPI\_WAIT} should block only the calling thread, allowing the thread
scheduler to schedule another thread for execution.
\end{implementors}

\begin{funcdef}{MPI\_TEST(request, flag, status)}
\funcarg{\INOUT}{request}{communication request (handle)}
\funcarg{\OUT}{flag}{{\sf true} if operation completed (logical)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Test(MPI\_Request~*request, int~*flag, MPI\_Status~*status)}

\mpifbind{MPI\_TEST(REQUEST, FLAG, STATUS, IERROR)\fargs LOGICAL  FLAG \\ INTEGER  REQUEST, STATUS(MPI\_STATUS\_SIZE), IERROR}



A call to \func{MPI\_TEST} returns \mpiarg{flag = true} if the
operation identified by \mpiarg{request} is complete.  In such a case, the
status object is set to contain information on the completed
operation;  if the communication object was created by a nonblocking
send or receive, then it is deallocated and the request handle is set to
\const{MPI\_REQUEST\_NULL}. The call returns
\mpiarg{flag = false}, otherwise. In this case, the value
of the status object is undefined.
\func{MPI\_TEST} is a local operation.

The return status object for a receive operation carries information that
can be accessed as described in
section~\ref{subsec:pt2pt-status}.
The status object for a send operation carries information that can be
accessed by
a call to \mpifunc{MPI\_TEST\_CANCELLED}
(see Section~\ref{sec:pt2pt-probe}).

\snir
One
is allowed to call \mpifunc{MPI\_TEST} with a null or inactive \mpiarg{request}
argument. In such a case the operation returns with \mpiarg{flag = true} and
null \mpiarg{status}.
\rins

The functions \func{MPI\_WAIT} and \func{MPI\_TEST} can be used to
complete both sends and receives.

\begin{users}
The use of
the nonblocking \func{MPI\_TEST} call allows the user to
schedule alternative activities within a single thread of execution.
An event-driven thread scheduler can be emulated with periodic calls to
\mpifunc{MPI\_TEST}.
\end{users}

\snir
\begin{rationale}
The function \mpifunc{MPI\_TEST} returns with \mpiarg{flag = true}
exactly in those situations where the function \mpifunc{MPI\_WAIT}
returns; both functions return in such case the same value in
\mpiarg{status}.  Thus, a blocking Wait can be easily replaced by a
nonblocking Test.
\end{rationale}
\rins

\begin{example} {\rm
\label{pt2pt-exJ}
Simple usage of nonblocking operations and \func{MPI\_WAIT}.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
    CALL MPI_ISEND(a(1), 10, MPI_REAL, 1, tag, comm, request, ierr)
    **** do some computation to mask latency ****
    CALL MPI_WAIT(request, status, ierr)
ELSE
    CALL MPI_IRECV(a(1), 15, MPI_REAL, 0, tag, comm, request, ierr)
    **** do some computation to mask latency ****
    CALL MPI_WAIT(request, status, ierr)
END IF
\end{verbatim}
} \end{example}

A request object can be deallocated without waiting for the associated
communication to complete, by using the following operation.

\begin{funcdef}{MPI\_REQUEST\_FREE(request)}
\funcarg{\INOUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Request\_free(MPI\_Request~*request)}

\mpifbind{MPI\_REQUEST\_FREE(REQUEST, IERROR)\fargs INTEGER  REQUEST, IERROR}

Mark the request object for deallocation and set \mpiarg{request} to
\const{MPI\_REQUEST\_NULL}.
An ongoing communication that is associated with the request will be allowed
to complete. The request will be deallocated only after its completion.

\begin{rationale}
The \mpifunc{MPI\_REQUEST\_FREE} mechanism is provided for
reasons of performance and convenience on the sending side.
\end{rationale}

\begin{users}
Once a request is freed by a call to \mpifunc{MPI\_REQUEST\_FREE}, it is not
possible to check for the successful completion of the associated
communication with calls to \mpifunc{MPI\_WAIT} or \mpifunc{MPI\_TEST}.  Also,
if an error occurs subsequently during the communication, an error code cannot
be returned to the user --- such an error must be treated as fatal.
Questions arise as to how one knows when the operations have completed
when using \mpifunc{MPI\_REQUEST\_FREE}.  Depending on the program logic,
there may be other ways in which the program knows that certain operations
have completed and this makes usage of \mpifunc{MPI\_REQUEST\_FREE}
practical.   For example, an active send request
could be freed when the logic of the program is such that the
receiver sends a reply to the message sent --- the arrival of the reply informs
the sender that the send has completed and the send buffer can be reused.
An active receive request should never be freed as the receiver will have no
way to verify that the receive has completed and the receive buffer can be
reused.
\end{users}

% odd. cannot use \mpifunc{MPI\_REQUEST\_FREE} on receives?
% What if I use ordering property and so make a wait call on only the last
% recv of a sequence of messages from a particular process?
% NO: "Consider an implementation where long messages are sent via DMA and short
% messages via PIO.If the first message is much longer than the
% second, then transfer of the first message may still be ongoing when
% the second receive completes."


\begin{example} {\rm
\label{pt2pt-exK}
An example using \mpifunc{MPI\_REQUEST\_FREE}.

\begin{verbatim}
CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank)
IF(rank.EQ.0) THEN
    DO i=1, n
      CALL MPI_ISEND(outval, 1, MPI_REAL, 1, 0, req, ierr)
      CALL MPI_REQUEST_FREE(req, ierr)
      CALL MPI_IRECV(inval, 1, MPI_REAL, 1, 0, req, ierr)
      CALL MPI_WAIT(req, status, ierr)
    END DO
ELSE    ! rank.EQ.1
    CALL MPI_IRECV(inval, 1, MPI_REAL, 0, 0, req, ierr)
    CALL MPI_WAIT(req, status)
    DO I=1, n-1
       CALL MPI_ISEND(outval, 1, MPI_REAL, 0, 0, req, ierr)
       CALL MPI_REQUEST_FREE(req, ierr)
       CALL MPI_IRECV(inval, 1, MPI_REAL, 0, 0, req, ierr)
       CALL MPI_WAIT(req, status, ierr)
    END DO
    CALL MPI_ISEND(outval, 1, MPI_REAL, 0, 0, req, ierr)
    CALL MPI_WAIT(req, status)
END IF
\end{verbatim}
} \end{example}

\subsection{Semantics of Nonblocking Communications}



The semantics of nonblocking communication is defined by suitably extending the
definitions in Section~\ref{sec:pt2pt-semantics}.

\paragraph*{Order}
Nonblocking communication operations are ordered according to the execution
order of the calls that initiate the communication.  The non-overtaking
requirement of Section~\ref{sec:pt2pt-semantics} is extended to
nonblocking communication, with this definition of order being used.

\begin{example} {\rm
\label{pt2pt-exL}
Message ordering for nonblocking operations.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (RANK.EQ.0) THEN
      CALL MPI_ISEND(a, 1, MPI_REAL, 1, 0, comm, r1, ierr)
      CALL MPI_ISEND(b, 1, MPI_REAL, 1, 0, comm, r2, ierr)
ELSE    ! rank.EQ.1
      CALL MPI_IRECV(a, 1, MPI_REAL, 0, MPI_ANY_TAG, comm, r1, ierr)
      CALL MPI_IRECV(b, 1, MPI_REAL, 0, 0, comm, r2, ierr)
END IF
CALL MPI_WAIT(r1,status)
CALL MPI_WAIT(r2,status)
\end{verbatim}
The first send of process zero will match the
first receive of process one, even
if both messages are sent before process one executes either receive.
} \end{example}

\paragraph*{Progress}
A call to \func{MPI\_WAIT} that completes a receive will eventually
terminate and return if a matching send has been started, unless
the send is satisfied by another receive.  In particular, if the matching send
is
nonblocking,
then the receive should complete even if no call is executed by the
sender to complete the send.   Similarly, a call to \func{MPI\_WAIT} that
completes a send will eventually
return if a matching receive has been started, unless
the receive is satisfied by another send, and even if no call is executed to
complete the receive.


\begin{example} {\rm
\label{pt2pt-exM}
An illustration of progress semantics.
\begin{verbatim}
CALL MPI_COMM_RANK(comm, rank, ierr)
IF (RANK.EQ.0) THEN
      CALL MPI_SSEND(a, 1, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_SEND(b, 1, MPI_REAL, 1, 1, comm, ierr)
ELSE    ! rank.EQ.1
      CALL MPI_IRECV(a, 1, MPI_REAL, 0, 0, comm, r, ierr)
      CALL MPI_RECV(b, 1, MPI_REAL, 0, 1, comm, ierr)
      CALL MPI_WAIT(r, status, ierr)
END IF
\end{verbatim}

This code should not deadlock in a correct \MPI/ implementation.  The first
synchronous send of process zero must complete after process one posts the
matching (nonblocking) receive even if process one has not yet reached
the completing wait call.  Thus, process zero will continue and execute
the second send, allowing process one to complete execution.
} \end{example}

If an \func{MPI\_TEST} that completes a receive is repeatedly called with the same
arguments, and
a matching send has been started, then the call will eventually
return \mpiarg{flag = true}, unless the send is satisfied by another
receive.
If an \func{MPI\_TEST} that completes a send is repeatedly called with the
same arguments, and
a matching receive has been started, then the call will eventually
return \mpiarg{flag = true}, unless the receive is satisfied by another
send.

\subsection{Multiple Completions}
\label{subsec:pt2pt-multiple}

It is convenient to be able to wait for the completion of any, some, or all the
operations in a list, rather than having to wait for a specific message.
A call to \func{MPI\_WAITANY} or \func{MPI\_TESTANY} can be used to
wait for the
completion of one out of several operations. A call to \func{MPI\_WAITALL}
or \func{MPI\_TESTALL} can be
used to wait for all pending operations in a list. A call to
\mpifunc{MPI\_WAITSOME} or \mpifunc{MPI\_TESTSOME} can be used to complete all
enabled operations in a list.

\begin{funcdef}{MPI\_WAITANY (count, array\_of\_requests, index, status)}
\funcarg{\IN}{count}{list length (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handles)}
\funcarg{\OUT}{index}{index of handle for operation that completed (integer)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Waitany(int~count, MPI\_Request~*array\_of\_requests, int~*index, MPI\_Status~*status)}

\mpifbind{MPI\_WAITANY(COUNT, ARRAY\_OF\_REQUESTS, INDEX, STATUS, IERROR)\fargs INTEGER  COUNT, ARRAY\_OF\_REQUESTS(*), INDEX, STATUS(MPI\_STATUS\_SIZE), IERROR}


Blocks until one of the operations associated with the active
requests in the array has completed.
If more then one operation is
enabled and can terminate, one is arbitrarily chosen.
Returns in \mpiarg{index} the index
of that request in the array and returns in \mpiarg{status} the status of the
completing communication.
(The array is indexed from zero in C, and from one in Fortran.)
If the request was allocated by a nonblocking communication operation, then it
is deallocated and the request handle is set to \const{MPI\_REQUEST\_NULL}.

\snir
The \mpiarg{array\_of\_requests} list may contain null or inactive
handles.
If the list contains no active handles (list has length zero or all
entries are null or inactive),
then the call  returns immediately with \mpiarg{index =
MPI\_UNDEFINED}, and a null \mpiarg{status}.
\rins

The execution of \func{MPI\_WAITANY(count, array\_of\_requests, index,
status)} has the same effect as the execution of \func{
MPI\_WAIT(\&amp;array\_of\_requests[i], status)}, where \mpiarg{i} is the value
returned by \mpiarg{index} (unless the value of \mpiarg{index}
is \const{MPI\_UNDEFINED}).
\mpifunc{MPI\_WAITANY} with an array containing one active entry
is equivalent to \mpifunc{MPI\_WAIT}.

\begin{funcdef}{MPI\_TESTANY(count, array\_of\_requests, index, flag, status)}
\funcarg{\IN}{count}{list length (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handles)}
\funcarg{\OUT}{index}{index of operation that completed,
or \const{MPI\_UNDE-} \linebreak\const{FINED} %mansplit
if none completed (integer)}
\funcarg{\OUT}{flag}{\const{true} if one of the operations is complete
(logical)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Testany(int~count, MPI\_Request~*array\_of\_requests, int~*index, int~*flag, MPI\_Status~*status)}

\mpifbind{MPI\_TESTANY(COUNT, ARRAY\_OF\_REQUESTS, INDEX, FLAG, STATUS, IERROR)\fargs LOGICAL  FLAG \\ INTEGER  COUNT, ARRAY\_OF\_REQUESTS(*), INDEX, STATUS(MPI\_STATUS\_SIZE), IERROR}


Tests for completion of
either one or none of the operations associated with active handles.
In the former case, it returns {\sf flag = true},
returns in {\sf index} the index of this request in the array,
and returns in {\sf status} the status of that operation; if the request was
allocated by a nonblocking communication call then the request is deallocated
and the handle is set to \const{MPI\_REQUEST\_NULL}.
(The array is indexed from zero in C, and from one in Fortran.)
In the latter case (no operation completed), it returns {\sf flag =
false}, returns a value
of \const{MPI\_UNDEFINED} in \mpiarg{index} and \mpiarg{status} is
undefined.  The array may contain null or inactive handles.

\snir
If the
array contains no active handles then the call returns
immediately with {\sf flag = true},
\mpiarg{index} = \const{MPI\_UNDEFINED}, and a null \mpiarg{status}.

If the array of requests contain active handles then
\rins
the execution of \func{MPI\_TESTANY(count, array\_of\_requests,
index, status)} has the same effect as the execution of
\func{MPI\_TEST( \&amp;array\_of\_requests[i], flag, status)},
for {\sf i=0, 1 ,..., count-1},
in some arbitrary order, until one call returns {\sf flag = true}, or
all fail.  In the former case, {\sf index} is set to the last value of {\sf i},
and in the latter case, it is set to \const{MPI\_UNDEFINED}.
\mpifunc{MPI\_TESTANY} with an array containing one active entry
is equivalent to \mpifunc{MPI\_TEST}.

\snir
\begin{rationale}
The function \mpifunc{MPI\_TESTANY} returns with \mpiarg{flag = true}
exactly in those situations where the function \mpifunc{MPI\_WAITANY}
returns; both functions return in such case the same values in
the remaining parameters.  Thus, a blocking \mpifunc{MPI\_WAITANY}
can be easily replaced by a
nonblocking \mpifunc{MPI\_TESTANY}.  The same relation holds for the
other pairs of Wait and Test functions defined in this section.
\end{rationale}
\rins


\begin{funcdef}{MPI\_WAITALL( count, array\_of\_requests, array\_of\_statuses)}
\funcarg{\IN}{count}{lists length (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handles)}
\funcarg{\OUT}{array\_of\_statuses}{array of status objects (array of Status)}
\end{funcdef}

\mpibind{MPI\_Waitall(int~count, MPI\_Request~*array\_of\_requests, MPI\_Status~*array\_of\_statuses)}

\mpifbind{MPI\_WAITALL(COUNT, ARRAY\_OF\_REQUESTS, ARRAY\_OF\_STATUSES, IERROR)\fargs INTEGER  COUNT, ARRAY\_OF\_REQUESTS(*) \\ INTEGER  ARRAY\_OF\_STATUSES(MPI\_STATUS\_SIZE,*), IERROR}


Blocks until all communication operations associated with active handles
in the list complete, and return the status of all these operations
(this includes the case where no handle in the list is active).
Both arrays have the same number of valid entries.  The \mpiarg{i}-th entry in
\mpiarg{array\_of\_statuses} is set to the return status of the
\mpiarg{i}-th operation.
Requests that were created by nonblocking communication operations are
deallocated and the corresponding handles in the array are set to
\const{MPI\_REQUEST\_NULL}.
The list may contain null or inactive handles.
The call sets to null the status of each such entry.

The execution of \func{MPI\_WAITALL(count, array\_of\_requests,
array\_of\_statuses)} has the same effect as the execution of
\func{MPI\_WAIT(\&amp;array\_of\_request[i], \&amp;array\_of\_statuses[i])},
for {\sf i=0 ,..., count-1}, in some arbitrary order.
\mpifunc{MPI\_WAITALL} with an array of length one
is equivalent to \mpifunc{MPI\_WAIT}.


\begin{funcdef}{MPI\_TESTALL(count, array\_of\_requests, flag,
array\_of\_statuses)}
\funcarg{\IN}{count}{lists length (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handles)}
\funcarg{\OUT}{flag}{(logical)}
\funcarg{\OUT}{array\_of\_statuses}{array of status objects (array of Status)}
\end{funcdef}

\mpibind{MPI\_Testall(int~count, MPI\_Request~*array\_of\_requests, int~*flag, MPI\_Status~*array\_of\_statuses)}

\mpifbind{MPI\_TESTALL(COUNT, ARRAY\_OF\_REQUESTS, FLAG, ARRAY\_OF\_STATUSES, IERROR)\fargs LOGICAL  FLAG \\ INTEGER  COUNT, ARRAY\_OF\_REQUESTS(*), ARRAY\_OF\_STATUSES(MPI\_STATUS\_SIZE,*), IERROR}


Returns \mpiarg{flag = true}
if all communications associated
with active handles in the array have completed (this includes the
case where no handle in the list is active).
In this case, each status entry that corresponds to an active handle
request
is set to the status of the corresponding communication; if the request was
allocated by a nonblocking communication call then it is deallocated, and
the handle is set to \const{MPI\_REQUEST\_NULL}.
Each status entry that corresponds to a null or inactive
handle is set to null.

Otherwise,
\mpiarg{flag = false} is returned, no request is modified
and the values of the status entries are undefined.
This is a local operation.


\begin{funcdef}{MPI\_WAITSOME(incount, array\_of\_requests, outcount,
array\_of\_indices, array\_of\_statuses)}
\funcarg{\IN}{incount}{length of array\_of\_requests (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handles)}
\funcarg{\OUT}{outcount}{number of completed requests (integer)}
\funcarg{\OUT}{array\_of\_indices}{array of indices of operations that
completed (array of integers)}
\funcarg{\OUT}{array\_of\_statuses}{array of status objects for
    operations that completed (array of Status)}
\end{funcdef}

\mpibind{MPI\_Waitsome(int~incount, MPI\_Request~*array\_of\_requests, int~*outcount, int~*array\_of\_indices, MPI\_Status~*array\_of\_statuses)}

\mpifbind{MPI\_WAITSOME(INCOUNT, ARRAY\_OF\_REQUESTS, OUTCOUNT, ARRAY\_OF\_INDICES, ARRAY\_OF\_STATUSES, IERROR)\fargs INTEGER INCOUNT, ARRAY\_OF\_REQUESTS(*), OUTCOUNT, ARRAY\_OF\_INDICES(*), ARRAY\_OF\_STATUSES(MPI\_STATUS\_SIZE,*), IERROR}

Waits until at least one of the operations associated with active
handles in the list have completed.
Returns in \mpiarg{outcount} the number of requests from the list
\mpiarg{array\_of\_requests} that have completed.  Returns in the first
\mpiarg{outcount} locations of the array \mpiarg{array\_of\_indices}
the indices of these operations (index within the
array \mpiarg{array\_of\_requests}; the array is indexed from zero in
C and from one in
Fortran).  Returns in the first \mpiarg{outcount}
locations of the array \mpiarg{array\_of\_status}
the status for these completed operations.  If a request that completed was
allocated by a nonblocking communication call, then it is deallocated, and the
associated handle is set to \const{MPI\_REQUEST\_NULL}.

% isn't the apparent "packing" of statuses wrong?  Apparently
% not...but I don't like it.

\snir
If the list contains no active handles, then the
call returns immediately with \mpiarg{outcount = MPI\_UNDEFINED}.
\rins


\begin{funcdef}{MPI\_TESTSOME(incount, array\_of\_requests, outcount,
array\_of\_indices, array\_of\_statuses)}
\funcarg{\IN}{incount}{length of array\_of\_requests (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handles)}
\funcarg{\OUT}{outcount}{number of completed requests (integer)}
\funcarg{\OUT}{array\_of\_indices}{array of indices of operations that
completed (array of integers)}
\funcarg{\OUT}{array\_of\_statuses}{array of status objects for
    operations that completed (array of Status)}
\end{funcdef}

\mpibind{MPI\_Testsome(int~incount, MPI\_Request~*array\_of\_requests, int~*outcount, int~*array\_of\_indices, MPI\_Status~*array\_of\_statuses)}

\mpifbind{MPI\_TESTSOME(INCOUNT, ARRAY\_OF\_REQUESTS, OUTCOUNT, ARRAY\_OF\_INDICES, ARRAY\_OF\_STATUSES, IERROR)\fargs INTEGER INCOUNT, ARRAY\_OF\_REQUESTS(*), OUTCOUNT, ARRAY\_OF\_INDICES(*), ARRAY\_OF\_STATUSES(MPI\_STATUS\_SIZE,*), IERROR}

Behaves like \mpifunc{MPI\_WAITSOME}, except that it returns
immediately. If no operation has completed it
returns \mpiarg{outcount = 0}.  If there is no active handle in the list it
returns \mpiarg{outcount = MPI\_UNDEFINED}.


\mpifunc{MPI\_TESTSOME} is a local operation, which returns immediately, whereas
\mpifunc{MPI\_WAIT-} \func{SOME} will %mansplit
block until a communication completes, if it was
passed a list that contains at least one active handle.  Both calls fulfil a
{\sf fairness} requirement:  If a request for a receive repeatedly
appears in a list of requests passed to \mpifunc{MPI\_WAITSOME} or
\mpifunc{MPI\_TESTSOME}, and a matching send has been posted, then the receive
will eventually succeed, unless the send is satisfied by another receive; and
similarly for send requests.

\begin{users}
The use of \mpifunc{MPI\_TESTSOME} is likely to be more efficient than the use
of \mpifunc{MPI\_TESTANY}. The former returns information on all
completed communications, with the latter, a new call is required for
each communication that completes.

A server with multiple clients can use \mpifunc{MPI\_WAITSOME} so as not to
starve any client.   Clients send messages to the server with service
requests. The server calls \mpifunc{MPI\_WAITSOME} with one receive request
for each client, and then handles all receives that completed.
If a call to \mpifunc{MPI\_WAITANY} is used instead, then one client
could starve while requests from another client always sneak in first.
\end{users}

\begin{implementors}
\mpifunc{MPI\_TESTSOME} should complete as many pending communications as
possible.
\end{implementors}



\begin{example} {\rm
\label{pt2pt-exN}
Client-server code (starvation can occur).
\begin{verbatim}

CALL MPI_COMM_SIZE(comm, size, ierr)
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank &gt; 0) THEN         ! client code
    DO WHILE(.TRUE.)
       CALL MPI_ISEND(a, n, MPI_REAL, 0, tag, comm, request, ierr)
       CALL MPI_WAIT(request, status, ierr)
    END DO
ELSE         ! rank=0 -- server code
       DO i=1, size-1
          CALL MPI_IRECV(a(1,i), n, MPI_REAL, 0, tag,
                   comm, request_list(i), ierr)
       END DO
       DO WHILE(.TRUE.)
          CALL MPI_WAITANY(size-1, request_list, index, status, ierr)
          CALL DO_SERVICE(a(1,index))  ! handle one message
          CALL MPI_IRECV(a(1, index), n, MPI_REAL, 0, tag,
                    comm, request_list(index), ierr)
       END DO
END IF
\end{verbatim}
} \end{example}
\label{page:pt2pt-waitany}




\begin{example} {\rm
\label{pt2pt-exO}
Same code, using \mpifunc{MPI\_WAITSOME}.
\begin{verbatim}

CALL MPI_COMM_SIZE(comm, size, ierr)
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank &gt; 0) THEN         ! client code
    DO WHILE(.TRUE.)
       CALL MPI_ISEND(a, n, MPI_REAL, 0, tag, comm, request, ierr)
       CALL MPI_WAIT(request, status, ierr)
    END DO
ELSE         ! rank=0 -- server code
    DO i=1, size-1
       CALL MPI_IRECV(a(1,i), n, MPI_REAL, 0, tag,
                      comm, request_list(i), ierr)
    END DO
    DO WHILE(.TRUE.)
       CALL MPI_WAITSOME(size, request_list, numdone,
                        index_list, status_list, ierr)
       DO i=1, numdone
          CALL DO_SERVICE(a(1, index_list(i)))
          CALL MPI_IRECV(a(1, index_list(i)), n, MPI_REAL, 0, tag,
                       comm, request_list(i), ierr)
       END DO
    END DO
END IF
\end{verbatim}
} \end{example}


\section{Probe and Cancel}
\label{sec:pt2pt-probe}


The \func{MPI\_PROBE} and \func{MPI\_IPROBE}
operations allow incoming messages to be checked for,
without
actually receiving them.  The user can then decide how to receive them, based
on the information returned by the probe (basically, the information
returned by \mpiarg{status}).  In particular, the user may allocate memory for
the receive buffer, according to the length of the probed message.

The \func{MPI\_CANCEL} operation allows pending communications to be canceled.
This
is required for cleanup.   Posting a send or a receive ties up user resources
(send or receive buffers), and a cancel may be needed to free these resources
gracefully.


\begin{funcdef}{MPI\_IPROBE(source, tag, comm, flag, status)}
\funcarg{\IN}{source}{source rank, or \const{ MPI\_ANY\_SOURCE} (integer)}
\funcarg{\IN}{tag}{tag value or \const{ MPI\_ANY\_TAG} (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{flag}{ (logical)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Iprobe(int~source, int~tag, MPI\_Comm~comm, int~*flag, MPI\_Status~*status)}

\mpifbind{MPI\_IPROBE(SOURCE, TAG, COMM, FLAG, STATUS, IERROR)\fargs LOGICAL  FLAG \\ INTEGER  SOURCE, TAG, COMM, STATUS(MPI\_STATUS\_SIZE), IERROR}



\func{MPI\_IPROBE(source, tag, comm, flag, status)}
returns \mpiarg{flag = true}
if there is a message that can be received
and that matches the pattern specified by the
arguments \mpiarg{source}, \mpiarg{tag}, and \mpiarg{comm}.
The call matches the same message
that would have been received by a call to \mpifunc{MPI\_RECV(..., source, tag,
comm, status)} executed at the same point in the program, and returns in
\mpiarg{status} the same
value that would have been returned by \func{MPI\_RECV()}.
Otherwise, the call returns \mpiarg{flag = false}, and leaves \mpiarg{status}
undefined.

If \func{MPI\_IPROBE} returns \mpiarg{flag = true},
then the content of the status object can be subsequently
accessed as described in section~\ref{subsec:pt2pt-status} to find the
source, tag and length of the probed message.

A subsequent
receive executed with the same context, and the source and tag
returned in \mpiarg{status} by
\func{MPI\_IPROBE} will receive the message
that was matched by the probe, if no other intervening receive occurs after
the probe.
If the receiving process is multi-threaded, it is the user's
responsibility to ensure that the last condition holds.

The \mpiarg{source} argument of \mpifunc{MPI\_PROBE} can be
\const{MPI\_ANY\_SOURCE}, and the \mpiarg{tag} argument can be
\const{MPI\_ANY\_TAG}, so that one can probe for messages from an arbitrary
source and/or with
an arbitrary tag.  However, a specific communication context
must be provided with the \mpiarg{comm} argument.

It is not necessary to receive a message immediately after it has been
probed for, and the
same message may be probed for several times before it is received.



\begin{funcdef}{MPI\_PROBE(source, tag, comm, status)}
\funcarg{\IN}{source}{source rank, or \const{ MPI\_ANY\_SOURCE} (integer)}
\funcarg{\IN}{tag}{tag value, or \const{ MPI\_ANY\_TAG} (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Probe(int~source, int~tag, MPI\_Comm~comm, MPI\_Status~*status)}

\mpifbind{MPI\_PROBE(SOURCE, TAG, COMM, STATUS, IERROR)\fargs INTEGER SOURCE, TAG, COMM, STATUS(MPI\_STATUS\_SIZE), IERROR}



\func{MPI\_PROBE} behaves like \func{MPI\_IPROBE} except that it is a blocking
call that returns only after a matching message has been found.

The \MPI/ implementation of \mpifunc{MPI\_PROBE} and \mpifunc{MPI\_IPROBE} needs
to guarantee progress:
if a call to \mpifunc{MPI\_PROBE} has been issued by a process, and a send that
matches the probe has been initiated by some process, then the call to
\mpifunc{MPI\_PROBE} will return, unless the message is received by another
concurrent receive operation (that is executed by another thread at the probing
process).   Similarly, if a process busy waits with
\mpifunc{MPI\_IPROBE} and a matching message has been issued,
then the call to
\mpifunc{MPI\_IPROBE} will eventually return {\sf flag = true}
unless the message is received by another concurrent receive
operation.


\begin{example} {\rm
\label{pt2pt-exP}

Use blocking probe to wait for an incoming message.

\begin{verbatim}
       CALL MPI_COMM_RANK(comm, rank, ierr)
       IF (rank.EQ.0) THEN
            CALL MPI_SEND(i, 1, MPI_INTEGER, 2, 0, comm, ierr)
       ELSE IF(rank.EQ.1) THEN
            CALL MPI_SEND(x, 1, MPI_REAL, 2, 0, comm, ierr)
       ELSE   ! rank.EQ.2
           DO i=1, 2
              CALL MPI_PROBE(MPI_ANY_SOURCE, 0,
                              comm, status, ierr)
              IF (status(MPI_SOURCE) = 0) THEN
100                CALL MPI_RECV(i, 1, MPI_INTEGER, 0, 0, status, ierr)
              ELSE
200                CALL MPI_RECV(x, 1, MPI_REAL, 1, 0, status, ierr)
              END IF
           END DO
       END IF
\end{verbatim}
Each message is received with the right type.
} \end{example}


\begin{example} {\rm
\label{pt2pt-exQ}
A similar program to the previous example, but now it
has a problem.
\begin{verbatim}
       CALL MPI_COMM_RANK(comm, rank, ierr)
       IF (rank.EQ.0) THEN
            CALL MPI_SEND(i, 1, MPI_INTEGER, 2, 0, comm, ierr)
       ELSE IF(rank.EQ.1) THEN
            CALL MPI_SEND(x, 1, MPI_REAL, 2, 0, comm, ierr)
       ELSE
           DO i=1, 2
              CALL MPI_PROBE(MPI_ANY_SOURCE, 0,
                              comm, status, ierr)
              IF (status(MPI_SOURCE) = 0) THEN
100                CALL MPI_RECV(i, 1, MPI_INTEGER, MPI_ANY_SOURCE,
                                 0, status, ierr)
              ELSE
200                CALL MPI_RECV(x, 1, MPI_REAL, MPI_ANY_SOURCE,
                                 0, status, ierr)
              END IF
           END DO
       END IF
\end{verbatim}

We slightly modified example \ref{pt2pt-exP}, using
\const{MPI\_ANY\_SOURCE} as the \mpiarg{source}
argument in the two receive calls in statements labeled 100 and 200.
The program is now incorrect: the receive operation may receive a message that
is distinct from the message probed by the preceding call to
\mpifunc{MPI\_PROBE}.
} \end{example}



\begin{implementors}
A call to \mpifunc{MPI\_PROBE(source, tag, comm, status)} will match the
message that would have been received by a call to \mpifunc{MPI\_RECV(...,
source, tag, comm, status)} executed at the same point.
Suppose
that this message has source {\sf s}, tag {\sf t} and communicator
{\sf c}.  If the
tag argument in the probe call has value \const{MPI\_ANY\_TAG}
then the message probed
will be the earliest pending message from source {\sf s}
with communicator {\sf c} and any tag; in any case, the message probed will be the
earliest pending message from source {\sf s} with tag {\sf t} and
communicator
{\sf c} (this is
the message that would have been received, so as to preserve message order).
This message continues as the earliest pending message
from source {\sf s} with tag {\sf t} and communicator {\sf c}, until it is received.
A receive operation subsequent to the probe that uses the same communicator as the
probe and uses the tag and source values returned by the probe, must receive
this message, unless it has already been received by another receive operation.
\end{implementors}



\begin{funcdef}{MPI\_CANCEL(request)}
\funcarg{\IN}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Cancel(MPI\_Request~*request)}

\mpifbind{MPI\_CANCEL(REQUEST, IERROR)\fargs INTEGER  REQUEST, IERROR}



A call to \func{MPI\_CANCEL} marks for cancellation a pending,
nonblocking communication operation (send or receive).  The cancel call is
local. It returns immediately, possibly before the
communication is actually canceled.
It is still necessary to complete a communication that has been marked
for cancellation, using a call to \func{MPI\_REQUEST\_FREE},
\func{MPI\_WAIT} or \func{MPI\_TEST} (or any of the derived operations).

If a communication is marked for cancellation, then a \func{MPI\_WAIT}
call for that communication is guaranteed to return, irrespective of
the activities of other processes (i.e., \func{MPI\_WAIT} behaves as a
local function); similarly if \func{MPI\_TEST} is
repeatedly called in a busy wait loop for a canceled communication,
then \mpifunc{MPI\_TEST} will eventually be successful.

\mpifunc{MPI\_CANCEL} can be used to cancel a communication that uses
a persistent request (see Sec. \ref{sec:pt2pt-persistent}), in
the same way it is used for nonpersistent requests.
A successful cancellation cancels
the active communication, but not the request itself.  After the call to
\mpifunc{MPI\_CANCEL} and the subsequent call to \mpifunc{MPI\_WAIT} or
\mpifunc{MPI\_TEST}, the request becomes inactive and
can be activated for a new communication.

The successful
cancellation of a buffered send frees the buffer space occupied by
the pending message.

Either the cancellation succeeds, or the communication succeeds, but
not both.
If a send is marked for cancellation, then it must be the case that
either the send completes normally, in which case the
message sent was received at the destination process, or that the send is
successfully
canceled, in which case no part of the message was received at the
destination.  Then, any matching receive has to be satisfied by another send.
If a receive is marked for cancellation, then it must be the case that
either the receive completes normally, or that the receive is
successfully canceled, in which case no part of the receive buffer
is altered.  Then, any matching send has to be satisfied by another receive.

If the operation has been
canceled, then information to that effect will be returned in the
status argument of the operation that completes the communication.


\begin{funcdef}{MPI\_TEST\_CANCELLED(status, flag)}
\funcarg{\IN}{status}{status object (Status)}
\funcarg{\OUT}{flag}{(logical)}
\end{funcdef}

\mpibind{MPI\_Test\_cancelled(MPI\_Status~status, int~*flag)}

\mpifbind{MPI\_TEST\_CANCELLED(STATUS, FLAG, IERROR)\fargs LOGICAL FLAG \\ INTEGER  STATUS(MPI\_STATUS\_SIZE), IERROR}



Returns \mpiarg{flag = true} if the communication associated with the
status object was canceled successfully.  In such a case, all
other fields of \mpiarg{status} (such as \mpiarg{count} or \mpiarg{tag}) are
undefined.  Returns \mpiarg{flag = false}, otherwise.  If a receive
operation might be canceled then one should call \func{MPI\_TEST\_CANCELLED}
first, to check whether the operation was
canceled, before checking on the other fields of the return status.

\begin{users}
Cancel can be an expensive operation that should be used only exceptionally.
\end{users}

\begin{implementors}
If a send operation uses an ``eager'' protocol (data is transferred to
the receiver
before a matching receive is posted), then the cancellation of this send
may require communication with the intended receiver in order to free
allocated
buffers.  On some systems this may require an interrupt to the
intended receiver.
Note that, while communication may be needed to implement
\mpifunc{MPI\_CANCEL},
this is still a local operation, since its completion does not
depend on the code executed by other processes.  If processing is required on
another process, this should be transparent to the application (hence the need
for an interrupt and an interrupt handler).
\end{implementors}


\section{Persistent communication requests}
\label{sec:pt2pt-persistent}

Often a communication with the same argument list is repeatedly
executed within the inner loop of a parallel computation.  In such a
situation, it may be possible to optimize the communication by
binding the list of communication arguments to a {\bf persistent} communication
request once and, then, repeatedly using
the request to initiate and complete messages.  The
persistent request thus created can be thought of as a
communication port or a ``half-channel.''
It does not provide the full functionality of a conventional channel,
since there is no binding of the send port to the receive port. This
construct allows reduction of the overhead for communication
between the process and communication controller, but not of the overhead for
communication between one communication controller and another.
It is not necessary that messages sent with a persistent request be received
by a receive operation using a persistent request, or vice versa.

A persistent communication request is created using one of the four following
calls.  These calls involve no communication.



\begin{funcdef}{MPI\_SEND\_INIT(buf, count, datatype, dest, tag,
comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements sent (integer)}
\funcarg{\IN}{datatype}{type of each element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Send\_init(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_SEND\_INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt;  BUF(*) \\ INTEGER  REQUEST, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Creates a persistent communication request
for a standard mode send operation, and binds to it all the
arguments of a send operation.



\begin{funcdef}{MPI\_BSEND\_INIT(buf, count, datatype, dest, tag,
comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements sent (integer)}
\funcarg{\IN}{datatype}{type of each element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Bsend\_init(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_BSEND\_INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt;  BUF(*) \\ INTEGER  REQUEST, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Creates a persistent communication request for a buffered mode send.




\begin{funcdef}{MPI\_SSEND\_INIT(buf, count, datatype, dest, tag,
comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements sent (integer)}
\funcarg{\IN}{datatype}{type of each element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Ssend\_init(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_SSEND\_INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt;  BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Creates a persistent communication object
for a synchronous mode send operation.



\begin{funcdef}{MPI\_RSEND\_INIT(buf, count, datatype, dest, tag,
comm, request)}
\funcarg{\IN}{buf}{initial address of send buffer (choice)}
\funcarg{\IN}{count}{number of elements sent (integer)}
\funcarg{\IN}{datatype}{type of each element (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{tag}{message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Rsend\_init(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_RSEND\_INIT(BUF, COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt;  BUF(*) \\ INTEGER  COUNT, DATATYPE, DEST, TAG, COMM, REQUEST, IERROR}

Creates a persistent communication object
for a ready mode send operation.



\begin{funcdef}{MPI\_RECV\_INIT(buf, count, datatype, source, tag,
comm, request)}
\funcarg{\OUT}{buf}{initial address of receive buffer (choice)}
\funcarg{\IN}{count}{number of elements received (integer)}
\funcarg{\IN}{datatype}{type of each element (handle)}
\funcarg{\IN}{source}{rank of source or MPI\_ANY\_SOURCE (integer)}
\funcarg{\IN}{tag}{message tag or MPI\_ANY\_TAG (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Recv\_init(void*~buf, int~count, MPI\_Datatype~datatype, int~source, int~tag, MPI\_Comm~comm, MPI\_Request~*request)}

\mpifbind{MPI\_RECV\_INIT(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR)\fargs &lt;type&gt;  BUF(*) \\ INTEGER  COUNT, DATATYPE, SOURCE, TAG, COMM, REQUEST, IERROR}

Creates a persistent communication request
for a receive operation.  The argument \mpiarg{buf} is marked as \mpiarg{OUT}
because the user gives permission to write on the receive buffer by passing the
argument to \func{MPI\_RECV\_INIT}.

A persistent communication request is inactive after it was created
--- no active communication is attached to the request.

A communication (send or receive) that uses a persistent request
is initiated by the function \mpifunc{MPI\_START}.

\begin{funcdef}{MPI\_START(request)}
\funcarg{\INOUT}{request}{communication request (handle)}
\end{funcdef}

\mpibind{MPI\_Start(MPI\_Request~*request)}

\mpifbind{MPI\_START(REQUEST, IERROR)\fargs INTEGER  REQUEST, IERROR}


The argument, \mpiarg{request}, is a handle returned
by one of the previous five calls.   The associated request should be
inactive.  The request becomes active once the call is made.

If the request is for a send with ready mode, then
a matching receive should be posted before the call is made.  The
communication buffer should not be accessed after the call, and
until the operation completes.

The call is local, with similar semantics to the nonblocking
communication operations described in
section~\ref{sec:pt2pt-nonblock}.  That is,
a call to \mpiarg{MPI\_START} with a
request created by \mpifunc{MPI\_SEND\_INIT}
starts a
communication in the same manner as a call to \mpifunc{MPI\_ISEND};
a call to \mpiarg{MPI\_START} with a
request created by \mpifunc{MPI\_BSEND\_INIT}
starts a
communication in the same manner as a call to
\mpifunc{MPI\_IBSEND};  and so on.


\begin{funcdef}{MPI\_STARTALL(count, array\_of\_requests)}
\funcarg{\IN}{count}{list length (integer)}
\funcarg{\INOUT}{array\_of\_requests}{array of requests (array of handle)}
\end{funcdef}

\mpibind{MPI\_Startall(int~count, MPI\_Request~*array\_of\_requests)}

\mpifbind{MPI\_STARTALL(COUNT, ARRAY\_OF\_REQUESTS, IERROR)\fargs INTEGER  COUNT, ARRAY\_OF\_REQUESTS(*), IERROR}



Start all communications associated with requests in
\mpiarg{array\_of\_requests}.  A call to
\linebreak
\func{MPI\_STARTALL(count, array\_of\_requests)} has the
same effect as calls to
\func{MPI\_START} \func{(\&amp;array\_of\_requests[i])},  %manpslit
executed for {\sf i=0 ,..., count-1}, in some arbitrary order.

A communication started with a call to \mpifunc{MPI\_START} or
\mpifunc{MPI\_STARTALL} is
completed by a call to \mpifunc{MPI\_WAIT}, \mpifunc{MPI\_TEST}, or
one of the derived functions described in
section~\ref{subsec:pt2pt-multiple}.  The request becomes inactive after
successful completion of such call.   The request is not deallocated
and it can be activated anew by an \mpifunc{MPI\_START} or
\mpifunc{MPI\_STARTALL} call.

A persistent request is deallocated by a call to
\mpifunc{MPI\_REQUEST\_FREE}
(Section~\ref{subsec:pt2pt-commend}).

The call to \mpifunc{MPI\_REQUEST\_FREE} can occur at any point in the program
after the persistent request was created.  However, the request will be
deallocated only after it becomes inactive.
Active receive requests should not be freed. Otherwise, it will not be
possible to check that the receive has completed.
It is preferable, in general, to free requests when they are inactive.  If this
rule is followed, then the functions
described in this section will be invoked
in a sequence of the form,

\vspace{.5cm}
\( \bf
Create \ (Start \ Complete)^* \ Free   \ ,
\)
\vspace{.5cm}
where $*$ indicates zero or more repetitions.
If the same communication object is used in several concurrent
threads, it is the user's responsibility to coordinate calls so that the
correct sequence is obeyed.

A send operation initiated with \func{MPI\_START} can be matched with
any receive operation and, likewise, a receive operation initiated
with \func{MPI\_START} can receive messages generated by any send
operation.



\section{Send-receive}
\label{sec:pt2pt-sendrecv}

The {\bf send-receive} operations combine in one call the sending of a
message to one destination and the receiving of another message, from
another process.  The two (source and destination) are possibly the same.
A send-receive operation is
very useful for executing a shift operation across a chain of
processes.  If blocking sends and receives are used for such a shift,
then one needs to order the sends and receives correctly (for
example, even processes
send, then receive, odd processes receive first, then send) so as to prevent
cyclic dependencies that may lead to deadlock.  When a send-receive
operation is used, the communication subsystem takes care of
these issues.  The send-receive operation can be used in conjunction
with the functions described in Chapter~\ref{chap:topol} in order to
perform shifts on various logical topologies.
Also, a send-receive operation is useful for implementing
remote procedure calls.


A message sent by a
send-receive operation can be received by a regular receive
operation or probed by a probe operation; a send-receive operation can
receive a message sent by a regular send operation.


\begin{funcdef}{MPI\_SENDRECV(sendbuf, sendcount, sendtype, dest,
sendtag, recvbuf, recvcount, recvtype, source, recvtag, comm, status)}
\funcarg{\IN}{sendbuf}{initial address of send buffer (choice)}
\funcarg{\IN}{sendcount}{number of elements in send buffer (integer)}
\funcarg{\IN}{sendtype}{type of elements in send buffer (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{sendtag}{send tag (integer)}
\funcarg{\OUT}{recvbuf}{initial address of receive buffer (choice)}
\funcarg{\IN}{recvcount}{number of elements in receive buffer (integer)}
\funcarg{\IN}{recvtype}{ type of elements in receive buffer (handle)}
\funcarg{\IN}{source}{rank of source (integer)}
\funcarg{\IN}{recvtag}{receive tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Sendrecv(void~*sendbuf, int~sendcount, MPI\_Datatype~sendtype, int~dest, int~sendtag, void~*recvbuf, int~recvcount, MPI\_Datatype~recvtype, int~source, MPI\_Datatype~recvtag, MPI\_Comm~comm, MPI\_Status~*status)}

\mpifbind{MPI\_SENDRECV(SENDBUF, SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVBUF, RECVCOUNT, RECVTYPE, SOURCE, RECVTAG, COMM, STATUS, IERROR)\fargs &lt;type&gt; SENDBUF(*), RECVBUF(*) \\ INTEGER  SENDCOUNT, SENDTYPE, DEST, SENDTAG, RECVCOUNT, RECVTYPE, SOURCE, RECV
TAG, COMM, STATUS(MPI\_STATUS\_SIZE), IERROR}



Execute a blocking send and receive operation.  Both send and receive
use the same communicator, but
possibly different tags.  The send buffer and receive buffers must be
disjoint, and may have different lengths and datatypes.

\begin{funcdef}{MPI\_SENDRECV\_REPLACE(buf, count, datatype, dest,
sendtag, source, recvtag, comm, status)}
\funcarg{\INOUT}{buf}{initial address of send and receive buffer (choice)}
\funcarg{\IN}{count}{number of elements in send and receive buffer (integer)}
\funcarg{\IN}{datatype}{type of elements in send and receive buffer (handle)}
\funcarg{\IN}{dest}{rank of destination (integer)}
\funcarg{\IN}{sendtag}{send message tag (integer)}
\funcarg{\IN}{source}{rank of source (integer)}
\funcarg{\IN}{recvtag}{receive message tag (integer)}
\funcarg{\IN}{comm}{communicator (handle)}
\funcarg{\OUT}{status}{status object (Status)}
\end{funcdef}

\mpibind{MPI\_Sendrecv\_replace(void*~buf, int~count, MPI\_Datatype~datatype, int~dest, int~sendtag, int~source, int~recvtag, MPI\_Comm~comm, MPI\_Status~*status)}

\mpifbind{MPI\_SENDRECV\_REPLACE(BUF, COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG, COMM, STATUS, IERROR)\fargs &lt;type&gt; BUF(*) \\ INTEGER COUNT, DATATYPE, DEST, SENDTAG, SOURCE, RECVTAG, COMM, STATUS(MPI\_STATUS\_SIZE), IERROR}



Execute a blocking send and receive. The same buffer is used both for
the send and for the receive, so that the message sent is replaced by
the message received.

The semantics of a send-receive operation is what would be obtained
if the caller forked two concurrent threads, one to execute the send,
and one to execute the receive, followed by a join of these two
threads.

\begin{implementors}
Additional intermediate buffering is needed for the
``replace'' variant.
\end{implementors}

\section{Null processes}
\label{sec:pt2pt-nullproc}

In many instances, it is convenient to specify a ``dummy'' source or
destination
for communication.  This simplifies the code that is needed for dealing with
boundaries, for example, in the case of a non-circular shift done with calls to
send-receive.

The special value \const{ MPI\_PROC\_NULL} can be used
instead of a rank wherever a
source or a destination argument is required in a call.   A communication
with process \const{ MPI\_PROC\_NULL} has no effect.
A send to \const{ MPI\_PROC\_NULL}
succeeds and returns as soon as possible.
A receive from \const{ MPI\_PROC\_NULL} succeeds and
returns as soon as possible with no modifications to the receive buffer.
When a receive with \mpiarg{source} = \const{ MPI\_PROC\_NULL} is executed then
the status object returns \mpiarg{
source} = \const{ MPI\_PROC\_NULL}, \mpiarg{tag} = \const{ MPI\_ANY\_TAG} and
\mpiarg{count = 0}.

\section{Derived datatypes}
\label{sec:pt2pt-datatype}

Up to here, all point to point communication have involved only contiguous
buffers containing a sequence of elements of the same type.
This is too constraining on two accounts.  One often wants to pass
messages that contain values with different datatypes (e.g., an integer count,
followed by a sequence of real numbers); and one often wants to send
noncontiguous data (e.g., a sub-block of a matrix).  One solution is to
pack noncontiguous data into a contiguous buffer
at the sender site and unpack it back at the receiver site.   This has
the disadvantage of requiring additional memory-to-memory copy operations
at both sites, even when the communication
subsystem has scatter-gather capabilities.   Instead, \MPI/ provides
mechanisms to specify more general, mixed, and noncontiguous
communication buffers. It is up to the implementation to decide
whether data should be first packed in a contiguous buffer before being
transmitted, or whether it can be collected directly from where it
resides.

The general mechanisms provided here allow one to transfer directly,
without copying, objects of various shape and size.  It is not assumed
that the \MPI/ library is cognizant of the objects declared in the host
language. Thus, if one wants to transfer a structure, or an array
section, it will be necessary to provide in \MPI/ a definition of a
communication buffer that mimics the definition of the structure or
array section in question.  These facilities can be used by library
designers to define communication functions that can transfer objects
defined in the host language --- by decoding their definitions as
available in a symbol table or a dope vector.  Such higher-level
communication functions are not part of \MPI/.

More general communication buffers are specified by replacing the
basic datatypes that have been used so far with derived datatypes that
are constructed from basic datatypes using the constructors described
in this section.  These methods of constructing derived datatypes can
be applied recursively.

A {\bf general datatype} is an opaque object that specifies two
things:
\begin{itemize}
\item
A sequence of basic datatypes
\item
A sequence of integer (byte) displacements
\end{itemize}

The displacements are not required to be positive, distinct, or
in increasing order. Therefore, the order of items need not
coincide with their order in store, and an item may appear more than
once.
We call such a pair of sequences (or sequence of pairs) a {\bf type map}.
The sequence of basic datatypes (displacements ignored) is the {\bf
type signature} of the datatype.

% ARM: need to index these.

Let
\[
Typemap = \{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\]
be such a type map, where $type_i$ are basic types, and
$disp_i$ are  displacements.
Let
\[
Typesig = \{ type_0 , ... , type_{n-1} \}
\]
be the associated type signature.
This type map, together with a base address {\em buf},
specifies a communication buffer: the communication buffer that consists of $n$
entries, where the $i$-th entry is at address $buf +
disp_i$ and has type $type_i$.
A message assembled from such a
communication buffer will consist of $n$ values, of the types defined
by $Typesig$.

We can use a handle to a general datatype as an argument in a send or
receive operation, instead of a basic datatype argument.  The
operation
\func{MPI\_SEND(buf, 1, datatype,...)} will use the send buffer
defined by the base address \mpiarg{buf} and the general datatype
associated with \mpiarg{datatype}; it will generate a message with the type
signature determined by the \mpiarg{datatype} argument.
\func{MPI\_RECV(buf, 1, datatype,...)} will use the receive buffer
defined by the base address \mpiarg{buf} and the general datatype
associated with \mpiarg{datatype}.

General datatypes can be used in all send and receive
operations.  We discuss, in Sec. \ref{subsec:pt2pt-datatypeuse}, the
case where the second argument \mpiarg{count} has value $&gt; 1$.

The basic datatypes presented in
section~\ref{subsec:pt2pt-messagedata}
are particular cases of a general datatype, and are predefined.
Thus, \type{MPI\_INT} is a predefined handle to a datatype with type
map $\{ ({\sf
int}, 0) \}$, with one entry of type {\sf int} and
displacement zero.  The other basic datatypes are similar.

The {\bf extent} of a datatype is defined to
be the span from the first byte to the last byte occupied by entries in this
datatype, rounded up to satisfy alignment requirements.
That is, if
\[
Typemap = \{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\]
then
\snir
\begin{eqnarray}
\label{eq:pt2pt-extent}
lb(Typemap) &amp; = &amp; \min_j disp_j , \nonumber \\
ub(Typemap) &amp; = &amp; \max_j (disp_j + sizeof(type_j)) + \epsilon , \mbox{ and}
\nonumber \\ extent(Typemap) &amp; = &amp; ub(Typemap) -lb(Typemap).
\end{eqnarray}
If $type_i$ requires alignment to a byte address that is is a multiple
of $k_i$,
then $\epsilon$ is the least nonnegative increment needed to round
$extent(Typemap)$ to the next multiple of $\max_i k_i$.
\rins

% ARM : Does epsilon really have to be this large?...depends upon how
% alignment works, in general...
% Do languages like C or Fortran specify this?


\begin{example} {\rm
\label{pt2pt-exR}
Assume that
$Type = \{ ({\sf double},0), ({\sf char}, 8) \}$
(a {\sf double} at
displacement zero, followed by a {\sf char} at displacement eight).
Assume, furthermore, that
doubles have to be strictly aligned at addresses that are multiples of eight.
Then, the extent of this datatype is 16 (9 rounded to the next multiple of 8).
A datatype that consists of a character immediately followed by a double will
also have an extent of 16.
} \end{example}

\begin{rationale}
The definition of extent is motivated by the assumption that
the amount of padding added at the end of each structure in an array of
structures is the least needed to fulfill alignment constraints.
More explicit control of the extent is provided in
section~\ref{subsec:pt2pt-markers}.  Such explicit control is needed
in cases where the assumption does not hold, for example, where union types
are used.
\end{rationale}

\subsection{Datatype constructors}
\label{subsec:pt2pt-datatypeconst}

\paragraph*{Contiguous}

The simplest datatype constructor is
\func{MPI\_TYPE\_CONTIGUOUS} which
allows replication of a datatype into contiguous locations.

\begin{funcdef}{MPI\_TYPE\_CONTIGUOUS(count, oldtype, newtype)}
\funcarg{\IN}{count}{replication count (nonnegative integer)}
\funcarg{\IN}{oldtype}{old datatype (handle)}
\funcarg{\OUT}{newtype}{new datatype (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_contiguous(int~count, MPI\_Datatype~oldtype, MPI\_Datatype~*newtype)}

\mpifbind{MPI\_TYPE\_CONTIGUOUS(COUNT, OLDTYPE, NEWTYPE, IERROR)\fargs INTEGER  COUNT, OLDTYPE, NEWTYPE, IERROR}



\mpiarg{newtype} is the datatype obtained by concatenating
\mpiarg{count} copies of
\mpiarg{oldtype}.   Concatenation is defined using {\em extent} as the size of
the concatenated copies.


\begin{example} {\rm
\label{pt2pt-exS}
Let \mpiarg{oldtype} have type map
\(
\{ ({\sf double}, 0), ({\sf char}, 8) \} ,
\)
with extent 16,
and let $\mpiarg{count} = 3$.  The type map of
the datatype returned by \mpiarg{newtype} is
\begin{displaymath}
\{ ({\sf double}, 0), ({\sf char}, 8), ({\sf double}, 16), ({\sf
char}, 24), ({\sf double}, 32), ({\sf char}, 40) \} ;
\end{displaymath}
i.e., alternating {\sf double} and {\sf char} elements, with displacements
$0, 8, 16, 24, 32, 40$.
} \end{example}

\vspace{.7cm}
In general,
assume that the type map of \mpiarg{oldtype} is
\begin{displaymath}
\{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\end{displaymath}
with extent $ex$.
Then \mpiarg{newtype} has a type map with $\sf count \cdot n$ entries
defined by:
\begin{displaymath}
\{ (type_0, disp_0), ..., (type_{n-1}, disp_{n-1}), (type_0, disp_0
+ex), ... ,(type_{n-1}, disp_{n-1} + ex) ,
\end{displaymath}
\begin{displaymath}
...,(type_0, disp_0 +ex \cdot({\sf count}-1) ), ... ,
(type_{n-1} , disp_{n-1} + ex \cdot ({\sf count}-1)) \} .
\end{displaymath}
\vspace{.7cm}

\paragraph*{Vector}

The function
\mpifunc{MPI\_TYPE\_VECTOR} is a more general constructor that
allows replication of a datatype
into locations that consist of equally spaced blocks.  Each block
is obtained by concatenating the same number of copies of the old datatype.
The spacing between blocks is a multiple of the extent of the old datatype.

\begin{funcdef}{MPI\_TYPE\_VECTOR( count, blocklength, stride, oldtype,
newtype)}
\funcarg{\IN}{count}{number of blocks (nonnegative integer)}
\funcarg{\IN}{blocklength}{number of elements in each block
(nonnegative integer)}
\funcarg{\IN}{stride}{number of elements between start of each block (integer)}
\funcarg{\IN}{oldtype}{old datatype (handle)}
\funcarg{\OUT}{newtype}{new datatype (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_vector(int~count, int~blocklength, int~stride, MPI\_Datatype~oldtype, MPI\_Datatype~*newtype)}

\mpifbind{MPI\_TYPE\_VECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)\fargs INTEGER  COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR}




\begin{example} {\rm
\label{pt2pt-exT}
Assume, again, that \mpiarg{oldtype} has type map
\(
\{ ({\sf double}, 0), ({\sf char}, 8) \} ,
\)
with extent 16.
A call to \mpifunc{MPI\_TYPE\_VECTOR( 2, 3, 4, oldtype, newtype)} will
create the datatype with type map,
\begin{displaymath}
\{
({\sf double}, 0), ({\sf char}, 8), ({\sf double}, 16), ({\sf char},
24), ({\sf double}, 32), ({\sf char}, 40),
\end{displaymath}
\begin{displaymath}
({\sf double}, 64), ({\sf char}, 72), ({\sf double}, 80), ({\sf char},
88), ({\sf double}, 96), ({\sf char}, 104)
\} .
\end{displaymath}
That is, two blocks with three copies each of the old
type, with a stride of 4 elements ($4 \cdot 16$ bytes) between the blocks.
} \end{example}

\begin{example} {\rm
\label{pt2pt-exTTT}
A call to \mpifunc{MPI\_TYPE\_VECTOR(3, 1, -2, oldtype, newtype)} will create
the datatype,
\begin{displaymath}
\{
({\sf double}, 0), ({\sf char}, 8), ({\sf double}, -32), ({\sf char},
-24), ({\sf double}, -64), ({\sf char}, -56)
\} .
\end{displaymath}
} \end{example}

\vspace{.7cm}
In general, assume that \mpiarg{oldtype} has type map,
\begin{displaymath}
\{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\end{displaymath}
with extent $ex$.  Let {\sf bl} be the {\sf blocklength}.
The newly created  datatype has a type map with
${\sf count} \cdot {\sf bl} \cdot n$
entries:
\begin{displaymath}
\{
(type_0, disp_0), ... , (type_{n-1} , disp_{n-1}),
\end{displaymath}
\begin{displaymath}
(type_0 ,disp_0 + ex) , ... ,
(type_{n-1} , disp_{n-1} + ex ), ...,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + ({\sf bl} -1) \cdot ex
) , ... , (type_{n-1} , disp_{n-1} + ({\sf bl} -1) \cdot ex ) ,
\end{displaymath}
\begin{displaymath}
(type_0 ,disp_0 + {\sf stride} \cdot ex ) , ... ,
(type_{n-1} , disp_{n-1} + {\sf stride} \cdot ex ), ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + ({\sf stride} + {\sf bl} -1) \cdot ex ) , ... ,
%\end{displaymath}
%\begin{displaymath}
(type_{n-1}, disp_{n-1} + ({\sf stride} + {\sf bl} -1) \cdot
ex ) , ....,
\end{displaymath}
\begin{displaymath}
(type_0 ,disp_0 + {\sf stride} \cdot ({\sf count}-1) \cdot ex ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_{n-1} , disp_{n-1} + {\sf stride} \cdot ({\sf count} -1) \cdot
ex )
, ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + ({\sf stride} \cdot ({\sf count} -1)
+ {\sf bl} -1) \cdot ex ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_{n-1}, disp_{n-1} + ({\sf stride} \cdot ({\sf count} -1)
+ {\sf bl} -1) \cdot ex )
\} .
\end{displaymath}
\vspace{.7cm}

A call to \mpifunc{MPI\_TYPE\_CONTIGUOUS(count, oldtype, newtype)} is
equivalent to a call to
\mpifunc{MPI\_TYPE\_VECTOR(count, 1, 1, oldtype, newtype)}, or to a call to
\mpifunc{MPI\_TYPE\_VECTOR(1, count, n, oldtype, newtype)}, {\sf n} arbitrary.

\paragraph*{Hvector}

The function \mpifunc{MPI\_TYPE\_HVECTOR} is identical to
\mpifunc{MPI\_TYPE\_VECTOR}, except that \mpiarg{stride} is given in bytes,
rather than in elements.  The use for both types of vector
constructors is illustrated in Sec. \ref{subsec:pt2pt-examples}.
({\sf H} stands for ``heterogeneous'').

\begin{funcdef}{MPI\_TYPE\_HVECTOR( count, blocklength, stride,
oldtype, newtype)}
\funcarg{\IN}{count}{number of blocks (nonnegative integer)}
\funcarg{\IN}{blocklength}{number of elements in each block
(nonnegative integer)}
\funcarg{\IN}{stride}{number of bytes between start of each block (integer)}
\funcarg{\IN}{oldtype}{old datatype (handle)}
\funcarg{\OUT}{newtype}{new datatype (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_hvector(int~count, int~blocklength, MPI\_Aint~stride, MPI\_Datatype~oldtype, MPI\_Datatype~*newtype)}

\mpifbind{MPI\_TYPE\_HVECTOR(COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR)\fargs INTEGER  COUNT, BLOCKLENGTH, STRIDE, OLDTYPE, NEWTYPE, IERROR}


\vspace{.7cm}
Assume that \mpiarg{oldtype} has type map,
\begin{displaymath}
\{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\end{displaymath}
with extent $ex$.  Let {\sf bl} be the {\sf blocklength}.
The newly created  datatype has a type map with
${\sf count} \cdot {\sf bl} \cdot n$
entries:
\begin{displaymath}
\{
(type_0, disp_0), ... , (type_{n-1} , disp_{n-1}),
\end{displaymath}
\begin{displaymath}
(type_0 ,disp_0 + ex) , ... ,
(type_{n-1} , disp_{n-1} + ex ), ...,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + ({\sf bl} -1) \cdot ex
) , ... , (type_{n-1} , disp_{n-1} + ({\sf bl} -1) \cdot ex ) ,
\end{displaymath}
\begin{displaymath}
(type_0 ,disp_0 + {\sf stride}  ) , ... ,
(type_{n-1} , disp_{n-1} + {\sf stride} ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + {\sf stride} + ( {\sf bl} -1) \cdot ex
) , ... ,
\end{displaymath}
\begin{displaymath}
(type_{n-1}, disp_{n-1} + {\sf stride} + ({\sf bl} -1) \cdot
ex ) ,
....,
\end{displaymath}
\begin{displaymath}
(type_0 ,disp_0 + {\sf stride} \cdot ({\sf count}-1) ) , ... ,
%\end{displaymath}
%\begin{displaymath}
(type_{n-1} , disp_{n-1} + {\sf stride} \cdot ({\sf count} -1)  )
, ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + {\sf stride} \cdot ({\sf count} -1)
+ ({\sf bl} -1) \cdot ex ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_{n-1}, disp_{n-1} + {\sf stride} \cdot ({\sf count} -1)
+ ({\sf bl} -1) \cdot ex )
\} .
\end{displaymath}
\vspace{.7cm}

\paragraph*{Indexed}

The function
\mpifunc{MPI\_TYPE\_INDEXED} allows
replication of an old datatype into a sequence of blocks (each block is
a concatenation of the old datatype), where
each block can contain a different number of copies and have a different
displacement.  All block displacements are multiples of the old type
extent.

\begin{funcdef}{MPI\_TYPE\_INDEXED( count, array\_of\_blocklengths,
array\_of\_displacements, oldtype, newtype)}
\funcarg{\IN}{count}{number of blocks -- also number of entries in
\linebreak \mpiarg{array\_of\_displacements} and
\mpiarg{array\_of\_blocklengths} (nonnegative integer)}
\funcarg{\IN}{array\_of\_blocklengths}{number of elements per block
(array of nonnegative integers)}
\funcarg{\IN}{array\_of\_displacements}{displacement for each block,
in multiples of \mpiarg{oldtype} extent (array of integer)}
\funcarg{\IN}{oldtype}{old datatype (handle)}
\funcarg{\OUT}{newtype}{new datatype (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_indexed(int~count, int~*array\_of\_blocklengths, int~*array\_of\_displacements, MPI\_Datatype~oldtype, MPI\_Datatype~*newtype)}

\mpifbind{MPI\_TYPE\_INDEXED(COUNT, ARRAY\_OF\_BLOCKLENGTHS, ARRAY\_OF\_DISPLACEMENTS, OLDTYPE, NEWTYPE, IERROR)\fargs INTEGER COUNT, ARRAY\_OF\_BLOCKLENGTHS(*), ARRAY\_OF\_DISPLACEMENTS(*), OLDTYPE, NEWTYPE, IERROR}


\begin{example} {\rm
\label{pt2pt-exU}

Let \mpiarg{oldtype} have type map
\(
\{ ({\sf double}, 0), ({\sf char}, 8) \} ,
\)
with extent 16.
Let  {\sf B = (3, 1)} and let {\sf D = (4, 0)}.  A call to
\mpifunc{MPI\_TYPE\_INDEXED(2, B, D, oldtype, newtype)} returns a datatype with
type map,
\begin{displaymath}
\{
({\sf double}, 64), ({\sf char}, 72), ({\sf double}, 80), ({\sf char},
88), ({\sf double}, 96), ({\sf char}, 104),
\end{displaymath}
\begin{displaymath}
({\sf double}, 0), ({\sf char}, 8)
\} .
\end{displaymath}
That is, three copies of the old type starting at displacement
64, and one copy starting at displacement 0.
} \end{example}

\vspace{.7cm}
In general,
assume that \mpiarg{oldtype} has type map,
\begin{displaymath}
\{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\end{displaymath}
with extent {\em ex}.
Let \mpiarg{B} be the \mpiarg{array\_of\_blocklength} argument and
\mpiarg{D} be the
\linebreak
\mpiarg{array\_of\_displacements} argument. The newly created datatype
has $n \cdot \sum_{i=0}^{{\sf count}-1}
\mpiarg{B[i]}$ entries:
\begin{displaymath}
\{
(type_0, disp_0 + \mpiarg{D[0]} \cdot ex ) , ... ,
(type_{n-1} , disp_{n-1} + \mpiarg{D[0]} \cdot ex ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + (\mpiarg{D[0]} + \mpiarg{B[0]} -1) \cdot ex) ,...,
(type_{n-1} , disp_{n-1} + (\mpiarg{D[0]} +\mpiarg{B[0]} -1) \cdot ex ) ,
...,
\end{displaymath}
\begin{displaymath}
(type_0, disp_0 + \mpiarg{D[count-1]} \cdot ex ) , ... ,
(type_{n-1} , disp_{n-1} + \mpiarg{D[count-1]} \cdot ex ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + (\mpiarg{D[count-1]} + \mpiarg{B[count-1]} -1) \cdot ex)
,...,
\end{displaymath}
\begin{displaymath}
(type_{n-1} , disp_{n-1} + (\mpiarg{D[count-1]} +\mpiarg{B[count-1]} -1)
\cdot ex )
\} .
\end{displaymath}
\vspace{.7cm}

A call to \mpifunc{MPI\_TYPE\_VECTOR(count, blocklength, stride, oldtype,
newtype)} is equivalent to a call to
\mpifunc{MPI\_TYPE\_INDEXED(count, B, D, oldtype, newtype)} where
\begin{displaymath}
\mpiarg{D[j]} = j \cdot \mpiarg{stride},\ j=0 ,..., {\sf count} -1 ,
\end{displaymath}
and
\begin{displaymath}
\mpiarg{B[j]} = \mpiarg{blocklength},\ j=0 ,..., {\sf count} -1 .
\end{displaymath}

\paragraph*{Hindexed}

The function \mpifunc{MPI\_TYPE\_HINDEXED} is identical to
\mpifunc{MPI\_TYPE\_INDEXED}, except that block displacements in
\mpiarg{array\_of\_displacements} are specified in
bytes, rather than in multiples of the \mpiarg{oldtype} extent.

\begin{funcdef}{MPI\_TYPE\_HINDEXED( count, array\_of\_blocklengths,
array\_of\_displacements, oldtype, newtype)}
\funcarg{\IN}{count}{number of blocks  -- also number of entries in
\linebreak \mpiarg{array\_of\_displacements} and
\mpiarg{array\_of\_blocklengths} (integer)}
\funcarg{\IN}{array\_of\_blocklengths}{number of elements in each block
(array of nonnegative integers)}
\funcarg{\IN}{array\_of\_displacements}{byte displacement of each block
(array of integer)}
\funcarg{\IN}{oldtype}{old datatype (handle)}
\funcarg{\OUT}{newtype}{new datatype (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_hindexed(int~count, int~*array\_of\_blocklengths, MPI\_Aint~*array\_of\_displacements, MPI\_Datatype~oldtype, MPI\_Datatype~*newtype)}

\mpifbind{MPI\_TYPE\_HINDEXED(COUNT, ARRAY\_OF\_BLOCKLENGTHS, ARRAY\_OF\_DISPLACEMENTS, OLDTYPE, NEWTYPE, IERROR)\fargs INTEGER COUNT, ARRAY\_OF\_BLOCKLENGTHS(*), ARRAY\_OF\_DISPLACEMENTS(*), OLDTYPE, NEWTYPE, IERROR}

\vspace{.7cm}
Assume that \mpiarg{oldtype} has type map,
\begin{displaymath}
\{ (type_0,disp_0), ..., (type_{n-1}, disp_{n-1}) \} ,
\end{displaymath}
with extent $ex$.
Let \mpiarg{B} be the \mpiarg{array\_of\_blocklength} argument and
\mpiarg{D} be the
\linebreak
\mpiarg{array\_of\_displacements} argument. The newly created datatype
has a type map with
$n \cdot \sum_{i=0}^{{\sf count}-1}
\mpiarg{B[i]}$ entries:
\begin{displaymath}
\{
(type_0, disp_0 + \mpiarg{D[0]}  ) , ... ,
(type_{n-1} , disp_{n-1} + \mpiarg{D[0]} ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + \mpiarg{D[0]} +(\mpiarg{B[0]} -1) \cdot ex) ,...,
\end{displaymath}
\begin{displaymath}
(type_{n-1} , disp_{n-1} + \mpiarg{D[0]} +(\mpiarg{B[0]} -1) \cdot ex ) ,
...,
\end{displaymath}
\begin{displaymath}
(type_0, disp_0 + \mpiarg{D[count-1]} ) , ... ,
(type_{n-1} , disp_{n-1} + \mpiarg{D[count-1]} ) , ... ,
\end{displaymath}
\begin{displaymath}
(type_0 , disp_0 + \mpiarg{D[count-1]} +(\mpiarg{B[count-1]} -1) \cdot ex)
,...,
\end{displaymath}
\begin{displaymath}
(type_{n-1} , disp_{n-1} + \mpiarg{D[count-1]} +(\mpiarg{B[count-1]} -1)
\cdot ex )
\} .
\end{displaymath}
\vspace{.7cm}


\paragraph*{Struct}

{\sf MPI\_TYPE\_STRUCT} is the most general type constructor.
It further generalizes
the previous one in that it allows each block to consist of replications of
different datatypes.


\begin{funcdef}{MPI\_TYPE\_STRUCT(count, array\_of\_blocklengths,
array\_of\_displacements, array\_of\_types, newtype)}
\funcarg{\IN}{count}{number of blocks (integer) -- also number of
entries in arrays \mpiarg{array\_of\_types},
\mpiarg{array\_of\_displacements} and \mpiarg{array\_of\_blocklengths}}
\funcarg{\IN}{array\_of\_blocklength}{number of elements in each block
(array of integer)}
\funcarg{\IN}{array\_of\_displacements}{byte displacement of each block
(array of integer)}
\funcarg{\IN}{array\_of\_types}{type of elements in each block (array
of handles to datatype objects)}
\funcarg{\OUT}{newtype}{new datatype (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_struct(int~count, int~*array\_of\_blocklengths, MPI\_Aint~*array\_of\_displacements, MPI\_Datatype~*array\_of\_types, MPI\_Datatype~*newtype)}

\mpifbind{MPI\_TYPE\_STRUCT(COUNT, ARRAY\_OF\_BLOCKLENGTHS, ARRAY\_OF\_DISPLACEMENTS, ARRAY\_OF\_TYPES, NEWTYPE, IERROR)\fargs INTEGER  COUNT, ARRAY\_OF\_BLOCKLENGTHS(*), ARRAY\_OF\_DISPLACEMENTS(*), ARRAY\_OF\_TYPES(*), NEWTYPE, IERROR}

\begin{example} {\rm
\label{pt2pt-exV}
Let \mpiarg{type1} have type map,
\[
\{ ({\sf double}, 0), ({\sf char}, 8) \} ,
\]
with extent 16.
Let {\sf B = (2, 1, 3)}, {\sf D = (0, 16, 26)},
and {\sf T = (MPI\_FLOAT, type1, MPI\_CHAR)}.  Then a call to
\mpifunc{MPI\_TYPE\_STRUCT(3, B, D, T, newtype)} returns
a datatype with type map,
\begin{displaymath}
\{
({\sf float}, 0), ({\sf float}, 4), ({\sf double}, 16), ({\sf char},
24), ({\sf char}, 26), ({\sf char}, 27), ({\sf char}, 28)
\} .
\end{displaymath}
That is, two copies of \type{MPI\_FLOAT} starting at 0, followed by
one copy of \mpiarg{type1} starting at 16, followed by three copies of
\type{MPI\_CHAR}, starting at 26.
(We assume that a float occupies four bytes.)
} \end{example}

\vspace{.7cm}
In general,
let \mpiarg{T} be the \mpiarg{array\_of\_types} argument, where \mpiarg{T[i]}
is a handle to,
\begin{displaymath}
typemap_i = \{ (type_0^i , disp_0^i ) , ... , (type_{n_{i}-1}^i ,
disp_{n_{i}-1}^i ) \} ,
\end{displaymath}
with extent $ex_i$.
Let
\mpiarg{B} be the \mpiarg{array\_of\_blocklength} argument and \mpiarg{D} be
the \mpiarg{array\_of\_displace-} \mpiarg{ments} argument.  %mansplit
Let {\sf c} be the
{\sf count} argument.
Then the newly created datatype has a type map with
$\sum_{i=0}^{{\sf c}-1}\mpiarg{B[i]} \cdot n_i$
entries:
\begin{displaymath}
\{
(type_0^0 , disp_0^0 +\mpiarg{D[0]}) , ... , (type_{n_0}^0 , disp_{n_0}^0 +
\mpiarg{D[0]} ) ,
... ,
\end{displaymath}
\begin{displaymath}
(type_0^0 , disp_0^0 + \mpiarg{D[0]} + (\mpiarg{B[0]}-1) \cdot ex_0 ) , ...
,
(type_{n_0}^0 , disp_{n_0}^0 + \mpiarg{D[0]} + (\mpiarg{B[0]-1)} \cdot
ex_0 )
, ... ,
\end{displaymath}
\begin{displaymath}
(type_0^{{\sf c}-1} , disp_0^{{\sf c}-1} +\mpiarg{D[c-1]}) , ... ,
(type_{{n_{{\sf c}-1}}-1}^{{\sf c}-1} ,
disp_{{n_{{\sf c}-1}}-1}^{{\sf c}-1} + \mpiarg{D[c-1]} ) ,
... ,
\end{displaymath}
\begin{displaymath}
(type_0^{{\sf c}-1} , disp_0^{{\sf c}-1} +
\mpiarg{D[c-1]} + (\mpiarg{B[c-1]}-1) \cdot ex_{{\sf c}-1} ) ,
... ,
\end{displaymath}
\begin{displaymath}
(type_{{n_{{\sf c}-1}}-1}^{{\sf c}-1} ,
disp_{{n_{{\sf c}-1}}-1}^{{\sf c}-1} + \mpiarg{D[c-1]} +
(\mpiarg{B[c-1]-1)} \cdot ex_{{\sf c}-1} )
\} .
\end{displaymath}
\vspace{.7cm}

A call to \mpifunc{MPI\_TYPE\_HINDEXED( count, B, D, oldtype, newtype)} is
equivalent to a call to
\mpifunc{MPI\_TYPE\_STRUCT( count, B, D, T, newtype)}, where each entry
of \mpiarg{T} is equal to \mpiarg{oldtype}.


\subsection{Address and extent functions}
\label{subsec:pt2pt-addfunc}

The displacements in a general datatype are relative to some initial buffer
address.  {\bf Absolute addresses} can be substituted for these
displacements: we treat them as displacements relative to ``address
zero,'' the start of the address space.  This initial address zero is
indicated by the constant \const{MPI\_BOTTOM}.  Thus, a datatype can
specify the absolute address of the entries in the communication
buffer, in which case the \mpiarg{buf} argument is passed the value
\const{MPI\_BOTTOM}.

The address of a location in memory can be found by invoking the
function \linebreak
\mpifunc{MPI\_ADDRESS}.

\begin{funcdef}{MPI\_ADDRESS(location, address)}
\funcarg{\IN}{location}{location in caller memory (choice)}
\funcarg{\OUT}{address}{address of location (integer)}
\end{funcdef}

\mpibind{MPI\_Address(void*~location, MPI\_Aint~*address)}

\mpifbind{MPI\_ADDRESS(LOCATION, ADDRESS, IERROR)\fargs &lt;type&gt; LOCATION(*) \\ INTEGER  ADDRESS, IERROR}



Returns the (byte) address of \mpiarg{location}.

\begin{example} {\rm
\label{pt2pt-exW}
Using \func{MPI\_ADDRESS} for an array.

\begin{verbatim}
   REAL A(100,100)
   INTEGER I1, I2, DIFF
   CALL MPI_ADDRESS(A(1,1), I1, IERROR)
   CALL MPI_ADDRESS(A(10,10), I2, IERROR)
   DIFF = I2 - I1
! The value of DIFF is 909*sizeofreal; the values of I1 and I2 are
! implementation dependent.
\end{verbatim}
} \end{example}

% ARM !!! I think the following &amp; cast expression is wrong, but
% I'll leave it for now.
\begin{users}
C users may be tempted to avoid the usage of \mpifunc{MPI\_ADDRESS} and rely on
the availability of the address operator \&amp;.  Note, however, that
\const{\&amp;}~{\em cast-expression} is a pointer, not an
address.  ANSI C does not require that the value of a pointer
(or the pointer cast to \const{int}) be
the absolute address of the object pointed at --- although this is
commonly the case.
Furthermore, referencing may not have a unique
definition on machines with a segmented address space.
The use of \mpifunc{MPI\_ADDRESS} to ``reference'' C
variables guarantees portability to such machines as well.
\end{users}

The following  auxiliary functions provide useful information on
derived datatypes.

\begin{funcdef}{MPI\_TYPE\_EXTENT(datatype, extent)}
\funcarg{\IN}{datatype}{datatype (handle)}
\funcarg{\OUT}{extent}{datatype extent (integer)}
\end{funcdef}

\snir
\mpibind{MPI\_Type\_extent(MPI\_Datatype~datatype, MPI\_Aint~*extent)}
\rins

\mpifbind{MPI\_TYPE\_EXTENT(DATATYPE, EXTENT, IERROR)\fargs INTEGER DATATYPE, EXTENT, IERROR}



Returns the extent of a datatype,
where extent is as defined in Eq.~\ref{eq:pt2pt-extent} on
page~\pageref{eq:pt2pt-extent}.


\begin{funcdef}{MPI\_TYPE\_SIZE(datatype, size)}
\funcarg{\IN}{datatype}{datatype (handle)}
\funcarg{\OUT}{size}{datatype size (integer)}
\end{funcdef}

\mpibind{MPI\_Type\_size(MPI\_Datatype~datatype, int~*size)}

\mpifbind{MPI\_TYPE\_SIZE(DATATYPE, SIZE, IERROR)\fargs INTEGER DATATYPE, SIZE, IERROR}

\func{MPI\_TYPE\_SIZE} returns the total size, in bytes, of the entries in
the type signature
associated with \mpiarg{datatype}; i.e., the total size of
the data in a message that would be created with this datatype.  Entries that
occur multiple times in the datatype are counted with their multiplicity.

\snir
%\begin{funcdef}{MPI\_TYPE\_COUNT(datatype, count)}
%\funcarg{\IN}{datatype}{datatype (handle)}
%\funcarg{\OUT}{count}{datatype count (integer)}
%\end{funcdef}

%\mpibind{MPI\_Type\_count(MPI\_Datatype~datatype, int~*count)}

%\mpifbind{MPI\_TYPE\_COUNT(DATATYPE, COUNT, IERROR)\fargs INTEGER DATATYPE,
%COUNT, IERROR}


%Returns the number of ``top-level'' entries in the datatype.
\rins

\subsection{Lower-bound and upper-bound markers}
\label{subsec:pt2pt-markers}

It is often convenient to define explicitly the lower bound and upper
bound of a type map, and override the definition given by
Equation~\ref{eq:pt2pt-extent} on page~\pageref{eq:pt2pt-extent}.
This allows one to define a datatype that
has ``holes'' at its beginning or its end, or a datatype with
entries that extend above the upper bound or below the lower bound.
Examples of
such usage are provided in Sec. \ref{subsec:pt2pt-examples}.
\snir
Also, the user may want to overide the alignment rules that are
used to compute upper bounds and extents.  E.g., a C compiler may allow
the user to overide default alignment rules for some of the
structures within
a program.  The user has to specify explicitly the bounds of the datatypes
that match these structures.
\rins

To achieve this, we add
two additional ``pseudo-datatypes,'' \type{MPI\_LB} and \type{MPI\_UB},
that can be used, respectively, to mark the lower bound or the upper
bound of a datatype.  These pseudo-datatypes occupy no space
($extent(\type{MPI\_LB}) = extent(\type{MPI\_UB}) =0$). They do not
affect the size or count of a datatype, and do not affect the
the content of a message created with this datatype.  However, they do
affect the definition of the
extent of a datatype and, therefore, affect the outcome of a replication of
this datatype by a datatype constructor.

\begin{example} {\rm
\label{pt2pt-exX}
Let {\sf D = (-3, 0, 6)}; {\sf T = (MPI\_LB, MPI\_INT, MPI\_UB)},
and {\sf B = (1, 1, 1)}.  Then a call to
\mpifunc{MPI\_TYPE\_STRUCT(3, B, D, T, type1)}
creates a new datatype that has an
extent of 9 (from -3 to 5, 5 included), and contains an integer at
displacement 0.   This is the datatype defined by the sequence
{\sf \{(lb, -3), (int, 0), (ub, 6)\} }.
If this type is replicated twice by a call to
\mpifunc{MPI\_TYPE\_CONTIGUOUS(2, type1, type2)} then the newly created
type can
be described by the sequence
{\sf \{(lb, -3), (int, 0), (int,9), (ub, 15)\} }.
\snir
(Entries of type
{\sf lb} or {\sf ub}
can be deleted if they are dominated by other such entries.)
\rins
} \end{example}

In general, if
\begin{displaymath}
Typemap = \{ (type_0 , disp_0 ) , ... , (type_{n-1} , disp_{n-1}) \} ,
\end{displaymath}
then the {\bf lower bound} of $Typemap$ is defined to be
\[
lb(Typemap) = \left\{ \begin{array}{ll}
\min_j disp_j &amp; \mbox{if no entry has basic type {\sf lb}} \\
\min_j \{ disp_j \ \mbox{such that}\ type_j = {\sf lb} \} &amp; \mbox{otherwise}
\end{array}
\right. \]

Similarly,
the {\bf upper bound} of $Typemap$ is defined to be
\snir
\[
ub(Typemap) = \left\{ \begin{array}{ll}
\max_j disp_j + sizeof(type_j) + \epsilon &amp; \mbox{if no entry has basic type
{\sf ub}}
\\ \max_j \{ disp_j \ \mbox{such that}\ type_j = {\sf ub} \} &amp; \mbox{otherwise}
\end{array}
\right. \]


Then
\[
extent(Typemap) = ub(Typemap) - lb(Typemap)
\]
\rins
If $type_i$ requires alignment to a byte address that is a multiple of $k_i$,
then $\epsilon$ is the least nonnegative increment needed to round
$extent(Typemap)$ to the next multiple of $\max_i k_i$.

The formal definitions given for the various datatype constructors
apply now, with the amended definition of {\bf extent}.

\snir
The extent of a datatype should alwyas be nonnegative.  It is erroneous to
provide upper or lower bound markers that result in a negative extent.
\rins

The two functions below can be used for finding the lower bound and
the upper bound of a datatype.

\begin{funcdef}{MPI\_TYPE\_LB( datatype, displacement)}
\funcarg{\IN}{datatype}{datatype (handle)}
\funcarg{\OUT}{displacement}{displacement of lower bound from origin,
                             in bytes (integer)}
\end{funcdef}

\snir
\mpibind{MPI\_Type\_lb(MPI\_Datatype~datatype, MPI\_Aint*~displacement)}
\rins

\mpifbind{MPI\_TYPE\_LB( DATATYPE, DISPLACEMENT, IERROR)\fargs INTEGER DATATYPE, DISPLACEMENT, IERROR}


\begin{funcdef}{MPI\_TYPE\_UB( datatype, displacement)}
\funcarg{\IN}{datatype}{datatype (handle)}
\funcarg{\OUT}{displacement}{displacement of upper bound from origin,
                             in bytes (integer)}
\end{funcdef}

\snir
\mpibind{MPI\_Type\_ub(MPI\_Datatype~datatype, MPI\_Aint*~displacement)}
\rins

\mpifbind{MPI\_TYPE\_UB( DATATYPE, DISPLACEMENT, IERROR)\fargs INTEGER DATATYPE, DISPLACEMENT, IERROR}

\snir
%\begin{rationale}
%Note that the
%rules given in Sec. \ref{subsec:pt2pt-segmented} imply that
%it is erroneous to call \func{MPI\_TYPE\_EXTENT},
%\func{MPI\_TYPE\_LB}, and \func{MPI\_TYPE\_UB}
%with a datatype argument that contains absolute addresses,
%unless all these addreses are within the same sequential storage.
%For this reason, the \mpiarg{displacement} for the C binding in
%\func{MPI\_TYPE\_UB} is an \ctype{int} and not of type \ctype{MPI\_Aint}.
%\end{rationale}

%\begin{example} {\rm
%\label{pt2pt-exY}
%\begin{verbatim}
%MPI_Type_extent(type, extent);
%MPI_Type_lb(type, lb);
%MPI_Type_ub(type, ub);
%if ( extent != ub - lb )
   %printf("MPI library failed validation test\n")
%\end{verbatim}
%} \end{example}
%





\subsection{Commit and free}
\label{subsec:pt2pt-comfree}

A datatype object has to be {\bf committed} before it can be used in a
communication.
A committed datatype can still be used as a argument in datatype constructors.
There is no need to commit basic datatypes. They are ``pre-committed.''

\begin{funcdef}{MPI\_TYPE\_COMMIT(datatype)}
\funcarg{\INOUT}{datatype}{datatype that is committed (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_commit(MPI\_Datatype~*datatype)}

\mpifbind{MPI\_TYPE\_COMMIT(DATATYPE, IERROR)\fargs INTEGER  DATATYPE, IERROR}

The commit operation commits the datatype, that is, the formal description of
a communication buffer, not the content of that buffer.  Thus, after a datatype
has been committed, it can be repeatedly reused to communicate the changing
content
of a buffer or, indeed, the content of different buffers, with different
starting addresses.

\begin{implementors}
The system may ``compile'' at commit time an internal
representation for the datatype that facilitates
communication, e.g.
change from a compacted representation to a flat representation of the
datatype, and select the most convenient transfer mechanism.
\end{implementors}

\begin{funcdef}{MPI\_TYPE\_FREE(datatype)}
\funcarg{\INOUT}{datatype}{datatype that is freed (handle)}
\end{funcdef}

\mpibind{MPI\_Type\_free(MPI\_Datatype~*datatype)}

\mpifbind{MPI\_TYPE\_FREE(DATATYPE, IERROR)\fargs INTEGER  DATATYPE, IERROR}

Marks the datatype object associated with \mpiarg{datatype} for
deallocation and sets \mpiarg{datatype} to \const{MPI\_DATATYPE\_NULL}.
Any communication that is currently using this datatype will complete normally.
Derived datatypes that were defined from the freed datatype are not
affected.

\begin{example} {\rm
\label{pt2pt-exZ}
The following code fragment gives examples of using
\func{MPI\_TYPE\_COMMIT}.
\begin{verbatim}
INTEGER type1, type2
CALL MPI_TYPE_CONTIGUOUS(5, MPI_REAL, type1, ierr)
              ! new type object created
CALL MPI_TYPE_COMMIT(type1, ierr)
              ! now type1 can be used for communication
type2 = type1
              ! type2 can be used for communication
              ! (it is a handle to same object as type1)
CALL MPI_TYPE_VECTOR(3, 5, 4, MPI_REAL, type1, ierr)
              ! new uncommitted type object created
CALL MPI_TYPE_COMMIT(type1, ierr)
              ! now type1 can be used anew for communication
\end{verbatim}
} \end{example}

Freeing a datatype does not affect any other datatype that was
built from the freed datatype.  The system behaves as if
input datatype arguments to derived datatype constructors are passed by value.

\begin{implementors}
The implementation may keep a reference count of active communications
that use the datatype, in order to decide when to free it.
Also, one may implement constructors of derived datatypes so that they keep
pointers to their datatype arguments, rather then copying them.  In this
case, one needs to keep track of active datatype definition references in order
to know when a datatype object can be freed.
\end{implementors}

\subsection{Use of general datatypes in communication}
\label{subsec:pt2pt-datatypeuse}

Handles to derived datatypes can be passed to a communication call wherever a
datatype argument is required.
A call of the form \mpifunc{MPI\_SEND(buf, count, datatype , ...)}, where
$\sf
count &gt; 1$, is interpreted as if the call was passed a new datatype
which is the
concatenation of \mpiarg{count} copies of \mpiarg{datatype}.
Thus,
\mpifunc{MPI\_SEND(buf, count, datatype, dest, tag, comm)} is equivalent to,
\begin{verbatim}
MPI_TYPE_CONTIGUOUS(count, datatype, newtype)
MPI_TYPE_COMMIT(newtype)
MPI_SEND(buf, 1, newtype, dest, tag, comm).
\end{verbatim}
Similar statements apply to all other communication functions that have a
\mpiarg{count} and \mpiarg{datatype} argument.

Suppose that a send operation \mpifunc{MPI\_SEND(buf, count,
datatype, dest, tag, comm)} is executed, where
\mpiarg{datatype} has type map,
\[
\{(type_0, disp_0),...,(type_{n-1}, disp_{n-1})\},
\]
and extent $extent$.  (Empty entries of ``pseudo-type'' \const{
MPI\_UB} and \const{ MPI\_LB} are not listed in the type map, but
they affect the value of $extent$.)
The send
operation sends $n \cdot \sf count$ entries, where entry $i
\cdot n + j$ is at location
$addr_{i,j} = {\sf buf} + extent \cdot i + disp_j$
and has type $type_j$,
for $i = 0 ,..., {\sf count}-1$ and $j = 0 ,..., n-1$.
These entries need not be contiguous, nor distinct; their order can be
arbitrary.

The variable stored at address $addr_{i,j}$ in the calling program
should be of a type that matches $type_j$, where
type matching is defined as in section~\ref{subsec:pt2pt-typematch}.
The message sent contains $n \cdot \sf count$ entries, where entry
$i \cdot n +j$ has type $type_j$.

Similarly, suppose that a receive operation
\mpifunc{MPI\_RECV(buf, count, datatype, source, tag, comm, status)} is
executed, where \mpiarg{datatype} has type map,
\begin{displaymath}
\{(type_0, disp_0) ,...,(type_{n-1}, disp_{n-1}) \},
\end{displaymath}
\noindent
with extent $extent$.  (Again, empty entries of ``pseudo-type'' \const{
MPI\_UB} and \const{ MPI\_LB} are not listed in the type map, but
they affect the value of $extent$.)
This receive operation receives
$n \cdot \sf count$ entries, where entry $i \cdot n + j$ is at location
${\sf buf} + extent \cdot i + disp_j$
and has type $type_j$.  If the incoming message consists of $k$
elements, then we must have $k \le n \cdot \sf count$; the $i \cdot n +
j$-th element of the message should have a type that matches $type_j$.

Type matching is defined according to the type signature of
the corresponding datatypes, that is, the sequence of basic type
components.  Type matching does not depend on some aspects of the
datatype definition, such as the displacements (layout in memory) or the
intermediate types used.

\begin{example} {\rm
\label{pt2pt-exAA}
This example shows that type matching is defined in terms of
the basic types that a derived type consists of.
\begin{verbatim}
...
CALL MPI_TYPE_CONTIGUOUS( 2, MPI_REAL, type2, ...)
CALL MPI_TYPE_CONTIGUOUS( 4, MPI_REAL, type4, ...)
CALL MPI_TYPE_CONTIGUOUS( 2, type2, type22, ...)
...
CALL MPI_SEND( a, 4, MPI_REAL, ...)
CALL MPI_SEND( a, 2, type2, ...)
CALL MPI_SEND( a, 1, type22, ...)
CALL MPI_SEND( a, 1, type4, ...)
...
CALL MPI_RECV( a, 4, MPI_REAL, ...)
CALL MPI_RECV( a, 2, type2, ...)
CALL MPI_RECV( a, 1, type22, ...)
CALL MPI_RECV( a, 1, type4, ...)
\end{verbatim}
Each of the sends matches any of the receives.
} \end{example}

A datatype may specify overlapping entries.  If such a datatype is used
in a receive operation, that is, if some part of the receive buffer
is written more than once by the receive operation, then the
call is erroneous.

Suppose that
\mpifunc{MPI\_RECV(buf, count, datatype, dest, tag, comm, status)} is
executed, where \mpiarg{datatype} has type map,
\begin{displaymath}
\{(type_0, disp_0) ,...,(type_{n-1}, disp_{n-1}) \}.
\end{displaymath}
The received message need not fill all the receive buffer, nor does it
need to fill a number of locations which is a multiple of $n$.
Any number, $k$, of basic
elements can be received, where $0 \le k \le {\sf count} \cdot n$.
The number of basic elements received can be retrieved from
\mpiarg{status} using the query function \mpifunc{MPI\_GET\_ELEMENTS}.

\begin{funcdef}{MPI\_GET\_ELEMENTS( status, datatype, count)}
\funcarg{\IN}{status}{return status of receive operation (Status)}
\funcarg{\IN}{datatype}{datatype used by receive operation (handle)}
\funcarg{\OUT}{count}{number of received basic elements (integer)}
\end{funcdef}

\mpibind{MPI\_Get\_elements(MPI\_Status~status, MPI\_Datatype~datatype, int~*count)}

\mpifbind{MPI\_GET\_ELEMENTS(STATUS, DATATYPE, COUNT, IERROR)\fargs INTEGER  STATUS(MPI\_STATUS\_SIZE), DATATYPE, COUNT, IERROR}

The previously defined
function, \func{MPI\_GET\_COUNT} (Sec. \ref{subsec:pt2pt-status}), has
a different behavior. It returns the number of ``top-level
elements'' received.  In the previous example, \func{MPI\_GET\_COUNT}
may return any integer value $k$, where $0 \le k \le {\sf count}$.
If \func{MPI\_GET\_COUNT} returns $k$, then the number of basic
elements received (and the value returned by \linebreak
\func{MPI\_GET\_ELEMENTS})
is $n \cdot k$.   If the number of basic elements received is not a
multiple of $n$, that is, if the receive operation has not received an
integral number of \mpiarg{datatype} ``copies,'' then
\func{MPI\_GET\_COUNT} returns the value \const{MPI\_UNDEFINED}.

\begin{example} {\rm
\label{pt2pt-exBB}
Usage of \func{MPI\_GET\_COUNT} and \func{MPI\_GET\_ELEMENT}.
\begin{verbatim}
...
CALL MPI_TYPE_CONTIGUOUS(2, MPI_REAL, Type2, ierr)
CALL MPI_TYPE_COMMIT(Type2, ierr)
...
CALL MPI_COMM_RANK(comm, rank, ierr)
IF(rank.EQ.0) THEN
      CALL MPI_SEND(a, 2, MPI_REAL, 1, 0, comm, ierr)
      CALL MPI_SEND(a, 3, MPI_REAL, 1, 0, comm, ierr)
ELSE
      CALL MPI_RECV(a, 2, Type2, 0, 0, comm, stat, ierr)
      CALL MPI_GET_COUNT(stat, Type2, i, ierr)     ! returns i=1
      CALL MPI_GET_ELEMENTS(stat, Type2, i, ierr)  ! returns i=2
      CALL MPI_RECV(a, 2, Type2, 0, 0, comm, stat, ierr)
      CALL MPI_GET_COUNT(stat, Type2, i, ierr)     ! returns i=MPI_UNDEFINED
      CALL MPI_GET_ELEMENTS(stat, Type2, i, ierr)  ! returns i=3
END IF
\end{verbatim}
} \end{example}

The function \func{MPI\_GET\_ELEMENTS} can also be used after a probe
to find the number of elements in the probed message.
Note that the two functions \func{MPI\_GET\_COUNT} and
\func{MPI\_GET\_ELEMENTS} return the same values when they are used
with basic datatypes.


\begin{rationale}
The extension given to the definition of \func{MPI\_GET\_COUNT} seems
natural: one would expect this function to return the value of the
\mpiarg{count} argument, when the receive buffer is filled.
Sometimes \mpiarg{datatype} represents
a basic unit of data one wants to transfer,
for example, a record in an array of records (structures).
One should be able to find out how many components were received
without bothering to divide by the number of elements in each
component.   However, on other occasions, \mpiarg{datatype} is used to
define a complex layout of data in the receiver memory, and does not represent
a basic unit of data for transfers.  In such cases, one needs to use
the function \func{MPI\_GET\_ELEMENTS}.
\end{rationale}

\begin{implementors}
The definition implies that a receive cannot change the value of
storage outside the entries defined to compose the communication
buffer.
In particular, the definition implies that padding space in a structure
should not be modified when such a structure is copied from one process to
another. This would
prevent the obvious optimization of copying the structure, together
with the padding, as one contiguous block.
The implementation is free to do this optimization when it does not
impact the outcome of the computation.
% How would it know?  ARM
The user can ``force'' this optimization by explicitly including
padding as part of the message.
% ARM How does the user do this?
\end{implementors}

\subsection{Correct use of addresses}
\label{subsec:pt2pt-segmented}

Successively declared variables in C or Fortran are not necessarily
stored at contiguous locations.  Thus, care must be exercised
that displacements do not cross from one variable
to another.  Also, in machines with a segmented address space,
addresses are not unique and address arithmetic has some peculiar
properties.   Thus, the use of {\bf addresses},
that is, displacements relative to the
start address \const{MPI\_BOTTOM}, has to be restricted.

Variables belong
to the same {\bf sequential storage} if they belong to the same
array,
to the same {\sf COMMON} block in Fortran, or to the same structure in C.
Valid addresses are defined recursively as follows:

\begin{enumerate}
\item
The function \mpifunc{MPI\_ADDRESS} returns a valid address, when
passed as argument a variable of the calling program.
\item
The \mpiarg{buf} argument of a communication function evaluates to a
valid address, when passed as argument a variable of the calling program.
\item
If \mpiarg{v} is a valid address, and \mpiarg{i} is an
integer, then \mpiarg{v+i} is a valid address, provided \mpiarg{v} and
\mpiarg{v+i} are in the same sequential storage.
\item
If \mpiarg{v} is a valid address then \const{MPI\_BOTTOM + v} is a valid
address.
\end{enumerate}

A correct program uses only valid addresses to identify the
locations of entries in communication buffers.
Furthermore, if \mpiarg{u} and \mpiarg{v} are two valid addresses, then
the (integer)
difference \mpiarg{u - v} can be computed only if both \mpiarg{u} and {\sf v}
are in the same sequential storage.
No other arithmetic operations can be meaningfully executed on addresses.

The rules above impose no constraints on the use of derived
datatypes, as long as they are used to define a communication buffer
that is wholly contained within the same sequential storage.
However, the construction of a communication buffer that contains
variables that are not within the same sequential storage must obey
certain restrictions.  Basically, a communication buffer with
variables that are not within the same sequential storage can be used
only by specifying in the communication call \mpiarg{buf =
MPI\_BOTTOM}, \mpiarg{count = 1}, and using a \mpiarg{datatype}
argument where all displacements are valid (absolute) addresses.

\begin{users}
It is not expected that \MPI/ implementations will be able to detect
erroneous, ``out of bound'' displacements --- unless those overflow the
user address space --- since the \MPI/ call may not know the extent of the
arrays and records in the host program.
\end{users}



\begin{implementors}
There is no need to distinguish (absolute) addresses and (relative)
displacements on a machine with contiguous address space:
\const{MPI\_BOTTOM} is
zero, and both addresses and displacements are integers.  On machines where the
distinction is required, addresses are recognized as expressions that involve
\const{MPI\_BOTTOM}.
\end{implementors}

\subsection{Examples}
\label{subsec:pt2pt-examples}

The following examples illustrate the use of derived datatypes.

\begin{example} {\rm
\label{pt2pt-exCC}
Send and receive a section of a 3D array.

\begin{verbatim}
      REAL a(100,100,100), e(9,9,9)
      INTEGER oneslice, twoslice, threeslice, sizeofreal, myrank, ierr
      INTEGER status(MPI_STATUS_SIZE)

C      extract the section a(1:17:2, 3:11, 2:10)
C      and store it in e(:,:,:).

      CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)

      CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)

C     create datatype for a 1D section
      CALL MPI_TYPE_VECTOR( 9, 1, 2, MPI_REAL, oneslice, ierr)

C     create datatype for a 2D section
      CALL MPI_TYPE_HVECTOR(9, 1, 100*sizeofreal, oneslice, twoslice, ierr)

C     create datatype for the entire section
      CALL MPI_TYPE_HVECTOR( 9, 1, 100*100*sizeofreal, twoslice, 1,
                             threeslice, ierr)

      CALL MPI_TYPE_COMMIT( threeslice, ierr)
      CALL MPI_SENDRECV(a(1,3,2), 1, threeslice, myrank, 0, e, 9*9*9,
                        MPI_REAL, myrank, 0, MPI_COMM_WORLD, status, ierr)
\end{verbatim}
} \end{example}


\begin{example} {\rm
\label{pt2pt-exDD}
Copy the (strictly) lower triangular part of a matrix.

\begin{verbatim}
      REAL a(100,100), b(100,100)
      INTEGER  disp(100), blocklen(100), ltype, myrank, ierr
      INTEGER status(MPI_STATUS_SIZE)

C     copy lower triangular part of array a
C     onto lower triangular part of array b

      CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)

C     compute start and size of each column
      DO i=1, 100
        disp(i) = 100*(i-1) + i
        block(i) = 100-i
      END DO

C     create datatype for lower triangular part
      CALL MPI_TYPE_INDEXED( 100, block, disp, MPI_REAL, ltype, ierr)

      CALL MPI_TYPE_COMMIT(ltype, ierr)
      CALL MPI_SENDRECV( a, 1, ltype, myrank, 0, b, 1,
                    ltype, myrank, 0, MPI_COMM_WORLD, status, ierr)
\end{verbatim}
} \end{example}

\begin{example} {\rm
\label{pt2pt-exEE}
Transpose a matrix.

\begin{verbatim}
      REAL a(100,100), b(100,100)
      INTEGER row, xpose, sizeofreal, myrank, ierr
      INTEGER status(MPI_STATUS_SIZE)

C     transpose matrix a onto b

      CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)

      CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)

C     create datatype for one row
      CALL MPI_TYPE_VECTOR( 100, 1, 100, MPI_REAL, row, ierr)

C     create datatype for matrix in row-major order
      CALL MPI_TYPE_HVECTOR( 100, 1, sizeofreal, row, xpose, ierr)

      CALL MPI_TYPE_COMMIT( xpose, ierr)

C     send matrix in row-major order and receive in column major order
      CALL MPI_SENDRECV( a, 1, xpose, myrank, 0, b, 100*100,
                MPI_REAL, myrank, 0, MPI_COMM_WORLD, status, ierr)
\end{verbatim}
} \end{example}

\begin{example} {\rm
\label{pt2pt-exFF}
Another approach to the transpose problem:
\begin{verbatim}
      REAL a(100,100), b(100,100)
      INTEGER  disp(2), blocklen(2), type(2), row, row1, sizeofreal
      INTEGER  myrank, ierr
      INTEGER status(MPI_STATUS_SIZE)

      CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank)

C     transpose matrix a onto b

      CALL MPI_TYPE_EXTENT( MPI_REAL, sizeofreal, ierr)

C     create datatype for one row
      CALL MPI_TYPE_VECTOR( 100, 1, 100, MPI_REAL, row, ierr)

C     create datatype for one row, with the extent of one real number
      disp(1) = 0
      disp(2) = sizeofreal
      type(1)  = row
      type(2)  = MPI_UB
      blocklen(1)  = 1
      blocklen(2)  = 1
      CALL MPI_TYPE_STRUCT( 2, blocklen, disp, type, row1, ierr)

      CALL MPI_TYPE_COMMIT( row1, ierr)

C     send 100 rows and receive in column major order
      CALL MPI_SENDRECV( a, 100, row1, myrank, 0, b, 100*100,
                MPI_REAL, myrank, 0, MPI_COMM_WORLD, status, ierr)
\end{verbatim}
} \end{example}


\begin{example} {\rm
\label{pt2pt-exGG}
We manipulate an array of structures.

\begin{verbatim}
struct Partstruct
   {
   int    class;  /* particle class */
   double d[6];   /* particle coordinates */
   char   b[7];   /* some additional information */
   };

struct Partstruct    particle[1000];

int                  i, dest, rank;
MPI_Comm     comm;


/* build datatype describing structure */

MPI_Datatype Particletype;
MPI_Datatype type[3] = {MPI_INT, MPI_DOUBLE, MPI_CHAR};
int          blocklen[3] = {1, 6, 7};
MPI_Aint     disp[3];
int          base;


/* compute displacements of structure components */

MPI_Address( particle, disp);
MPI_Address( particle[0].d, disp+1);
MPI_Address( particle[0].b, disp+2);
base = disp[0];
for (i=0; i &lt;3; i++) disp[i] -= base;

MPI_Type_struct( 3, blocklen, disp, type, &amp;Particletype);

   /* If compiler does padding in mysterious ways,
   the following may be safer */

MPI_Datatype type1[4] = {MPI_INT, MPI_DOUBLE, MPI_CHAR, MPI_UB};
int          blocklen1[4] = {1, 6, 7, 1};
MPI_Aint     disp1[4];

/* compute displacements of structure components */

MPI_Address( particle, disp1);
MPI_Address( particle[0].d, disp1+1);
MPI_Address( particle[0].b, disp1+2);
MPI_Address( particle+1, disp1+3);
base = disp1[0];
for (i=0; i &lt;4; i++) disp1[i] -= base;

/* build datatype describing structure */

MPI_Type_struct( 4, blocklen1, disp1, type1, &amp;Particletype);


              /* 4.1:
        send the entire array */

MPI_Type_commit( &amp;Particletype);
MPI_Send( particle, 1000, Particletype, dest, tag, comm);


              /* 4.2:
        send only the entries of class zero particles,
        preceded by the number of such entries */

MPI_Datatype Zparticles;   /* datatype describing all particles
                              with class zero (needs to be recomputed
                              if classes change) */
MPI_Datatype Ztype;

MPI_Aint     zdisp[1000];
int zblock[1000], j, k;
int zzblock[2] = {1,1};
MPI_Aint     zzdisp[2];
MPI_Datatype zztype[2];

/* compute displacements of class zero particles */
j = 0;
for(i=0; i &lt; 1000; i++)
  if (particle[i].class==0)
     {
     zdisp[j] = i;
     zblock[j] = 1;
     j++;
     }

/* create datatype for class zero particles  */
MPI_Type_indexed( j, zblock, zdisp, Particletype, &amp;Zparticles);

/* prepend particle count */
MPI_Address(&amp;j, zzdisp);
MPI_Address(particle, zzdisp+1);
zztype[0] = MPI_INT;
zztype[1] = Zparticles;
MPI_Type_struct(2, zzblock, zzdisp, zztype, &amp;Ztype);

MPI_Type_commit( &amp;Ztype);
MPI_Send( MPI_BOTTOM, 1, Ztype, dest, tag, comm);


       /* A probably more efficient way of defining Zparticles */

/* consecutive particles with index zero are handled as one block */
j=0;
for (i=0; i &lt; 1000; i++)
  if (particle[i].index==0)
    {
    for (k=i+1; (k &lt; 1000)&amp;&amp;(particle[k].index == 0) ; k++);
    zdisp[j] = i;
    zblock[j] = k-i;
    j++;
    i = k;
    }
MPI_Type_indexed( j, zblock, zdisp, Particletype, &amp;Zparticles);


                /* 4.3:
          send the first two coordinates of all entries */

MPI_Datatype Allpairs;      /* datatype for all pairs of coordinates */

MPI_Aint sizeofentry;

MPI_Type_extent( Particletype, &amp;sizeofentry);

     /* sizeofentry can also be computed by subtracting the address
        of particle[0] from the address of particle[1] */

MPI_Type_hvector( 1000, 2, sizeofentry, MPI_DOUBLE, &amp;Allpairs);
MPI_Type_commit( &amp;Allpairs);
MPI_Send( particle[0].d, 1, Allpairs, dest, tag, comm);

      /* an alternative solution to 4.3 */

MPI_Datatype Onepair;   /* datatype for one pair of coordinates, with
                          the extent of one particle entry */
MPI_Aint disp2[3];
MPI_Datatype type2[3] = {MPI_LB, MPI_DOUBLE, MPI_UB};
int blocklen2[3] = {1, 2, 1};

MPI_Address( particle, disp2);
MPI_Address( particle[0].d, disp2+1);
MPI_Address( particle+1, disp2+2);
base = disp2[0];
for (i=0; i&lt;2; i++) disp2[i] -= base;

MPI_Type_struct( 3, blocklen2, disp2, type2, &amp;Onepair);
MPI_Type_commit( &amp;Onepair);
MPI_Send( particle[0].d, 1000, Onepair, dest, tag, comm);

\end{verbatim}
} \end{example}

\begin{example} {\rm
\label{pt2pt-exHH}
The same manipulations as in the previous example, but use absolute
addresses in datatypes.

\begin{verbatim}
struct Partstruct
   {
   int class;
   double d[6];
   char b[7];
   };

struct Partstruct particle[1000];

           /* build datatype describing first array entry */

MPI_Datatype Particletype;
MPI_Datatype type[3] = {MPI_INT, MPI_DOUBLE, MPI_CHAR};
int          block[3] = {1, 6, 7};
MPI_Aint     disp[3];

MPI_Address( particle, disp);
MPI_Address( particle[0].d, disp+1);
MPI_Address( particle[0].b, disp+2);
MPI_Type_struct( 3, block, disp, type, &amp;Particletype);

/* Particletype describes first array entry -- using absolute
   addresses */

                  /* 5.1:
            send the entire array */

MPI_Type_commit( &amp;Particletype);
MPI_Send( MPI_BOTTOM, 1000, Particletype, dest, tag, comm);


                 /* 5.2:
         send the entries of class zero,
         preceded by the number of such entries */

MPI_Datatype Zparticles, Ztype;

MPI_Aint zdisp[1000]
int zblock[1000], i, j, k;
int zzblock[2] = {1,1};
MPI_Datatype zztype[2];
MPI_Aint     zzdisp[2];

j=0;
for (i=0; i &lt; 1000; i++)
  if (particle[i].index==0)
    {
    for (k=i+1; (k &lt; 1000)&amp;&amp;(particle[k].index = 0) ; k++);
    zdisp[j] = i;
    zblock[j] = k-i;
    j++;
    i = k;
    }
MPI_Type_indexed( j, zblock, zdisp, Particletype, &amp;Zparticles);
/* Zparticles describe particles with class zero, using
   their absolute addresses*/

/* prepend particle count */
MPI_Address(&amp;j, zzdisp);
zzdisp[1] = MPI_BOTTOM;
zztype[0] = MPI_INT;
zztype[1] = Zparticles;
MPI_Type_struct(2, zzblock, zzdisp, zztype, &amp;Ztype);

MPI_Type_commit( &amp;Ztype);
MPI_Send( MPI_BOTTOM, 1, Ztype, dest, tag, comm);

\end{verbatim}
} \end{example}

\begin{example} {\rm
\label{pt2pt-exII}
Handling of unions.

\begin{verbatim}
union {
   int     ival;
   float   fval;
      } u[1000]

int     utype;

/* All entries of u have identical type; variable
   utype keeps track of their current type */

MPI_Datatype   type[2];
int            blocklen[2] = {1,1};
MPI_Aint       disp[2];
MPI_Datatype   mpi_utype[2];
MPI_Aint       i,j;

/* compute an MPI datatype for each possible union type;
   assume values are left-aligned in union storage. */

MPI_Address( u, &amp;i);
MPI_Address( u+1, &amp;j);
disp[0] = 0; disp[1] = j-i;
type[1] = MPI_UB;

type[0] = MPI_INT;
MPI_Type_struct(2, blocklen, disp, type, &amp;mpi_utype[0]);

type[0] = MPI_FLOAT;
MPI_Type_struct(2, blocklen, disp, type, &amp;mpi_utype[1]);

for(i=0; i&lt;2; i++) MPI_Type_commit(&amp;mpi_utype[i]);

/* actual communication */

MPI_Send(u, 1000, mpi_utype[utype], dest, tag, comm);
\end{verbatim}
} \end{example}


\section{Pack and unpack}
\label{sec:pt2pt-packing}

Some existing communication libraries provide pack/unpack functions for sending
noncontiguous data. In these, the user explicitly packs
data into a contiguous buffer
before sending it, and unpacks it from a contiguous buffer after receiving it.
Derived datatypes, which are described in
Section~\ref{sec:pt2pt-datatype}, allow one, in most cases, to avoid
explicit packing and unpacking.  The user specifies the layout of
the data to be sent or received, and the communication library directly
accesses a noncontiguous buffer.  The pack/unpack routines are provided for
compatibility with previous libraries.   Also, they provide some functionality
that is not otherwise available in \MPI/.
For instance, a message can be received in several
parts, where the receive operation done on
a later part may depend on the content of a former part.
Another use is that outgoing messages
may be explicitly buffered in user supplied space, thus overriding the system
buffering policy.   Finally, the availability of pack and unpack operations
facilitates the development of additional communication libraries layered on top
of \MPI/.


\begin{funcdef}{MPI\_PACK(inbuf, incount, datatype, outbuf, outcount,
position,  comm)}
\funcarg{\IN}{inbuf}{input buffer start (choice)}
\funcarg{\IN}{incount}{number of input data items (integer)}
\funcarg{\IN}{datatype}{datatype of each input data item (handle)}
\funcarg{\OUT}{outbuf}{output buffer start (choice)}
\funcarg{\IN}{outcount}{output buffer size, in bytes (integer)}
\funcarg{\INOUT}{position}{current position in buffer, in bytes (integer)}
\funcarg{\IN}{comm}{communicator for packed message (handle)}
\end{funcdef}

\mpibind{MPI\_Pack(void*~inbuf, int~incount, MPI\_Datatype~datatype, void~*outbuf, int~outcount, int~*position,  MPI\_Comm~comm)}

\mpifbind{MPI\_PACK(INBUF, INCOUNT, DATATYPE, OUTBUF, OUTCOUNT, POSITION, COMM, IERROR)\fargs &lt;type&gt; INBUF(*), OUTBUF(*)\\ INTEGER INCOUNT, DATATYPE, OUTCOUNT, POSITION,  COMM, IERROR}

Packs the message in the send buffer specified by \mpiarg{inbuf, incount,
datatype} into the buffer
space specified by \mpiarg{outbuf} and \mpiarg{outcount}.  The
input buffer can
be any communication buffer allowed in \mpifunc{MPI\_SEND}.  The output buffer
is a contiguous storage area containing \mpiarg{outcount} bytes, starting at
the address \mpiarg{outbuf} (length is counted in {\sf bytes}, not elements,
as if it were a communication buffer for a message of type \type{MPI\_PACKED}).

The input value of \mpiarg{position} is the first
location in the output buffer to be used for packing.  \mpiarg{position} is
incremented by the size of the packed message, and the output value of
\mpiarg{position} is the first location in the output buffer
following the locations occupied by the packed message.
The \mpiarg{comm} argument
is the communicator that will be subsequently used for sending the packed
message.

\begin{funcdef}{MPI\_UNPACK(inbuf, insize, position, outbuf, outcount,
datatype, comm)}
\funcarg{\IN}{inbuf}{input buffer start (choice)}
\funcarg{\IN}{insize}{size of input buffer, in bytes (integer)}
\funcarg{\INOUT}{position}{current position in bytes (integer)}
\funcarg{\OUT}{outbuf}{output buffer start (choice)}
\funcarg{\IN}{outcount}{number of items to be unpacked (integer)}
\funcarg{\IN}{datatype}{datatype of each output data item (handle)}
\funcarg{\IN}{comm}{communicator for packed message (handle)}
\end{funcdef}

\mpibind{MPI\_Unpack(void*~inbuf, int~insize, int~*position, void~*outbuf, int~outcount, MPI\_Datatype~datatype, MPI\_Comm~comm)}

\mpifbind{MPI\_UNPACK(INBUF, INSIZE, POSITION, OUTBUF, OUTCOUNT, DATATYPE, COMM, IERROR)\fargs  &lt;type&gt; INBUF(*), OUTBUF(*) \\ INTEGER INSIZE, POSITION, OUTCOUNT, DATATYPE, COMM, IERROR}

Unpacks a message into the receive buffer specified by \mpiarg{outbuf,
outcount,
datatype} from the buffer
space specified by \mpiarg{inbuf} and \mpiarg{insize}.  The output buffer can
be any communication buffer allowed in \mpifunc{MPI\_RECV}.  The input
buffer is a contiguous storage area containing \mpiarg{insize} bytes,
starting at address \mpiarg{inbuf}.
The input value of \mpiarg{position} is the first location in
the output buffer occupied by the packed message.
\mpiarg{position} is incremented
by the size of the packed message, so that the output value of
\mpiarg{position} is the first location in the output buffer
after the locations occupied by the message that was unpacked.
\mpiarg{comm} is the communicator used to receive the packed message.

\begin{users}
Note the difference between \mpifunc{MPI\_RECV} and \mpifunc{MPI\_UNPACK}:  in
\mpifunc{MPI\_RECV}, the \mpiarg{count} argument specifies the maximum
number of
items that can be received.  The actual number of items received is determined
by the length of the incoming message.  In \mpiarg{MPI\_UNPACK}, the
\mpiarg{count} argument specifies the actual number
of items that are unpacked;
the ``size'' of the corresponding message is the increment in
\mpiarg{position}.
The reason for this change is that the ``incoming message size'' is not
predetermined since the user decides how much to unpack; nor is it easy to
determine
the ``message size'' from the number of items to be unpacked.  In fact, in a
heterogeneous system, this number may not be determined {\em a priori}.
\end{users}


To understand the behavior of pack and unpack, it is convenient to think of the
data part of a message as being the sequence obtained by concatenating the
successive values sent in that message.   The pack operation stores this
sequence in the buffer space, as if sending the message to that buffer.  The
unpack operation retrieves this sequence from buffer space, as if receiving a
message from that buffer.  (It is helpful to think of internal Fortran files or
{\sf sscanf} in C, for a similar function.)

Several messages can be successively packed into one {\bf packing unit}.  This
is effected by several successive {\bf related} calls to \mpiarg{MPI\_PACK},
where the first
call provides \mpiarg{position = 0}, and each successive call inputs the value
of \mpiarg{position} that was output by the previous call, and the same values
for \mpiarg{outbuf, outcount} and \mpiarg{comm}.   This packing unit
now contains
the equivalent information that would have been stored in a message by one send
call with a send buffer that is the ``concatenation'' of the individual send
buffers.

A  packing unit can
be sent using type \type{MPI\_PACKED}.  Any point to point
or collective communication function can be used to move the sequence of bytes
that forms the packing unit from one process to another.  This packing unit
can now be
received using any receive operation, with any datatype:  the
type matching rules are relaxed for messages sent with type \type{MPI\_PACKED}.

A message sent with any type (including \type{MPI\_PACKED}) can be
received using the type \type{MPI\_PACKED}.  Such a message can then be
unpacked by calls to \mpifunc{MPI\_UNPACK}.

A packing unit (or a message created by a regular, ``typed'' send)
can be unpacked into several successive messages.  This is
effected by several successive related calls to \mpifunc{MPI\_UNPACK}, where
the first
call provides \mpiarg{position = 0}, and each successive call inputs the value
of \mpiarg{position} that was output by the previous call, and the same values
for \mpiarg{inbuf, insize} and \mpiarg{comm}.


The concatenation of two packing units is not necessarily a packing unit; nor is
a substring of a packing unit necessarily a packing unit.  Thus, one cannot
concatenate two packing units and then unpack the result as one packing
unit; nor can one unpack a substring of a packing unit as a separate
packing unit.  Each packing unit, that was created by a related
sequence of pack
calls, or by a regular send,
must be unpacked as a unit, by a sequence of related unpack calls.



\begin{rationale}
The restriction on ``atomic'' packing and unpacking of packing units allows
the implementation to add at the head of packing units additional
information, such as
a description of the sender architecture (to be used for type conversion, in a
heterogeneous environment)
\end{rationale}



The following call allows the user to find out how much space is
needed to pack a message and, thus, manage space allocation for
buffers.

\begin{funcdef}{MPI\_PACK\_SIZE(incount, datatype, comm, size)}
\funcarg{\IN}{incount}{count argument to packing call (integer)}
\funcarg{\IN}{datatype}{datatype argument to packing call (handle)}
\funcarg{\IN}{comm}{communicator argument to packing call (handle)}
\funcarg{\OUT}{size}{upper bound on size of packed message, in bytes (integer)}
\end{funcdef}

\mpibind{MPI\_Pack\_size(int~incount, MPI\_Datatype~datatype, MPI\_Comm~comm, int~*size)}

\mpifbind{MPI\_PACK\_SIZE(INCOUNT, DATATYPE, COMM, SIZE, IERROR)\fargs INTEGER INCOUNT, DATATYPE, COMM, SIZE, IERROR}

A call to \mpifunc{MPI\_PACK\_SIZE(incount, datatype, comm, size)}
returns in \mpiarg{size} an upper bound on the increment in \mpiarg{position}
that is effected by a call to \mpifunc{MPI\_PACK(inbuf, incount, datatype,
outbuf, outcount, position, comm)}.

\begin{rationale}
The call returns an upper bound, rather than an exact bound, since the
exact amount of space needed to pack the message may depend on the
context (e.g., first message packed in a packing unit may take more
space).
\end{rationale}


\begin{example} {\rm
\label{pt2pt-exJJ}
An example using \func{MPI\_PACK}.
\begin{verbatim}
int position, i, j, a[2];
char buff[1000];

....

MPI_Comm_rank(MPI_COMM_WORLD, &amp;myrank);
if (myrank == 0)
{
   / * SENDER CODE */

  position = 0;
  MPI_Pack(&amp;i, 1, MPI_INT, buff, 1000, &amp;position, MPI_COMM_WORLD);
  MPI_Pack(&amp;j, 1, MPI_INT, buff, 1000, &amp;position, MPI_COMM_WORLD);
  MPI_Send( buff, position, MPI_PACKED, 1, 0, MPI_COMM_WORLD);
}
else  /* RECEIVER CODE */
  MPI_Recv( a, 2, MPI_INT, 0, 0, MPI_COMM_WORLD)

}
\end{verbatim}
} \end{example}



\begin{example} {\rm
\label{pt2pt-exKK}
A elaborate example.
\begin{verbatim}
int position, i;
float a[1000];
char buff[1000]

....

MPI_Comm_rank(MPI_Comm_world, &amp;myrank);
if (myrank == 0)
{
  / * SENDER CODE */

  int len[2];
  MPI_Aint disp[2];
  MPI_Datatype type[2], newtype;

  /* build datatype for i followed by a[0]...a[i-1] */

  len[0] = 1;
  len[1] = i;
  MPI_Address( &amp;i, disp);
  MPI_Address( a, disp+1);
  type[0] = MPI_INT;
  type[1] = MPI_FLOAT;
  MPI_Type_struct( 2, len, disp, type, &amp;newtype);
  MPI_Type_commit( &amp;newtype);

  /* Pack i followed by a[0]...a[i-1]*/

  position = 0;
  MPI_Pack( MPI_BOTTOM, 1, newtype, buff, 1000, &amp;position, MPI_COMM_WORLD);

  /* Send */

  MPI_Send( buff, position, MPI_PACKED, 1, 0,
            MPI_COMM_WORLD)

/* *****
   One can replace the last three lines with
   MPI_Send( MPI_BOTTOM, 1, newtype, 1, 0, MPI_COMM_WORLD);
   ***** */
}
else /* myrank == 1 */
{
   /* RECEIVER CODE */

  MPI_Status status;

  /* Receive */

  MPI_Recv( buff, 1000, MPI_PACKED, 0, 0, &amp;status);

  /* Unpack i */

 position = 0;
 MPI_Unpack(buff, 1000, &amp;position, &amp;i, 1, MPI_INT, MPI_COMM_WORLD);

 /* Unpack a[0]...a[i-1] */
 MPI_Unpack(buff, 1000, &amp;position, a, i, MPI_FLOAT, MPI_COMM_WORLD);
}
\end{verbatim}
} \end{example}

\begin{example} {\rm
\label{pt2pt-exLL}
Each process sends a count, followed by count characters to the root;
the root concatenate all characters into one string.
\begin{verbatim}
int count, gsize, counts[64], totalcount, k1, k2, k,
    displs[64], position, concat_pos;
char chr[100], *lbuf, *rbuf, *cbuf;
...
MPI_Comm_size(comm, &amp;gsize);
MPI_Comm_rank(comm, &amp;myrank);

      /* allocate local pack buffer */
MPI_Pack_size(1, MPI_INT, comm, &amp;k1);
MPI_Pack_size(count, MPI_CHAR, &amp;k2);
k = k1+k2;
lbuf = (char *)malloc(k);

      /* pack count, followed by count characters */
position = 0;
MPI_Pack(&amp;count, 1, MPI_INT, &amp;lbuf, k, &amp;position, comm);
MPI_Pack(chr, count, MPI_CHAR, &amp;lbuf, k, &amp;position, comm);

if (myrank != root)
      /* gather at root sizes of all packed messages */
   MPI_Gather( &amp;position, 1, MPI_INT, NULL, NULL,
             NULL, root, comm);

      /* gather at root packed messages */
  MPI_Gatherv( &amp;buf, position, MPI_PACKED, NULL,
           NULL, NULL, NULL, root, comm);

else {   /* root code */
      /* gather sizes of all packed messages */
   MPI_Gather( &amp;position, 1, MPI_INT, counts, 1,
             MPI_INT, root, comm);

      /* gather all packed messages */
  displs[0] = 0;
  for (i=1; i &lt; gsize; i++)
    displs[i] = displs[i-1] + counts[i-1];
  totalcount = dipls[gsize-1] + counts[gsize-1];
  rbuf = (char *)malloc(totalcount);
  cbuf = (char *)malloc(totalcount);
  MPI_Gatherv( lbuf, position, MPI_PACKED, rbuf,
           counts, displs, MPI_PACKED, root, comm);

      /* unpack all messages and concatenate strings */
  concat_pos = 0;
  for (i=0; i &lt; gsize; i++) {
    position = 0;
    MPI_Unpack( rbuf+displs[i], totalcount-displs[i],
          &amp;position, &amp;count, 1, MPI_INT, comm);
    MPI_Unpack( rbuf+displs[i], totalcount-displs[i],
          &amp;position, cbuf+concat_pos, count, MPI_CHAR, comm);
    concat_pos += count;
    }
  cbuf[concat_pos] = `\0';
  }
\end{verbatim}
} \end{example}
</pre></body></html>