mpi.qbk

Boost provides free peer-reviewed portable C++ source libraries. We emphasize libraries that work

text could come out completely garbled, because one process can start
writing "I am a process" before another process has finished writing
"of 7.".

[section:point_to_point Point-to-Point communication]

As a message passing library, MPI's primary purpose is to routine
messages from one process to another, i.e., point-to-point. MPI
contains routines that can send messages, receive messages, and query
whether messages are available. Each message has a source process, a
target process, a tag, and a payload containing arbitrary data. The
source and target processes are the ranks of the sender and receiver
of the message, respectively. Tags are integers that allow the
receiver to distinguish between different messages coming from the
same sender. 

The following program uses two MPI processes to write "Hello, world!"
to the screen (`hello_world.cpp`):

  #include <boost/mpi.hpp>
  #include <iostream>
  #include <boost/serialization/string.hpp>
  namespace mpi = boost::mpi;

  int main(int argc, char* argv[]) 
  {
    mpi::environment env(argc, argv);
    mpi::communicator world;

    if (world.rank() == 0) {
      world.send(1, 0, std::string("Hello"));
      std::string msg;
      world.recv(1, 1, msg);
      std::cout << msg << "!" << std::endl;
    } else {
      std::string msg;
      world.recv(0, 0, msg);
      std::cout << msg << ", ";
      std::cout.flush();
      world.send(0, 1, std::string("world"));
    }

    return 0;
  }

The first processor (rank 0) passes the message "Hello" to the second
processor (rank 1) using tag 0. The second processor prints the string
it receives, along with a comma, then passes the message "world" back
to processor 0 with a different tag. The first processor then writes
this message with the "!" and exits. All sends are accomplished with
the [memberref boost::mpi::communicator::send
communicator::send] method and all receives use a corresponding
[memberref boost::mpi::communicator::recv
communicator::recv] call.

[section:nonblocking Non-blocking communication]

The default MPI communication operations--`send` and `recv`--may have
to wait until the entire transmission is completed before they can
return. Sometimes this *blocking* behavior has a negative impact on
performance, because the sender could be performing useful computation
while it is waiting for the transmission to occur. More important,
however, are the cases where several communication operations must
occur simultaneously, e.g., a process will both send and receive at
the same time.

Let's revisit our "Hello, world!" program from the previous
section. The core of this program transmits two messages:

    if (world.rank() == 0) {
      world.send(1, 0, std::string("Hello"));
      std::string msg;
      world.recv(1, 1, msg);
      std::cout << msg << "!" << std::endl;
    } else {
      std::string msg;
      world.recv(0, 0, msg);
      std::cout << msg << ", ";
      std::cout.flush();
      world.send(0, 1, std::string("world"));
    }

The first process passes a message to the second process, then
prepares to receive a message. The second process does the send and
receive in the opposite order. However, this sequence of events is
just that--a *sequence*--meaning that there is essentially no
parallelism. We can use non-blocking communication to ensure that the
two messages are transmitted simultaneously
(`hello_world_nonblocking.cpp`): 

  #include <boost/mpi.hpp>
  #include <iostream>
  #include <boost/serialization/string.hpp>
  namespace mpi = boost::mpi;

  int main(int argc, char* argv[]) 
  {
    mpi::environment env(argc, argv);
    mpi::communicator world;

    if (world.rank() == 0) {
      mpi::request reqs[2];
      std::string msg, out_msg = "Hello";
      reqs[0] = world.isend(1, 0, out_msg);
      reqs[1] = world.irecv(1, 1, msg);
      mpi::wait_all(reqs, reqs + 2);
      std::cout << msg << "!" << std::endl;
    } else {
      mpi::request reqs[2];
      std::string msg, out_msg = "world";
      reqs[0] = world.isend(0, 1, out_msg);
      reqs[1] = world.irecv(0, 0, msg);
      mpi::wait_all(reqs, reqs + 2);
      std::cout << msg << ", ";
    }

    return 0;
  }

We have replaced calls to the [memberref
boost::mpi::communicator::send communicator::send] and
[memberref boost::mpi::communicator::recv
communicator::recv] members with similar calls to their non-blocking
counterparts, [memberref boost::mpi::communicator::isend
communicator::isend] and [memberref
boost::mpi::communicator::irecv communicator::irecv]. The
prefix *i* indicates that the operations return immediately with a
[classref boost::mpi::request mpi::request] object, which
allows one to query the status of a communication request (see the
[memberref boost::mpi::request::test test] method) or wait
until it has completed (see the [memberref
boost::mpi::request::wait wait] method). Multiple requests
can be completed at the same time with the [funcref
boost::mpi::wait_all wait_all] operation. 

If you run this program multiple times, you may see some strange
results: namely, some runs will produce:

  Hello, world!

while others will produce:

  world!
  Hello,

or even some garbled version of the letters in "Hello" and
"world". This indicates that there is some parallelism in the program,
because after both messages are (simultaneously) transmitted, both
processes will concurrent execute their print statements. For both
performance and correctness, non-blocking communication operations are
critical to many parallel applications using MPI.

[endsect]

[section:user_data_types User-defined data types]

The inclusion of `boost/serialization/string.hpp` in the previous
examples is very important: it makes values of type `std::string`
serializable, so that they can be be transmitted using Boost.MPI. In
general, built-in C++ types (`int`s, `float`s, characters, etc.) can
be transmitted over MPI directly, while user-defined and
library-defined types will need to first be serialized (packed) into a
format that is amenable to transmission. Boost.MPI relies on the
_Serialization_ library to serialize and deserialize data types. 

For types defined by the standard library (such as `std::string` or
`std::vector`) and some types in Boost (such as `boost::variant`), the
_Serialization_ library already contains all of the required
serialization code. In these cases, you need only include the
appropriate header from the `boost/serialization` directory. 

For types that do not already have a serialization header, you will
first need to implement serialization code before the types can be
transmitted using Boost.MPI. Consider a simple class `gps_position`
that contains members `degrees`, `minutes`, and `seconds`. This class
is made serializable by making it a friend of
`boost::serialization::access` and introducing the templated
`serialize()` function, as follows:

  class gps_position
  {
  private:
      friend class boost::serialization::access;

      template<class Archive>
      void serialize(Archive & ar, const unsigned int version)
      {
          ar & degrees;
          ar & minutes;
          ar & seconds;
      }

      int degrees;
      int minutes;
      float seconds;
  public:
      gps_position(){};
      gps_position(int d, int m, float s) :
          degrees(d), minutes(m), seconds(s)
      {}
  };

Complete information about making types serializable is beyond the
scope of this tutorial. For more information, please see the
_Serialization_ library tutorial from which the above example was
extracted. One important side benefit of making types serializable for
Boost.MPI is that they become serializable for any other usage, such
as storing the objects to disk to manipulated them in XML.

Some serializable types, like `gps_position` above, have a fixed
amount of data stored at fixed field positions. When this is the case,
Boost.MPI can optimize their serialization and transmission to avoid
extraneous copy operations. To enable this optimization, users should
specialize the type trait [classref
boost::mpi::is_mpi_datatype `is_mpi_datatype`], e.g.:

  namespace boost { namespace mpi {
    template <>
    struct is_mpi_datatype<gps_position> : mpl::true_ { };
  } }

For non-template types we have defined a macro to simplify declaring a type 
as an MPI datatype

  BOOST_IS_MPI_DATATYPE(gps_position)

For composite traits, the specialization of [classref
boost::mpi::is_mpi_datatype `is_mpi_datatype`] may depend on
`is_mpi_datatype` itself. For instance, a `boost::array` object is
fixed only when the type of the parameter it stores is fixed:

  namespace boost { namespace mpi {
    template <typename T, std::size_t N>
    struct is_mpi_datatype<array<T, N> > 
      : public is_mpi_datatype<T> { };
  } }
  
The redundant copy elimination optimization can only be applied when
the shape of the data type is completely fixed. Variable-length types
(e.g., strings, linked lists) and types that store pointers cannot use
the optimiation, but Boost.MPI will be unable to detect this error at
compile time. Attempting to perform this optimization when it is not
correct will likely result in segmentation faults and other strange
program behavior.

Boost.MPI can transmit any user-defined data type from one process to
another. Built-in types can be transmitted without any extra effort;
library-defined types require the inclusion of a serialization header;
and user-defined types will require the addition of serialization
code. Fixed data types can be optimized for transmission using the
[classref boost::mpi::is_mpi_datatype `is_mpi_datatype`]
type trait.

[endsect]
[endsect]

[section:collectives Collective operations]

[link mpi.point_to_point Point-to-point operations] are the
core message passing primitives in Boost.MPI. However, many
message-passing applications also require higher-level communication
algorithms that combine or summarize the data stored on many different
processes. These algorithms support many common tasks such as
"broadcast this value to all processes", "compute the sum of the
values on all processors" or "find the global minimum." 

[section:broadcast Broadcast]
The [funcref boost::mpi::broadcast `broadcast`] algorithm is
by far the simplest collective operation. It broadcasts a value from a
single process to all other processes within a [classref
boost::mpi::communicator communicator]. For instance, the
following program broadcasts "Hello, World!" from process 0 to every
other process. (`hello_world_broadcast.cpp`)

  #include <boost/mpi.hpp>
  #include <iostream>
  #include <boost/serialization/string.hpp>
  namespace mpi = boost::mpi;

  int main(int argc, char* argv[])
  {
    mpi::environment env(argc, argv);
    mpi::communicator world;

    std::string value;
    if (world.rank() == 0) {
      value = "Hello, World!";
    }

    broadcast(world, value, 0);

    std::cout << "Process #" << world.rank() << " says " << value 
              << std::endl;
    return 0;
  } 

Running this program with seven processes will produce a result such
as:

[pre
Process #0 says Hello, World!
Process #2 says Hello, World!
Process #1 says Hello, World!
Process #4 says Hello, World!
Process #3 says Hello, World!
Process #5 says Hello, World!
Process #6 says Hello, World!
]
[endsect]

[section:gather Gather]
The [funcref boost::mpi::gather `gather`] collective gathers
the values produced by every process in a communicator into a vector
of values on the "root" process (specified by an argument to
`gather`). The /i/th element in the vector will correspond to the
value gathered fro mthe /i/th process. For instance, in the following
program each process computes its own random number. All of these
random numbers are gathered at process 0 (the "root" in this case),
which prints out the values that correspond to each processor. 
(`random_gather.cpp`)

  #include <boost/mpi.hpp>
  #include <iostream>
  #include <cstdlib>
  namespace mpi = boost::mpi;

  int main(int argc, char* argv[])
  {
    mpi::environment env(argc, argv);
    mpi::communicator world;