users-guide.tex

fortran并行计算包

% \documentstyle[11pt,psfig]{article}
\documentstyle[11pt]{article}
\hoffset=-.7in
\voffset=-.6in
\textwidth=6.5in
\textheight=8.5in
\begin{document}
\vspace*{-1in}
\thispagestyle{empty}
\begin{center}
ARGONNE NATIONAL LABORATORY \\
9700 South Cass Avenue \\
Argonne, IL 60439
\end{center}
\vskip .5 in
\begin{center}
\rule{1.75in}{.01in} \\
\vspace{.1in}

ANL/MCS-TM-234 \\

\rule{1.75in}{.01in} \\

\vskip 1.3 in
{\Large\bf Users Guide for ROMIO: A High-Performance, \\ [1ex]
Portable MPI-IO Implementation} \\ [4ex]
by \\ [2ex]
{\large\it Rajeev Thakur, Robert Ross, Ewing Lusk, and William Gropp}
\vspace{1in}

Mathematics and Computer Science Division

\bigskip

Technical Memorandum No.\ 234


\vspace{1.4in}
Revised May 2004

\end{center}

\vfill

{\small
\noindent
This work was supported by the Mathematical, Information, and
Computational Sciences Division subprogram of the Office of Advanced
Scientific Computing Research, U.S. Department of Energy, under
Contract W-31-109-Eng-38; and by the Scalable I/O Initiative, a
multiagency project funded by the Defense Advanced Research Projects
Agency (Contract DABT63-94-C-0049), the Department of Energy, the
National Aeronautics and Space Administration, and the National
Science Foundation.}

\newpage


%%  Line Spacing (e.g., \ls{1} for single, \ls{2} for double, even \ls{1.5})
%%

\newcommand{\ls}[1]
   {\dimen0=\fontdimen6\the\font 
    \lineskip=#1\dimen0
    \advance\lineskip.5\fontdimen5\the\font
    \advance\lineskip-\dimen0
    \lineskiplimit=.9\lineskip
    \baselineskip=\lineskip
    \advance\baselineskip\dimen0
    \normallineskip\lineskip
    \normallineskiplimit\lineskiplimit
    \normalbaselineskip\baselineskip
    \ignorespaces
   }
\renewcommand{\baselinestretch}{1}
\newcommand {\ix} {\hspace*{2em}}
\newcommand {\mc} {\multicolumn}


\tableofcontents
\thispagestyle{empty}
\newpage

\pagenumbering{arabic}
\setcounter{page}{1}
\begin{center}
{\bf Users Guide for ROMIO:  A High-Performance,\\[1ex]
Portable MPI-IO Implementation} \\ [2ex]
by \\ [2ex]
{\it Rajeev Thakur, Robert Ross, Ewing Lusk, and William Gropp}

\end{center}
\addcontentsline{toc}{section}{Abstract}
\begin{abstract}
\noindent
ROMIO is a high-performance, portable implementation of MPI-IO (the
I/O chapter in \mbox{MPI-2}). This document describes how to install and use
ROMIO version~1.2.4 on various machines.
\end{abstract}

\section{Introduction} 

ROMIO\footnote{\tt http://www.mcs.anl.gov/romio} is a
high-performance, portable implementation of MPI-IO (the I/O chapter in 
MPI-2~\cite{mpi97a}). This document describes how to install and use
ROMIO version~1.2.4 on various machines.


%
% MAJOR CHANGES IN THIS VERSION
%
\section{Major Changes in This Version}
\begin{itemize}
\item Added section describing ROMIO \texttt{MPI\_FILE\_SYNC} and
      \texttt{MPI\_FILE\_CLOSE} behavior to User's Guide
\item Bug removed from PVFS ADIO implementation regarding resize operations
\item Added support for PVFS listio operations (see Section \ref{sec:hints})
\item Added the following working hints:
      \texttt{romio\_pvfs\_listio\_read}, \texttt{romio\_pvfs\_listio\_write}
\end{itemize}

%
% GENERAL INFORMATION
%
\section{General Information}

This version of ROMIO includes everything defined in the MPI-2 I/O
chapter except support for file interoperability (\S~9.5 of MPI-2) and
user-defined error handlers for files (\S~4.13.3).  The subarray and
distributed array datatype constructor functions from Chapter 4
(\S~4.14.4 \& \S~4.14.5) have been implemented. They are useful for
accessing arrays stored in files. The functions {\tt MPI\_File\_f2c}
and {\tt MPI\_File\_c2f} (\S~4.12.4) are also implemented.  C,
Fortran, and profiling interfaces are provided for all functions that
have been implemented.

This version of ROMIO runs on at least the following machines: IBM SP; Intel
Paragon; HP Exemplar; SGI Origin2000; Cray T3E; NEC SX-4; other
symmetric multiprocessors from HP, SGI, DEC, Sun, and IBM; and networks of
workstations (Sun, SGI, HP, IBM, DEC, Linux, and FreeBSD).
Supported file systems are IBM PIOFS, Intel PFS, HP/Convex
HFS, SGI XFS, NEC SFS, PVFS, NFS, NTFS, and any Unix file system (UFS).

This version of ROMIO is included in MPICH 1.2.4; an earlier version
is included in at least the following MPI implementations: LAM, HP
MPI, SGI MPI, and NEC MPI. 

Note that proper I/O error codes and classes are returned and the status
variable is filled only when used with MPICH revision 1.2.1 or later.

You can open files on multiple file systems in the same program. The
only restriction is that the directory where the file is to be opened
must be accessible from the process opening the file. For example, a
process running on one workstation may not be able to access a
directory on the local disk of another workstation, and therefore
ROMIO will not be able to open a file in such a directory. NFS-mounted
files can be accessed.

An MPI-IO file created by ROMIO is no different from any other file
created by the underlying file system. Therefore, you may use any of
the commands provided by the file system to access the file, for example,
{\tt ls}, {\tt mv}, {\tt cp}, {\tt rm}, {\tt ftp}.

Please read the limitations of this version of ROMIO that are listed
in Section~\ref{sec:limit} of this document (e.g., restriction to homogeneous
environments). 

\subsection{ROMIO Optimizations}
\label{sec:opt}

ROMIO implements two I/O optimization techniques that in general
result in improved performance for applications.  The first of these
is \emph{data sieving}~\cite{choudhary:passion}.  Data sieving is a
technique for efficiently accessing noncontiguous regions of data in files
when noncontiguous accesses are not provided as a file system primitive.
The naive approach to accessing noncontiguous regions is to use a separate
I/O call for each contiguous region in the file.  This results in a large
number of I/O operations, each of which is often for a very small amount
of data.  The added network cost of performing an I/O operation across the
network, as in parallel I/O systems, is often high because of latency.
Thus, this naive approach typically performs very poorly because of
the overhead of multiple operations.  
% 
In the data sieving technique, a number of noncontiguous regions are
accessed by reading a block of data containing all of the regions,
including the unwanted data between them (called ``holes'').  The regions
of interest are then extracted from this large block by the client.
This technique has the advantage of a single I/O call, but additional
data is read from the disk and passed across the network.

There are four hints that can be used to control the application of
data sieving in ROMIO: \texttt{ind\_rd\_buffer\_size},
\texttt{ind\_wr\_buffer\_size}, \texttt{romio\_ds\_read},
and \texttt{romio\_ds\_write}.  These are discussed in
Section~\ref{sec:hints}.

The second optimization is \emph{two-phase
I/O}~\cite{bordawekar:primitives}.  Two-phase I/O, also called collective
buffering, is an optimization that only applies to collective I/O
operations.  In two-phase I/O, the collection of independent I/O operations
that make up the collective operation are analyzed to determine what
data regions must be transferred (read or written).  These regions are
then split up amongst a set of aggregator processes that will actually
interact with the file system.  In the case of a read, these aggregators
first read their regions from disk and redistribute the data to the
final locations, while in the case of a write, data is first collected
from the processes before being written to disk by the aggregators.

There are five hints that can be used to control the application
of two-phase I/O: \texttt{cb\_config\_list}, \texttt{cb\_nodes},
\texttt{cb\_buffer\_size}, \texttt{romio\_cb\_read},
and \texttt{romio\_cb\_write}.  These are discussed in
Subsection~\ref{sec:hints}.

\subsection{Hints}
\label{sec:hints}

The following hints control the data sieving optimization and are
applicable to all file system types:
\begin{itemize}
\item \texttt{ind\_rd\_buffer\_size} -- Controls the size (in bytes) of the
intermediate buffer used by ROMIO when performing data sieving during
read operations.  Default is \texttt{4194304} (4~Mbytes).
\item \texttt{ind\_wr\_buffer\_size} -- Controls the size (in bytes) of the
intermediate buffer used by ROMIO when performing data sieving during
write operations.  Default is \texttt{524288} (512~Kbytes).
\item \texttt{romio\_ds\_read} -- 
Determines when ROMIO will choose to perform data sieving.
Valid values are \texttt{enable}, \texttt{disable}, or \texttt{automatic}.
Default value is \texttt{automatic}.  In \texttt{automatic} mode ROMIO
may choose to enable or disable data sieving based on heuristics.
\item \texttt{romio\_ds\_write} -- Same as above, only for writes.
\end{itemize}

The following hints control the two-phase (collective buffering)
optimization and are applicable to all file system types:
\begin{itemize}
\item \texttt{cb\_buffer\_size} -- Controls the size (in bytes) of the
intermediate buffer used in two-phase collective I/O.  If the amount
of data that an aggregator will transfer is larger than this value,
then multiple operations are used.  The default is \texttt{4194304} (4~Mbytes).
\item \texttt{cb\_nodes} -- Controls the maximum number of aggregators
to be used.  By default this is set to the number of unique hosts in the
communicator used when opening the file.
\item \texttt{romio\_cb\_read} -- Controls when collective buffering is
applied to collective read operations.  Valid values are
\texttt{enable}, \texttt{disable}, and \texttt{automatic}.  Default is
\texttt{automatic}.  When enabled, all collective reads will use
collective buffering.  When disabled, all collective reads will be
serviced with individual operations by each process.  When set to
\texttt{automatic}, ROMIO will use heuristics to determine when to
enable the optimization.
\item \texttt{romio\_cb\_write} -- Controls when collective buffering is
applied to collective write operations.  Valid values are 
\texttt{enable}, \texttt{disable}, and \texttt{automatic}.  Default is 
\texttt{automatic}.  See the description of \texttt{romio\_cb\_read} for
an explanation of the values.
\item \texttt{romio\_no\_indep\_rw} -- This hint controls when ``deferred
open'' is used.  When set to \texttt{true}, ROMIO will make an effort to avoid
performing any file operation on non-aggregator nodes.  The application is
expected to use only collective operations.  This is discussed in further
detail below.
\item \texttt{cb\_config\_list} -- Provides explicit control over 
aggregators.  This is discussed in further detail below.
\end{itemize}

For some systems configurations, more control is needed to specify which
hardware resources (processors or nodes in an SMP) are preferred for
collective I/O, either for performance reasons or because only certain
resources have access to storage.  The additional MPI\_Info key name
\texttt{cb\_config\_list} specifies a comma-separated list of strings,
each string specifying a particular node and an optional limit on the
number of processes to be used for collective buffering on this node.

This refers to the same processes that \texttt{cb\_nodes} refers to,
but specifies the available nodes more precisely.

The format of the value of \texttt{cb\_config\_list} is given by the
following BNF:
\begin{verbatim}
cb_config_list => hostspec [ ',' cb_config_list ]
hostspec => hostname [ ':' maxprocesses ]
hostname => <alphanumeric string>
         |  '*' 
maxprocesses => <digits>
         |  '*'
\end{verbatim}

The value \texttt{hostname} identifies a processor. This name must match
the name returned by \texttt{MPI\_Get\_processor\_name}~\footnote{The
MPI standard requires that the output from this routine identify a
particular piece of hardware; some MPI implementations may not conform
to this requirement. MPICH does conform to the MPI standard.}
%
for the specified hardware. The value \texttt{*} as a hostname matches all
processors. The value of maxprocesses may be any nonnegative integer
(zero is allowed).

The value \texttt{maxprocesses} specifies the maximum number of
processes that may be used for collective buffering on the specified
host. If no value is specified, the value one is assumed. If \texttt{*}
is specified for the number of processes, then all MPI processes with
this same hostname will be used..

Leftmost components of the info value take precedence.

Note: Matching of processor names to \texttt{cb\_config\_list} entries
is performed with string matching functions and is independent of the
listing of machines that the user provides to mpirun/mpiexec.  In other
words, listing the same machine multiple times in the list of hosts to
run on will not cause a \texttt{*:1} to assign the same host four
aggregators, because the matching code will see that the processor name
is the same for all four and will assign exactly one aggregator to the
processor.

The value of this info key must be the same for all processes (i.e., the
call is collective and each process must receive the same hint value for
these collective buffering hints).  Further, in the ROMIO implementation
the hint is only recognized at \texttt{MPI\_File\_open} time.

The set of hints used with a file is available through the routine
\texttt{MPI\_File\_get\_info}, as documented in the MPI-2 standard. 
As an additional feature in the ROMIO implementation, wildcards will
be expanded to indicate the precise configuration used with the file,
with the hostnames in the rank order used for the collective buffering
algorithm (\emph{this is not implemented at this time}).

Here are some examples of how this hint might be used:
\begin{itemize}
\item \texttt{*:1} One process per hostname (i.e., one process per node)
\item \texttt{box12:30,*:0} Thirty processes on one machine, namely
      \texttt{box12}, and none anywhere else.
\item \texttt{n01,n11,n21,n31,n41} One process on each of these specific
      nodes only.
\end{itemize}

When the values specified by \texttt{cb\_config\_list} conflict with
other hints (e.g., the number of collective buffering nodes specified by
\texttt{cb\_nodes}), the implementation is encouraged to take the minimum