users-guide.tex

fortran并行计算包

of the two values.  In other words, if \texttt{cb\_config\_list} specifies
ten processors on which I/O should be performed, but \texttt{cb\_nodes}
specifies a smaller number, then an implementation is encouraged to use
only \texttt{cb\_nodes} total aggregators. If \texttt{cb\_config\_list}
specifies fewer processes than \texttt{cb\_nodes}, no more than the
number in \texttt{cb\_config\_list} should be used.

The implementation is also encouraged to assign processes in the order
that they are listed in \texttt{cb\_config\_list}.

The following hint controls the deferred open feature of romio and are also
applicable to all file system types: 
\begin{itemize}
\item \texttt{no\_indep\_rw} -- If the application plans on performing only 
   collecitve operations and this hint is set to ``true'', then ROMIO can
   have just the aggregators open a file.   The \texttt{cb\_config\_list} and
   \texttt{cb\_nodes} hints can be given to further control which nodes are
   aggregators.  
\end{itemize}

For PVFS, PIOFS, and PFS:
\begin{itemize}
\item \texttt{striping\_factor} -- Controls the number of I/O devices to
stripe across.  The default is file system dependent, but for PVFS it is
\texttt{-1}, indicating that the file should be striped across all I/O
devices.
\item \texttt{striping\_unit} --  Controls the striping unit (in bytes).
For PVFS the default will be the PVFS file system default strip size.
\item \texttt{start\_iodevice} -- Determines what I/O device data will
first be written to.  This is a number in the range of 0 ...
striping\_factor - 1.
\end{itemize}

Also for PFS:
\begin{itemize}
\item \texttt{pfs\_svr\_buf} -- Turns on PFS server buffering.  Valid
values are \texttt{true} and \texttt{false}.  Default is \texttt{false}.
\end{itemize}

For XFS control is provided for the direct I/O optimization:
\begin{itemize}
\item \texttt{direct\_read} -- Controls direct I/O for reads.  Valid
values are \texttt{true} and \texttt{false}.  Default is \texttt{false}.
\item \texttt{direct\_write} -- Controls direct I/O for writes.  Valid
values are \texttt{true} and \texttt{false}.  Default is \texttt{false}.
\end{itemize}

For PVFS control is provided for the use of the listio interface.  This
interface to PVFS allows for a collection of noncontiguous regions to be
requested (for reading or writing) with a single operation.  This can result
in substantially higher performance when accessing noncontiguous regions.
Support for these operations in PVFS exists after version 1.5.4, but has not
been heavily tested, so use of the interface is disabled in ROMIO by default
at this time.  The hints to control listio use are:
\begin{itemize}
\item \texttt{romio\_pvfs\_listio\_read} -- Controls use of listio for reads.
Valid values are \texttt{enable}, \texttt{disable}, and \texttt{automatic}.
Default is \texttt{disable}.
\item \texttt{romio\_pvfs\_listio\_write} -- Controls use of listio for writes.
Valid values are \texttt{enable}, \texttt{disable}, and \texttt{automatic}.
Default is \texttt{disable}.
\end{itemize}

If ROMIO doesn't understand a hint, or if the value is invalid, the hint
will be ignored. The values of hints being used by ROMIO for a file
can be obtained at any time via {\tt MPI\_File\_get\_info}.

\subsubsection{Systemwide Hints}
\label{sec:system_hints}

A site administrator with knowledge of the storage and networking capabilities
of a machine might be able to come up with a set of hint values that work
better for that machine than the ROMIO default values.  As an extention to the
standard, ROMIO will consult a ``hints file''.  This file provides an
additional mechanism for setting MPI-IO hints, albeit in a ROMIO-specific
manner.  The hints file contains a list of hints and their values.  ROMIO will
use these initial hint settings, though programs are free to override any of
them.

The format of the hints file is a list of hints and their values, one per line.
A \# character in the first column indicates a comment, and ROMIO will ignore
the entire line.  Here's an example:

\begin{verbatim}
# this is a comment describing the following setting
cb_nodes 32
# these nodes happen to have the best connection to storage
cb_config_list n01,n11,n21,n31,n41
\end{verbatim}

ROMIO will look for these hints in the file \texttt{/etc/romio-hints}.  A user
can set the environment variable \texttt{ROMIO\_HINTS} to the name of a file
which ROMIO will use instead.   

\subsection{Using ROMIO on NFS}

It is worth first mentioning that in no way do we encourage the use
of ROMIO on NFS volumes.  NFS is not a high-performance protocol, nor
are NFS servers typically very good at handling the types of concurrent
access seen from MPI-IO applications.  Nevertheless, NFS is a very popular
mechanism for providing access to a shared space, and ROMIO does support
MPI-IO to NFS volumes, provided that they are configured properly.

To use ROMIO on NFS, file locking with {\tt fcntl} must work correctly
on the NFS installation. On some installations, fcntl locks don't
work.  To get them to work, you need to use Version~3 of NFS, ensure
that the lockd daemon is running on all the machines, and have the system
administrator mount the NFS file system with the ``{\tt noac}'' option
(no attribute caching). Turning off attribute caching may reduce
performance, but it is necessary for correct behavior.

The following are some instructions we received from Ian Wells of HP
for setting the {\tt noac} option on NFS. We have not tried them
ourselves. We are including them here because you may find 
them useful. Note that some of the steps may be specific to HP
systems, and you may need root permission to execute some of the
commands. 

\begin{verbatim}   
   >1. first confirm you are running nfs version 3
   >
   >rpcnfo -p `hostname` | grep nfs
   >
   >ie 
   >    goedel >rpcinfo -p goedel | grep nfs
   >    100003    2   udp   2049  nfs
   >    100003    3   udp   2049  nfs
   >
   >
   >2. then edit /etc/fstab for each nfs directory read/written by MPIO
   >   on each  machine used for multihost MPIO.
   >
   >    Here is an example of a correct fstab entry for /epm1:
   >
   >   ie grep epm1 /etc/fstab
   > 
   >      ROOOOT 11>grep epm1 /etc/fstab
   >      gershwin:/epm1 /rmt/gershwin/epm1 nfs bg,intr,noac 0 0
   >
   >   if the noac option is not present, add it 
   >   and then remount this directory
   >   on each of the machines that will be used to share MPIO files
   >
   >ie
   >
   >ROOOOT >umount /rmt/gershwin/epm1
   >ROOOOT >mount  /rmt/gershwin/epm1
   >
   >3. Confirm that the directory is mounted noac:
   >
   >ROOOOT >grep gershwin /etc/mnttab 
   >gershwin:/epm1 /rmt/gershwin/epm1 nfs
   >noac,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0 0 0 899911504
\end{verbatim}

\subsubsection{ROMIO, NFS, and Synchronization}

NFS has a ``sync'' option that specifies that the server should put data on
the disk before replying that an operation is complete.  This means that
the actual I/O cost on the server side cannot be hidden with caching,
etc. when this option is selected.
                                                                                
In the ``async'' mode the server can get the data into a buffer (and
perhaps put it in the write queue; this depends on the implementation)
and reply right away.  Obviously if the server were to go down after the
reply was sent but before the data was written, the system would be in
a strange state, which is why so many articles suggest the "sync" option.

Some systems default to ``sync'', while others default to ``async'',
and the default can change from version to version of the NFS software.  If
you find that access to an NFS volume through MPI-IO is particularly slow,
this is one thing to check out.


\subsection{Using testfs}
The testfs ADIO implementation provides a harness for testing components
of ROMIO or discovering the underlying I/O access patterns of an
application.  When testfs is specified as the file system type, no
actual files will be opened.  Instead debugging information will be
displayed on the processes opening the file.  Subsequent I/O operations
on this testfs file will provide additional debugging information.

The intention of the testfs implementation is that it serve as a
starting point for further instrumentation when debugging new features
or applications.  As such it is expected that users will want to modify
the ADIO implementation in order to get the specific output they desire.

\subsection{ROMIO and {\tt MPI\_FILE\_SYNC}}

The MPI-2 specification notes that a call to {\tt MPI\_FILE\_SYNC} ``causes
all previous writes to {\tt fh} by the calling process to be transferred to
the storage device.''  Likewise, calls to {\tt MPI\_FILE\_CLOSE} have this
same semantic.  Further, ``if all processes have made updates to the storage
device, then all such updates become visible to subsequent reads of {\tt fh}
by the calling process.''

The intended use of {\tt MPI\_FILE\_SYNC} is to allow all processes in the
communicator used to open the file to see changes made to the file by each
other (the second part of the specification).  The definition of ``storage
device'' in the specification is vague, and it isn't necessarily the case that
calling {\tt MPI\_FILE\_SYNC} will force data out to permanent storage.

Since users often use {\tt MPI\_FILE\_SYNC} to attempt to force data out to
permanent storage (i.e. disk), the ROMIO implementation of this call enforces
stronger semantics for most underlying file systems by calling the appropriate
file sync operation when {\tt MPI\_FILE\_SYNC} is called (e.g. {\tt fsync}).
However, it is still unwise to assume that the data has all made it to disk
because some file systems (e.g. NFS) may not force data to disk when a client
system makes a sync call.

For performance reasons we do \emph{not} make this same file system call at
{\tt MPI\_FILE\_CLOSE} time.  At close time ROMIO ensures any data has been
written out to the ``storage device'' (file system) as defined in the
standard, but does not try to push the data beyond this and into physical
storage. Users should call {\tt MPI\_FILE\_SYNC} before the close if they wish
to encourage the underlying file system to push data to permanent storage.

\subsection{ROMIO and {\tt MPI\_FILE\_SET\_SIZE}}

{\tt MPI\_FILE\_SET\_SIZE} is a collective routine used to resize a file.  It
is important to remember that a MPI-IO routine being collective does not imply
that the routine synchronizes the calling processes in any way (unless this is
specified explicitly).

As of 1.2.4, ROMIO implements {\tt MPI\_FILE\_SET\_SIZE} by calling {\tt
ftruncate} from all processes.  Since different processes may call the
function at different times, it means that unless external synchronization is
used, a resize operation mixed in with writes or reads could have unexpected
results.

In short, if synchronization after a set size is needed, the user should add a
barrier or similar operation to ensure the set size has completed.


%
% INSTALLATION INSTRUCTIONS
%
\section{Installation Instructions}
Since ROMIO is included in MPICH, LAM, HP MPI, SGI MPI, and NEC MPI, you don't
need to install it separately if you are using any of these MPI
implementations.  If you are using some other MPI, you
can configure and build ROMIO as follows: 

Untar the tar file as
\begin{verbatim}
    gunzip -c romio.tar.gz | tar xvf -
\end{verbatim}
{\noindent or}
\begin{verbatim}
    zcat romio.tar.Z | tar xvf -
\end{verbatim}

{\noindent then}

\begin{verbatim}
    cd romio
    ./configure
    make
\end{verbatim}

Some example programs and a Makefile are provided in the {\tt romio/test}
directory.  Run the examples as you would run any MPI program. Each
program takes the filename as a command-line argument ``{\tt -fname
filename}''.

The {\tt configure} script by default configures ROMIO for the file
systems most likely 
to be used on the given machine. If you wish, you can explicitly specify the file
systems by using the ``{\tt -file\_system}'' option to configure. Multiple file
systems can be specified by using `+' as a separator, e.g., \\
\hspace*{.4in} {\tt ./configure -file\_system=xfs+nfs} \\
For the entire list of options to configure, do\\ 
\hspace*{.4in} {\tt ./configure -h | more} \\
After building a specific version, you can install it in a
particular directory with \\
\hspace*{.4in} {\tt make install PREFIX=/usr/local/romio    (or whatever directory you like)} \\
or just\\
\hspace*{.4in} {\tt make install          (if you used -prefix at configure time)}

If you intend to leave ROMIO where you built it, you should {\it not}
install it; {\tt make install} is used only to move the necessary
parts of a built ROMIO to another location. The installed copy will
have the include files, libraries, man pages, and a few other odds and
ends, but not the whole source tree.  It will have a {\tt test}
directory for testing the installation and a location-independent
Makefile built during installation, which users can copy and modify to
compile and link against the installed copy.

To rebuild ROMIO with a different set of configure options, do\\
\hspace*{.4in} {\tt make distclean}\\
to clean everything, including the Makefiles created by {\tt
configure}.  Then run {\tt configure} again with the new options,
followed by {\tt make}.

\subsection{Configuring for Linux and Large Files }

32-bit systems running linux kernel version 2.4.0 or newer and glibc
version 2.2.0 or newer can support files greater than 2 GBytes in size.
This support is currently automaticly detected and enabled.  We document the
manual steps should the automatic detection not work for some reason.

The two macros {\tt\_FILE\_OFFSET\_BITS=64} and
{\tt\_LARGEFILE64\_SOURCE} tell gnu libc it's ok to support large files
on 32 bit platforms.  The former changes the size of {\tt off\_t} (no
need to change source.  might affect interoperability with libraries
compiled with a different size of {\tt off\_t}).   The latter exposes
the gnu libc functions open64(), write64(), read64(), etc.   ROMIO does
not make use of the 64 bit system calls directly at this time, but we
add this flag for good measure.  

If your linux system is relatively new, there is an excellent chance it
is running kernel 2.4.0 or newer and glibc-2.2.0 or newer.  Add the
string
\begin{verbatim}
"-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE"
\end{verbatim}
to your CFLAGS environment variable before runnint {\tt./configure}

%
% TESTING ROMIO
%
\section{Testing ROMIO}
To test if the installation works, do\\
\hspace*{.4in} {\tt make testing}\\
in the {\tt romio/test} directory. This calls a script that runs the test
programs and compares the results with what they should be. By
default, {\tt make testing} causes the test programs to create files in
the current directory and use whatever file system that corresponds
to. To test with other file systems, you need to specify a filename in
a directory corresponding to that file system as follows:\\
\hspace*{.4in} {\tt make testing TESTARGS="-fname=/foo/piofs/test"}


%
% COMPILING AND RUNNING MPI-IO PROGRAMS
%
\section{Compiling and Running MPI-IO Programs}
If ROMIO is not already included in the MPI implementation, you need
to include the file {\tt mpio.h} for C or {\tt mpiof.h} for Fortran in