source-guide.tex

fortran并行计算包

... what else is in there?

\subsection{Calling ADIO Functions}

Throughout the code you will see calls to functions such as ADIO\_ReadContig.
There is no such function - this is actually a macro defined in
adio/include/adioi.h that calls the particular function out of the correct
ADIOI\_Fns\_struct for the file being accessed.  This is done for convenience.

Exceptions!!!  ADIO\_Open, ADIO\_Close...

\section{ROMIO Implementation Details}

The ROMIO Implementation relies on some basic concepts in order to operate and
to optimize I/O access.  In this section we will discuss these concepts and
how they are implemented within ROMIO.  Before we do that though, we will
discuss the core data structure of ROMIO, the ADIO\_File structure.

\subsection{ADIO\_File}

... discussion ...

\subsection{I/O Aggregation and Aggregators}

When performing collective I/O operations, it is often to our advantage to
combine operations or eliminate redundant operations altogether.  We call this
combining process "aggregation", and processes that perform these combined
operations aggregators.

Aggregators are defined at the time the file is opened.  A collection of MPI
hints can be used to tune what processes become aggregators for a given file
(see ROMIO User's Guide).  The aggregators will then interact with the file
system during collective operations.

Note that it is possible to implement a system where ALL I/O operations pass
exclusively through aggregators, including independent I/O operations from
non-aggregators.  However, this would require a guarantee of progress from the
aggregators that for portability would mean adding a thread to manage I/O.  We
have chosen not to pursue this path at this time, so independent operations
continue to be serviced by the process making the call.

... how implemented ...

Rank 0 in the communicator opening a file \emph{always} processes the
cb\_config\_list hint using ADIOI\_cb\_config\_list\_parse.  A previous call to
ADIOI\_cb\_gather\_name\_array had collected the processor names from all hosts
into an array that is cached on the communicator (so we don't have to gather
it more than once).  This creates an ordered array of ranks (relative to the
communicator used to open the file) that will be aggregators.  This array is
distributed to all processes using ADIOI\_cb\_bcast\_rank\_map.  Aggregators are
referenced by their rank in the communicator used to open the file.  These
ranks are stored in fd->hints->ranklist[].

Note that this could be a big list for very large runs.  If we were to
restrict aggregators to a rank order subset, we could use a bitfield instead.

If the user specified hints and met conditions for deferred open, then a
separate communicator is also set up (fd->agg\_comm) that contains all the
aggregators, in order of their original ranks (not their order in the rank
list).  Otherwise this communicator is set to MPI\_COMM\_NULL, and in any case
it is set to this for non-aggregators.  This communicator is currently only
used at ADIO\_Close (adio/common/ad\_close.c), but could be useful in two-phase
I/O as well (discussed later).


\subsection{Deferred Open}

We do not always want all processes to attempt to actually open a file when
MPI\_File\_open is called.  We might want to avoid this open because in fact
some processes (non-aggregators) cannot access the file at all and would get
an error, or we might want to avoid this open to avoid a storm of system calls
hitting the file system all at once.  In either case, ROMIO implements a
"deferred open" mode that allows some processes to avoid opening the file
until such time as they perform an independent I/O operation on the file (see
ROMIO User's Guide).

Deferred open has a broad impact on the ROMIO implementation, because with its
addition there are now many places where we must first check to see if we have
called the file system specific ADIO Open call before performing I/O.  This
impact is limited to the MPI-IO layer by semantically guaranteeing the FS ADIO
Open call has been made by the process prior to calling a read or write
function.

... how implemented ...

\subsection{Two-Phase I/O}

Two-Phase I/O is a technique for increasing the efficiency of I/O operations
by reordering data between processes, either before writes, or after reads.

ROMIO implements two-phase I/O as part of the generic implementations of
ADIO\_WriteStridedColl and ADIO\_ReadStridedColl.  These implementations in turn
rely heavily on the aggregation code to determine what processes will actually
perform I/O on behalf of the application as a whole.



\subsection{Data Sieving}

Data sieving is a single-process technique for reducing the number of I/O
operations used to service a MPI read or write operation by accessing a
contiguous region of the file that contains more than one desired region at
once.  Because often I/O operations require data movement across the network,
this is usually a more efficient way to access data.

Data sieving is implemented in the common strided I/O routines
(adio/common/ad\_write\_str.c and adio/common/ad\_read\_str.c).  These functions
use the contig read and write routines to perform actual I/O.  In the case of
a write operation, a read/modify/write sequence is used.  In that case, as
well as in the atomic mode case, locking is required on the region.  Some of
the ADIO implementations do not currently support locking, and in those cases
it would be erroneous to use the generic strided I/O routines.

\subsection{Shared File Pointers}

Because no file systems supported by ROMIO currently support a shared file
pointer mode, ROMIO must implement shared file pointers under the covers on
its own.

Currently ROMIO implements shared file pointers by storing the file pointer
value in a separate file...

Note that the ROMIO team has devised a portable method for implementing shared
file pointers using only MPI-1 and MPI-2 functions.  However, this method has
not yet been implemented in ROMIO.

file name is selected at end of mpi-io/open.c.

\subsection{Error Handling}

\subsection{MPI and MPIO Requests}

\section*{Appendix A: ADIO Functions and Semantics}

ADIOI\_Open(ADIO\_File fd, int *error\_code)

Open is used in a strange way in ROMIO, as described previously.

The Open function is used to perform whatever operations are necessary prior
to actually accessing a file using read or write.  The file name for the file
is stored in fd->filename prior to Open being called.

Note that when deferred open is in effect, all processes may not immediately
call Open at MPI\_File\_open time, but instead call open if they perform
independent I/O.  This can result in somewhat unusual error returns to
processes (e.g. learning that a file is not accessible at write time).

ADIOI\_ReadContig(ADIO\_File fd, void *buf, int count, MPI\_Datatype datatype,
int file\_ptr\_type, ADIO\_Offset offset, ADIO\_Status *status, int *error\_code)

ReadContig is used to read a contiguous region from a file into a contiguous
buffer.  The datatype (which refers to the buffer) can be assumed to be
contiguous.  The offset is in bytes and is an absolute offset if
ADIO\_EXPLICIT\_OFFSET was passed as the file\_ptr\_type or relative to the
current individual file pointer if ADIO\_INDIVIDUAL was passed as
file\_ptr\_type.  Open has been called by this process prior to the call to
ReadContig.  There is no guarantee that any other processes will call this
function at the same time.

ADIOI\_WriteContig(ADIO\_File fd, void *buf, int count, MPI\_Datatype datatype,
int file\_ptr\_type, ADIO\_Offset offset, ADIO\_Status *status, int *error\_code)

WriteContig is used to write a contiguous region to a file from a contiguous
buffer.  The datatype (which refers to the buffer) can be assumed to be
contiguous.  The offset is in bytes and is an absolute offset if
ADIO\_EXPLICIT\_OFFSET was passed as the file\_ptr\_type or relative to the
current individual file pointer if ADIO\_INDIVIDUAL was passed as
file\_ptr\_type.  Open has been called by this process prior to the call to
WriteContig.  There is no guarantee that any other processes will call this
function at the same time.

ADIOI\_ReadStridedColl

ADIOI\_WriteStridedColl

ADIOI\_SeekIndividual

ADIOI\_Fcntl

ADIOI\_SetInfo

ADIOI\_ReadStrided

ADIOI\_WriteStrided

ADIOI\_Close(ADIO\_File fd, int *error\_code)

Close is responsible for releasing any resources associated with an open file.
It is called on all processes that called the corresponding ADIOI Open, which
might not be all the processes that opened the file (due to deferred open).
Thus it is not safe to perform collective communication among all processes in
the communicator during Close, although collective communication between
aggregators would be safe (if desired).

For performance reasons ROMIO does not guarantee that all file data is written
to "storage" at MPI\_File\_close, instead only performing synchronization
operations at MPI\_File\_sync time.  As a result, our Close implementations do
not typically call a sync.  However, any locally cached data, if any, should
be passed on to the underlying storage system at this time.

Note that ADIOI\_GEN\_Close is implemented in adio/common/adi\_close.c;
ad\_close.c implements ADIO\_Close, which is called by all processes that opened
the file.

ADIOI\_IreadContig

ADIOI\_IwriteContig

ADIOI\_ReadDone

ADIOI\_WriteDone

ADIOI\_ReadComplete

ADIOI\_WriteComplete

ADIOI\_IreadStrided

ADIOI\_IwriteStrided

ADIOI\_Flush

ADIOI\_Resize(ADIO\_File fd, ADIO\_Offset size, int *error\_code)

Resize is called collectively by all processes that opened the file referenced
by fd.  It is not required that the Resize implementation block until all
processes have completed resize operations, but each process should be able to
see the correct size with a corresponding MPI\_File\_get\_size operation (an
independent operation that results in an ADIO Fcntl to obtain the file size).

ADIOI\_Delete(char *filename, int *error\_code)

Delete is called independently, and because only a filename is passed, there
is no opportunity to coordinate deletion if an application were to choose to
have all processes call MPI\_File\_delete.  That's not likely to be an issue
though.

\section*{Appendix B: Status of ADIO Implementations}

... who wrote what, status, etc.

Appendix C: Adding a New ADIO Implementation

References

\end{document}