node110.html

Design and building parallel program

user-specified instrumentation into programs.  In addition, Pablo
calls can be incorporated into communication libraries or compilers to
generate trace files automatically.  When logging an event, Pablo can
be requested to invoke a user-defined event handler that may perform
on-the-fly data reduction.  For example, a user-defined handler can
compute communication statistics rather than logging every message or
can combine procedure entry and exit events to determine procedure
execution times.  This very general mechanism provides great
flexibility.  A disadvantage is that the overhead associated with
logging an event is greater than in other, less general systems.
<P>
A novel feature of Pablo is its support for automatic <em>
<A NAME=14474>&#160;</A>
throttling
 </em> of event data generation.  The user can specify a
threshold data rate for each type of event. If events are generated at
a greater rate, event recording is disabled or replaced by periodic
logging of event counts, thereby enabling a variety of
events to be logged without the danger that one will unexpectedly swamp the system.
<P>
Pablo provides a variety of data reduction and display modules that
can be plugged together to form specialized data analysis and
visualization networks.  For example, most displays provided by
Paragraph can be constructed using Pablo modules.  This feature is
illustrated in
<A HREF="#pablo1">Plate 13</A>,

<P>
which shows a variety of Paragraph-like displays and the Pablo network
used to generate them.
<A NAME=14479>&#160;</A>
As noted earlier, Pablo uses its own SDDF.
<P>
<P><HR>
<A NAME=pablo1 HREF="PICL.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/PICL.gif"> <img
ALIGN=MIDDLE src="PICL_small.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/PICL_small.gif"></A>
<P>
(GIF <A HREF="PICL.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/PICL.gif">67844</A> bytes; RGB <A
HREF="javascript:if(confirm('http://www.dit.hcmut.edu.vn/books/system/par_anl/PICL.rgb  \n\nThis file was not retrieved by Teleport Pro, because the server reports that this file cannot be found.  \n\nDo you want to open it from the server?'))window.location='http://www.dit.hcmut.edu.vn/books/system/par_anl/PICL.rgb'" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/PICL.rgb">635350</A> bytes.)
Plate 13: Pablo display of performance data collected from a
numerical solver.  Image courtesy of D.~Reed.
<P><HR>

<P>
An interesting feature of the Pablo environment is its support for
novel ``display'' technologies, such as sound and immersive virtual
environments.  Sound appears to be particularly effective for alerting
the
<A NAME=14482>&#160;</A>
user to unusual events, while immersive virtual environments can be
used to display higher-dimensional data, as illustrated in
<A HREF="#pablo2">Plate 14</A>.

<P>
In this plate, each cube represents a different performance metric,
and the spheres within the cubes represent processors moving within a
three-dimensional metric space.  While both approaches are still
experimental at present, they are suggestive of future directions.
<P>
<P><HR>
<A NAME=pablo2 HREF="msgs0.htm#27" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/tppmsgs/msgs0.htm#27"> <img
ALIGN=MIDDLE src="cubescape_small.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/cubescape_small.gif"></A>
<P>
(GIF <A HREF="msgs0.htm#27" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/tppmsgs/msgs0.htm#27">184245</A> bytes; RGB <A
HREF="javascript:if(confirm('http://www.dit.hcmut.edu.vn/books/system/par_anl/cubescape.rgb  \n\nThis file was not retrieved by Teleport Pro, because the server reports that this file cannot be found.  \n\nDo you want to open it from the server?'))window.location='http://www.dit.hcmut.edu.vn/books/system/par_anl/cubescape.rgb'" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/cubescape.rgb">622138</A> bytes.)
Plate 14: Pablo virtual
reality display of performance data.  Image courtesy of D.  Reed.
<P><HR>

<H2><A NAME=SECTION03644000000000000000>9.4.4 Gauge</A></H2>
<P>
<A NAME=14490>&#160;</A>
The Gauge performance tool developed at the California Institute of
Technology is distinguished by its focus on profiles and counters
<A NAME=14491>&#160;</A>
rather than execution traces.  The Gauge display tool allows the user
to examine a multidimensional performance data set in a variety of
ways, collapsing along different dimensions and computing various
higher-order statistics.  For example, a three-dimensional view of an
execution profile uses color to indicate execution time per processor
and per routine; corresponding two-dimensional displays provide
histograms for time per routine summed over all processors or for time
per processor for all routines.  Idle time is also measured on a
per-processor basis and associated with program components by
determining which task is enabled by arrival of a message.  Some of
these displays are illustrated in
<A HREF="#gauge">Plate 7</A>.

<P>
<H2><A NAME=SECTION03645000000000000000>9.4.5 ParAide</A></H2>
<P>
<A NAME=14497>&#160;</A>
The ParAide system developed by Intel's Supercomputer Systems Division
<A NAME=14498>&#160;</A>
is specialized for the Paragon parallel computer.  It incorporates a
variety of different tools.  Modified versions of the standard Unix
<tt> prof</tt> and <tt> gprof</tt> tools provide profiling on a per-node
basis.  An enhanced version of Paragraph provides various data
reduction and display mechanisms.  The System Performance
Visualization system uses displays specialized for the Paragon's
two-dimensional mesh architecture to show data collected by hardware
performance monitors.  These provide detailed low-level information
regarding the utilization of the processor, communication network, and
memory bus.  This fine level of detail is made possible by hardware
and operating system support in the Paragon computer.
<P>
<H2><A NAME=SECTION03646000000000000000>9.4.6 IBM's Parallel Environment</A></H2>
<P>
<A NAME=14502>&#160;</A>
The IBM AIX Parallel Environment is specialized for IBM computers, in
<A NAME=14503>&#160;</A>
particular the SP multicomputer.  It incorporates a variety of
different tools.  A variant of the standard Unix <tt> prof</tt> and <tt>
gprof</tt> commands can be used to generate and process multiple profile
files, one per task involved in a computation.  The Visualization Tool
(VT) can be used to display a variety of different trace data.  Three
types of trace data are supported, as follows:
<OL><LI>
A communication record represents a low-level communication event,
such as a send, receive, or call to a global (collective)
communication routine.
<P>
<LI>
A system statistics record samples low-level statistics such as CPU
utilization, disk traffic, and virtual memory statistics.  The
sampling rate can be selected by the programmer.
<P>
<LI>
An application marker record is generated manually by a programmer and
may be used to delineate distinct stages in a program's execution.
</OL>
Communication trace records and system statistics are generated
automatically, although the programmer can turn them on and off and
can control their frequency.
<P>
VT displays are similar to those provided by Paragraph in many respects, but
they give the programmer greater flexibility in how data are displayed
and can deal with a wider range of data.
<A HREF="#ibmvt">Plate 15</A>

<P>
shows one display, in this case a space-time diagram.
<P>
<H2><A NAME=SECTION03647000000000000000>9.4.7 AIMS</A></H2>
<P>
The Automated Instrumentation and Monitoring System (AIMS) developed
<A NAME=14513>&#160;</A>
at the NASA Ames Research Center provides both instrumentation tools
and a variety of trace visualization mechanisms for message-passing
programs.  Users can either specify trace events manually or request
AIMS to log communication events and procedure calls automatically.
The resulting traces can be visualized by using the AIMS View Kernel (VK),
Pablo, or Paragraph.  A strength of AIMS is its tight
integration with a source code browser that allows the user both to
mark code blocks for tracing and to relate communication events with
source code.  For example, the user can click on a line representing a
communication in a space-time diagram to identify the corresponding
communication operation in the source code.  AIMS also provides
statistical analysis functions that can be used to determine average
resource utilization and message latencies.
<P>
<H2><A NAME=SECTION03648000000000000000>9.4.8 Custom Tools</A></H2>
<P>
We conclude this section by noting that while general-purpose tools
<A NAME=14515>&#160;</A>
have the advantage of being easy to use, custom performance tools can
also be valuable, particularly in
understanding the performance of a complex parallel program.
Extensible tools
such as Pablo can be useful in this regard.  So can text manipulation
systems such as awk and PERL, statistical packages such as Mathematica
and Matlab, and general-purpose graphics packages such as AVS.
<P>
As an example of this approach,
<A HREF="#ccm2">Plate 5</A>

<P>
shows an image generated by a tool developed specifically to help
understand load imbalances in a parallel climate model.  This tool
collects timing data by using interval timers and counters inserted
manually into the parallel climate model.  The data are postprocessed
to compensate for timer overhead and are then displayed by using a
general-purpose graphics package.  Sequences of such images provide
insights into how computational load varies over time, and have
motivated the design of load-balancing algorithms.
<P>

<BR> <HR><a href="msgs0.htm#2" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/tppmsgs/msgs0.htm#2"><img ALIGN=MIDDLE src="asm_color_tiny.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/asm_color_tiny.gif" alt="[DBPP]"></a>    <A NAME=tex2html3288 HREF="node109.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node109.html"><IMG ALIGN=MIDDLE ALT="previous" SRC="previous_motif.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/previous_motif.gif"></A> <A NAME=tex2html3296 HREF="node111.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node111.html"><IMG ALIGN=MIDDLE ALT="next" SRC="next_motif.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/next_motif.gif"></A> <A NAME=tex2html3294 HREF="node106.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node106.html"><IMG ALIGN=MIDDLE ALT="up" SRC="up_motif.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/up_motif.gif"></A> <A NAME=tex2html3298 HREF="node1.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node1.html"><IMG ALIGN=MIDDLE ALT="contents" SRC="contents_motif.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/contents_motif.gif"></A> <A NAME=tex2html3299 HREF="node133.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node133.html"><IMG ALIGN=MIDDLE ALT="index" SRC="index_motif.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/index_motif.gif"></A> <a href="msgs0.htm#3" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/tppmsgs/msgs0.htm#3"><img ALIGN=MIDDLE src="search_motif.gif" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/search_motif.gif" alt="[Search]"></a>   <BR>
<B> Next:</B> <A NAME=tex2html3297 HREF="node111.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node111.html">9.5 Summary</A>
<B>Up:</B> <A NAME=tex2html3295 HREF="node106.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node106.html">9 Performance Tools</A>
<B> Previous:</B> <A NAME=tex2html3289 HREF="node109.html" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/node109.html">9.3 Data Transformation and Visualization</A>
<BR><HR><P>
<P><ADDRESS>
<I>&#169 Copyright 1995 by <A href="msgs0.htm#6" tppabs="http://www.dit.hcmut.edu.vn/books/system/par_anl/tppmsgs/msgs0.htm#6">Ian Foster</a></I>
</ADDRESS>
</BODY>