Optimizing Programs on the IA-64 Linux Cluster
Timing
There are gettimeofday (c), getrusage (c), cputime (fortran), times (os) which are routines and commands that you can measure the run time of whole or segments of the code. In addition, mpi_wtime may be used to time portions of MPI code.
Profiling
gnu gprof is available for a quick information on functions and routines. The procedure involves compiling with "-p" for the Intel compilers, then executing the binary to generate a profile output file "gmon.out". Run the gprof utility for the binary and profile output to view the gmon.out output Please see the gprof man pages for complete details.
Intel compiler optimization and other flags:
| Option | Description |
| -O0 | Disables all optimization |
| -O1 | reordering of functions, inlining with _inline keyword |
| -O2 | -O1 + inline expansion, constant propagation, predication, speculation, software pipelining |
| -O3 | -O2 + loop unrolling, memory hierarchy( data pre-fetching, loop and register blocking, linear loop transformations, scalar replacements) |
| -Op | improves the predictability of applications by enabling only those optimizations that preserve numerical accuracy and floating-point state of the machine |
| -Oi | enables inline expansion of intrinsic or standard library functions |
| -Ob0 | disables inline expansion |
| -Ow | tells the compiler that no aliasing occurs within function bodies but might occur across function calls |
| -mp | Maintain floating point precision (disables some optimizations) |
Other Profiling and Performance Tools
VAMPIR/VAMPIRTRACE
The Vampirtrace profiling tool for MPI applications produces tracefiles that can be analyzed with the Vampir performance analysis tool.
VAMPIR records all calls to the MPI library and all transmitted messages, and allows arbitrary user defined events to be recorded. Instrumentation can be switched on or off at runtime, and a filtering mechanism helps to limit the amount of the generated trace data. Vampirtrace is an add-on for existing MPI implementations; using it merely requires relinking the application with the Vampirtrace profiling library (see below and the VAMPIRTRACE and VAMPIR documentation in the References section). This will enable the tracing of all calls to MPI routines, as well as all explicit message-passing. On some platforms, calls to user-level subroutines and functions will also be recorded.
Tracing MPI Applications
Using Vampirtrace for MPI is straightforward: just relink your MPI application with the appropriate profiling library and execute it (see example below). This will generate a tracefile suitable for use with Vampir, including records of all calls to MPI routines as well as all point-to-point and collective communication operations performed by the application. The VAMPIR program can then be used to view the trace file(s).
If you wish to get more detailed information about your application, you can instrument the application source code with calls to the Vampirtrace API and recompile. This will allow arbitrary user-defined events to be traced; in practice, it is often very useful to record your applications entry and exit to/from subroutines or regions within large subroutines. For detailed information on how to do this, please consult the VAMPIR/VAMIRTRACE User Guides (see References).
Compiling MPI Programs with Vampirtrace on SDSC TeraGrid IA-64
To use VAMPIR/VAMPIRTRACE on SDSC TeraGrid IA-64, users must define an environment variable in order to use VAMPIR/VAMPIRTRACE. The following example is for tcshell:
setenv PAL_LICENSEFILE /usr/local/apps/vampir/etc/license.dat
setenv PAL_ROOT /usr/local/apps/vampir
In addition, the VT_ROOT directory must be defined:
setenv VT_ROOT /usr/local/apps/vampirtrace
Linking MPI Programs with Vampirtrace
The Vampirtrace library libVT.a contains entry points for all MPI routines. They must be linked against your application object files before your system’s MPI library, which can be done:
mpicc ctest.c -L$(VT ROOT)/lib -lVT -lnsl -lm -lpthread -o ctest
Output of VAMPIR traces can be viewed with VAMPIR:
/usr/local/apps/vampir/bin/vampir
VAMPIR/VAMPIRTRACE References
VAMPIR User Guide [ PDF ]
VAMPIRTRACE User Guide [ PDF ]
