Skip to content

Compiling on the IA-64 Linux Cluster

Serial and MPI Programs

If you have an existing serial or MPI-based parallel application program already running on a distributed-memory platform:

  • Copy your application file(s) to the SDSC TeraGrid local disk space - either to your $HOME directory or to the $WORK area associated with your user account - /work/username, where "username" is your TeraGrid user name. If this directory does not exist, you will have to create it yourself. Please note that the /work area is purged regularly and is not backed up. Files for long-term storage should be stored either in your $HOME directory or on HPSS.

  • MPI source code should be recompiled for the TeraGrid system with the following exemplary compiler commands:
    • mpicc [options] file.c	(C and C++)
mpif90 [options] file.f	(fixed form Fortran source code)
  • Serial source code should be recompiled for the TeraGrid system with the following exemplary compiler commands:
    • icc [options] file.c        (C and C++)
ifort [options] file.f      (fixed form Fortran source code)
ifort [options] file.f90    (free format Fortran source code)

The following compilers are available on the IA-64 cluster:

Compiler Commands Description
Intel icc (C, C++), ifort (Fortran 77/90) Default compiler on SDSC's TeraGrid system. icc (or ifort) -help for usage information
Gnu gcc (C), g++ (C++), g77(Fortran 77) Code's performance will not be as good compared to the Intel compilers.
mpich-gm mpicc (MPI C, C++), mpif90 (MPI Fortran 77/90) Default MPI compiler. Uses Myrinet network for communication
mpich-vmi mpicc (MPI C, C++), mpif90 (MPI Fortran 77/90) Installed in the /usr/local/apps/mpich-vmi-intel/ directory. Use mpich-vmi for single-site or multiple-site runs using the vmi interface (http://vmi.ncsa.uiuc.edu/).

Useful Intel Fortran Compiler/Linker Options

Compatibility Options

  • -fpp[n]
    runs the Fortran preprocessor on source files prior to compilation.
  • -Dname[=value]
    specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).
  • -w90, -w95
    suppress messages about use of non-standard Fortran
  • -i8
    set default KIND of integer variables is 8
  • -integer_size size
    specifies the default size of integer and logical variables (size: 16, 32, 64)
  • -r8
    set default size of REAL to 8 bytes
  • -real_size size
    specify the size of REAL and COMPLEX declarations, constants, functions, and intrinsics (size: 32, 64, 128)
  • -save
    save all variables (static allocation)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

  • -g
    produce symbolic debug information in object file (implies -O0)
  • -CB
    array bounds checking
  • -fpen
    specifies behavior on floating point exceptions (n = 0, 1 or 3)
  • -zero
    implicitly initialize all data to zero
  • -u
    equivalent to having "IMPLICIT NONE" throughout the code
  • -traceback
    (Fortran 8.X compiler) allow the display of source file traceback information at runtime when a severe error occurs.

Optimization

  • -O2
    enable optimizations (DEFAULT)
  • -O1
    optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit
  • -O3
    enable -O2 plus more aggressive optimizations that may not improve performance for all programs
  • -O0
    disable optimizations
  • -O
    same as -O2
  • -ipo
    enable multi-file IP optimizations (between files)
  • -ftz
    flush denormal values to zero when the application is in the gradual underflow mode. By default, the compiler lets results to gradually underflow. Flushing the denormal values to zero may improve performance of your application. Use this option if the denormal values are not critical to application behavior.
  • -mp
    maintain floating point precision (disables some optimizations)
  • -mp1
    improve floating-point precision (speed impact is less than -mp)
  • -tpp2
    Optimize for Intel Itanium 2 processors. (default)

Back to Top

Numerical Libraries

Intel has developed Math Kernel Library (MKL) which contains many linear algebra, FFT and other useful numerical routines.

  • Basic linear algebra subprograms (BLAS) with additional sparse routines.
  • Fast Fourier Transforms (FFT) in 1 and 2 dimensions, complex and real.
  • The linear algebra package, LAPACK
  • A C interface to BLAS
  • Vector Math Library (VML)
  • Vector Statistical Library (VSL)
  • Multi-dimensional Discrete Fourier Transforms (DFTs)

Documentation is available in HTML and PDF formats in ${MKL_HOME}/doc.

The libraries for Itanium processors are installed under ${MKL_HOME}/lib/64.

To link the MKL libraries dynamically:

For blas only: -L${MKL_HOME}/lib/64 -lmkl -lguide -lpthread
For lapack and blas: -L${MKL_HOME}/lib/64 -lmkl_lapack64 -lmkl -lguide -lpthread

To link the MKL libraries statically:

For blas only: -L${MKL_HOME}/lib/64 -lmkl_ipf -lguide -lpthread
For lapack and blas: -L${MKL_HOME}/lib/64 -lmkl_lapack -lmkl_ipf -lguide -lpthread

Notes:

  • Some MKL routines can run in parallel (using OpenMP) if the environment variable OMP_NUM_THREADS is set to the number of threads. OMP_NUM_THREADS is not set by default.
  • To force Intel MKL to serial mode, environment variable MKL_SERIAL should be set to YES. It works regardless of OMP_NUM_THREADS value. MKL_SERIAL is not set by default.
  • If you get unresolved symbols when calling MKL routines from C/C++ programs with the non-default Intel 7.1 compilers you will also need to link with:
    -lPEPCF90 -lCEPCF90 -lF90 -lintrins.

Users are encouraged to use these routines where applicable instead of their own because they generally produce faster programs and have been tested for accuracy and correctness. The following Math library is available on SDSC's IA-64 cluster:

Intel Math Kernel Library This library contains Blas, Lapack and 1,2D FFT routines. It is installed in /usr/local/apps/intel/mkl/lib/64 directory. The library can be linked by "-L/usr/local/apps/intel/mkl/lib/64 -lmkl_lapack -lmkl_ipf -lguide -lpthread" for any fortran program.

All third-party applications can be found in /usr/local/apps, or view the complete description of TeraGid software packages installed on all TG systems.

Back to Top


Did You Get
What You
Wanted?
Yes No
Comments