Skip to content

Trestles User Guide: Compiling

Trestles provides the Portland Group (PGI), Intel, and GNU compilers along with multiple MPI implementations (MVAPICH2 and OpenMPI). Most applications will achieve the best performance on Trestles using the PGI compilers and MVAPICH2, and the majority of libraries installed on Trestles have been built using this combination. Although other compilers and MPI implementations are available, we suggest using these only for compatibility purposes.

Using the PGI Compilers (Default/Recommended)

The PGI compilers and the MVAPICH2 MPI implementation will be loaded by default. If you have modified your environment, you can reload them by executing the following commands at the Linux prompt or by placing them in your startup file (~/.cshrc or ~/.bashrc):

module purge 
module load gnubase pgi mvapich2_ib
Serial MPI OpenMP
Fortran pgf90 mpif90 pgf90 -mp
C pgcc mpicc pgcc -mp
C++ pgCC mpicxx pgCC -mp

 

Using the Intel Compilers

The Intel compilers can be loaded by executing the following commands at the Linux prompt or placing them in your startup file (~/.cshrc or ~/.bashrc):

module purge
module load gnubase intel mvapich2_ib

Intel MKL libraries are available as part of the "intel" modules on Trestles. Once this module is loaded, the environment variable MKL_ROOT points to the location of the mkl libraries. The MKL link advisor can be used to ascertain the link line (change the MKL_ROOT aspect appropriately).

For example to compile a C program statically linking 64 bit scalapack libraries on Trestles:

module unload mvapich2_ib
module unload pgi
module load intel
module load mvapich2_ib
mpicc -o pdpttr.exe pdpttr.c \
    -I$MKL_ROOT/include ${MKL_ROOT}/lib/intel64/libmkl_scalapack_lp64.a \
    -Wl,--start-group ${MKL_ROOT}/lib/intel64/libmkl_intel_lp64.a \
    ${MKL_ROOT}/lib/intel64/libmkl_core.a ${MKL_ROOT}/lib/intel64/libmkl_sequential.a \
    -Wl,--end-group ${MKL_ROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm
Serial MPI OpenMP
Fortran ifort mpif90 ifort -openmp
C icc mpicc icc -openmp
C++ icpc mpicxx icpc -openmp

 

Using the GNU Compilers

The GNU compilers can be loaded by executing the following commands at the Linux prompt or placing them in your startup file (~/.cshrc or ~/.bashrc):

module purge
module load gnubase gnu openmpi_ib
Serial MPI OpenMP
Fortran gfortran mpif90 gfortran -fopenmp
C gcc mpicc gcc -fopenmp
C++ g++ mpicxx g++ -fopenmp

Notes and hints

  • The mpif90, mpicc, and mpicxx commands are actually wrappers that call the appropriate serial compilers and load the correct MPI libraries. While the same names are used for the Intel, PGI and GNU compilers, keep in mind that these are completely independent scripts.
  • If you use the Intel or GNU compilers or switch between compilers for different applications, make sure that you load the appropriate modules before running your executables.
  • When building OpenMP applications and moving between different compilers, one of the most common errors is to use the wrong flag to enable handling of OpenMP directives. Note that PGI, Intel, and GNU compilers use the -mp, -openmp, and -fopenmp flags, respectively.
  • Explicitly set the optimization level in your makefiles or compilation scripts. Most well written codes can safely use the highest optimization level (-O3), but many compilers set lower default levels (e.g. GNU compilers use the default -O0, which turns off all optimizations).

Turn off debugging, profiling, and bounds checking when building executables intended for production runs as these can seriously impact performance. These options are all disabled by default. The flag used for bounds checking is compiler dependent, but the debugging (-g) and profiling (-pg) flags tend to be the same for all major compilers.


Did You Get
What You
Wanted?
Yes No
Comments