Migrating Code to Other NSF Resources
Introduction
The Track 2 machines at TACC and NICS are two of the next generation computers funded by NSF. These new resources are designed to support a wide range of science and engineering computing applications and to provide significantly larger capabilities than the current NSF machines, including DataStar and Blue Gene. SDSC staff will be available to aid current DataStar and Blue Gene users with the migration to these new machines, along with other NSF resources. Revisit this page for updates and additional details as they become available. Please note that DataStar and Blue Gene are scheduled to remain in operation at least until the end of September, 2008.
- Porting Recommendations: Suggestions based on code requirements and the available system details for the track 2 machines
- Common Porting Issues: Issues encountered during porting with some details specific to DataStar and Blue Gene. Please email us at consult@sdsc.edu if you have further questions and if you need help with porting your code.
Porting Recommendations
The suggestions below represent some possible migration scenarios based on SDSC experience with capabilities of and codes running on DataStar and BlueGene. For a full listing of other NSF resources, including recommended uses, capabilities, and specifications, please see the TeraGrid Resource Catalog.
The primary factor to consider is the memory per node that is available on the system. It is always easier to migrate to a platform with the same (or larger) memory per node than to a system with less. Another consideration is the scalability required by the applications. Applications that require extreme scalability (exemplified by BG/L applications or the largest DataStar applications) may need to migrate to systems with superior interconnects, such as the Cray XT systems at NICS/ORNL.
On the basis of the above considerations we have the following recommendations:
- BlueGene/L users: Existing BG/L applications have small per-node memory requirements and potentially high scalability requirements. Both of these constraints are most easily satisfied on Cray XT systems at NICS/ORNL (Kraken).
- DataStar P655 users: Existing P655 and P655+ applications mst run in <=12 GB per node (P655) or <27 GB per node (P655+) using up to 8 processors per node. Large scale applications (up to 2048 processors) may require good interconnect performance. These constraints are most easily met by the SUN Constellation system at TACC (Ranger).
- DataStar P690 users: DataStar P690 users may fall into one of 3 categories, or potentially
more than one of these simultaneously:
- OpenMP shared memory programs requiring >32 GB and/or <=32 cpus. The only realistic NSF allocated alternative platform is the NCSA SGI Altix platform (Cobalt), which can support up to 1.4TB shared memory applications.
- MPI applications requiring >4GB per MPI task. These applications may run on the TACC SUN Constellation system (Ranger) by idling one or more cpus per node in order to meet the per-task memory required.
- Large shared-memory serial tasks (requiring >32 GB/task). The only realistic NSF allocated alternative platform is the NCSA SGI Altix platform (Cobalt), which can support up to 1.4TB shared memory applications.
- Users with significant requirements for I/O performance: (serial or parallel) should consider the TACC SUN Constellation system (Ranger).
- Users with small scale applications or workflows: Consider Linux clusters such as NCSA/LONI Abe/Queen Bee.
Common porting issues
Compilers and Linkers
DataStar and BlueGene support IBM compilers and linkers. The IBM compiler environment supports a rich set of capabilities (particularly arithmetic support), some of which are not available on other platforms. To mitigate issues with addressing modes, arithmetic consistency, etc., make sure your code conforms to the following:
- Complies with ANSI/ISO language standards on DataStar
- Runs with 64-bit addressing on DataStar
- Runs with re-entrancy (i.e. use xlf90_r, mpcc_r, etc.)
Set up a reasonably demanding test case and run it on DataStar to establish correct results that can be used as a benchmark later. Run on DataStar with strict arithmetic ordering and an acceptable level of optimization. E.g.:
xlf90_r -q64 -qstrict -O3
The Cray XT systems utilize PGI compilers (Version 7.x) and the MPI is MPICH-based. The SUN Constellation utilizes PGI compilers (Version 7.1) and MVAPICH-2. In addition Intel 10.1 and Sun compilers are also available.
The SDSC IA-64 Teragrid cluster utilizes Intel Version 9.1 and may be a useful local migration platform (see below concerning Endian-ness issues).
Compilers and mathematical functions, arithmetic issues
Some common IBM specific features/issues are listed below. Your code may behave differently on the new Track 2 machines based on the how the new machines handle these cases.
- IBM does not flush denormals to zero.
- IBM typically provides better options for handling floating point errors.
- IBM provides strict arithmetic compliance, e.g., IBM -qstrict
- IBM supports 128-bit arithmetic in Fortran, C and C++. NO other platforms support 128-bit arithmetic in C/C++. If you have a mixed-language code you may need to replace 128-bit arithmetic with function calls to library routines.
- Uninitialized variables in memory, or on the stack or the heap may or may not be initialized to zero by the loader/linker.
- Math functions may not obey all ANSI/ISO standards. IBM always conforms to the strict interpretation of IEEE. C library functions such as labs and llabs are not always consistent with type. In general, in C/C++ it is best to use macros to ensure consistent types (remember C/C++ functions are generally double).
Endian-ness and File Conversion
IBM processors, such as the Power4 and PowerPC, use big-endian byte ordering. AMD and Intel processors use little-endian byte ordering. This is of no consequence to a running program, but unformatted (binary) data written using an IBM platform cannot be read directly on a little-endian platform. Various methods can be used to convert from big-endian to little-endian (or vice versa), possibly even at run time, but there are strict conditions which must be met:
- It is not possible to write a generic automatic tool for converting big endian to little endian format or vice versa if a file contains more than a single standard data type (e.g., all 4 byte reals, all 8 byte reals, logical or character data, etc.).
- It is therefore essential that users have access to the original source code which generated the data on the IBM.
Implicit Conversion for Fortran I/O
When reading binary files that were created on DataStar or BlueGene, use the -byteswapio option for the PGI compiler to convert simple intrinsic types from big-endian to little-endian format. For example:
pgf77/pgf90 -o testio -byteswapio test.f
Explicit conversion
Explicit conversion may be necessary in C or Fortran when dealing with generic binary data. Read the data as unformatted binary and convert the data in memory by swapping byte order. For example:
// 2-byte
int SHORT_little_endian_TO_big_endian(int i)
{
return ((i>>8)&0xff)+((i << 8)&0xff00);
}
// 4-byte
int INT_little_endian_TO_big_endian(int i)
{
return((i&0xff)<<24)+((i&0xff00)<<8)+((i&0xff0000)>>8)+((i>>24)&0xff);
}
Cross-Language Linking
Linking is different on practically every machine. This is often most exposed in mixed-language codes where each compiler environment may have its own method of linking C to/from Fortran. Remember, in these cases that the burden of linking is handled on the C side of the interface, most often by adding leading or trailing underscores to the external routine names. This is most easily handled using C/C++ macros.
Library Software
Please notify SDSC Consulting if you link 3rd-party software with your application code, especially if you use IBM ESSL or PESSL routines. ESSL/PESSL subroutine calls will require explicit conversion to more portable libraries supported on the target site (e.g. NAG, IMSL, LAPACK) or source-level alternative code.
Users can expect LAPACK, BLAS, and SCALAPACK to be supported everywhere.
Some systems may not support the system() subroutine call which forks a shell process. It may be necessary to code the explicit C library calls directly within the application code.
Archival Storage
Both the Track 2 sites will have archival storage available (HPSS at NICS/ORNL and SAM/QFS at TACC). The archival storage at SDSC will continue to be available and accessible via hsi, SRB or gridftp
.
File Transfer
- Small files (<100 GByte) can be moved using scp.
- Large files (>100 GByte) should be moved using gridftp, bbftp, or bbcp.
- Stage HPSS recalls through GPFSWAN.
- Use md5sum to checksum on both ends of a file transfer operation.
Runtime issues & Debugging
By default, IBM systems do not flag floating point errorssuch as divide-by-zero resulting in a NaN. IBM provides a comprehensive error trap mechanism through compiler options. The following options can be used with the PGI compilers:
-Ktrap : Processed by the compilers when compiling main functions' programs. The options inv, denorm, divz, ovf, unf, and inexact correspond to the processor's exception mask bits invalid operation, denormalized operand, divide-by-zero, overflow, underflow, and precision, respectively. Normally, the processor's exception mask bits are on (floating-point exceptions are masked the processor recovers from the exceptions and continues). If a floating-point exception occurs and its corresponding mask bit is off (or unmasked ), execution terminates with an arithmetic exception (C's SIGFPE signal). -Ktrap=fp is equivalent to -Ktrap=inv,divz,ovf.
The debugging switches on the PGI compilers are as follows:
| -M[no]bounds | Generate code to check array bounds |
| -Mchkfpstk | Check consistency of floating point stack at subprogram calls (32-bit only) Note: This switch only works on 32-bit. On 64-bit, the switch is ignored. |
| -Mchkstk | Check for sufficient stack space upon subprogram entry |
| -Mcoff | Generate COFF format object |
| -Mdwarf1 | Generate DWARF1 debug information with -g |
| -Mdwarf2 | Generate DWARF2 debug information with -g |
| -Mdwarf3 | Generate DWARF3 debug information with -g |
| -Melf | Generate ELF format object |
| -g | Generate information for debugger |
| -gopt | Generate information for debugger without disabling optimizations |
Modules
Both the CRAY XT systems and Ranger will support modules to build a user specific application environment. The environment for compilers, tools, and libraries can be set by modules commands. Available modules can be listed using:
module available
To list environment changes performed for a given application:
module help <app>
