TSCC User Guide

System Information

Hardware Specifications

There are three kinds of compute nodes in the cluster: General Computing Nodes, GPU Nodes, and Large Memory Nodes. The TSCC group will periodically update the hardware choices for general computing and GPU condo node purchases, to keep abreast of technological and cost advances. Please see the TSCC Home section for current specifications of each node type.

Network

TSCC nodes can have either a single port 10GbE network interface or an optional EDR InfiniBand/FDR Infiniband (IB) connect  for low latency parallel applications. 

TSCC has 2 ibgroups available for the FDR IB network and the EDR IB network. All Sandy nodes are on the FDR network with ibgroup0. All Cascade/Skylake nodes are on the EDR network with ibgroup4. Please use the command pbsnodes -a to verify the node properties such as the processor type and the ibgroup.

Storage

TSCC users will receive 100GB of backed-up home file storage, and shared access to the 800+ TB Data Oasis Lustre-based high performance parallel file system. There is a 90–day purge policy on Data Oasis. Data Oasis storage is not backed up and files stored on this system may be lost or destroyed without recourse to restore them. Long-term file storage should be maintained in your $HOME directory or project storage. Project storage can be purchased by sending e-mail to tscc-info@ucsd.edu

Additional persistent storage can be mounted from lab file servers over the campus network or purchased from SDSC.  For more information, contact tscc-info@ucsd.edu.

Back to top

System Access

Acceptable Use Policy

All users on the Triton Shared Computing Cluster and associated resources must agree to comply with the Acceptable Use Policy.

Logging In

TSCC supports command line authentication using your UCSD AD password, or via ssh keys (for non-UCSD users).

To login to the TSCC, use the following hostname:

tscc-login.sdsc.edu

Following are examples of Secure Shell (ssh) commands that may be used to login to the TSCC:

ssh <your_username>@tscc-login.sdsc.edu
ssh -l <your_username> tscc-login.sdsc.edu

Generating SSH Keys

TSCC allows all users to access TSCC via RSA keys. Please feel free to append your public RSA key to your ~/.ssh/authorized_keys file to enable access from authorized hosts without having to enter your password. Make sure you have a password on the private key on your local machine. You can use ssh-agent or keychain to avoid repeatedly typing the private key password. To set up ssh keypairs, please follow the instructions for generating ssh keys for Linux/Mac  or Windows using Putty.  For non-UCSD users please send public key to tscc-support@ucsd.edu

Back to top

Environment Modules

Managing Your Shell Environment

TSCC uses the Environment Modules package to control user environment settings. Below is a brief discussion of its common usage. You can learn more at the Modules home page.

Overview

The Environment Modules package provides for dynamic modification of a shell environment. Module commands set, change, or delete environment variables, typically in support of a particular application. They also let the user choose between different versions of the same software or different combinations of related codes.

For example, if the pgi module and openmpi_ib module are loaded and the user compiles with mpif90, the generated code is compiled with the Portland Group Fortran 90 compiler and MPI libraries utilizing openmpi are linked. By unloading the openmpi_ib module, loading the mvapich2_ib module, and compiling with mpif90, the Portland compiler is used but linked with mvapich2 support.

Back to top

Default Modules

Several modules that determine the default TSCC environment are loaded at login time. These include the intel and openmpi_ib modules to set the default compiler environment.

Useful Modules Commands

Here are some common module commands and their descriptions:

Note that the order in which modules are loaded is significant. For example, if the pgi module is loaded and subsequently the intel module is loaded, the intel compiler will be used. Also, some modules depend on others so may be loaded or unloaded as a consequence of another module command. For example, if intel and mvapich2_ib modules are both loaded, running the command unload intel will automatically unload mvapich2_ib. Subsequently issuing the load intel command will not automatically reload mvapich2_ib.

If you find yourself regularly using a set of module commands, you may want to add these to your configuration files (.bashrc for bash users, .cshrc for C shell users).  Complete documentation is available in the module(1) and modulefile(4) manpages.

Module: command not found

The error message module: command not found is sometimes encountered when switching from one shell to another or attempting to run the module command from within a shell script or batch job.  The reason that the module command may not be inherited as expected is that it is defined as a function for your login shell. If you encounter this error execute the following from the command line (interactive shells) or add to your shell script.

source /etc/profile.d/modules.sh 

Back to top

Compiling

TSCC  provides Intel, Portland Group (PGI), and GNU compilers along with MPI implementations, such as MVAPICH2 and OpenMPI.  Intel and OpenMPI are loaded as default modules.

Using the Intel Compilers (Default)

The Intel compilers and the openmpi MPI implementation will be loaded by default. If you have modified your environment, you can reload by executing the following commands at the Linux prompt or placing in your startup file (~/.cshrc or ~/.bashrc)

module purge
module load intel openmpi_ib

For AVX2 support, compile with the -xHOST option. Note that -xHOST alone does not enable aggressive optimization, so compilation with -O3 is also suggested. The -fast flag invokes -xHOST, but should be avoided since it also turns on interprocedural optimization (-ipo), which may cause problems in some instances.

Intel MKL libraries are available as part of the "intel" modules on TSCC. Once this module is loaded, the environment variable MKL_ROOT points to the location of the mkl libraries. The MKL link advisor can be used to ascertain the link line (change the MKL_ROOT aspect appropriately). To link the MKL libraries, please refer to the Intel MKL Link Line Advisor Web page. This tool accepts inputs for several variables based on your environment and automatically generates a link line for you. When using the output generated by this site, substitute the TSCC path of the Intel MKL for the value $MKLPATH in the generated script. That value is ${MKL_ROOT}/lib/em64t.

For example to compile a C program statically linking 64 bit scalapack libraries on TSCC:

mpicc -o pdpttr.exe pdpttr.c \
    -I$MKL_ROOT/include ${MKL_ROOT}/lib/intel64/libmkl_scalapack_lp64.a \
    -Wl,--start-group ${MKL_ROOT}/lib/intel64/libmkl_intel_lp64.a \
    ${MKL_ROOT}/lib/intel64/libmkl_core.a ${MKL_ROOT}/lib/intel64/libmkl_sequential.a \
    -Wl,--end-group ${MKL_ROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm

For more information on the Intel compilers: [ifort | icc | mpif77 | mpif90 | mpicc ] -help

Back to top

Using the PGI Compilers

The PGI compilers can be loaded by executing the following commands at the Linux prompt or placing in your startup file (~/.cshrc or ~/.bashrc)

module purge
module load pgi mvapich2_ib

For AVX support, compile with -fast

For more information on the PGI compilers: man [pgf77 | pgf90 | pgcc | mpif90 | mpicc]

The Portland Group compilers come with the Optimized ACML library (LAPACK/BLAS/FFT).

To link:

pg90/pgf77 myprog.f -llapack -lblas

Back to top

Using the GNU Compilers

The GNU compilers can be loaded by executing the following commands at the Linux prompt or placing in your startup files (~/.cshrc or ~/.bashrc)

module purge
module load gnu openmpi_ib

For AVX support, compile with -mavx. Note that AVX support is only available in version 4.7 or later, so it is necessary to explicitly load the gnu/4.9.2 module until such time that it becomes the default.

For more information on the GNU compilers: man [gfortran | gcc | g++ | mpif90 | mpicc | mpicxx]

Notes and Hints

• The mpif90 and mpicc commands are actually wrappers that call the appropriate serial compilers and load the correct MPI libraries. While the same names are used for the Intel, PGI and GNU compilers, keep in mind that these are completely independent scripts.
• If you use the PGI or GNU compilers or switch between compilers for different applications, make sure that you load the appropriate modules before running your executables.
• When building OpenMP applications and moving between different compilers, one of the most common errors is to use the wrong flag to enable handling of OpenMP directives. Note that Intel, PGI, and GNU compilers use the -openmp, -mp, and -fopenmp flags, respectively.
• Explicitly set the optimization level in your makefiles or compilation scripts. Most well written codes can safely use the highest optimization level (-O3), but many compilers set lower default levels (e.g. GNU compilers use the default -O0, which turns off all optimizations).
• Turn off debugging, profiling, and bounds checking when building executables intended for production runs as these can seriously impact performance. These options are all disabled by default. The flag used for bounds checking is compiler dependent, but the debugging (-g) and profiling (-pg) flags tend to be the same for all major compilers.

Back to top

Running Jobs on TSCC

Important Guidelines for Running Jobs

• Please do not write job output to your home directory ( /home/$USER). NFS filesystems have a single server which handles all the metadata and storage requirements. This means that if a job writes from multiple compute nodes and cores, the load is focused on this one server.
• The Lustre parallel filesystem (/oasis/tscc/scratch) is optimized for efficient handling of large files, however it doesn't work nearly as well when writing many small files. We recommend using this filesystem only if your metadata load is modest – i.e., you have O(10)-O(200) files open simultaneously. The Lustre parallel filesysetm has a 90-day purge policy, please move your files to a more persistent storage location.
• Use local scratch $TMPDIR  if your job writes a lot of files from each task. The local scratch filesystem is purged at the end of each job, so you will need to copy out files that you want to retain after the job completes.

Back to top

Running Jobs with TORQUE

TSCC uses the TORQUE Resource Manager (also known by its historical name Portable Batch System, or PBS) with the Maui Cluster Scheduler to define and manage job queues. TORQUE allows the user to submit one or more jobs for execution, using parameters specified in a job script.

Job Queue Characteristics

The default walltime for all queues is now one hour. Max cores has been updated on some queues as well. Max walltimes are still in force per the below list.

The limits are provided for each partition in the table below.

The intended uses for the submit queues are as follows:

• hotel The hotel queue supports all non-contributor users of TSCC. Jobs submitted to this queue will use only the hotel nodes.
• home The home queue is a routing queue intended for all submissions to group-specific condo clusters.  Users intending to run only within the nodes their group has contributed, should submit to this queue. Users may belong to more than one home group; in this case, a default will be in effect, and using a non-default group needs to be specified in the job submission.  The home queue does not have a maximum time limit.  The home queue will route jobs to specific queues on the submitter's group membership.   
• condo The condo queue is exclusive to contributors, but allows jobs to run on nodes in addition to those purchased. This means that a project can use more cores can be used than were contributed by the project.   The queue limits the run time to eight hours to allow the node owners to have access per their contracted agreement.
• glean The glean queue will allow jobs to run, free of charge, on any idle condo nodes. These jobs will be terminated whenever the other queues receive job submissions that can use the idle nodes. This queue is exclusive to condo participants.
• pdafm The pdafm queue will allow jobs to run on the large memory nodes. As of 7/24/2019, there are two pdafm nodes with 512gb and 756gb.   Current system configuration can be found on the Condo Details page.

For the hotel, condo, pdafm and home queues, jobs charges are based on the number of cores allocated. Memory is allocated in proportion to the number of cores on the node. 

Queue Usage Policies and Restrictions

All nodes in the system are shared.  The number of cores on a given node dictates the maximum number of jobs that can be run on each.  For example a node with 16 cores can run up to 16 jobs.

Jobs are scheduled by priorty.  Jobs submitted to home queues have higher priority then those submitted to condo or glean.  If the system is sufficiently busy that all available processors are in use and both the home and condo queues have jobs waiting, the home jobs will run first . 

A maximium of 10 jobs per user are eligible for scheduling per scheduler iteration.

All TSCC nodes have slightly less  than the specificed  amount of memory available, due to system overhead. Jobs that attempt to use more than the specified proportion will be killed.

Back to top

Submitting a Job

To submit a script to TORQUE:

qsub <batch_script>

The following is an example of a TORQUE batch script for running an MPI job. The script lines are discussed in the comments that follow.

#!/bin/csh
#PBS -q <queue name>
#PBS -N <job name>
#PBS -l nodes=10:ppn=2
#PBS -l walltime=0:50:00
#PBS -o <output file>
#PBS -e <error file>
#PBS -V
#PBS -M <email address list>
#PBS -m abe
#PBS -A <account>
cd /oasis/tscc/scratch/<user name>
mpirun -v -machinefile $PBS_NODEFILE -np 20 <./mpi.out>

Comments for the above script:

#PBS -q <queue name>

# Specify queue to which job is being submitted, one of:

• hotel
• gpu-hotel
• condo
• gpu-condo
• pdafm
• glean

#PBS -N <job name>

Specify the name of the job.

#PBS -l nodes=10:ppn=2

Request 10 nodes and 2 processors per node.

#PBS -l walltime=0:50:00

Reserve the requested nodes for 50 minutes.

#PBS -o <output file>

Redirect standard output to a file.

#PBS -e <error file>

Redirect standard error to a file.

#PBS -V

Export all user environment variables to the job.

#PBS -M <email address list>

List users, separated by commas, to receive email from this job.

#PBS -m abe

Set of conditions under which the execution server will send email about the job: (a)bort, (b)egin, (e)nd.

#PBS -A <account>

Specify account to be charged for running the job; optional if user has only one account.

If more than one account is available and this line is omitted, job will be charged to default account. To ensure the correct account is charged, it is recommended that the -A option always be used.

cd /oasis/tscc/scratch/<user name>

Change to user's working directory in the Lustre filesystem.

mpirun -v -machinefile $PBS_NODEFILE -np 20 <./mpi.out>

Run as a parallel job, in verbose output mode, using 20 processors, on the nodes specified by the list contained in the file referenced by $PBS_NODEFILE, and send the output to file mpi.out in current working directory.

To reduce email load on the mailservers, please specify an email address in your TORQUE script. For example:

#PBS -M <your_username@ucsd.edu>

#PBS -m mail_options

or using the command line:

qsub -m mail_options -M <your_username@ucsd.edu>

These mail_options are available:

no mail is sent

mail is sent when the job is aborted by the batch system.

mail is sent when the job begins execution.

mail is sent when the job terminates.

Example Scripts for Applications

SDSC User Services staff have developed sample TORQUE scripts for common applications. They are available in the

/projects/builder-group/examples directory on TSCC.

Back to top

GPU Queue Details

The GPU nodes have their own queues, named gpu-hotel and gpu-condo. The GPUs on a GPU node are allocated proportionally to the total number of cores present  e.g. if a GPU node has 12 CPU cores and 4 GPUs, then to access one gpu three cores should be requested. 

In general total number of cores/total number of gpus = number of cores that needs to be requested per GPU. 

To use 1 gpu on a GPU node with 12 cores and 4 GPUs, use a command similar to

# qsub -I -q gpu-hotel -l nodes=1:ppn=3

Allocated GPUs are referenced by the CUDA_VISIBLE_DEVICES environment variable. Applications using the CUDA libraries will discover GPU allocations through that variable. Users should never set CUDA_VISIBLE_DEVICES, it is set automatically. 

The condo-based GPUs are accessible through the gpu-condo queue, which provides users who have contributed GPU nodes to the cluster with access to each other's nodes. Like the general computing condo queue, jobs submitted to this queue have an 8-hour time limit.

Condo owners can glean cycles on condo GPU node(s) via the general glean queue. To run on gpu nodes in glean queue,  add :gpu to the node resource specification. For example:

# qsub -I -q glean -l nodes=1:ppn=3:gpu

Back to top

qsub -I -l nodes=2:ppn=10 -l walltime=0:50:00

Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
90.tscc-46                PBStest          hocks                  0 R hotel
91.tscc-46                PBStest          hocks                  0 Q hotel
92.tscc-46                PBStest          hocks                  0 Q hotel

active jobs------------------------
JOBID              USERNAME      STATE PROCS   REMAINING            STARTTIME
94                    hocks    Running     8    00:09:53  Fri Apr  3 13:40:43
1 active job               8 of 16 processors in use by local jobs (50.00%)
                            8 of 8 nodes active      (100.00%)

eligible jobs----------------------
JOBID              USERNAME      STATE PROCS     WCLIMIT              QUEUETIME
95                    hocks       Idle     8    00:10:00  Fri Apr  3  13:40:04
96                    hocks       Idle     8    00:10:00  Fri Apr  3  13:40:05
2 eligible jobs

blocked jobs-----------------------
JOBID              USERNAME      STATE PROCS     WCLIMIT             QUEUETIME
0 blocked jobs
Total jobs:  3

Partition     Tasks  Nodes      Duration   StartOffset       StartDate
---------     -----  -----  ------------  ------------  --------------
ALL               8      8      INFINITY      00:00:00  13:45:30_04/03

$ yqd 13976617
13976617 (xxxxx home-abc 1x3 ['gpu1080ti'] 4:48:23): No nodes available

$ lsjobs --property=xxx-node 
tscc-0-40: 13969396/yyyyy/47:00:44/home-YXXx24 
tscc-0-41: 13969434/yyyyy/50:23:20/home-YXXx24

$ checkjob 13976617
>Reservation '13976617' (9:19:34:24 -> 12:19:34:24  Duration: 
>3:00:00:00)
>PE:  3.00  StartPriority:  203174
>job cannot run in partition DEFAULT (idle procs do not meet 
>requirements
>: 0 of 3 procs found)
>idle procs: 3995  feasible procs:   0
>....

gbalance -u <user_name>

gbalance -p <group_name>

gstatement -u <user_name> -s 2019-01-01 -e 2019-02-28

gstatement -p <group_name> -s 2019-01-01 -e 2019-02-28

Note:

You can download the DDT User Guide from the Allinea web site.

Back to top

Bundling Serial Jobs

How to submit multiple serial jobs with a single script

Occasionally, a group of serial jobs need to be run on TSCC. Rather than submit each job individually, they can be grouped together and submitted using a single batch script procedure such as the one described below.

Back to top

Overview

Although it's preferable to run parallel codes whenever possible, sometimes that is not cost-effective, or the tasks are simply not parallelizable. In that case, using a procedure like this can save time and effort by organizing multiple serial jobs into a single input file and submitting them all in one step.

The code for this process is given below in a very simple example that uses basic shell commands as the serial tasks. Your complex serial tasks can easily be substituted for those commands and submitted using a modified version of these scripts and run from your own home directory.

Note that the /home filesystem on TSCC uses autofs. Under autofs, filesystems are not always visible to the ls command. If you cd to the/home/beta directory, for example, it will get mounted and become accessible. You can also reference it explicitly, e.g. ls /home/beta, to verify its availability. Autofs is used to minimize the number of mounts visible to active nodes. All users have their own filesystem for their home directory.

The components used in this example include:

• submit.qsub (batch script to run to submit the job)
• my_script.pl (perl script invoked by batch job)
• jobs-list (input to perl script with names of serial jobs)
• getid (executable to obtain the processor number, called by perl script)

Back to top

Example Batch File

The following is an example script that can be modified to suit users with similar needs. This file is named submit.qsub.

#!/bin/sh 
# 
#PBS -q hotel 
#PBS -m e 
#PBS -o outfile 
#PBS -e errfile 
#PBS -V
  
################################################################### 
### Update the below variables with correct values  

### Name your job here again 
#PBS -N jobname  

### Put your node count and time here 
#PBS -l nodes=1:np=5 
#PBS -l walltime=00:10:00  

### Put your notification E-mail ID here 
#PBS -M username@some.domain  

### Set this to the working directory 
cd /home/beta/scripts/bundling  

####################################################################  

## Run my parallel job 
/opt/openmpi_pgimx/bin/mpirun -machinefile $PBS_NODEFILE -np 5 \ 
./my_script.pl jobs-list

Back to top

Example Script and Input Files

The above batch script refers to this script file, named my_script.pl.

#!/usr/bin/perl 
# 
# This script executes a command from a list of files, 
# based on the current MPI id. 
# 
# Last modified: Mar/11/2005 
#  

# call getid to get the MPI id number  

($myid,$numprocs) = split(/\s+/,`./getid`); 
$file_id = $myid; 
$file_id++;   

# open file and execute appropriate command  
$file_to_use = $ARGV[0]; 
open (INPUT_FILE, $file_to_use) or &showhelp;  

for ($i = 1; $i <= $file_id; $i++) { 
    $buf = <INPUT_FILE>; 
}  

system("$buf");  

close INPUT_FILE;   

sub showhelp { 
        print "\nUsage: mpiscript.pl <filename>\n\n"; 
        print "<filename> should contain a list of executables,"; 
        print " one-per-line, including the path.\n\n"; 
}
</filename></filename>

The batch script refers to this input file, named jobs-list.

hostname; date 
hostname; ls 
hostname; uptime 
uptime 
uptime > line-5

Back to top

Sample Output

Running the above script writes output like this to the file <outfile>. Notice that the output lines are non-sequential and may be written to the same file.

12:20:53 up 3 days, 5:41, 0 users, load average: 0.92, 1.00, 0.99 
compute-0-51.local 
compute-0-51.local 
Wed Aug 19 12:20:53 PDT 2009 
12:20:53 up 3 days, 5:41, 0 users, load average: 0.92, 1.00, 0.99 
compute-0-51.local 
getid getid.c jobs-list line-5 my_script.pl submit.qsub

Line 5 in the above script writes output like this to the file <line-5>. This output does not appear in the shared output file.

12:20:53 up 3 days, 5:41, 0 users, load average: 0.92, 1.00, 0.99

Back to top

TSCC Software

Installed and Supported Software

The TSCC runs CentOS version 7.3. Over 50 additional software applications and libraries are installed on the system, and system administrators regularly work with researchers to extend this set as time/costs allow. To check for currently availalbe versions please use the command:

$ module avail

Back to top

Applications Software

This list is subject to change. Specifics of installed location, version and other details may change as the packages are updated. Please contact tscc-support@ucsd.edu for details.

Singularity

Singularity is a platform to support users that have different environmental needs then what is provided by the resource or service provider.  Singularity leverages a workflow and security model that makes it a very reasonable candidate for shared or multi-tenant HPC resources like Comet without requiring any modifications to the scheduler or system architecture. Additionally, all typical HPC functions can be leveraged within a Singularity container (e.g. InfiniBand, high performance file systems, GPUs, etc.). While Singularity supports MPI running in a hybrid model where you invoke MPI outside the container and it runs the MPI programs inside the container, we have not yet tested this.

Singlularity in the New environment

• User running  GPU-accelerated Singularitity containers with the older drivers will need to use the —nv switch.  The —nv switch will import the host system drivers and override the ones in the container, allowing users to run with the containers they’ve been using.
• The /oasis  filesystems will not automatically  mounted within Singluarity containers at runtime.  Users will need to manually --bind mount them at runtime.

Example:

tscc ~]$ singularity shell --bind /oasis  ....../pytorch/pytorch-cpu.simg ......

DDT

DDT is a debugging tool for scalar, multithreaded and parallel applications.

DDT Debugging Guide from TACC

You can download the DDT User Guide from the Allinea web site.

Back to top

Requesting Additional Software

Users can install software in their home directories. If interest is shared with other users, requested installations can become part of the core software repository. Please submit new software requests to tscc-support@ucsd.edu.

For general information, contact tscc-info@ucsd.edu.

Back to top