Skip to content

Dash User Guide: vSMP Jobs

The vSMP node is accessible through the large queue. In order to ensure optimal performance on the vSMP node for both you and other users sharing the node, it is important to follow the scripts listed below depending on the type of job you are running:

Note: There is currently only 1 vSMP node. The number of nodes should always be set to 1, ppn set to the number of processors desired, and the large queue specified.

OpenMP Jobs

#!/bin/bash 
#PBS -A <allocation> 
#PBS -q large 
#PBS -l nodes=1:ppn=16 

export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so 
export PATH=/opt/ScaleMP/numabind/bin:$PATH 
export OMP_NUM_THREADS=16 
export KMP_AFFINITY=compact,verbose,0,`numabind --offset=16` 
./openmp-app

MKL Jobs

#!/bin/bash 
#PBS -A <allocation> 
#PBS -q large 
#PBS -l nodes=1:ppn=16 

export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so 
export PATH=/opt/ScaleMP/numabind/bin:$PATH 
export OMP_NUM_THREADS=16 
export KMP_AFFINITY=compact,verbose,0,`numabind --offset=16` 
export MKL_DYNAMIC=FALSE 
export MKL_VSMP=1 
./openmp_mkl-app

pThreads Jobs

pThreads codes on Dash require an additional configure file in order to bind to the cpus under vSMP. Create a file, config, with the following information all on a single line:

$ cat config 
name=myconfig pattern=pthread.exe verbose=0 process_allocation=multi task_affinity=cpu rule=RULE-procgroup.so flags=ignore_idle
       

Where pattern=pthread.exe is the name of your executable. For your submit script:

#!/bin/bash
#PBS -A <allocation>
#PBS -q large
#PBS -l nodes=1:ppn=16

export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so
export PATH=/opt/ScaleMP/numabind/bin:$PATH
./pthread.exe &            # start the executable 'pthread.exe', name must match in config file
sleep 10                   # wait until all threads have been created
numabind --config config   # numabind will bind the threads to individual cpus.

MPI Jobs

#!/bin/bash 
#PBS -A <allocation> 
#PBS -q large 
#PBS -l nodes=1:ppn=16 

export PATH=/opt/ScaleMP/mpich2/1.3.2/bin:$PATH 
export VSMP_PLACEMENT=PACKED 
export VSMP_VERBOSE=YES 
export VSMP_MEM_PIN=YES 
vsmputil --unpinall 
mpirun -np 16 ./mpi.ex

Serial Jobs

#!/bin/bash 
#PBS -A <allocation> 
#PBS -q large 
#PBS -l nodes=1:ppn=1 

export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so 
export PATH=/opt/ScaleMP/numabind/bin:$PATH 
taskset -c `numabind --offset=1` ./serial.ex
     

Did You Get
What You
Wanted?
Yes No
Comments