Dash User Guide: vSMP Jobs
The vSMP node is accessible through the large queue. In order to ensure optimal performance on the vSMP node for both you and other users sharing the node, it is important to follow the scripts listed below depending on the type of job you are running:
Note: There is currently only 1 vSMP node. The number of nodes should always be set to 1, ppn set to the number of processors desired, and the large queue specified.
OpenMP Jobs
#!/bin/bash #PBS -A <allocation> #PBS -q large #PBS -l nodes=1:ppn=16 export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so export PATH=/opt/ScaleMP/numabind/bin:$PATH export OMP_NUM_THREADS=16 export KMP_AFFINITY=compact,verbose,0,`numabind --offset=16` ./openmp-app
MKL Jobs
#!/bin/bash #PBS -A <allocation> #PBS -q large #PBS -l nodes=1:ppn=16 export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so export PATH=/opt/ScaleMP/numabind/bin:$PATH export OMP_NUM_THREADS=16 export KMP_AFFINITY=compact,verbose,0,`numabind --offset=16` export MKL_DYNAMIC=FALSE export MKL_VSMP=1 ./openmp_mkl-app
pThreads Jobs
pThreads codes on Dash require an additional configure file in order to bind to the cpus under vSMP. Create a file, config, with the following information all on a single line:
$ cat config
name=myconfig pattern=pthread.exe verbose=0 process_allocation=multi task_affinity=cpu rule=RULE-procgroup.so flags=ignore_idle
Where pattern=pthread.exe is the name of your executable. For your submit script:
#!/bin/bash #PBS -A <allocation> #PBS -q large #PBS -l nodes=1:ppn=16 export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so export PATH=/opt/ScaleMP/numabind/bin:$PATH ./pthread.exe & # start the executable 'pthread.exe', name must match in config file sleep 10 # wait until all threads have been created numabind --config config # numabind will bind the threads to individual cpus.
MPI Jobs
#!/bin/bash #PBS -A <allocation> #PBS -q large #PBS -l nodes=1:ppn=16 export PATH=/opt/ScaleMP/mpich2/1.3.2/bin:$PATH export VSMP_PLACEMENT=PACKED export VSMP_VERBOSE=YES export VSMP_MEM_PIN=YES vsmputil --unpinall mpirun -np 16 ./mpi.ex
Serial Jobs
#!/bin/bash
#PBS -A <allocation>
#PBS -q large
#PBS -l nodes=1:ppn=1
export LD_PRELOAD=/opt/ScaleMP/libvsmpclib/0.1/lib64/libvsmpclib.so
export PATH=/opt/ScaleMP/numabind/bin:$PATH
taskset -c `numabind --offset=1` ./serial.ex
