Dash User Guide: Flash Storage
Dash features 4 TB of flash storage, also known as solid state disk (SSD). Read/write performance of SSDs is an order of magnitude better than spinning disk, and latencies are essentially zero by comparison. Users have direct access to the flash resource as described below.
Dash is currently partitioned into two distinct clusters: one 16-node partition that runs the vSMP memory aggregation software, and one 16-node partition that does not. For the vSMP partition there are 15 64GB SSDs that are presented to the user an aggregated 960GB resource. Accordingly, a job running under vSMP can access the 960 GB of flash as single file system.
For the non-vSMP partition there are 16 64GB SSDs (1.024 TB), each of which is accessible by a user process local to that node.
How to Access Flash
The flash storage-based filesystem is mounted as /scratch on each node. A user accessible directory named /scratch/$PBS_JOBID is created when a job is submitted to the queuing system. This directory on the SSD is a temporary storage resource and once a job completes, the directory is purged and space released to the system. If the data written to SSD is needed after the run has completed, users should incorporate instructions in their job script to copy the data from SSD to persistent storage (such as GPFS). Failure to do this will result in the loss of all data that was on the SSD.
#!/bin/bash #PBS -l nodes=1:ppn=8 #PBS -l walltime=HH:MM:SS #PBS -q batch cd /scratch/$PBS_JOBID # copy the execution environment to /scratch cp $HOME/run/data . mpirun -np 8 ./mpi.ex # copy the simulation output back to persistent storage, e.g. GPFS cp -r /scratch/$PBS_JOBID /gpfs-wan/scratch/$USER/out/
Using Flash in combination with vSMP
Users may also consider using flash in conjunction with memory to improve performance. Under vSMP, there is a total of 768 GB (48GB/node) of memory (650 GB of which is available to any users on the machine at that time). Combining this with the 960GB of flash gives users access to a combined 1.728TB resource, with 1.610 TB as the maximum available to the user.
Users are encouraged to work with the SDSC consultants to explore ways to take maximum advantage of both the memory and SSD resources of Dash.