Help Desk: Programming Tips
Queues on DataStar
DataStar has 272 (8-way) P655+ and 5 (32-way) P690 compute nodes. The 1.5 GHz 8-way nodes (176 in number) have 16 GB, the
1.7 GHz 8-way nodes (96 in number) have 32 GB, while the 32-way nodes have 128 GB of memory. There is also one 32-way
node with 256 GB of memory for applications requiring large memory space. Users can use a combination of the class name
and the resource specifications to selectively choose the nodes required for their runs. Some common questions regarding
the queue structure on DataStar are addressed below.
What are the classes available on the 8-way p655 nodes?
All the p655 nodes have "normal" and "high" classes defined on them. Hence, you can run on the p655 nodes by
specifying the class as "normal" or "high" in the loadleveler submit script.
How do I run my code using only the new 1.7 GHz, 32 GB p655 nodes?
The scheduler on DataStar (Catalina) uses memory requirement as one of the criteria for selecting the nodes for a
particular job. Currently, the consumable memory is set to 13GB on the old nodes and 24GB on the new nodes. Hence if your
job requests more than 13GB of memory per node, Catalina will schedule your job on the new nodes. You can set the
consumable memory in your Loadleveler script. For example:
#@node = 4
#@tasks_per_node = 8
#@resources = ConsumableCpus(1) ConsumableMemory(3gb)
In the above case, the total memory requested is 8*3=24GB (tasks_per_node*ConsumableMemory). The resource requirement is
more than 13GB per node. Hence, this job will run on four of the new nodes. Please note that some part of the memory is reserved for system operations. If encountered with "BADRESOURCE LIST" while using the #@resources statement, please lower the memory limit specified
How do I run my jobs in the interactive queues?
The interactive queues are accessible
from dspoe.sdsc.edu. There are three 8-way p655 nodes with 72 slots (8 on one node
and 32 each on the other two) available in the interactive queue. You can access this queue by setting the class to
"interactive" in your job script or by running from the command prompt on dspoe.sdsc.edu.
How do I use the express queues?
There are 4 p655 nodes (with 16 slots each) available for short (<2hrs) benchmarking or debug runs.
They can be accessed from dspoe.sdsc.edu. You can use your regular batch script and change
the class to "express".
What are the classes available on the 32-way p690 nodes?
All the batch p690 nodes have the "normal32" and "high32" classes defined on them. Hence,
you can run on the p690 nodes by specifiying the class as "normal32" or "high32" in your
Loadleveler submit script. One crucial difference on the p690 nodes is that the nodes are
shared (as opposed to the p655 nodes which are exclusive). Hence, users *must* use the
resources line to specify how many processors and how much memory their job needs. Examples
of the resources line for various scenarios (MPI, OPENMP, Mixed programs) can be found in
the Datastar guide
Mahidhar Tatineni is reachable via e-mail at email@example.com