SDSC Thread Graphic Issue 6, June 2006

RSS RSS Feed (What is this?)

User Services Director:
Anke Kamrath

Subhashini Sivagnanam

Graphics Designer:
Diana Diehl

Application Designer:
Fariba Fana

Help Desk: Programming Tips

Queues on DataStar

—Mahidhar Tatineni

DataStar has 272 (8-way) P655+ and 5 (32-way) P690 compute nodes. The 1.5 GHz 8-way nodes (176 in number) have 16 GB, the 1.7 GHz 8-way nodes (96 in number) have 32 GB, while the 32-way nodes have 128 GB of memory. There is also one 32-way node with 256 GB of memory for applications requiring large memory space. Users can use a combination of the class name and the resource specifications to selectively choose the nodes required for their runs. Some common questions regarding the queue structure on DataStar are addressed below.

  • What are the classes available on the 8-way p655 nodes?
    All the p655 nodes have "normal" and "high" classes defined on them. Hence, you can run on the p655 nodes by specifying the class as "normal" or "high" in the loadleveler submit script.
  • How do I run my code using only the new 1.7 GHz, 32 GB p655 nodes?
    The scheduler on DataStar (Catalina) uses memory requirement as one of the criteria for selecting the nodes for a particular job. Currently, the consumable memory is set to 13GB on the old nodes and 24GB on the new nodes. Hence if your job requests more than 13GB of memory per node, Catalina will schedule your job on the new nodes. You can set the consumable memory in your Loadleveler script. For example:

    #@node = 4
    #@tasks_per_node = 8
    #@resources = ConsumableCpus(1) ConsumableMemory(3gb)

    In the above case, the total memory requested is 8*3=24GB (tasks_per_node*ConsumableMemory). The resource requirement is more than 13GB per node. Hence, this job will run on four of the new nodes. Please note that some part of the memory is reserved for system operations. If encountered with "BADRESOURCE LIST" while using the #@resources statement, please lower the memory limit specified
  • How do I run my jobs in the interactive queues?
    The interactive queues are accessible from There are three 8-way p655 nodes with 72 slots (8 on one node and 32 each on the other two) available in the interactive queue. You can access this queue by setting the class to "interactive" in your job script or by running from the command prompt on
  • How do I use the express queues?
    There are 4 p655 nodes (with 16 slots each) available for short (<2hrs) benchmarking or debug runs. They can be accessed from You can use your regular batch script and change the class to "express".
  • What are the classes available on the 32-way p690 nodes?
    All the batch p690 nodes have the "normal32" and "high32" classes defined on them. Hence, you can run on the p690 nodes by specifiying the class as "normal32" or "high32" in your Loadleveler submit script. One crucial difference on the p690 nodes is that the nodes are shared (as opposed to the p655 nodes which are exclusive). Hence, users *must* use the resources line to specify how many processors and how much memory their job needs. Examples of the resources line for various scenarios (MPI, OPENMP, Mixed programs) can be found in the Datastar guide

Mahidhar Tatineni is reachable via e-mail at

Did you know ..?

that Home directories should not be used for Batch jobs
Home directories are good places for login scripts and source files.Because it is NFS-mounted (slower than GPFS), this area is not suitable for storing large amounts of output from batch jobs. Increased home directory quotas for non-I/O intensive work and CVSROOT area for storing CVS directory trees are provided upon request. Users can request the CVSROOT area by emailing Larry Diegel