Getting Started on the OnDemand Cluster
ATTENTION: OnDemand is in pre-production. View or subscribe to SDSC News for updates. To request access to this resource, contact consult@sdsc.edu.
UNIX Login
To log on from a UNIX command line:
- Log on to OnDemand via Secure Shell (SSH) using the
hostname rocks-131.sdsc.edu.
% ssh username@rocks-131.sdsc.edu
OR
ssh -l username rocks-131.sdsc.edu - When prompted, enter your UNIX password (which you originally received in your account packet). You may change your password at any time using https://passive.sdsc.edu. (Note: It is strongly recommended for first-time users to reset their passwords before logging in.)
Please do NOT run compute-intensive programs on the frontend node, rocks-131.sdsc.edu. Also, do not leave orphaned background processes. This node should be used only for the purpose of editing, compiling, and submitting your programs as well as transferring data. Submit batch scripts for production jobs.
SSH configuration on OnDemand (Rocks-131)
The head node will generate the ssh-key for connecting to the other compute nodes when you first login. Do not provide any passwords during this set-up procedure, or you will not be able to access the compute nodes from the head node without a password.
Moving Files to OnDemand
There are several ways to transfer files to OnDemand. From UNIX systems, secure copy (scp) is recommended. The following is an example of an scp from a local machine to OnDemand:
% scp orig_file username@rocks-131.sdsc.edu:/to_dir/new_file
Porting MPI Programs to OnDemand
If you have an existing MPI-based parallel application program already running on a distributed-memory computer, porting to the OnDemand should be straightforward.
- Copy your application file(s) to the OnDemand local disk space - the $HOME or /ibrix/username/ directory associated with your user account. If this directory does not exist, you will have to create it. Please note that the work areas are not backed up. Files for long-term storage should be on either the user's $HOME directory or on HPSS.
- Source code should be recompiled for the OnDemand's X86_64 architecture and relinked with the appropriate libraries, e.g., MPI.
System Configuration
The OnDemand cluster consists of 64 4-core nodes with the peak performance of 37.3 GFlops each for a total of 2.4TFlops. Currently 48 nodes (with the head node of rocks-131) are assigned for the users and the 16 other nodes (with the head node of rocks-132 and rocks-133) are assigned for other projects related to the cluster development. The node distribution for the batch queues is tabulated below.
The use of this 4-way node is exclusive: users will have exclusive access to any requested 4-way node(s) during job execution, and they will be charged for using 4 CPUs per node whether they use 4 CPUs per node or not.
Acceptable Use of OnDemand Nodes
OnDemand has one head (login) node, rocks-131.sdsc.edu. This node should be used only for auxiliary tasks such as transferring and editing your files; compiling and submitting batch jobs. You may launch batch jobs from the rocks-131 node to the many compute nodes.
Tasks not allowed on ANY node:
- NO Background detached jobs (i.e., gvim)
- NO Touch scripts
Note: Your $HOME directory has a 1GB quota.
How the Nodes are Connected
The nodes are connected by the Infiniband switch. The IBRIX parallel file space is also connected through the Infiniband switch.
Software Environment
The operating system (OS) on OnDemand is currently RedHat's 64bit CentOS release 4.4 installed by Rocks cluster software developed by SDSC.
File Storage: Disk
It is your responsibility to back up critical data. Please maintain your own copy of important data stored on SDSC file systems.
Each user has several areas of disk space for storing files for immediate use on OnDemand. These areas may have size or time limits for how long disk files may reside. To request increased disk capabilities (which may mean both larger quotas in home directories and periods without purging in /ibrix) on OnDemand please send justifications to consult@sdsc.edu.
Filesystem Characteristics $HOME $HOME directories are mounted on a NFS (network file system). They are best used for storing source files and scripts. They should not be used for storing large files or ouput from batch programs. 1 GB quota per user will be enforced soon. For your convenience, the environment variable $HOME is defined in your .login and .cshrc files. Regular backups are also performed. /ibrixfs /ibrixfs is the IBRIX area (IBRIX's parallel file system). It is recommended for use with most jobs, especially where large amounts of data are produced or read, where I/O performance is important, or where parallel I/O is performed. To use IBRIX, create a subdirectory with your username, such as /ibrixfs/my_username. This space should only be used for staging your data for calculations. Such data is considered temporary and is NOT backed up. Files untouched for more than 4 days will be purged. Users are responsible for moving their important long-term data to archival storage. If you plan to generate multiple TB of data in a short period of time or you have particular storage needs, contact SDSC Consulting so that special arrangements can be made.
/scratch /scratch is a local file system on each compute node of approximately 140 GB. It provides the fastest I/O and is best used for storing small data within the timeframe of the job's execution. This space is cleaned up after each job, therefore, it should not be used for anything other than temporary storage. Please do not use /scratch from the login nodes.
Note: The $HOME directory is mainly for account configuration files; the /ibrix directory is for working files; and safeguard important files from being purged by moving them to archival storage. Do NOT store files in /tmp.
It is your responsibility to archive important files to prevent data loss due to SDSC's automated file purging. Purge times are based on date of last access. After your files are purged, SDSC has no way of retrieving them.
File Storage: Archival
It is your responsibility to back up critical data. Please maintain your own copy of important data stored in SDSC archival systems. Because of the enormous amount of data involved, SDSC does not back up files in archival storage. Although the SDSC storage systems are very reliable, data can be lost or damaged due to media failures, system software bugs, hardware failures, and user mistakes.
Archival storage for SDSC production systems is provided through SDSC's High Performance Storage System (HPSS).
The recommended interface to HPSS is HSI. HSI supports wildcards for local and HPSS pathname pattern matching. It also provides recursion for many commands, including the ability to store, retrieve, and list entire directory trees, or change permissions on entire trees. It may be used interactively or in batch mode and may be included in UNIX pipes. HSI is especially useful for SDSC users with accounts on multiple platforms, as it provides an interface to the SDSC HPSS system from most SDSC machines.
To move a directory from a local machine to OnDemand via HPSS:
- Create a copy of the local directory with tar (the time to do this depends
upon the sizes and number of files, etc.):
% tar -cf your_tar_file
- Compress tar file with gzip (this step may not be necessary if your tar
file is small). This creates a tar file with the name your_tar_file.gz:
% gzip your_tar_file
- Access HPSS from local machine with HSI with the hsi command. Client
binaries are available for download on the HSI
site:
% hsi
- Store the compressed file in HPSS:
- Login to OnDemand and access HPSS with HSI:
- Download the compressed tar file from HPSS:
- Uncompress the tar file:
- Move the tar file to the desired location on OnDemand and untar:
% put your_tar_file.gz
% ssh rocks-131.sdsc.edu
% hsi
% get your_tar_file.gz
gunzip your_tar_file.gz
tar -xf your_tar_file
For more details on using HPSS, refer to the HPSS User Guide.
Adding Users to an Account (for PIs only)
To request additional login names, Please contact Dongju Choi via consult@sdsc.edu,
