Getting Started on the IA-64 Linux Cluster
Help and Information
Help from TeraGrid Consultants:
Please send email to help@teragrid.org, or call 866-336-2357 (toll free), to report problems or ask questions related to any TeraGrid systems or services. Please see the TeraGrid Help Desk Web page for more information.
User News and Information:
Information about system downtimes, maintenance periods, upgrades, etc., is available at http://news.teragrid.org. Updates can be received via email or via the website. Users can manage their news subscriptions to receive information only on the platforms of interest.
System Access
Logging In
To login to the TeraGrid system, use one of the following hostnames or IP addresses:
tg-login.sdsc.teragrid.org (an alias for tg-login1.sdsc.teragrid.org)
tg-login1.sdsc.teragrid.org
(198.202.112.33)
tg-login2.sdsc.teragrid.org (198.202.112.34)
Currently, SDSC's DNS does not round-robin between the two login nodes. Please note that while tg-login1.sdsc.teragrid.org and tg-login2.sdsc.teragrid.org are the same for general login and pbs qsub job submission, globus jobs must be submitted to tg-login.sdsc.teragrid.org.
Users can login to the SDSC IA-64 Cluster using SSH as follows:
ssh username@tg-login.sdsc.teragrid.org
or
ssh -l username tg-login.sdsc.teragrid.org
More information about Secure Shell (SSH) may be found in the TeraGrid User Guide. SDSC security policy may be found at the SDSC Security site.
Please note that certificates issued by the SDSC CA are automatically added to the gx-map system and propagated to SDSC internal systems and the SDSC IA-64 TG cluster. They should appear in the grid-mapfiles for those systems within 5 or 10 minutes. There is no need for users to run gx-map themselves on SDSC systems for these certificates.
Computational grid users may also use X.509 certificates for authentication.
Mail is now available from either of the login nodes, tg-login1 and tg-login2. For help using standard UNIX mail, please refer to the application's man pages. For limited mail capacity within qsub while doing batch jobs, see the qsub section under batch jobs. [Currently mail fom PBS batch jobs is disabled. We hope to have this function restored soon.]
Changing Your Password
SDSC users can change their password on the TG cluster by going to the SDSC Password Changing Service. For details on using SSH key pairs on the TeraGrid, please go to the User Guide. For questions email help@teragrid.org.
File Transfer
There are several ways to transfer files to the TeraGrid. From UNIX systems, secure copy (scp) is recommended for small files. The following is an example of an scp transfer from a local machine to SDSC's IA-64 Cluster (the user command is in red text, with italicized variables):
% scp original_file username@tg-login1.sdsc.teragrid.org:/to_dir/copied_file
To use secure copy from Windows platforms, download a copy of WinSCP (freeware).
The scp command is appropriate for transfers of either small files or a small number of files. Information on other utilities for file transfer, including cross-site data transfer across multiple TeraGrid sites and into and out of mass storage, please view the Data Transfer documentation.
Transferring large files with bbftp
For transferring large data files, typically those bigger than 2GB in size, we have installed bbftp software on our IA-64 Cluster machine. This uses multiple transfer streams and compression in order to speed up the transfer process, making transfers up to 10x faster than can be achieved using regular scp.
Installing
Linux/UNIX:
To use bbftp from a within a Linux/UNIX environment, please download the latest version of the bbftp-client source code from the bbftp downloads site. Installation instructions can be found here. (You may need the assistance of your local system administrator to complete this step.)
Windows:
To use bbftp from within a windows environment a client compiled with Cygwin is available (the file bbftp-client-cygwin-3.2.0.zip from http://doc.in2p3.fr/bbftp/download.html). In order to use it you need a working cygwin installation on your windows machine with ssh installed. (You may need the assistance of your local system administrator at this stage.)
NOTE: On the SDSC IA-64 machine you need to add +bbftp to your .soft file to use bbftp. To do this execute echo +bbftp >> .soft.
Usage
Bbftp provides an FTP-like interface for transferring files. For example, in order to download the file output.dat in the directory /gpfs/ux12345/data/ from the IA-64 and put it in /home/username/output.dat (using 10 streams), one would execute the following command :
bbftp -s -e 'get /gpfs/ux12345/data/output.dat /home/username/output.dat' -u ux12345 -p 10 -V tg-login.sdsc.teragrid.org
In order to upload the file input.dat from your local machine to your gpfs directory on the IA-64 one would execute:
bbftp -s -e 'put input.dat/gpfs/ux12345/input.dat' -u ux12345 -p 10 -V tg-login.sdsc.teragrid.org
In both examples replace ux12345 with your SDSC username. More documentation can be found at the bbftp website, or by typing 'man bbftp' on the SDSC machines.
File Storage: Disk
WARNING: It is your responsibility to back up critical data! Please maintain your own copy of important data stored on SDSC file systems.
Each user has several areas of disk space for storing files for immediate use on SDSC's IA-64 Cluster. These areas may have size or time limits for how long disk files may stay resident.
TeraGrid GPFS-WAN (Global Parallel File System-Wide Area Network) is a file system mounted on the SDSC IA-64 resource, as well as on the NCSA and ANL TeraGrid IA-64 systems. The recommended use for this file system is for users/groups with the need for a common storage space for results from large multi-site runs and/or for those with large data libraries. See the TeraGrid GPFS-WAN page for file system details and information on requesting an account.
Any users that require the GPFS-WAN filesystem should use the gpfs-wan node property for their jobs (this will prevent jobs from running during scheduled outages of the gpfs-wan filesystem). To select nodes with the gpfs-wan property for your job, add the string "gpfs-wan" to your node specification in your PBS script. For example, to request 16 gpfs-wan nodes, put the following in your job script: "-l nodes=16:gpfs-wan".
| Filesystem Path | Environment Variable | Characteristics |
| /users/your_username ($HOME) |
TG_CLUSTER_HOME | 1 GB per user; regular backup |
| /scratch/local ($SCRATCH) |
TG_NODE_SCRATCH | 72 GB local disk space. $SCRATCH is shared among all users and is NOT backed up. /scratch directories on compute nodes are purged upon completion of batch jobs |
| /gpfs/your_username | TG_CLUSTER_GPFS | 34 TB parallel file system (supports MPI I/O) shared among all users; NOT backed up, 4 day purge policy |
| /gpfs-wan/ (GPFS-WAN) |
TBD | 250 TB shared, not backed up, user quotas, supports MPI I/O |
File Storage: Archival
WARNING: It is your responsibility to back up critical data! Please maintain your own copy of important data stored in NPACI archival systems. Because of the enormous amount of data involved, SDSC does not back up files in archival storage. Although the TeraGrid storage systems are very reliable, data can be lost or damaged due to media failures, system software bugs, hardware failures, and user mistakes.
Archival storage for SDSC systems is provided through SDSC's High Performance Storage System (HPSS). SDSC's Storage Resource Broker (SRB), a data management tool, may also be used to store large data sets across distributed, heterogeneous storage systems. More information about SRB is available in the TeraGrid User Guide and the SRB User Guide.
TeraGrid users may move entire directory structures from one system to another via HPSS. The following example illustrates a directory move from a local machine to SDSC's IA-64 Cluster via HPSS (user commands follow the % symbol):
- Create a copy of the local directory with tar
(the time to do this depends upon the sizes and number of files, etc.):
% tar -cf name-of-your-tar-file . - Compress tar file with gzip (this step may not
be necessary if your tar file is small):
% gzip name-of-your-tar-file
This creates a compressed tar file with the name name-of-your-tar-file.gz - Access HPSS from local machine with HSI* (client
binaries are available for download at the
HSI site):
% hsi - Store the compressed file in HPSS:
% put name-of-your-tar-file.gz - Login to SDSC's IA-64 Cluster and access HPSS with HSI:
% ssh tg-login.sdsc.teragrid.org
% hsi - Download compressed tar file from HPSS:
% get name-of-your-tar-file.gz - Uncompress tar file:
% gunzip name-of-your-tar-file.gz - Move tar file to the desired location on SDSC's IA-64 Cluster and untar:
% tar -xf name-of-your-tar-file
The htar utility provides an easy way to handle the archiving/retrieval of multiple files/directories.
- To write the file1 and file2 files to a new archive called
"files.tar" in the current HPSS home directory, enter:
htar -cf files.tar file1 file2 - To extract all files from the project1/src directory in the
Archive file called proj1.tar, and use the time of extraction as the
modification time, enter:
htar -xm -f proj1.tar project1/src - To display the names of the files in the out.tar archive file
within the HPSS home directory, enter:
htar -vtf out.tar
More detailed information on using HPSS and htar can be found at: SDSC User Support for HPSS.
Frequently Asked Questions
- How do I change my password?
If you have concerns regarding password changes please go to the SDSC Password Changing Service. For details on using SSH key pairs on the TeraGrid, please go to the User Guide. - How can I find my account number(s) and monitor my account usage?
In order to view current account usage at SDSC, use the command tgusage. see the Monitoring Usage page for additional information. - How do I add new users to my TeraGrid account(s)?
Adding Users to Existing Allocations
If you are a PI who wishes to authorize additional users, you can add users on the TeraGrid User Portal . After your application has been reviewed and accepted, your users will receive a packet via conventional mail that contains their account information. They should receive the packet in about two weeks.
As with the original allocation request, the packet contains account information from each of the sites that you requested and the User Responsibility Form. The last page of the form must be signed and returned by the new user within 30 days. (View a sample of this form as either HTML or PDF). You may start using your account as soon as you receive the packet.
