SAM-QFS User Guide: Computation Allocations
Introduction
SAM-QFS is a high-performance archival storage system primarily for users housing data collections. See the main SAM-QFS User Guide page for information about using SAM-QFS with a data allocation. Users with computation allocations whose projects require long-term disk cache residency may contact SDSC Consulting to request access to SAM-QFS. However, SAM-QFS is considered to be in experimental mode for this use. This page contains login and data transfer information for those special computation projects.
It is your responsibility to back up critical data. This storage system is very reliable; however, data can be lost or damaged due to media failures, system software bugs, hardware failures, and user mistakes. Because of the enormous amount of data involved, SDSC maintains only one copy of SAM-QFS data on tape. For dual-copy capabilities or offsite backups, send a request to datacentral-allocations@sdsc.edu.
Access From DataStar
On DataStar, SAM-QFS acts as a mounted drive with the root name /archive. The environment variable $ARCHIVE is defined in your SDSC .login and .cshrc files. To transfer data, use standard UNIX commands such as copy (cp) and move (mv). For high-performance file transfers, use a GridFTP client.
For special projects involving more than 1TB of data or requiring long-term disk cache residency, contact SDSC Consulting.
Access From the IA-64 Linux Cluster (TeraGrid Cluster)
Authentication: GSI-SSH
Grid computing users must use GSI-SSH authentication to access SAM-QFS. To begin using gsissh, you must first set up your certificates and DNs:
- Log in to any SDSC machine. To obtain a direct login to SAM-QFS, send a request to SDSC Data Central.
- Obtain a certificate from the machine:
% cacl
- Add a Distinguished Name (DN) entry to the machine's gridmap file:
% gx-map -interactive
- For each session, create a temporary proxy certificate:
% grid-proxy-init
For more detailed information on TeraGrid authentication methods and using gsissh, visit the TeraGrid User Guide.
Access From SRB
SDSC's Storage Resource Broker (SRB) is a client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and accessing replicated data sets. SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on their attributes and/or logical names rather than their names or physical locations. For more detailed information about SRB, visit the SRB User Guide.
To begin using SRB:
- Obtain a SRB account.
- For any non-SDSC machines, download and install the latest version of SRB.
- From UNIX, check your environment file for the
proper settings:
% cat ~/.srb/.MdasEnv
- Your .MdasEnv file should look similar to:
mdasCollectionHome <home_collection_name>
mdasDomainHome <MCAT_user_home_domain_name>
srbUser <MCAT_username>
srbHost archive.sdsc.edu
AUTH_SCHEME <PASSWD_AUTH | ENCRYPT1 | SEA_AUTH | GSI_AUTH>
SERVER_DN <server_user_distinguished_name> (for GSI authentication only) - Check to see that you have a password:
% cat ~/.srb/.MdasAuth
For each SRB session:
- Initiate a session:
% Sinit - Use Scommands such as Scp, Sget, Sput, etc. to transfer files.
- Transfer using the host name archive.sdsc.edu.
- For SAM-QFS only, you may also use Sstage to stage data.
Transferring Data to and from Compute Nodes
Many methods are available for transferring data. A brief summary is presented below. For more information on TeraGrid data transfer, see the help on tgcp and globus.
Moving & Porting Data: GridFTP
GridFTP is a high-performance secure File Transfer Protocol optimized for high-bandwidth wide-area networks. The following table describes some GridFTP clients you may use to access SAM-QFS, a.k.a. archive.sdsc.edu.
| Client | Description |
| uberftp |
|
| tgcp |
|
| globus-url-copy |
|
