Getting Started with HSI (HPSS Interface)
Archive Migration Notice
SDSC has migrated from HPSS to the Storage and Archive Manager-Quick File System (SAM-QFS). SDSC staff will be responsible for moving all the data from HPSS to SAM-QFS.
Users are requested NOT to move their own data, as this will significantly delay the migration effort.
HSI is a FTP-like interface to the High Performance Storage System (HPSS). HSI supports wildcards for local and HPSS pathname pattern matching, and provides recursion for many commands, including the ability to store, retrieve, list, or change permissions on entire trees. It may be used interactively or in batch mode, and may be included in UNIX pipes.
HSI is especially useful for SDSC users with accounts on multiple platforms, as it provides an interface to the SDSC HPSS system from a variety of SDSC machines. For more information on HSI commands, type help while running HSI interactively.
Currently, Kerberos is the default authentication method, implemented along with an HPSS software upgrade in April 2008. Please visit https://passive.sdsc.edu/, login with any of your SDSC machine passwords, follow the instructions to create your new HPSS Kerberos principal and set a password for the principal. Then using this password you can access HPSS in one of the following ways:
- More secure way and default way starting April 30th: Use this password every time you access HPSS, and hsi & htar programs will prompt you to enter your password.
- Password-less (ideal for scripts & automation): Use this password to obtain your Kerberos keytab (by running get_hpss_kb_keytab script on SDSC machines), and then onwards hsi & htar programs will use the keytab and not ask for the password.
If you have an $HOME/.hsirc configuration file, which defaults your authentication to DCE Keytabs (with the line 'authmethod = keytab'), you will have to comment out this line or remove the $HOME/.hsirc file.
DCE Keytabs was used as the default authentication method for hsi in the past. If you have not accessed HPSS since April 2008 you will need to set up your Kerberos Principle.
Remote Site Execution
The old executables that were previously available for download in this section do not work with Kerberos. New versions will be available shortly.
Please contact firstname.lastname@example.org for additional help.
To begin a session:
In order to use HSI interactively, users must log on to one of the SDSC hosts (for example, IA-64 or an SDSC workstation) and type hsi at the command prompt. Your Kerberos principal and password will be required, if you have not set up a Kerberos keytab.
After successful authentication, the HSI utility responds with a confirming message and a "?" default HSI prompt. For a list of HSI Commands, see the SDSC Storage Group's HSI Commands FAQ. Remember to wait for the ? prompt before entering commands.
To end a session:
To end an interactive session, enter one of the following commands from the HSI prompt:
HSI also allows users to enter commands on the initial execute line. After pressing <RETURN>, each of the commands is executed, and HSI automatically ends. The bye or quit commands are not required to terminate HSI in single-line execution. Command line for single-line execution is:
hsi command1 [;command2;command3;...]
It may be necessary to use single or double quotes around metacharacters to avoid having the shell expand the metacharacters (such as the semicolon, which the shell uses to separate its own commands). Normal conventions for metacharacters apply. For example:
hsi get *.c
will not work, but
hsi get "*.c"
will retrieve all files ending in .c.
Note that for get and put operations, HSI uses a different syntax than FTP to identify the local file name. The syntax uses a ':' (colon character) to separate the local pathname from the HPSS pathname. For example:
hsi put local_file : hpss_file
Moving Large Files
To retrieve files larger than 100 GB, users may wish to use the -S option for the get or mget commands. The -S option allows the user to bypass staging large files on the disk cache. The syntax is:
get -S [other options] local : hpss [local : hpss ...]
For more information on the -S flag and other options, type get.
Archiving Files With HTAR
Htar is a utility for storing large numbers of files and or directories. It can be viewed as an HPSS-aware version of the familiar UNIX utility "tar". One way to move a large number of files and/or directories into HPSS is a multi-step process:
- Create a tar file:
tar -cvf mytar.tar file_or_directory
- Store tar file in HPSS:
hsi put mytar.tar
Htar combines this process into one command:
htar -c -f mytar.tar -L file_or_directory
Htar has been optimized for creation of archive files directly in HPSS. In most cases, it will be significantly faster to use htar to create a tar file in HPSS than to create it on local disk and then copy it to HPSS, or to use tar piped into ftp (or hsi) to create the tar file directly in HPSS. The program uses multiple threads and a sophisticated buffering scheme in order to package member files into in-memory buffers, while making use of the high-speed network striping capabilities of HPSS.
The complete documentation for htar is at: http://www.sdsc.edu/us/resources/hpss/htar.html.
More information on HSI and HSI commands is available at http://www.sdsc.edu/us/resources/hpss/command_faq.html or contact SDSC Consulting.