Skip to content

User Support banner image

SDSC > User Support > Compute and Data Resource Guides > HPSS User Guide > HPSS User Guide: HTAR Utility

HTAR Utility

Archive Migration Notice

SDSC has migrated from HPSS to the Storage and Archive Manager-Quick File System (SAM-QFS). SDSC staff will be responsible for moving all the data from HPSS to SAM-QFS.

Users are requested NOT to move their own data, as this will significantly delay the migration effort.

More Information

Availibility

HTAR is installed and supported on the following platforms at SDSC (use at your own risk):

  • SUSE Linux: SDSC TeraGrid IA-64

Back to Top

Purpose

The htar command manipulates HPSS-resident tar-format archives by writing files to, or retrieving files from, the High Performance Storage System (HPSS). Files written to HPSS are in the POSIX 1003.1 "tar" format, and may be retrieved from HPSS, and read by native "tar" programs.

The local files used by the htar command are represented by the Filespec parameter. If the Filespec parameter refers to a directory, then that directory, and, recursively, all files and directories within it, are referenced as well.

Unlike the standard UNIX "tar" command, there is no default archive device; the "-f Archive" flag is required.

Back to Top

Why use htar?

htar has been optimized for creation of archive files directly in HPSS, without having to go through the intermediate step of first creating the archive file on local disk storage, and then copying the archive file to HPSS via some other process such as ftp or hsi. The program uses multiple threads and a sophisticated buffering scheme in order to package member files into in-memory buffers, while making use of the high-speed network striping capabilities of HPSS.

In most cases, it will be significantly faster to use htar to create a tar file in HPSS than to either create a local tar file and then copy it to HPSS, or to use tar piped into ftp (or hsi) to create the tar file directly in HPSS.

In addition, htar creates a separate index file, which contains the names and locations of all of the member files in the archive (tar) file. Individual files and directories in the archive can be randomly retrieved without having to read through the archive file. Because the index file is usually smaller than the archive file, it is possible that the index file may reside in HPSS disk cache even though the archive file has been moved offline to tape. Since htar uses the index file for listing operations, it may be possible to list the contents of the archive file without having to incur the time delays of reading the archive file back onto disk cache from tape.

It is also possible to create an index file for a tar file that was not originally created by htar or to recreate an index that has been unintentionally deleted.

Back to Top

Command Format

htar -{c|t|x|X} -f Archive [-?] [-B] [-E] [-L inputlist] 
[-h] [-m] [-o] [-d debuglevel] [-p] [-v] [-V] [-w] 
[-I {IndexFile | .suffix}] [-Y [Archive COS ID]
                           [:Index File COS ID]] 
[-S Bufsize] [-T Max Threads] [Filespec | Directory ...]

Back to Top

Options

Two groups of flags exist for the htar command; "action" flags and "optional" flags. Action flags specify the operation to be performed by the htar command, and are specified by one of the following: -c, -t, -x, -X

At least one action flag must be selected in order for the htar command to perform any useful function (note: in the initial implementation, one and only one action can be specified per execution).


Action Flags

-c
Creates a new HPSS-resident archive, and writes the local files specified by one or more File parameters into the archive. Warning: any preexisting archive file will be overwritten without prompting. This behavior mimics that of the AIX tar utility.
-t
Lists the files in the order in which they appear in the HPSS- resident archive. Listable output is written to standard output; all other output is written to standard error.
-x

Extracts the files specified by one or more File parameters from the HPSS-resident archive. If the File parameter refers to a directory, the htar command recursively extracts that directory and all of its subdirectories from the archive.

If the File parameter is not specified, htar extracts all of the files from the archive. If an archive contains multiple copies of the same file, the last copy extracted overwrites all previously extracted copies. If the file being extracted does not already exist on the system, it is created. If you have the proper permissions, then htar command restores all files and directories with the same owner and group IDs as they have on the HPSS tar file. If you do not have the proper permissions, then files and directories are restored with your owner and group IDs.

-X
Builds a new index file by reading the entire tar file. This operation is used either to reconstruct an index for tar files whose Index File is unavailable (e.g., accidentally deleted), or for tar files that were not originally created by htar.

Back to Top

Optional Flags

-?
Displays htar's verbose help
-B
Displays block numbers as part of the listing (-t option). This is normally used only for debugging.
-d debuglevel
Sets debug level (0 - N) for htar. 0 disables debug, 1 - n enable progressively higher levels of debug output. 5 is the highest level; anything > 5 is silently mapped to 5.
-E
If present, specifies that a local file should be used for the file specified by the "-f Archive" option. If not specified, then the archive file will reside in HPSS.
-f Archive

Uses "Archive" as the name of archive to be read or written. Note: This is a required parameter for htar, unlike the standard tar utility, which uses a built-in default name.

If the Archive variable specified is - (minus sign), the tar command writes to standard output or reads from standard input. If you write to standard output, the -I option is mandatory, in order to specify an Index File, which is copied to HPSS if the Archive file is successfully written to standard output. [Note: this behavior is deferred - reading from or writing to pipes is not supported in the initial version of htar].

-h
Forces the tar command to follow symbolic links as if they were normal files or directories. Normally, the tar command does not follow symbolic links.
-I index_name

Specifies the index file name or suffix. If the first character of the index_name is a period, then index_name is appended to the Archive name, e.g. "-f the_htar -I .xndx" would create an index file called "the_htar.xndx". If the first character is not a period, then index_name is treated as a relative pathname for the index file (relative to the Archive file directory) if the pathname does not start with "/", or an absolute pathname otherwise.

The default directory for the Index file is the same as for the Archive file. If a relative Index file pathname is specified, then it is appended to the directory path for the Archive file. For example, if the Archive file resides in HPSS in the directory "projects/prj/ and is called files.tar, then an Index file specification of "-I projects/prj/files.old.idx" would fail, because htar would look for the file in the directory "projects/prj/projects/prj". The correct specification in this case is "-I files.old.idx".

-L InputList
Writes the files and directories listed in the "InputList" file to the archive. Directories named in the InputList file are not treated recursively. For directory names contained in the InputList file, the tar command writes only the directory entry to the archive, not the files and subdirectories rooted in the directory. Note that "home directory" notation ("~") is not expanded for pathnames contained in the InputList file, nor are wildcard characters, such as "*" and "?".
-m
Uses the time of extraction as the modification time. The default is to preserve the modification time of the files. Note that the modification time of directories is not guaranteed to be preserved, since the operating system may change the timestamp as the directory contents are changed by extracting other files and/or directories. htar will explicitly set the timestamp on directories that it extracts from the Archive, but not on intermediate directories that are created during the process of extracting files.
-o
Provides backwards compatibility with older versions (non-AIX) of the tar command. When this flag is used for reading, it causes the extracted file to take on the User and Group ID (UID and GID) of the user running the program, rather than those on the archive. This is the default behavior for the ordinary user. If htar is being run as root, use of this option causes files to be owned by root rather than the original user.
-p
Says to restore fields to their original modes, ignoring the present umask. The setuid, setgid, and tacky bit permissions are also restored to the user with root user authority.
-S bufsize
Specifies the buffer size to use when reading or writing the HPSS tar file. The buffer size can be specified as a value, or as kilobytes by appending any of "k","K","kb", or "KB" to the value. It can also be specified as megabytes by appending any of "m" or "M" or "mb" or "MB" to the value, for example, 23mb.
-T Max Threads

Specifies the maximum number of threads to use when copying local member files to the Archive file. The default is defined when htar is built; the release value is 15. The maximum number of threads actually used is dependent upon the local file sizes, and the size of the I/O buffers. A good approximation is usually

buffer size/average file size

If the -v or -V option is specified, then the maximum number of local file threads used while writing the Archive file to HPSS is displayed when the transfer is complete.

-V
"Slightly verbose" mode. If selected, file transfer progress will be displayed in interactive mode. This option should normally not be selected if verbose (-v) mode is enabled, as the outputs for the two different options are generated by separate threads, and may be intermixed on the output.
-v
"Verbose" mode. For each file processed, displays a one-character operation flag, and lists the name of each file. The flag values displayed are: "a" - file was added to the archive "x" - file was extracted from the archive "i" - index file entry was created (Build Index operation)
-w
Displays the action to be taken, followed by the file name, and then waits for user confirmation. If the response is affirmative, the action is performed. If the response is not affirmative, the file is ignored.
-Y auto | 
[Archive CosID]
[:IndexCosID]
           
Specifies the HPSS Class of Service ID to use when creating a new Archive and/or Index file. If the keyword "auto" is specified, then the HPSS "hints" mechanism is used to select the archive COS, based upon file size. If "-Y cosID" is specified, then "cosID" is the numeric COS ID to be used for the Archive File. If "-Y :IndexCosID" is specified, then "IndexCosID" is the numeric COS ID to be used for the Index File. The default COS ID (or "auto") is a site-specific option that is defined when HTAR is built. If both COS IDs are specified, the entire parameter must be specified as a single string with no embedded spaces, e.g. "-Y 40:30". This option may also be specified by the "HTAR_COS" environment variable. The environment variable is overridden by the -Y command line option, if both are used.

Also note that the files to be "htarred" [Filespec | Directory] can be specified in one of the following forms:

WildcardPath or Pathname or Filename

"WildcardPath" is a path specification that includes standard filename pattern-matching characters, as specified for the shell that is being used to invoke htar. The pattern-matching characters are expanded by the shell and passed to htar as command line arguments.

Note that using wildcard characters for the -t and -x actions may not work as expected unless there are existing local files that match the pattern. For example, "htar -xf someFile.tar a*" will only extract files beginning with "a" in "someFile.tar" that also already exist in the current local working directory.

Back to Top

Usage

"Archive" and "Member" files
Throughout the htar documentation, the term "archive file" is used to refer to the tar-format file, which is named by the "-f filename" command line option. The term "member file" is used to refer to individual files contained within the archive file.

HTAR Index File (.idx)
As part of the process of creating an archive file on HPSS, htar also creates an index file, which is a directory of the files contained in the archive. The Index File includes the position of member files within the archive, so that files and/or directories can be randomly retrieved from the archive without having to read through it sequentially. The index file is usually significantly smaller in size than the archive file, and may often reside in HPSS disk cache even though the archive file resides on tape. All htar operations make use of an index file.

It is also possible to create an index file for an archive file that was not created by htar, by using the "Build Index" [-X] function (see description of -X flag above).

By default, the index filename is created by adding ".idx" as a suffix to the Archive name specified by the -f parameter. A different suffix or index filename may be specified by the "-I " option, as described above.

By default, the Index File is assumed to reside in the same directory as the Archive File. This can be changed by specifying a relative or absolute pathname via the -I option. The Index file's relative pathname is relative to the Archive File directory unless an absolute pathname is specified.

Use of Absolute Pathnames
Although htar does not restrict the use of absolute pathnames (pathnames that begin with a leading "/") when the archive is created, it will remove the leading / when files are extracted from the archive. All extracted files use pathnames that are relative to the current working directory.

HTAR Consistency File
HTAR writes an extra file as the last member file of each Archive, with a name similar to:

/usr/tmp/HTAR_CF_CHK_64474_982644481

This file is used to verify the consistency of the Archive File and the Index File. Unless the file is explicitly specified, HTAR does not extract this file from the Archive when the -x action is selected. The file is listed, however, when the -t action is selected.

Tar File Restrictions
When specifying path names that are greater than 100 characters for a file (POSIX 1003.1 USTAR) format, remember that the path name is composed of a prefix buffer, a / (slash), and a name buffer.

The prefix buffer can be a maximum of 155 bytes and the name buffer can hold a maximum of 100 bytes. Since some implementations of TAR require the prefix and name buffers to terminate with a null ('\0') character, htar enforces the restriction that the effective prefix buffer length is 154 characters (+ trailing zero byte), and the name buffer length is 99 bytes (+ trailing zero byte). If the path name cannot be split into these two parts by a slash, it cannot be archived. This limitation is due to the structure of the tar archive headers, and must be maintained for compliance with standards and backwards compatibility. In addition, the length of a destination for a hard or symbolic link (the 'link name') cannot exceed 100 bytes (99 characters + zero-byte terminator).

Back to Top

HPSS Default Directories
The default directory for the Archive file is the user's HPSS home directory. An absolute or relative HPSS path can optionally be specified for either the Archive file or the Index file. By default, the Index file is created in the same HPSS directory as the Archive file.

Local Temporary Directory
HTAR makes use of the TMPDIR environment variable when creating temporary files. If TMPDIR is not set in the environment, then "/tmp" is used.

HTAR Memory Restrictions

When writing to an HPSS archive, the tar command uses a temporary file (normally in /tmp) and maintains in memory a table of files; you receive an error message if htar cannot create the temporary file, or if there is not enough memory available to hold the internal tables.

Authentication
HTAR uses Kerberos authentication in order to grant access to HPSS. For most SDSC systems on which htar is supported, you obtain the necessary credentials automatically when you login to the system. If Kerberos credentials are not available when HTAR is started, it will run the kinit program to obtain them, and these credentials may be used for subsequent HTAR invocations, until such time as they expire.

HTAR Execution Environment
At SDSC, HTAR is actually a wrapper script, which sets the proper environment variables and then execs the htar executable. HTAR makes use of the following HPSS environment variables, if they are available:

HTAR_COS - set to the default COS ID for the archive file, or the string "auto" to force automatic COS selection based upon file size hints. This environment variable is overridden by the -Y command line option.

HPSS_SERVER_HOST - contains the server hostname and optional port number of the HTAR server.

HPSS_HOSTNAME - contains the hostname or IP address of the network interface to which HPSS mover(s) should connect when transferring data. This is overridden by the file specified in the PFTP_CONFIG_FILENAME environment variable. The default interface is the one specified by the "hostname" command. Note that this is often a slow interface, such as the control ethernet on an IBM SP2.

HPSS_PATH_ETC - pathname of a local directory containing the HPSS network options file

PFTP_CONFIG_FILENAME - pathname of a file containing the list of HPSS network interfaces to be used

HTAR also references the following non-HPSS environment variables:

TMPDIR - used when creating temporary files HOME - used when searching for the network options file (normally only used by HPSS system administrators).


Notes

  1. The maximum size of a single Member file within the Archive is approximately 8 GB, due to restrictions in the format of the tar header. HTAR does not impose any restriction on the size of the Archive File when it is written to HPSS; however, space quotas or other system restrictions may limit the size of the Archive File when it is written to a local file (-E option).
  2. HTAR will optionally write to a local file; however, it will not write to any file type except "regular files". In particular, it is not suitable for writing to magnetic tape. To write to a magnetic tape device, use the "tar" or "cpio" utility.
  3. Deleting or manipulating files to be "htarred" with any other application (e.g. emacs) while HTAR is running will cause HTAR to fail.

Exit Status

The command "echo $status" (csh, tcsh) or "echo $?" (ksh, bash) returns the following exit values:

0 Successful completion.

>0 An error occurred.


Files

  • /usr/local/bin/htar - Specifies the name of the htar wrapper script.
  • /usr/local/bin/htar.exe - Contains the htar executable.
  • /tmp/tar* - Specifies a temporary file.

Related Information

For file archivers: the cat command, dd command, pax command.

For HPSS file transfer programs: pftp, nft, hsi

File Systems Overview for System Management in AIX Version 4 System Management Guide: Operating System and Devices explains file system types, management, structure, and maintenance.

Directory Overview in AIX Version 4 Files Reference explains working with directories and path names.

Files Overview in AIX Version 4 System User's Guide: Operating System and Devices provides information on working with files.


Limitations

  • There is no way to specify relative Index file pathnames that are not rooted in the Archive file directory without specifying an absolute path.
  • The initial implementation of HTAR does not provide the ability to append, update or remove files. These features, and others, are planned enhancements for future versions.
  • There is an unresolved AIX or HTAR problem on Blue Horizon that may be related to files being read via NFS.
  • Individual file sizes cannot exceed approximately 8 GB (the filesize is contained in a 12-byte, zero-terminated octal field in the header). 2^11-1=8589934591 (Note that there is no limit on the size of the archive file that htar creates)
  • There are pathname limitations that are described in the htar man page.
    • 99 characters max for the filename
    • 255 characters for the pathname, except that the pathname prefix can be at most 155 characters.
  • For symbolic links, there is a 99 character limit on the length of the path that the symlink contains.

Back to Top

Examples

  1. To write the file1 and file2 files to a new archive called "files.tar" in the current HPSS home directory, enter:

    htar -cf files.tar file1 file2

  2. To extract all files from the project1/src directory in the Archive file called proj1.tar, and use the time of extraction as the modification time, enter:

    htar -xm -f proj1.tar project1/src

  3. To display the names of the files in the out.tar archive file within the HPSS home directory, enter:

    htar -vtf out.tar

Back to Top

Need Help?

For help with htar, email consult@sdsc.edu or use the consulting help ticket form.


Did You Get
What You
Wanted?
Yes No
Comments