Jargon Tutorial

From SRB

Synergizing the paradigm with your core skills.

JARGON is a pure java API for developing programs with a data grid interface. The API currently handles file I/O for local and SRB file systems, as well as querying and modify SRB metadata. The API is also easily extensible to other file systems. File handling with JARGON closely matches file handling in Sun's java.io API, hopefully a familiar API to most java programmers.

The top level of the tree has a general class, which contains all the methods independent of the peculiarities of particular file systems. For example, GeneralFile which has available all the methods from the java.io.File class, or GeneralRandomAccessFile with all the methods of the java.io.RandomAccessFile.

The next level down splits into local and remote classes. The local classes are basically wrappers of the java.io classes, e.g., LocalFile which wraps java.io.File. The remote classes add functionality common to remote systems, e.g. RemoteRandomAccessFile.

Subclassed off the Remote classes are the remote file system specific classes, e.g. SRBFile and SRBRandomAccessFile. These classes add the functionality that is only supported by the SRB. For example, SRBFile adds a replicate method.

Class structure in JARGON conforms to the following hierarchy.

java.lang.Object
   |
   +-edu.sdsc.grid.io.General
      |
      +-edu.sdsc.grid.io.Local
      |
      +-edu.sdsc.grid.io.Remote
         |
         +-edu.sdsc.grid.io.srb.SRB
         |
         +-...(other filesystems)

The various General classes are abstract classes that get instantiated by one of the subclasses. However factory methods can be used to hide those complexities when not performing tasks special to a particular file system, such as SRBFile.replicate. But I always prefer to just look at example code, so let's move on to that. (More example code is available in the following files: MetaDataTest.java, FactoryTest.java and Test.java.)

Connecting to a File System

This simplest way is if the user tells us what file they want as a uri string, "srb://myName.sdsc@srb.sdsc.edu:5544/srbtest" or "file:/C:/test", This is a good way to instantiate a file object, because the uri contains all the necessary login information and especially as then you (the developer) don't have to pay attention to the specific requirements of the particular file system you are trying to access. (You might want to look at java.net.URI and the SRB URI protocol.)

  URI uri = new URI( "file:/C:/test" ); 
  GeneralFile file = FileFactory.newFile( uri );

  uri = new URI( "srb://myName.sdsc@srb.sdsc.edu:5544/srbtest" ); 
  GeneralFile file2 = FileFactory.newFile( uri );

Of course, it is also possible to connect directly to a file system. For example, the SRBFileSystem object represents a connection to the SRB. Certain information is used to establish a connection to the SRB, like username and password. The default constructor of the SRBFileSystem tries to find this information in the "home directory"/.srb/ on the local machine. This information may not always be available, in that case an account object must be constructed to hold the necessary information, e.g.

  SRBAccount account = new SRBAccount( 
     host, port, userName, password, homeDirectory, mdasDomainName, defaultStorageResource );
  SRBFileSystem srbFileSystem = new SRBFileSystem( account );

This SRB filesystem object can be used to locate files on that system, the constructors for SRBFile objects are the same as java.io.File plus a variable to pass the filesystem object, e.g.,

  SRBFile srbFile = new SRBFile( srbFileSystem, "filepath", "file.ext" );
  SRBFile srbFile2 = new SRBFile( srbFile, "file2.ext" ); 

For srbFile2, the filesystem is contained within the parent file.

GSI authentication to the SRB is also possible with the JARGON API. In place of the standard password in the SRBAccount, pass in the filepath of the user's proxy file. Be sure to set the SRBAccount options to GSIAUTH and the locations of the certificate authorities (CA).

  account = new SRBAccount();
  account.setPassword( "/tmp/x509up_u26477" );
  account.setOptions( SRBAccount.GSI_AUTH );
  account.setCertificateAuthority(
     "/etc/grid-security/certificates/b89793e4.0, "+
     "/etc/grid-security/certificates/42864e48.0" );
  GeneralFileSystem fileSystem = new SRBFileSystem( account );

Common File Operations

Once you have a GeneralFile object, all the methods common to the java.io.File class are available. Such as making a directory, checking read access, or listing the files in the directory.

  file.mkdir();
  boolean read = file.canRead();
  String[] list = file.list();

All of which can be used without knowing if the file is local or remote. Of course, to use one of the SRB specific functions would require an SRB declaration or casting the General object to an SRB object.

  ((SRBFile) file).replicate( );

Reading and Writing a File

There are a couple ways to manipulate the file's data. You can copy the whole file using the GeneralFile method GeneralFile.copyTo( GeneralFile ). This method will copy this general file object to the general file given as the argument.

  uri = new URI( args[1] );
  file.copyTo( FileFactory.newFile( uri ) );

I'm sure you'll be happy to know, that by using the copyTo/From methods certain optimizations have been include which speed up transfers to and from the SRB using the SRB parallel file transfer protocol. As well as a bulk load method which copies a directory of files much faster then can be done in serial.

The GeneralRandomAccessFile and subclasses provide random access support, read, write, seek, modeled after the java.io.RandomAccessFile.

  GeneralRandomAccessFile randomAccessFile = FileFactory.newRandomAccessFile( file, "rw" );
  randomAccessFile.write( new String( "This is a test file.\n" ) );

Querying the Metadata

Metadata querying for the SRB was added to the JARGON1.1 release. Here is an example program on metadata querying, MetaDataTest.java.

Queries are roughly like a very simplified SQL, only the equivalents of SELECT and WHERE are used. For a query like, "For all files with size between 0 and 10,000 return the filename."

WHERE:

  MetaDataSet.newCondition( SRBMetaDataSet.SIZE, MetaDataCondition.BETWEEN, 0, 10000 );

SELECT:

  MetaDataSet.newSelection(SRBMetaDataSet.FILE_NAME );

An array of these conditions and selections forms the query.

A query returns the results as MetaDataRecordList objects.

  MetaDataRecordList[] recordList = fileSystem.query( conditions, selects );

By default a query will only return the first 300 values. The various MetaDataRecordList methods are used to retrieve further results from the query.

The metadata can be changed by creating a new, or modifying an existing, MetaDataRecordList and then sending it back to the file system.

  recordList.addRecord( 
     SRBMetaDataSet.getField( SRBMetaDataSet.FILE_COMMENTS ),"new comments go here." );

  file.modifyMetaData( recordList );

Querying the definable metadata is simliar to querying other metadata, but the MetaDataTable class is used instead of a scalar value.

  String[][] definableMetaDataValues = new String[2][2];
  definableMetaDataValues[0][0] = "zxcv";
  definableMetaDataValues[0][1] = "123";
  definableMetaDataValues[1][0] = "asdf";
  definableMetaDataValues[1][1] = "999";
  int[] operators = new int[definableMetaDataValues.length]; 
  operators[0] = MetaDataCondition.EQUAL;
  operators[1] = MetaDataCondition.LESS_THAN;

This query would find those items with user defined metadata matching: zxcv = 123 asdf < 999

  MetaDataTable metaDataTable = new MetaDataTable( operators, definableMetaDataValues );
  MetaDataSet.newCondition( 
     SRBMetaDataSet.DEFINABLE_METADATA_FOR_FILES, metaDataTable );

Setting the definable metadata is the same as other metadata.

  recordList = new SRBMetaDataRecordList( SRBMetaDataSet.getField( SRBMetaDataSet.DEFINABLE_METADATA_FOR_FILES ), metaDataTable );
  
  file.modifyMetaData( recordList );

Be sure to take a look at the example code in, MetaDataTest.java, FactoryTest.javaand Test.java.