Jargon
From SRB
Contents |
Jargon, A Java client API for the DataGrid
A new grid API, that you already know.
The data grid is a new and exciting development in computing, utilizing transparent replication, archiving, caching heterogenous storage, aggregated data movement, shadow objects and a lot of other things most people don't know, or want to know, much about. That's why JARGON has been designed from the ground up to make programming for the grid as straightforward as possible.
Downloads
Example programs
Scommand-like example utilities
ImageJ Plugins
A set of 2 simple plugins designed to support an ImageJ connection to the Storage Resource Broker.CVS
How To: Using Jargon
Synergizing the paradigm with your core skills.
JARGON is a pure java API for developing programs with a data grid interface. The API currently handles file I/O for local and SRB file systems, as well as querying and modify SRB metadata. The API is also easily extensible to other file systems. File handling with JARGON closely matches file handling in Sun's java.io API, hopefully a familiar API to most java programmers.
The top level of the tree has a general class, which contains all the methods independent of the peculiarities of particular file systems. For example, GeneralFile which has available all the methods from the java.io.File class, or GeneralRandomAccessFile with all the methods of the java.io.RandomAccessFile.
The next level down splits into local and remote classes. The local classes are basically wrappers of the java.io classes, e.g., LocalFile which wraps java.io.File. The remote classes add functionality common to remote systems, e.g. RemoteRandomAccessFile.
Subclassed off the Remote classes are the remote file system specific classes, e.g. SRBFile and SRBRandomAccessFile. These classes add the functionality that is only supported by the SRB. For example, SRBFile adds a replicate method.
Class structure in JARGON conforms to the following hierarchy.
java.lang.Object
|
+-edu.sdsc.grid.io.General
|
+-edu.sdsc.grid.io.Local
|
+-edu.sdsc.grid.io.Remote
|
+-edu.sdsc.grid.io.srb.SRB
|
+-...(other filesystems)
The various General classes are abstract classes that get instantiated by one of the subclasses. However factory methods can be used to hide those complexities when not performing tasks special to a particular file system, such as SRBFile.replicate. But I always prefer to just look at example code, so let's move on to that. (More example code is available in the following files: MetaDataTest.java, FactoryTest.java and Test.java.)
Connecting to a File System
This simplest way is if the user tells us what file they want as a uri string, "srb://myName.sdsc@srb.sdsc.edu:5544/srbtest" or "file:/C:/test", This is a good way to instantiate a file object, because the uri contains all the necessary login information and especially as then you (the developer) don't have to pay attention to the specific requirements of the particular file system you are trying to access. (You might want to look at java.net.URI and the SRB URI protocol.)
URI uri = new URI( "file:/C:/test" ); GeneralFile file = FileFactory.newFile( uri );
uri = new URI( "srb://myName.sdsc@srb.sdsc.edu:5544/srbtest" ); GeneralFile file2 = FileFactory.newFile( uri );
Of course, it is also possible to connect directly to a file system. For example, the SRBFileSystem object represents a connection to the SRB. Certain information is used to establish a connection to the SRB, like username and password. The default constructor of the SRBFileSystem tries to find this information in the "home directory"/.srb/ on the local machine. This information may not always be available, in that case an account object must be constructed to hold the necessary information, e.g.
SRBAccount account = new SRBAccount(
host, port, userName, password, homeDirectory, mdasDomainName, defaultStorageResource );
SRBFileSystem srbFileSystem = new SRBFileSystem( account );
This SRB filesystem object can be used to locate files on that system, the constructors for SRBFile objects are the same as java.io.File plus a variable to pass the filesystem object, e.g.,
SRBFile srbFile = new SRBFile( srbFileSystem, "filepath", "file.ext" ); SRBFile srbFile2 = new SRBFile( srbFile, "file2.ext" );
For srbFile2, the filesystem is contained within the parent file.
GSI authentication to the SRB is also possible with the JARGON API. In place of the standard password in the SRBAccount, pass in the filepath of the user's proxy file. Be sure to set the SRBAccount options to GSIAUTH and the locations of the certificate authorities (CA).
account = new SRBAccount();
account.setPassword( "/tmp/x509up_u26477" );
account.setOptions( SRBAccount.GSI_AUTH );
account.setCertificateAuthority(
"/etc/grid-security/certificates/b89793e4.0, "+
"/etc/grid-security/certificates/42864e48.0" );
GeneralFileSystem fileSystem = new SRBFileSystem( account );
Common File Operations
Once you have a GeneralFile object, all the methods common to the java.io.File class are available. Such as making a directory, checking read access, or listing the files in the directory.
file.mkdir(); boolean read = file.canRead(); String[] list = file.list();
All of which can be used without knowing if the file is local or remote. Of course, to use one of the SRB specific functions would require an SRB declaration or casting the General object to an SRB object.
((SRBFile) file).replicate( );
Reading and Writing a File
There are a couple ways to manipulate the file's data. You can copy the whole file using the GeneralFile method GeneralFile.copyTo( GeneralFile ). This method will copy this general file object to the general file given as the argument.
uri = new URI( args[1] ); file.copyTo( FileFactory.newFile( uri ) );
I'm sure you'll be happy to know, that by using the copyTo/From methods certain optimizations have been include which speed up transfers to and from the SRB using the SRB parallel file transfer protocol. As well as a bulk load method which copies a directory of files much faster then can be done in serial.
The GeneralRandomAccessFile and subclasses provide random access support, read, write, seek, modeled after the java.io.RandomAccessFile.
GeneralRandomAccessFile randomAccessFile = FileFactory.newRandomAccessFile( file, "rw" ); randomAccessFile.write( new String( "This is a test file.\n" ) );
Querying the Metadata
Metadata querying for the SRB was added to the JARGON1.1 release. Here is an example program on metadata querying, MetaDataTest.java.
Queries are roughly like a very simplified SQL, only the equivalents of SELECT and WHERE are used. For a query like, "For all files with size between 0 and 10,000 return the filename."
WHERE:
MetaDataSet.newCondition( SRBMetaDataSet.SIZE, MetaDataCondition.BETWEEN, 0, 10000 );
SELECT:
MetaDataSet.newSelection(SRBMetaDataSet.FILE_NAME );
An array of these conditions and selections forms the query.
A query returns the results as MetaDataRecordList objects.
MetaDataRecordList[] recordList = fileSystem.query( conditions, selects );
By default a query will only return the first 300 values. The various MetaDataRecordList methods are used to retrieve further results from the query.
The metadata can be changed by creating a new, or modifying an existing, MetaDataRecordList and then sending it back to the file system.
recordList.addRecord(
SRBMetaDataSet.getField( SRBMetaDataSet.FILE_COMMENTS ),"new comments go here." );
file.modifyMetaData( recordList );
Querying the definable metadata is simliar to querying other metadata, but the MetaDataTable class is used instead of a scalar value.
String[][] definableMetaDataValues = new String[2][2]; definableMetaDataValues[0][0] = "zxcv"; definableMetaDataValues[0][1] = "123"; definableMetaDataValues[1][0] = "asdf"; definableMetaDataValues[1][1] = "999"; int[] operators = new int[definableMetaDataValues.length]; operators[0] = MetaDataCondition.EQUAL; operators[1] = MetaDataCondition.LESS_THAN;
This query would find those items with user defined metadata matching: zxcv = 123 asdf < 999
MetaDataTable metaDataTable = new MetaDataTable( operators, definableMetaDataValues );
MetaDataSet.newCondition(
SRBMetaDataSet.DEFINABLE_METADATA_FOR_FILES, metaDataTable );
Setting the definable metadata is the same as other metadata.
recordList = new SRBMetaDataRecordList( SRBMetaDataSet.getField( SRBMetaDataSet.DEFINABLE_METADATA_FOR_FILES ), metaDataTable ); file.modifyMetaData( recordList );
Be sure to take a look at the example code in, MetaDataTest.java, FactoryTest.javaand Test.java.
JavaDocs
License Info
The SRB Clients are distributed by the University of California under the BSD License:
Copyright (c) 2006, Regents of the University of California All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the University of California, San Diego (UCSD) nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Debugging Help
To get extra debugging information launch your JVM with the jargon.debug variable set from 0-6, e.g.
java -Djargon.debug=1
Or add the following line of code before instantiating any Jargon java objects.
System.setProperty("jargon.debug", "1");
The extent of debugging information increases with higher values.
0: No debuging information.
1: A number of methods in GeneralFile can actually result in IOExceptions due to failures communicating to the server. These exceptions are caught and handled within that GeneralFile method. However sometimes it can be useful to see what exceptions did occur. The stack trace of those exceptions are printed to System.err. As well as the name of each low level method call to the server.
2: Adds information about connection and authentication. Helps to insure you are connecting to the expected server as the correct user.
- Higher debugging levels get especially technical...
3: Adds information about the query return value.
4: Adds even more information about the query result and the actual query that was sent to the server.
- Debugging levels beyond here are probably only useful to a developer expanding the Jargon API.
5: Adds the actual bytes sent to the sever by a query and the bytes returned from the server after a query.
6: Adds every byte sent on the socket connection to the server and every byte returned by the server. You'll probably want to read the section on the SRB protocol.
