Very Old Releases
(This was readme.dir/release.history.)
These are notes for some of the older SRB releases. See ReleaseNotes2.0 and README.first.htm for descriptions of more recent versions. =====================New Features of release 1.1.8===================== This is the release note for SRB release 1.1.8. This release is preceded by release 1.1.7. New features of this release includes the inclusion of encrypted password authentication as one of the four authentication scheme supported by the SRB, incorporation of transparency so that regular UNIX command can be used to manipulate SRB data, support for files larger 2GB, prespawning server processes to improve interactivity support for non-dce authentication for HPSS drivers. The SRB system has also been ported to two additional systems: RedHat Linux (6.2) and Macintosh OSX. Additional metadata support for resources and collections have also been added. The MCAT system has also been speeded up for faster access to the metadata catalog. New Features of release 1.1.8 ----------------------------- 1) Encrypted password authentication - This authentication scheme is similar to the plain text password scheme except that the password passed from a client to server will be encrypted with a self contained encryption scheme. 2) UNIX Transparency - This involves using the PRELOAD feature of the Solaris and Linux platform to trap UNIX I/O system calls and replaced them with SRB I/O calls so that regular UNIX commands and applications can be used to access and manipulate data stored in SRB without recompiling. Please read README.transparency for more info on this subject. 3) Support file size larger than 2GB - The SRB software will now support file size larger than 2 GB for the Solaris, Linix, Aix and C90 platforms. To build the SRB software with this feature, enable the SRB_LARGEFILE64 flag in the mk/mk/config file. 4) Prespawning server processes - To improve interactivity, the SRB servers can now be configured to prespawn a specific number of server processes. A SRB administrator can use the env variable PRE_SPAWN_CNT in the bin/runsrb file to specify the number of server process to be prespawn. However, to prevent spawning a large number of idle servers, it is recommended that only the MCAT enabled servers should be prespawned. 5) Non-DCE authentication for HPSS - The HPSS driver can use either the normally used DCE authentication or the NON-DCE implementation developed by Mike Gleicher. If the NON-DCE authentication is used, the NO_DCE flag in addition to the HPSS flag in mk/mk.config should be set before the build. If it is not set, DCE authentication is assumed. Please contact Mike at email@example.com for informations regarding the licensing of the Non-DCE HPSS client/server software. 6) Metadata Catalog Additions - Several additions to the MetaCatalog has been made to enhance the capabilities of the SRB/MCAT system. SRB resources can now be access controlled by the individual SRBAdmins so that they can permit limited number of users to write to their resources. Reading from the resources are not controlled. Also, when a new resource is registered in the SRB, the permission to write is given to public. The SRBAdmin have to override this if they want to restrict access. A user can now associate a container with a collection. Once such a container is associated, data created in the collection are automatically placed in this (default) container. The user can override this by specifying either a new ccontainer or a resource name, in which case these will be used. Also, when a new sub-collection is created it inherits the container from its parent collection. Again, the user can override this. Currently only one container can be associated with a collection. If more containers are associated, only one of them will be used and the user has no control about which one will be used. The user-defined metadata (UDM) has been enhanced to include collection-level user-defined metadata (UDM). The same set of rules and Scommands apply to this case as was done for UDM for data objects. Please review the release notes for version 1.1.7 (see below) for further information. The collection-level UDM can be used to store metadata that are (optionally) required of data-level UDM. This can be done by storing the string "COLLMETAATTR" in UDSMD_COLL0 and the name of the data-level UDM in UDSMD_COLL1. Then, when a dataset is created in the colection, then one can store the data-level UDMs and their associated values in UDSMD0 and UDSMD1 respectively at the data-level UDM. Querying appropriately will help in searching for data based on the metadata attributes. The MCAT has been ported to Oracle 8.1.6. New Features of release 1.1.7 ----------------------------- 1) GSI authentication - Integrated the GSI authentication (based on X.509 certificates) scheme into the SRB client and server environment. The SRB can now support three authentication schemes - GSI, SEA (developed at SDSC) and plain text password. A new parameter for the ~/.srb/.MdasEnv file - AUTH_SCHEME has been created to allow a SRB client to specify which scheme to use for authentication. The AUTH_SCHEME parameter replaces the SEA_OPT parameter which has been used in previous releases for authentication scheme specification. The README.client file gives a more detailed descriptions on setting up the client SRB environment and the files README.build and README.serverconfig files describe the building and configuring the SRB servers to support GSI authentication. The README.gsi file gives links to GSI background information as well as procedures for obtaining a NPACI X.509 certificate and using it in the SRB environment. 2) DataCutter integration - Completed integrating the dataCutter code from UMD into SRB. The 1.1.7 SRB server can now perform datacutter proxy operations on behalf on the client. Nine datacutter specific client APIs have been added to the SRB client library for index creation, spatial query and proxy spatial filter operations. Documentation of the DataCutter software is given it the postscript file datacutter.ps. 3) Container - Redesigned the container system to support multiple archival and cache storage resources. Each resource group (logical resource) may have a primary cache resource and a primary archival resource. By default, the SRB server will attempt to stage a container from the primary archival resource to the primary cache resource first. Other resources will be used if the primary resources are not available at the time. Synchronization of a modified cache copy to the archival resource can be carried out to only the primary archival resource or to all the archival resources. A new API - srbReplContainer() and a new utility - Sreplcont have been created to support staging/replicating a container copy on a arbitrary storage resource within the resource group where the container was created. In addition, a flag - PRIMARY_FLAG has been added to the srbSyncContainer() and a "-p" option has been added to the Ssyncont command to specify the synchronization will be done to the primary archival resource only. The default is to synchronize all archival resources. 4) Remote Proxy Command - A new API - srbExecCommand() and a new utility - Spcommand have been created to support the remote execution of arbitrary commands installed in a specific predefined directory on the remote host. 5) A new functionality that facilitates annotating data objects has been included. The annotation facility allows an owner of the data (or a user with 'all' permission) to annontate the data object with textual information that will be stored in the MCAT catalog. The owner can also grant permission to other users to annotate the data object. The annotator can also provide a 'position' for the annotation (a string) which may be used by a viewer (tbd) that can pin the annotation to that position. The annotation can also be retrieved and displayed, and can also be deleted or updated. Any user with a 'read' (or above) permission can retrieve the annotations on a data object. Access to read the annotations are through the 'srbGetDataDirInfo' function and the write, update and delete operations are through the 'srbModifyDataset' function. We have developed a new utility called Sannotate that facilitates creation, update, deletion and display of annotations. A man page for Sannotate provides additional details on the command. Some examples of the Sannotate command are given below: Insert new annotations: Sannotate -w 100 'tis is a new annotation' Srm.c Sannotate -w 200 'this is a second annotation' Srm.c Display annotations: Sannotate Srm.c Sannotate . Update an old annotation: Sannotate -u '2000-03-17-15.59.07' 'this is a new annotation' Srm.c Delete an annotation: Sannotate -d '2000-03-17-15.59.07' Srm.c 6) A set of user-definable metadata attributes for metadata inside MCAT. The set consists of ten string attributes visible as UDSMD0, UDSMD1, ... UDSMD9 and two integer attributes visible as UDIMD0 and UDIMD1. These metadata can be used to store any meta information that the user wants to provide for his/her data object. Users can define semantics to these meta attributes as wanted. For exampe one user may define UDSMD0 to mean 'tag' , UDSMD1 to mean 'value for tag' , UDIMD0 to mean 'begin location for tag' and UDIMD1 to mean 'end location for tag'. Another user may decide that in his interpretation UDSMD0 means 'links to other objects' and store object names for data objects linked from the current data object. The user may also store the 'location of the link' in UDSMD0. The same user in another collection, possibly containing simulation data, may use a different interpretations to these attributes. For example he/she may use the first five string attributes to hold the values of their experimental parameters and the first integer attribute to hold some return value of the simulation. Hence there is a wide variety of interpretation that one can apply to these metadata attributes. Data object owner and users with 'all' permission are permitted to insert, delete and update the metadata metadata. Access to read the metadata are through the 'srbGetDataDirInfo' function and the write, update and delete operations are through the 'srbModifyDataset' function. We have provided a new utility called Smeta that facilitates insertion, update, deletion and display of these metadata. A man page for Smeta provides additional details on the command. Some examples of the Smeta command are given below: Insert a new set of metadata for the dataObject Srm.c in current collection: Smeta -i -I "UDSMD1=abcedeff" -I UDSMD5 = 'rrrr'" -I "UDIMD1=5577" Srm.c Insert another set of metadata for the same object: Smeta -i -I "UDIMD0=2222" -I "UDIMD1=5555" Srm.c Display all the metadata for the object: Smeta Display all the metadata for all objects in current collection: Smeta . Delete metadata for the object which has the metadata index = 1 (all sets of metadata for an object are given integral numbers for identification): Smeta -d 1 Srm.c Update the value of the 7th string metadata for the object: Smeta -u 0 "UDSMD6='trert'" Srm.c Display the first and second string metadata for all objects in collection where the second metadata has a 'c' (possibly later) followed by 'e'. Smeta -I "UDSMD1 like '*c*e*'" -I UDSMD0 . New Features of release 1.1.5 ----------------------------- This release adds a new type of access functionality to SRB. It provides a stream-oriented access (both read and write) to information stored in databases. This release also provides a porting of all database functionality to Oracle 8i, including the MCAT package. It also provides a driver for DPSS. 1) The main feature of this release is the implementation of the database access interface (DAI) facility. Previous versions of SRB provided access to Large Objects (LOBs) stored in databases. In this version, we added the functionality to access to generic tabular data. The API of SRB release 1.1.4 did not change with the inclusion of this feature; but there are some important usage guidelines that need to be followed for using the DAI facility. These guidelines are discussed in the file README.DAI found in the source package readme.dir directory. An interesting feature of DAI is that it isolates the ingestion and presentation issues from storage issues. Normally, with databases, one expects to ingest data into it either through a formatted load files or through SQL statements. Similarly, queries are in SQL and results in form of tables. In DAI, one can ingest data into a SRB database, in any form, eg., HTML pages, SGML or XML documents, formatted files, packed strings, or any other structured or semi-structured formats. Similarly, the output of a query to the DAI can be in any of the above forms. The reason for this flexibility is that one can associate a 'template' file with the input which can be used to interpret the data in the input file and convert them into SQL statements for inserting into the database. On the way out, the 'template' file can be used to construct forms and marked-up documents using the tabular data produced by SQL statements. These conversion operations occur on the fly and inside the SRB and is conveniently transparent to the user. We have implemented a template language called T-language which is used for building these templates. As an example, consider that one has astronomical data in the form of an XML document conforming to the Astronomical Data Center's DTD. Then a ADC template can be used to convert the input data into multiple SQL statements that can be inserted in blocks into the database. The ADC DTD-compliant input is read as a stream of bytes (through block reads) as reading any file and the data is converted and inserted into the database. On the output side, the scenario is revered and the template is used for populating the database information into the ADC DTD compliant document. Note that, the feature can be used for XML-form conversion, since a data ingested in one XML format can be presented in another XML format. 2) A new driver has been developed for storing and retrieving data out of DPSS. DPSS is a Distributed parallel storage system developed by the Data Intensive Distributed Computing Project at Lawrence Berkeley National Laboratory. DPSS software provides a high-speed parallel cache system (can be used in WANs also) that can be used to cache files that can be retrieved for high-performance computing. See http://www-didc.lbl.gov/DPSS/ for more information. 3) The Metadata catalog has been ported to work with Oracle 8i database. This version is released as MCAT Version 1.1. With this port, MCAT can now be deployed on IBM's DB2v2, DB2v5 (UDB) database systems as well as in Oracle's 7.3.3 and 8i database systems. For using the new port, there are three more flags that need to be defined in the mk.config files. The new flags are named: ORACLE_VERSION, ORACLE_CONNECT and MCAT_VERSION. The ORACLE_VERSION flag differentiates between the Oracle 7 and Oracle 8 versions, ORACLE_CONNECT provides for connection to Oracle using either the Oracle name service or the TNS name service, and the MCAT_VERSION flag delineates between versions 1.0 and 1.1. The last flag is needed because of one important change made in the definition of the character string fields in MCAT. In version 1.1, many of the fixed character string fields occurring in version 1.0 were changed to variable length character string fields to save space. New Features of release 1.1.4 ----------------------------- 1) The main feature of this release is the implementation of the container abstraction. Due to the relative high overhead of creating/opening files in archival storage systems such as HPSS, they are not suitable for storing large number of small files typically found in digital library systems. The container concept was specifically created to handle this type of limitation. Through the use of containers, many small files can be aggregated before storage in the archieval storage system. The metadata for translating an inContainer object to file path, offset and size are kept in the MCAT and the SRB I/O drivers have been adapted to handle inContainer objects. The following software was developed to support containers: API - Added four new APIs, srbContainerCreate(), srbRmContainer(), srbGetContainerInfo() and srbSyncContainer() to create, remove, query container info and synchronize copies of a container. In addition, a new CONTAINER keyword has been added to the srbObjCreate() call to allow clients to specify the storing of objects in containers. When a client wishes to store objects in a container, the following step should be carried out: a) Create a container by specifying the name of the container to be created and the logical resource to be used using the srbContainerCreate() call. The logical resource should contain two physical resources: a "permanent" (e.g., HPSS) and a "temporary" or "cache" (e.g., UNIX disk file system) resource. b) To put a data object in a container, the srbObjCreate() call is used and the CONTAINER keyword should be used to specify the name of the container for data storage. c) Once the object has been created, the normal srbObjWrite() call is used to store the object in the container. d) Internally, all I/O are done only to the "cache" copy of the container. If the "cache" copy does not exist (e.g., purged), a replicate of the "permanent" copy will be made to the "cache" resource before any I/O operation can be made. The srbSyncContainer() call is used to synchronize the "cache" copy with the permanent copy. A purge flag is provided in the srbSyncContainer() call to allow the purging of the "cache" copy once the copies has been synchronized. When the container is full (max container size exceeded), the SRB server automatically renames the full container by appending a unique integer (clock in seconds) to the container name and creates a new container with the original name. This way, the client does not have to worry about filling up containers. Having a limit on container size is needed because of the finite space in the "cache" resource. Command line utilities - Added 4 command line utilities: Smkcont, Srmcont, Ssyncont and Slscont to create, remove, sync and list meta-data of container. The Slscont command allows the listing of all containers owned or accessible by a user as well as listing all inContainer objects stored in a given container. Java GUI - The Java srbBrowser has been modified to provide several additional functionalities: Creating a new container, Deleting an existing container, Synchronizing a container, copying objects using SRB containers and importing local files and directories to SRB containers. 2) Java GUI for for SRB sys admin tasks. Functions include: . Add and modify users . Add physical and logical resources. . Add user groups. . Add locations. . Add physical resources to logical resource. This Java GUI codes can be found in the MCAT/java directory. 3) Round Robin resource selection capability for the SRB server. When creating an object using the srbObjCreate() call, a client may specify the logical resource (the "resourceName" input) in which this object is to be created. This logical resource may contains more than one physical resources. Previously, a client may use the keyword "COPIES=n" where n is an integer, to specify the number of replica to be produced among the various physical resources. A new keyword "COPIES=RR" allows the creation of a single copy of the object with the physical resource chosen in a Round-Robin among the physical resources in the logiccal resource. New Features of release 1.1.3 ----------------------------- 1) New java srbBrowser features - The srbBrowser has been enhenced to include the following new features: a) The "Replicate" operation now can be recursive. i.e., replicating a collection recursively. b) New functions "Register" and "Unregister" of local files have been added. These functions may be recursive. Comparing to the "Import" and "Export" functions, registering a local file does not involve the copying of the file physically. Rather, the file is registered with MCAT and hence becomes visible to the SRB world. c) The new "Display" function allows graphical (jpeg, gif, etc) and text (C, html, etc) files to be displayed in the srbBrowser window. This can also be achieved by double clicking the file displayed in the metadata window. d) The new "Rename" function allows the renaming of datasets. e) The new "Metadata" menu allows the manipulation and querying of system level metadata. Manipulation of two attributes - "Access Control" and "Comments" has been implemented thus far. Users now can use these interfaces to grant access permission to individual users or group of users as well as listing the access permission of datasets and collections. Recursive operation is allowed for setting the access permission of a collection. In addition, users can now use this interface to attach a new "comment" to a dataset or modify an existing comment of a dataset. The content of these comments can be used for dataset querying. 2) The recursive option (-r) was added to the Sreplicate command making it possible to replicate all datasets in a collection recursively. 3) The testsuite script has been modified making it easier to configure. 4) A few bug fixes. New Features of release 1.1.2 ----------------------------- Features of this release are summarized in the following : 1) The java srbBrowser - This is a WINDOW-like browser. In addition to browsing collections and datasets, a number of basic operations such as copying of collections and datasets, exporting/importing of files and directories to/from local file systems, deleting datasets and collections, replicating datasets, creating new collections and renaming datasets, etc are provided. For more details, please read the README.srbBrowser file. 2) Many of the utilities (S commands) have been re-written to include the -r (recursive) option. These utilities include Sget, Sput, Scp, Srm, Sls and Schmod. The -l (long form) option has been added to the Sls command to list some important metadata associated with the datasets being listed. 3) srbMon daemon - This is a SRB monitoring daemon which remotely monitors and logs all SRB servers belonging to a federation, and attempts to restart any servers that were found to be down. For more details, please read the README.srbMon file. 4) HPSS Class Of Service (COS) configuration - Previously, the HPSS driver uses a single default COS for all file creation in HPSS. The current release uses the data/hpssCosConfig file for COS configuration. Based on the "dataSize" input parameter of the srbObjCreate() call, the HPSS driver picks the appropriate COS using the table given in the data/hpssCosConfig file. 5) Version checking - The SRB server now checks whether connecting clients run the same version of the SRB software. An error will be returned if there is a mismatch. 6) Easier Host configuration - Previously, every hosts belonging to the SRB federation must be configured in the data/hostConfig file. Now, only the MCAT enabled host is needed in the data/mcatHost file. Through this MCAT enabled SRB server, all other hosts can be obtained from the MCAT. The data/hostConfig file is now optional and is used to change the default Port Number associated with a host and/or add aliases to hostNames. 7) The data/vaultConfig is no longer needed. Info on storage vault are now obtained from MCAT. 8) A number of minor bug fixes. Content of this release ----------------------- Directory Comments --------- --------------------------------------------------------- ./src The source the the SRB code ./obj The SRB object files, the SRB client library. ./bin The SRB server binary and startup script. ./data The SRB server configuration files ./mk The SRB server make configuration ./readme.dir The README directory ./MCAT MCAT installation, configuration and management. ./utilities The SRB client utilities. ./test Client programs, test suites. ./java The java srbBrowser. ./admin The SRB server monitoring program. Other README files ------------------ faq.html - Frequently Asked Questions. README.build - Notes on how to build the SRB server and client library. README.serverconfig - Notes on how to configure and run the SRB server. README.client - Notes on how to configure a SRB client environment and use the SRB client API. README.clientAPI - Gives a summary of the SRB client API. README.utilities - Notes on Installation and Use of Client Utilities. README.MCAT.INSTALL - Notes on Installation and Management of the MCAT catalog. readme.sea directory - This directory contains several SEA authentication documentations. README.srbBrowser - Notes on Installation and Use of the java srbBrowser. README.srbMon - Notes on Installation and Use of the SRB server monitoring daemon. README.tapeMassStor - Notes on Installation of the SRB's own Tape Mass Storage system. README.DAI - Notes on usage of the database access interface (DAI). TLANG.PRIMER - A small primer for the Template Language (T Language). html/SRB1_1_3.htm - SRB Version 1.1.3 Manual in HTML format. html/installExample.htm - Example of SRB software installation in HTML format.