Very Old Releases

From SRB

(This was readme.dir/release.history.)


These are notes for some of the older SRB releases.  See
ReleaseNotes2.0 and README.first.htm for descriptions of more recent
versions.


=====================New Features of release 1.1.8=====================
This is the release note for SRB release 1.1.8. This release is preceded
by release 1.1.7. New features of this release includes the inclusion of 
encrypted password authentication as one of the four authentication scheme 
supported by the SRB, incorporation of transparency so that regular UNIX
command can be used to manipulate  SRB data, support for files larger 
2GB, prespawning server processes to improve interactivity
support for non-dce authentication for HPSS drivers. 
The SRB system has also been ported to two additional systems: 
RedHat Linux (6.2) and Macintosh OSX. Additional metadata support for 
resources and collections have also been added. The MCAT system has also 
been speeded up for faster access to the metadata catalog.

New Features of release 1.1.8
-----------------------------

1) Encrypted password authentication - This authentication scheme is 
similar to the plain text password scheme except that the password passed
from a client to server will be encrypted with a self contained encryption
scheme.

2) UNIX Transparency - This involves using the PRELOAD feature of the Solaris
and Linux platform to trap UNIX I/O system calls and replaced them with
SRB I/O calls so that regular UNIX commands and applications can be used
to access and manipulate data stored in SRB without recompiling. Please read 
README.transparency for more info on this subject.

3) Support file size larger than 2GB - The SRB software will now support
file size larger than 2 GB for the Solaris, Linix, Aix and C90 platforms.
To build the SRB software with this feature, enable the SRB_LARGEFILE64
flag in the mk/mk/config file.

4) Prespawning server processes - To improve interactivity, the SRB servers
can now be configured to prespawn a specific number of server processes.
A SRB administrator can use the env variable PRE_SPAWN_CNT in the bin/runsrb
file to specify the number of server process to be prespawn. However, to 
prevent spawning a large number of idle servers, it is recommended that 
only the MCAT enabled servers should be prespawned.

5) Non-DCE authentication for HPSS - The HPSS driver can use either the 
normally used DCE authentication or the NON-DCE implementation developed by
Mike Gleicher.  If the NON-DCE authentication is used, the NO_DCE flag in
addition to the HPSS flag in mk/mk.config should be set before the build. 
If it is not set, DCE authentication is assumed.
Please contact Mike at mkg@san.rr.com for informations regarding the licensing 
of the Non-DCE HPSS client/server software.

6) Metadata Catalog Additions - Several additions to the MetaCatalog has been
made to enhance the capabilities of the SRB/MCAT system. 

SRB resources can now be access controlled by the individual SRBAdmins so 
that they can permit limited number of users to write to their resources. 
Reading from the resources  are not controlled. Also, when a new resource is 
registered in the SRB, the permission  to write is given to public. The 
SRBAdmin have to override this if they want to restrict access.

A user can now associate a container with a collection. Once such a container 
is associated, data created in the collection are automatically placed in this 
(default) container. The user can override this by specifying either a new
ccontainer or a resource name, in which case these will be used. Also, when 
a new sub-collection  is created it inherits the container from its parent 
collection. Again, the user can override this. Currently only one container 
can be associated with a collection. If more containers are associated, only 
one of them will be used and the user has no control about which one will be 
used.

The user-defined metadata (UDM) has been enhanced to include  collection-level 
user-defined metadata (UDM). The same set of rules and Scommands apply to this
case as was done for UDM for data objects. Please review the release notes 
for version 1.1.7  (see below) for further information. The collection-level 
UDM can be used to store metadata that are (optionally) required of data-level
UDM. This can be done by storing the string "COLLMETAATTR" in  UDSMD_COLL0  
and the name of the data-level UDM in UDSMD_COLL1. Then, when a dataset is 
created in the colection, then one can store the data-level UDMs and their 
associated  values in UDSMD0  and  UDSMD1 respectively at the data-level UDM. 
Querying appropriately will help in searching for data based on the metadata 
attributes.

The MCAT has been ported to Oracle 8.1.6.




New Features of release 1.1.7
-----------------------------

1) GSI authentication - Integrated the GSI authentication (based on X.509 
certificates) scheme into the SRB client and server environment. The SRB 
can now support three authentication schemes - GSI, SEA (developed at SDSC) 
and plain text password. A new parameter for the ~/.srb/.MdasEnv file -
AUTH_SCHEME has been created to allow a SRB client to specify which 
scheme to use for authentication. The AUTH_SCHEME parameter replaces the 
SEA_OPT parameter which has been used in previous releases for authentication 
scheme specification. 

The README.client file gives a more detailed descriptions on setting up 
the client SRB environment and the files README.build and README.serverconfig 
files describe the building and configuring the SRB servers to support GSI 
authentication. The README.gsi file gives links to GSI background information
as well as procedures for obtaining a NPACI X.509 certificate and using it
in the SRB environment.
   

2) DataCutter integration - Completed integrating the dataCutter code
from UMD into SRB. The 1.1.7 SRB server can now perform datacutter proxy 
operations on behalf on the client. Nine datacutter specific client APIs 
have been added to the SRB client library for index creation, spatial query 
and proxy spatial filter operations. Documentation of the DataCutter
software is given it the postscript file datacutter.ps.

3) Container - Redesigned the container system to support multiple archival 
and cache storage resources. Each resource group (logical resource) may have 
a primary cache resource and a primary archival resource. By default, the 
SRB server will attempt to stage a container from the primary archival 
resource to the primary cache resource first. Other resources will be used 
if the primary resources are not available at the time. Synchronization of 
a modified cache copy to the archival resource can be carried out to only 
the primary archival resource or to all the archival resources.

A new API - srbReplContainer() and a new utility - Sreplcont have been created
to support staging/replicating a container copy on a arbitrary storage 
resource within the resource group where the container was created. 
In addition, a flag - PRIMARY_FLAG has been added to the srbSyncContainer()
and a "-p" option has been added to the Ssyncont command to specify the
synchronization will be done to the primary archival resource only. The 
default is to synchronize all archival resources. 

4) Remote Proxy Command - A new API - srbExecCommand() and a new utility -
Spcommand have been created to support the remote execution of arbitrary
commands installed in a specific predefined directory on the remote host. 

5) A new functionality that facilitates annotating data objects has been 
included. The annotation facility allows an owner of the data (or a 
user with 'all' permission) to annontate the data object with textual
information that will be stored in the MCAT catalog. The owner can also 
grant permission to other users to annotate the data object. The annotator
can also provide a 'position' for the annotation (a string) which
may be used by a viewer (tbd) that can pin the annotation to that
position. The annotation can also be retrieved and displayed, and 
can also be deleted or updated. Any user with a 'read' (or above) permission 
can retrieve the annotations on a data object. Access to read the
annotations are through the 'srbGetDataDirInfo' function and the
write, update and delete operations are through the 'srbModifyDataset' function.
We have developed a new utility
called Sannotate that facilitates creation, update, deletion and display of
annotations. A man page for Sannotate provides additional details on the command.
Some examples of the Sannotate command are given below:
 Insert new annotations:
   Sannotate -w 100 'tis is a new annotation' Srm.c  
   Sannotate -w 200 'this is a second annotation' Srm.c 
 Display annotations:
   Sannotate Srm.c
   Sannotate .
 Update an old annotation:
   Sannotate -u '2000-03-17-15.59.07' 'this is a new annotation' Srm.c 
 Delete an annotation:
   Sannotate -d '2000-03-17-15.59.07'  Srm.c 

6) A set of user-definable metadata attributes for metadata inside MCAT. The set
consists of ten  string attributes visible as UDSMD0, UDSMD1, ... UDSMD9 and
two integer  attributes visible as UDIMD0 and UDIMD1. These metadata can be used 
to store any meta information that the user wants to provide for his/her data 
object.  Users can define semantics to these meta attributes as wanted.
For exampe one user may define  UDSMD0 to mean 'tag' , UDSMD1 to mean 
'value for tag' , UDIMD0 to mean 'begin location for tag' and UDIMD1 to
mean 'end location for tag'. Another user may decide that in his
interpretation UDSMD0 means 'links to other objects' and store object names
for data objects linked from the current data object. The user may also
store the 'location of the link' in  UDSMD0. The same user in another
collection, possibly containing simulation data, may use a different 
interpretations to these attributes. For example he/she may use the first 
five string attributes to hold the values of their experimental parameters 
and the first integer attribute to hold some return value of the simulation. 
Hence there is a wide variety of interpretation that one can apply to these 
metadata attributes.

Data object  owner and  users with 'all' permission are permitted to insert, 
delete and update the metadata metadata. Access to read the
metadata are through the 'srbGetDataDirInfo' function and the
write, update and delete operations are through the 'srbModifyDataset' function.
We have provided a new utility called Smeta that  facilitates insertion, update,
deletion and display of these metadata. A man page for Smeta provides 
additional details on the command. Some examples of the Smeta command
are given below:
  Insert a new set of metadata for the dataObject Srm.c in current collection:
    Smeta -i -I "UDSMD1=abcedeff" -I UDSMD5 = 'rrrr'"  -I "UDIMD1=5577" Srm.c
  Insert another set of metadata for the same object:
    Smeta -i -I "UDIMD0=2222" -I "UDIMD1=5555" Srm.c
  Display all the metadata  for the object:
    Smeta
  Display all the metadata  for  all objects in current collection:
    Smeta .
  Delete metadata for the object which has the metadata index =  1 (all
sets of metadata for an object are given integral numbers for identification):
    Smeta -d 1 Srm.c
  Update the value of the 7th string metadata for the object:
    Smeta -u 0 "UDSMD6='trert'" Srm.c
  Display the first and second string metadata for all objects in collection 
       where the second metadata has a 'c' (possibly later) followed by 'e'.
    Smeta -I "UDSMD1 like '*c*e*'"  -I UDSMD0 .





New Features of release 1.1.5
-----------------------------
This release adds a new type of access functionality to SRB. It provides a 
stream-oriented access (both read and write) to information stored in 
databases. This release also provides a porting of all database functionality 
to Oracle 8i, including the MCAT package. It also provides a driver for DPSS.

1) The main feature of this release is the implementation of the database 
access interface (DAI) facility. Previous versions of SRB provided access to 
Large Objects (LOBs) stored in databases. In this version, we added the 
functionality to access to generic tabular data.

The API of SRB release 1.1.4 did not change with the inclusion of this feature;
but there are some important usage guidelines that need to be followed for
using the DAI facility. These guidelines are discussed in the file
README.DAI found in the source package readme.dir directory.

An interesting feature of DAI is that it isolates the ingestion and 
presentation issues from storage issues. Normally, with databases, one expects
to ingest data into it either through a formatted load files or through SQL 
statements. Similarly, queries are in SQL and results in form of tables. 
In DAI, one can ingest data into a SRB database, in any form, eg., HTML
pages, SGML or XML documents, formatted files, packed strings, or any other
structured or semi-structured formats. Similarly, the output of a query
to the DAI can be in any of the above forms. The reason for this
flexibility is that one can associate a 'template' file with the input
which can be used to interpret the data in the input file and convert them 
into  SQL statements for inserting into the database. On the way out,
the 'template' file can be used to construct forms and marked-up documents
using the tabular data produced by SQL statements. These conversion operations
occur on the fly and inside the SRB and is conveniently transparent to the
user.  We have implemented a template language called T-language which is
used for building these templates.

As an example, consider that one has astronomical data
in the form of an XML document conforming to the Astronomical Data Center's
DTD. Then a ADC template can be used to convert the  input data into
multiple SQL statements that can be inserted in blocks into the database.
The ADC DTD-compliant input is read as a stream of bytes (through block reads)
as reading any file and the data is converted and inserted into the database.
On the output side, the scenario is revered and the template  is used for 
populating the database information into the ADC DTD compliant document.
Note that, the feature can be used for XML-form conversion, since a data 
ingested in one XML format can be presented in another XML format. 

2) A new driver has been developed for storing and retrieving data
out of DPSS. DPSS is a Distributed parallel storage system developed
by the Data Intensive Distributed Computing Project at 
Lawrence Berkeley National Laboratory. DPSS software
provides a high-speed parallel  cache system (can be used in WANs also)
that can be used to cache  files that can be retrieved for high-performance
computing. See http://www-didc.lbl.gov/DPSS/ for more information. 

3) The Metadata catalog has been ported to work with Oracle 8i database.
This version is released as MCAT Version 1.1. 
With this port, MCAT can now be deployed on IBM's DB2v2, DB2v5 (UDB) database
systems as well as in Oracle's 7.3.3 and 8i database systems. For using the
new port, there are three more flags that need to be defined in the mk.config
files. The new flags are named: ORACLE_VERSION, ORACLE_CONNECT and 
MCAT_VERSION. The  ORACLE_VERSION flag differentiates between the Oracle
7 and Oracle 8 versions,  ORACLE_CONNECT provides for connection to Oracle 
using either the Oracle name service or the TNS name service, and the
MCAT_VERSION flag delineates between versions 1.0 and 1.1. The last
flag is needed because of one important change made in the definition of
the character string fields in MCAT. In version 1.1, many of the fixed 
character string fields occurring in version 1.0 were changed to variable
length character string fields to save space.





New Features of release 1.1.4
-----------------------------
1) The main feature of this release is the implementation of the container 
abstraction. Due to the relative high overhead of creating/opening files in 
archival storage systems such as HPSS, they are not suitable for storing 
large number of small files typically found in digital library systems. 
The container concept was specifically created to handle this type of 
limitation. Through the use of containers, many small files can be aggregated 
before storage in the archieval storage system. The metadata for translating 
an inContainer object to file path, offset and size are kept in the MCAT and
the SRB I/O drivers have been adapted to handle inContainer objects.

The following software was developed to support containers:

API - Added four new APIs, srbContainerCreate(), srbRmContainer(),
srbGetContainerInfo() and srbSyncContainer() to create, remove, query
container info and synchronize copies of a container. In addition, a new
CONTAINER keyword has been added to the srbObjCreate() call to allow
clients to specify the storing of objects in containers.

When a client wishes to store objects in a container, the
following step should be carried out:

a) Create a container by specifying the name of the container to be created
and the logical resource to be used using the srbContainerCreate() call.
The logical resource should contain two physical resources: a "permanent"
(e.g., HPSS) and a "temporary" or "cache" (e.g., UNIX disk file system)
resource.

b) To put a data object in a container, the srbObjCreate() call is used and
the CONTAINER keyword should be used to specify the name of the container
for data storage.

c) Once the object has been created, the normal srbObjWrite() call is used
to store the object in the container.

d) Internally, all I/O are done only to the "cache" copy of the container.
If the "cache" copy does not exist (e.g., purged), a replicate of the
"permanent" copy will be made to the "cache" resource before any I/O
operation can be made.  The srbSyncContainer() call is used to synchronize
the "cache" copy with the permanent copy. A purge flag is provided in the
srbSyncContainer() call to allow the purging of the "cache" copy once the
copies has been synchronized.  When the container is full (max container
size exceeded), the SRB server automatically renames the full container by
appending a unique integer (clock in seconds) to the container name and
creates a new container with the original name. This way, the client does
not have to worry about filling up containers. Having a limit on container
size is needed because of the finite space in the "cache" resource.

Command line utilities - Added 4 command line utilities: Smkcont, Srmcont,
Ssyncont and Slscont to create, remove, sync and list meta-data of
container.  The Slscont command allows the listing of all containers owned
or accessible by a user as well as listing all inContainer objects stored
in a given container.

Java GUI - The Java srbBrowser has been modified to provide several
additional functionalities: Creating a new container, Deleting an existing
container, Synchronizing a container, copying objects using SRB containers
and importing local files and directories to SRB containers.


2) Java GUI for for SRB sys admin tasks. Functions include:
                . Add and modify users
                . Add physical and logical resources.
                . Add user groups.
                . Add locations.
                . Add physical resources to logical resource.

This Java GUI codes can be found in the MCAT/java directory.


3) Round Robin resource selection capability for the SRB server. When 
creating an object using the srbObjCreate() call, a client may specify
the logical resource (the "resourceName" input) in which this object is to
be created. This logical resource may contains more than one physical
resources. Previously, a client may use the keyword "COPIES=n" where n is
an integer, to specify the number of replica to be produced among the various 
physical resources. A new keyword "COPIES=RR" allows the creation of a single
copy of the object with the physical resource chosen in a Round-Robin among
the physical resources in the logiccal resource.

New Features of release 1.1.3
-----------------------------

1) New java srbBrowser features - The srbBrowser has been enhenced to include
the following new features:

	a) The "Replicate" operation now can be recursive. i.e., replicating
	a collection recursively.

	b) New functions "Register" and "Unregister" of local files have been 
	added. These functions may be recursive. Comparing to the "Import" 
	and "Export" functions, registering a local file does not involve 
	the copying of the file physically. Rather, the file is 
	registered with MCAT and hence becomes visible to the SRB world.

	c) The new "Display" function allows graphical (jpeg, gif, etc) and 
	text (C, html, etc) files to be displayed in the srbBrowser window.
	This can also be achieved by double clicking the file displayed in 
	the metadata window. 

	d) The new "Rename" function allows the renaming of datasets.

	e) The new "Metadata" menu allows the manipulation and querying
	of system level metadata. Manipulation of two attributes -
	"Access Control" and "Comments" has been implemented thus far.
	Users now can use these interfaces to grant access permission to
	individual users or group of users as well as listing the access
	permission of datasets and collections. Recursive operation is
	allowed for setting the access permission of a collection. In
	addition, users can now use this interface to attach a new "comment"
	to a dataset or modify an existing comment of a dataset.  The 
	content of these comments can be used for dataset querying.


2) The recursive option (-r) was added to the Sreplicate command making it
possible to replicate all datasets in a collection recursively.
 
3) The testsuite script has been modified making it easier to configure. 

4) A few bug fixes.

New Features of release 1.1.2
-----------------------------

Features of this release are summarized in the following :

1) The java srbBrowser - This is a WINDOW-like browser. In addition to
browsing collections and datasets, a number of basic operations such as
copying of collections and datasets, exporting/importing of files and
directories to/from local file systems, deleting datasets and collections,
replicating datasets, creating new collections and renaming datasets, etc 
are provided. For more details, please read the README.srbBrowser file. 

2) Many of the utilities (S commands) have been re-written to include
the -r (recursive) option. These utilities include Sget, Sput, Scp,
Srm, Sls and Schmod. The -l (long form) option has been added to the
Sls command to list some important metadata associated with the datasets
being listed. 

3) srbMon daemon - This is a SRB monitoring daemon which remotely monitors 
and logs all SRB servers belonging to a federation, and attempts to
restart any servers that were found to be down. For more details, please 
read the README.srbMon file.

4) HPSS Class Of Service (COS) configuration - Previously, the HPSS driver
uses a single default COS for all file creation in HPSS. The current
release uses the data/hpssCosConfig file for COS configuration. Based
on the "dataSize" input parameter of the srbObjCreate() call, the HPSS
driver picks the appropriate COS using the table given in the 
data/hpssCosConfig file.

5) Version checking - The SRB server now checks whether connecting clients 
run the same version of the SRB software. An error will be returned if
there is a mismatch.

6) Easier Host configuration - Previously, every hosts belonging to the SRB 
federation must be configured in the data/hostConfig file. Now, only the
MCAT enabled host is needed in the data/mcatHost file. Through this MCAT
enabled SRB server, all other hosts can be obtained from the MCAT. The
data/hostConfig file is now optional and is used to change the default Port
Number associated with a host and/or add aliases to hostNames.

7) The data/vaultConfig is no longer needed. Info on storage vault are now 
obtained from MCAT.

8) A number of minor bug fixes.

Content of this release
-----------------------

Directory                              Comments
---------           ---------------------------------------------------------
./src			The source the the SRB code
./obj                   The SRB object files, the SRB client library. 
./bin			The SRB server binary and startup script.
./data			The SRB server configuration files
./mk			The SRB server make configuration
./readme.dir		The README directory
./MCAT			MCAT installation, configuration and management.
./utilities		The SRB client utilities.
./test                  Client programs, test suites.
./java			The java srbBrowser.
./admin			The SRB server monitoring program.



Other README files
------------------

faq.html - Frequently Asked Questions.

README.build - Notes on how to build the SRB server and client library.

README.serverconfig - Notes on how to configure and run the SRB server.

README.client - Notes on how to configure a SRB client environment and use
the SRB client API.

README.clientAPI - Gives a summary of the SRB client API.

README.utilities - Notes on Installation and Use of Client Utilities.

README.MCAT.INSTALL - Notes on Installation and Management of the MCAT
catalog.

readme.sea directory - This directory contains several SEA authentication
documentations.

README.srbBrowser - Notes on Installation and Use of the java srbBrowser.

README.srbMon - Notes on Installation and Use of the SRB server monitoring
daemon.

README.tapeMassStor - Notes on Installation of the SRB's own Tape
Mass Storage system.
README.DAI - Notes on usage of the database access interface (DAI).

TLANG.PRIMER - A small primer for the Template Language (T Language).

html/SRB1_1_3.htm - SRB Version 1.1.3 Manual in HTML format.

html/installExample.htm - Example of SRB software installation in HTML format.