Very Old Releases
From SRB
(This was readme.dir/release.history.)
These are notes for some of the older SRB releases. See
ReleaseNotes2.0 and README.first.htm for descriptions of more recent
versions.
=====================New Features of release 1.1.8=====================
This is the release note for SRB release 1.1.8. This release is preceded
by release 1.1.7. New features of this release includes the inclusion of
encrypted password authentication as one of the four authentication scheme
supported by the SRB, incorporation of transparency so that regular UNIX
command can be used to manipulate SRB data, support for files larger
2GB, prespawning server processes to improve interactivity
support for non-dce authentication for HPSS drivers.
The SRB system has also been ported to two additional systems:
RedHat Linux (6.2) and Macintosh OSX. Additional metadata support for
resources and collections have also been added. The MCAT system has also
been speeded up for faster access to the metadata catalog.
New Features of release 1.1.8
-----------------------------
1) Encrypted password authentication - This authentication scheme is
similar to the plain text password scheme except that the password passed
from a client to server will be encrypted with a self contained encryption
scheme.
2) UNIX Transparency - This involves using the PRELOAD feature of the Solaris
and Linux platform to trap UNIX I/O system calls and replaced them with
SRB I/O calls so that regular UNIX commands and applications can be used
to access and manipulate data stored in SRB without recompiling. Please read
README.transparency for more info on this subject.
3) Support file size larger than 2GB - The SRB software will now support
file size larger than 2 GB for the Solaris, Linix, Aix and C90 platforms.
To build the SRB software with this feature, enable the SRB_LARGEFILE64
flag in the mk/mk/config file.
4) Prespawning server processes - To improve interactivity, the SRB servers
can now be configured to prespawn a specific number of server processes.
A SRB administrator can use the env variable PRE_SPAWN_CNT in the bin/runsrb
file to specify the number of server process to be prespawn. However, to
prevent spawning a large number of idle servers, it is recommended that
only the MCAT enabled servers should be prespawned.
5) Non-DCE authentication for HPSS - The HPSS driver can use either the
normally used DCE authentication or the NON-DCE implementation developed by
Mike Gleicher. If the NON-DCE authentication is used, the NO_DCE flag in
addition to the HPSS flag in mk/mk.config should be set before the build.
If it is not set, DCE authentication is assumed.
Please contact Mike at mkg@san.rr.com for informations regarding the licensing
of the Non-DCE HPSS client/server software.
6) Metadata Catalog Additions - Several additions to the MetaCatalog has been
made to enhance the capabilities of the SRB/MCAT system.
SRB resources can now be access controlled by the individual SRBAdmins so
that they can permit limited number of users to write to their resources.
Reading from the resources are not controlled. Also, when a new resource is
registered in the SRB, the permission to write is given to public. The
SRBAdmin have to override this if they want to restrict access.
A user can now associate a container with a collection. Once such a container
is associated, data created in the collection are automatically placed in this
(default) container. The user can override this by specifying either a new
ccontainer or a resource name, in which case these will be used. Also, when
a new sub-collection is created it inherits the container from its parent
collection. Again, the user can override this. Currently only one container
can be associated with a collection. If more containers are associated, only
one of them will be used and the user has no control about which one will be
used.
The user-defined metadata (UDM) has been enhanced to include collection-level
user-defined metadata (UDM). The same set of rules and Scommands apply to this
case as was done for UDM for data objects. Please review the release notes
for version 1.1.7 (see below) for further information. The collection-level
UDM can be used to store metadata that are (optionally) required of data-level
UDM. This can be done by storing the string "COLLMETAATTR" in UDSMD_COLL0
and the name of the data-level UDM in UDSMD_COLL1. Then, when a dataset is
created in the colection, then one can store the data-level UDMs and their
associated values in UDSMD0 and UDSMD1 respectively at the data-level UDM.
Querying appropriately will help in searching for data based on the metadata
attributes.
The MCAT has been ported to Oracle 8.1.6.
New Features of release 1.1.7
-----------------------------
1) GSI authentication - Integrated the GSI authentication (based on X.509
certificates) scheme into the SRB client and server environment. The SRB
can now support three authentication schemes - GSI, SEA (developed at SDSC)
and plain text password. A new parameter for the ~/.srb/.MdasEnv file -
AUTH_SCHEME has been created to allow a SRB client to specify which
scheme to use for authentication. The AUTH_SCHEME parameter replaces the
SEA_OPT parameter which has been used in previous releases for authentication
scheme specification.
The README.client file gives a more detailed descriptions on setting up
the client SRB environment and the files README.build and README.serverconfig
files describe the building and configuring the SRB servers to support GSI
authentication. The README.gsi file gives links to GSI background information
as well as procedures for obtaining a NPACI X.509 certificate and using it
in the SRB environment.
2) DataCutter integration - Completed integrating the dataCutter code
from UMD into SRB. The 1.1.7 SRB server can now perform datacutter proxy
operations on behalf on the client. Nine datacutter specific client APIs
have been added to the SRB client library for index creation, spatial query
and proxy spatial filter operations. Documentation of the DataCutter
software is given it the postscript file datacutter.ps.
3) Container - Redesigned the container system to support multiple archival
and cache storage resources. Each resource group (logical resource) may have
a primary cache resource and a primary archival resource. By default, the
SRB server will attempt to stage a container from the primary archival
resource to the primary cache resource first. Other resources will be used
if the primary resources are not available at the time. Synchronization of
a modified cache copy to the archival resource can be carried out to only
the primary archival resource or to all the archival resources.
A new API - srbReplContainer() and a new utility - Sreplcont have been created
to support staging/replicating a container copy on a arbitrary storage
resource within the resource group where the container was created.
In addition, a flag - PRIMARY_FLAG has been added to the srbSyncContainer()
and a "-p" option has been added to the Ssyncont command to specify the
synchronization will be done to the primary archival resource only. The
default is to synchronize all archival resources.
4) Remote Proxy Command - A new API - srbExecCommand() and a new utility -
Spcommand have been created to support the remote execution of arbitrary
commands installed in a specific predefined directory on the remote host.
5) A new functionality that facilitates annotating data objects has been
included. The annotation facility allows an owner of the data (or a
user with 'all' permission) to annontate the data object with textual
information that will be stored in the MCAT catalog. The owner can also
grant permission to other users to annotate the data object. The annotator
can also provide a 'position' for the annotation (a string) which
may be used by a viewer (tbd) that can pin the annotation to that
position. The annotation can also be retrieved and displayed, and
can also be deleted or updated. Any user with a 'read' (or above) permission
can retrieve the annotations on a data object. Access to read the
annotations are through the 'srbGetDataDirInfo' function and the
write, update and delete operations are through the 'srbModifyDataset' function.
We have developed a new utility
called Sannotate that facilitates creation, update, deletion and display of
annotations. A man page for Sannotate provides additional details on the command.
Some examples of the Sannotate command are given below:
Insert new annotations:
Sannotate -w 100 'tis is a new annotation' Srm.c
Sannotate -w 200 'this is a second annotation' Srm.c
Display annotations:
Sannotate Srm.c
Sannotate .
Update an old annotation:
Sannotate -u '2000-03-17-15.59.07' 'this is a new annotation' Srm.c
Delete an annotation:
Sannotate -d '2000-03-17-15.59.07' Srm.c
6) A set of user-definable metadata attributes for metadata inside MCAT. The set
consists of ten string attributes visible as UDSMD0, UDSMD1, ... UDSMD9 and
two integer attributes visible as UDIMD0 and UDIMD1. These metadata can be used
to store any meta information that the user wants to provide for his/her data
object. Users can define semantics to these meta attributes as wanted.
For exampe one user may define UDSMD0 to mean 'tag' , UDSMD1 to mean
'value for tag' , UDIMD0 to mean 'begin location for tag' and UDIMD1 to
mean 'end location for tag'. Another user may decide that in his
interpretation UDSMD0 means 'links to other objects' and store object names
for data objects linked from the current data object. The user may also
store the 'location of the link' in UDSMD0. The same user in another
collection, possibly containing simulation data, may use a different
interpretations to these attributes. For example he/she may use the first
five string attributes to hold the values of their experimental parameters
and the first integer attribute to hold some return value of the simulation.
Hence there is a wide variety of interpretation that one can apply to these
metadata attributes.
Data object owner and users with 'all' permission are permitted to insert,
delete and update the metadata metadata. Access to read the
metadata are through the 'srbGetDataDirInfo' function and the
write, update and delete operations are through the 'srbModifyDataset' function.
We have provided a new utility called Smeta that facilitates insertion, update,
deletion and display of these metadata. A man page for Smeta provides
additional details on the command. Some examples of the Smeta command
are given below:
Insert a new set of metadata for the dataObject Srm.c in current collection:
Smeta -i -I "UDSMD1=abcedeff" -I UDSMD5 = 'rrrr'" -I "UDIMD1=5577" Srm.c
Insert another set of metadata for the same object:
Smeta -i -I "UDIMD0=2222" -I "UDIMD1=5555" Srm.c
Display all the metadata for the object:
Smeta
Display all the metadata for all objects in current collection:
Smeta .
Delete metadata for the object which has the metadata index = 1 (all
sets of metadata for an object are given integral numbers for identification):
Smeta -d 1 Srm.c
Update the value of the 7th string metadata for the object:
Smeta -u 0 "UDSMD6='trert'" Srm.c
Display the first and second string metadata for all objects in collection
where the second metadata has a 'c' (possibly later) followed by 'e'.
Smeta -I "UDSMD1 like '*c*e*'" -I UDSMD0 .
New Features of release 1.1.5
-----------------------------
This release adds a new type of access functionality to SRB. It provides a
stream-oriented access (both read and write) to information stored in
databases. This release also provides a porting of all database functionality
to Oracle 8i, including the MCAT package. It also provides a driver for DPSS.
1) The main feature of this release is the implementation of the database
access interface (DAI) facility. Previous versions of SRB provided access to
Large Objects (LOBs) stored in databases. In this version, we added the
functionality to access to generic tabular data.
The API of SRB release 1.1.4 did not change with the inclusion of this feature;
but there are some important usage guidelines that need to be followed for
using the DAI facility. These guidelines are discussed in the file
README.DAI found in the source package readme.dir directory.
An interesting feature of DAI is that it isolates the ingestion and
presentation issues from storage issues. Normally, with databases, one expects
to ingest data into it either through a formatted load files or through SQL
statements. Similarly, queries are in SQL and results in form of tables.
In DAI, one can ingest data into a SRB database, in any form, eg., HTML
pages, SGML or XML documents, formatted files, packed strings, or any other
structured or semi-structured formats. Similarly, the output of a query
to the DAI can be in any of the above forms. The reason for this
flexibility is that one can associate a 'template' file with the input
which can be used to interpret the data in the input file and convert them
into SQL statements for inserting into the database. On the way out,
the 'template' file can be used to construct forms and marked-up documents
using the tabular data produced by SQL statements. These conversion operations
occur on the fly and inside the SRB and is conveniently transparent to the
user. We have implemented a template language called T-language which is
used for building these templates.
As an example, consider that one has astronomical data
in the form of an XML document conforming to the Astronomical Data Center's
DTD. Then a ADC template can be used to convert the input data into
multiple SQL statements that can be inserted in blocks into the database.
The ADC DTD-compliant input is read as a stream of bytes (through block reads)
as reading any file and the data is converted and inserted into the database.
On the output side, the scenario is revered and the template is used for
populating the database information into the ADC DTD compliant document.
Note that, the feature can be used for XML-form conversion, since a data
ingested in one XML format can be presented in another XML format.
2) A new driver has been developed for storing and retrieving data
out of DPSS. DPSS is a Distributed parallel storage system developed
by the Data Intensive Distributed Computing Project at
Lawrence Berkeley National Laboratory. DPSS software
provides a high-speed parallel cache system (can be used in WANs also)
that can be used to cache files that can be retrieved for high-performance
computing. See http://www-didc.lbl.gov/DPSS/ for more information.
3) The Metadata catalog has been ported to work with Oracle 8i database.
This version is released as MCAT Version 1.1.
With this port, MCAT can now be deployed on IBM's DB2v2, DB2v5 (UDB) database
systems as well as in Oracle's 7.3.3 and 8i database systems. For using the
new port, there are three more flags that need to be defined in the mk.config
files. The new flags are named: ORACLE_VERSION, ORACLE_CONNECT and
MCAT_VERSION. The ORACLE_VERSION flag differentiates between the Oracle
7 and Oracle 8 versions, ORACLE_CONNECT provides for connection to Oracle
using either the Oracle name service or the TNS name service, and the
MCAT_VERSION flag delineates between versions 1.0 and 1.1. The last
flag is needed because of one important change made in the definition of
the character string fields in MCAT. In version 1.1, many of the fixed
character string fields occurring in version 1.0 were changed to variable
length character string fields to save space.
New Features of release 1.1.4
-----------------------------
1) The main feature of this release is the implementation of the container
abstraction. Due to the relative high overhead of creating/opening files in
archival storage systems such as HPSS, they are not suitable for storing
large number of small files typically found in digital library systems.
The container concept was specifically created to handle this type of
limitation. Through the use of containers, many small files can be aggregated
before storage in the archieval storage system. The metadata for translating
an inContainer object to file path, offset and size are kept in the MCAT and
the SRB I/O drivers have been adapted to handle inContainer objects.
The following software was developed to support containers:
API - Added four new APIs, srbContainerCreate(), srbRmContainer(),
srbGetContainerInfo() and srbSyncContainer() to create, remove, query
container info and synchronize copies of a container. In addition, a new
CONTAINER keyword has been added to the srbObjCreate() call to allow
clients to specify the storing of objects in containers.
When a client wishes to store objects in a container, the
following step should be carried out:
a) Create a container by specifying the name of the container to be created
and the logical resource to be used using the srbContainerCreate() call.
The logical resource should contain two physical resources: a "permanent"
(e.g., HPSS) and a "temporary" or "cache" (e.g., UNIX disk file system)
resource.
b) To put a data object in a container, the srbObjCreate() call is used and
the CONTAINER keyword should be used to specify the name of the container
for data storage.
c) Once the object has been created, the normal srbObjWrite() call is used
to store the object in the container.
d) Internally, all I/O are done only to the "cache" copy of the container.
If the "cache" copy does not exist (e.g., purged), a replicate of the
"permanent" copy will be made to the "cache" resource before any I/O
operation can be made. The srbSyncContainer() call is used to synchronize
the "cache" copy with the permanent copy. A purge flag is provided in the
srbSyncContainer() call to allow the purging of the "cache" copy once the
copies has been synchronized. When the container is full (max container
size exceeded), the SRB server automatically renames the full container by
appending a unique integer (clock in seconds) to the container name and
creates a new container with the original name. This way, the client does
not have to worry about filling up containers. Having a limit on container
size is needed because of the finite space in the "cache" resource.
Command line utilities - Added 4 command line utilities: Smkcont, Srmcont,
Ssyncont and Slscont to create, remove, sync and list meta-data of
container. The Slscont command allows the listing of all containers owned
or accessible by a user as well as listing all inContainer objects stored
in a given container.
Java GUI - The Java srbBrowser has been modified to provide several
additional functionalities: Creating a new container, Deleting an existing
container, Synchronizing a container, copying objects using SRB containers
and importing local files and directories to SRB containers.
2) Java GUI for for SRB sys admin tasks. Functions include:
. Add and modify users
. Add physical and logical resources.
. Add user groups.
. Add locations.
. Add physical resources to logical resource.
This Java GUI codes can be found in the MCAT/java directory.
3) Round Robin resource selection capability for the SRB server. When
creating an object using the srbObjCreate() call, a client may specify
the logical resource (the "resourceName" input) in which this object is to
be created. This logical resource may contains more than one physical
resources. Previously, a client may use the keyword "COPIES=n" where n is
an integer, to specify the number of replica to be produced among the various
physical resources. A new keyword "COPIES=RR" allows the creation of a single
copy of the object with the physical resource chosen in a Round-Robin among
the physical resources in the logiccal resource.
New Features of release 1.1.3
-----------------------------
1) New java srbBrowser features - The srbBrowser has been enhenced to include
the following new features:
a) The "Replicate" operation now can be recursive. i.e., replicating
a collection recursively.
b) New functions "Register" and "Unregister" of local files have been
added. These functions may be recursive. Comparing to the "Import"
and "Export" functions, registering a local file does not involve
the copying of the file physically. Rather, the file is
registered with MCAT and hence becomes visible to the SRB world.
c) The new "Display" function allows graphical (jpeg, gif, etc) and
text (C, html, etc) files to be displayed in the srbBrowser window.
This can also be achieved by double clicking the file displayed in
the metadata window.
d) The new "Rename" function allows the renaming of datasets.
e) The new "Metadata" menu allows the manipulation and querying
of system level metadata. Manipulation of two attributes -
"Access Control" and "Comments" has been implemented thus far.
Users now can use these interfaces to grant access permission to
individual users or group of users as well as listing the access
permission of datasets and collections. Recursive operation is
allowed for setting the access permission of a collection. In
addition, users can now use this interface to attach a new "comment"
to a dataset or modify an existing comment of a dataset. The
content of these comments can be used for dataset querying.
2) The recursive option (-r) was added to the Sreplicate command making it
possible to replicate all datasets in a collection recursively.
3) The testsuite script has been modified making it easier to configure.
4) A few bug fixes.
New Features of release 1.1.2
-----------------------------
Features of this release are summarized in the following :
1) The java srbBrowser - This is a WINDOW-like browser. In addition to
browsing collections and datasets, a number of basic operations such as
copying of collections and datasets, exporting/importing of files and
directories to/from local file systems, deleting datasets and collections,
replicating datasets, creating new collections and renaming datasets, etc
are provided. For more details, please read the README.srbBrowser file.
2) Many of the utilities (S commands) have been re-written to include
the -r (recursive) option. These utilities include Sget, Sput, Scp,
Srm, Sls and Schmod. The -l (long form) option has been added to the
Sls command to list some important metadata associated with the datasets
being listed.
3) srbMon daemon - This is a SRB monitoring daemon which remotely monitors
and logs all SRB servers belonging to a federation, and attempts to
restart any servers that were found to be down. For more details, please
read the README.srbMon file.
4) HPSS Class Of Service (COS) configuration - Previously, the HPSS driver
uses a single default COS for all file creation in HPSS. The current
release uses the data/hpssCosConfig file for COS configuration. Based
on the "dataSize" input parameter of the srbObjCreate() call, the HPSS
driver picks the appropriate COS using the table given in the
data/hpssCosConfig file.
5) Version checking - The SRB server now checks whether connecting clients
run the same version of the SRB software. An error will be returned if
there is a mismatch.
6) Easier Host configuration - Previously, every hosts belonging to the SRB
federation must be configured in the data/hostConfig file. Now, only the
MCAT enabled host is needed in the data/mcatHost file. Through this MCAT
enabled SRB server, all other hosts can be obtained from the MCAT. The
data/hostConfig file is now optional and is used to change the default Port
Number associated with a host and/or add aliases to hostNames.
7) The data/vaultConfig is no longer needed. Info on storage vault are now
obtained from MCAT.
8) A number of minor bug fixes.
Content of this release
-----------------------
Directory Comments
--------- ---------------------------------------------------------
./src The source the the SRB code
./obj The SRB object files, the SRB client library.
./bin The SRB server binary and startup script.
./data The SRB server configuration files
./mk The SRB server make configuration
./readme.dir The README directory
./MCAT MCAT installation, configuration and management.
./utilities The SRB client utilities.
./test Client programs, test suites.
./java The java srbBrowser.
./admin The SRB server monitoring program.
Other README files
------------------
faq.html - Frequently Asked Questions.
README.build - Notes on how to build the SRB server and client library.
README.serverconfig - Notes on how to configure and run the SRB server.
README.client - Notes on how to configure a SRB client environment and use
the SRB client API.
README.clientAPI - Gives a summary of the SRB client API.
README.utilities - Notes on Installation and Use of Client Utilities.
README.MCAT.INSTALL - Notes on Installation and Management of the MCAT
catalog.
readme.sea directory - This directory contains several SEA authentication
documentations.
README.srbBrowser - Notes on Installation and Use of the java srbBrowser.
README.srbMon - Notes on Installation and Use of the SRB server monitoring
daemon.
README.tapeMassStor - Notes on Installation of the SRB's own Tape
Mass Storage system.
README.DAI - Notes on usage of the database access interface (DAI).
TLANG.PRIMER - A small primer for the Template Language (T Language).
html/SRB1_1_3.htm - SRB Version 1.1.3 Manual in HTML format.
html/installExample.htm - Example of SRB software installation in HTML format.
