SRB User Manual

From SRB

Contents

SRB - The Storage Resource Broker

This User Manual for release 3.4.0.


Introduction

The Storage Resource Broker (SRB) is a client/server middleware that provides clients with a set of uniform interfaces to distributed and heterogeneous storage resources Storage resources handled by the the SRB include the UNIX file system, archival storage systems such as UNITREE and HPSS, and database Large Objects managed by various DBMS including DB2, Oracle and Illustra.

Client applications are provided with a set of API for sending requests and receiving response to/from the SRB servers. It supports a rich set of client interfaces and tools including:

   C API Library
   Scommands -UNIX like utilities (e.g., ls, cp, chmod, etc) for manipulating and 
   querying collections and datasets in the SRB space.
   inQ - a graphical SRB client for Windows 98/Me/NT/2k/XP.
   MySRB - a web-based browse and search interface to the SRB.
   JARGON - a pure Java API for developing SRB (or other) datagrid interfaces.
   SDSC Matrix - a data grid workflow management system.
   Java based Admin Tool - a java-based graphical (GUI) srb-mcat 
   Administration tool.
   Java SRB_Browser - A Java based browse and search interface to the SRB.
 

The meta data of the SRB system is managed by the MCAT server which is based on the relational database technology. The MCAT server stores meta data associated with data sets, users and resources managed by the SRB. Detailed descriptions on the design of MCAT are given in http://www.npaci.edu/DICE/Software/SRB/mcat.html. Meta data includes information for access control, and data structures required for implementing "collection" (directory) abstraction. Since the MCAT is implemented with relational database technology, it can be extended beyond the capability of traditional file system, such as implementing a more complex access control system, proxy operations, and information discovery based on system level and application level meta data.


The MCAT presents clients with a logical view of data sets stored in the SRB. Similar to the file name in the file system paradigm, each data set stored in SRB has a logical name, which may be used as a handle for data operation. Unlike the file system where the physical location of a file is implicitly implied in its path name through its mount point, the physical location of a data set in the SRB environment is logically mapped to the data sets. Therefore, the actual data of data sets belonging to the same collection may physically reside in different storage systems. A client does not need to remember the physical mapping of a data set. It is stored as the meta data associated with the data set.

Similar to file system paradigm, data sets can be arranged in a directory-like structure, which we call collection. Collections provide a logical grouping mechanism where each collection may contain a group of physically distributed data sets and/or sub-collections.


A number of performance enhencements were made to improve the data access performance:

i) Parallel I/O - designed to improve the performance of transferring large files by using multiple streams for the data transfer.

ii) Bulk transfer - designed to improve the performance of transferring a large number of small files Small files are concatenated into a large buffer first and this buffer is transferred in a single operation to reduce overhead.

iii) Third party transfer - designed to improve the performance of server to server transfer such as data replication which data transfer is carried out directly between the two servers without involving the client.

iv) Container implementation - designed to improve the performance of archiving a large number of small files. Due to the relative high latency of writing a large number of small files to tape, these small files are concatenated into a single container file before writing to tapes.


The SRB supports a rich and fine grain access control mechanism. Owner of a data set can grant access to individual users or groups of users.


The SRB also supports a rich user level metadata scheme including Extensible_Schema that can be associated with data sets so that it can be used to query and locate data.

The Logical resource implementation allows grouping of two or more physical resources into a resource group or logical resource. Logical resources can be used in more than one way. A logical resource can be used to group together a pool of disk cache and appears as one large disk cache, or it can be used to automatically replicate data in each of the physical resource belonging to the logical resource.

The data replication capability of SRB allows data to be replicated sseamlessly across multiples resources. The following gives some details of the implementation:

i) Automatic replication of new data - When creating a data set, a client can specify a logical resource rather than a physical resource as the resource input. With the COPIES=ALL flag set, a copy of the data will be created in each of the physical resource belonging to the logical resource.

ii) Replication of existing data - A user can use the Sreplicate command or the srbObjReplicate() API to replicate existing data to the specified resource.

iii) Replica synchronization - Copies of replica can modified individually and replica of an object can be out of sync. Now, the most recently modified copy will be marked as "dirty". A new command - "Ssyncd" and API - srbSyncData() has been created to synchronize all copies (replica) of an SRB object with the dirty copy. A "%" character to the left of the file name of a "Sls -l" output indicates the given copy is "dirty".

The Proxy operation capability of the SRB allows clients to request cetain operations on data to be performed on the servers to improve efficiency. An example is the copy operation where it is more efficient for the server to do all the read and write operations on behalf of the client than passing the data read to the client and then for the client passing it back to the server for the write operation. A framework for handling proxy operations has been set up to facilitate the building of new proxy functions. These proxy functions can be in the form of either direct codes in the SRB server or individual executables which are accessible by users the the Spcommand utility or the srbExecCommand() API

The Auditing capability of the SRB provides a facility to audit the usage of data sets.


TOP

The SRB Architecture

An Overview of the SRB architecture

The Storage Resource Broker (SRB) is a middleware that provides distributed clients with uniform access to diverse storage resources in a heterogeneous computing Environment.

Image:img001.GIF

Figure 1 gives a simplified view of the SRB architecture. The model consists of three components: the meta data catalog (MCAT) service, SRB servers and SRB clients, connected to each other via network.

The MCAT stores meta data associated with data sets, users and resources managed by the SRB. The MCAT server handles requests from the SRB servers. These requests include information queries as well as instructions for meta data creation and update.

Client applications are provided with a set of API for sending requests and receiving response to/from the SRB servers. The SRB server is responsible for carrying out tasks to satisfy the client requests. These tasks include interacting with the MCAT service, and performing I/O on behalf of the clients. A client uses the same common API to access every storage systems managed by the SRB. The complex tasks of interacting with various types of storage system and OS/hardware architecture, are handled by the SRB server.

TOP '

Implementation details

'

The SRB process model

Image:img002.GIF

Figure 2 depicts the SRB process model. The design of the SRB server is based on the traditional network connected client/server model. It is composed of two separate servers, SRB Master and SRB Server. The SRB Master is the main daemon listening continuously on a well-known port for connection requests from clients. Once a connection from a client is established and authenticated, it forks and execs a copy of the SRB Server, which we call SRB agent, to service the connection. From that point onward, the client and the SRB agent communicate using a different port and the SRB Master goes back to listening for more connections. A client can use the same SRB agent to service multiple requests.

Client applications communicate with the SRB agent using a set of API via TCP/IP sockets. The client library sends requests using pre-defined request stubs to the SRB agent, and receives and parses replies from the SRB agent. The model is distributed in the sense that clients and servers may be running on different hosts.

TOP


Federation of SRB servers

A group of distributed SRB servers coordinating with each other to service client requests, can be configured to form a federation. The advantages of a federated SRB are as follows:

o For technical and policy reasons, various storage systems must run on different hosts.

o Improved performance.

o Improved data reliability and availability ? Replica of data may be stored in different storage systems and on different hosts.

Image:img003.GIF

Figure 3 depicts the working of a federated SRB, which consists of two SRB masters, running on hosts A and B. The SRB server on A is MCAT enabled meaning that it can talk to the MCAT server. In this example, a client "open data set" request may result in the following steps:

1) After completing the connection sequence with the SRB master on host A, the client sends a request to the SRB agent on host A to open a data set for reading.

2) The SRB agent makes a MCAT call passing the client?s user ID and the data set name to the MCAT server to check if the client has the proper access permission to the data set. If it checks out, the MCAT call returns a data structure that contains the physical location where the data set is stored. The physical location data structure includes the host name, the storage system type (e.g., UNIX, HPSS, DB2 Large Object, etc) and the path name (e.g. a unix file path).

3) The SRB agent on host A realizes the requested data set is on host B and carries a remote open on behalf on the client passing along the storage system type and the path name to the SRB agent on host B.

4) The SRB agent on host B uses the storage system type to invoke the appropriate low level storage driver to invoke to handle the open call, passing along the path name to be opened. Upon completion of the open call, the SRB agent on host B returns the opened file descriptor or error code to the SRB agent on host A.

5) If the open call is successful, the SRB agent on host A stores the returned file descriptor and other information such as host name, etc in a internal data structure and pass back the pointer to this data structure to the client. The client can then use this pointer in subsequent read calls; if the open call is not successful, an error code is returned instead.

TOP


Federation of MCAT Zones

A zone is defined as a federation of SRB resources/servers controlled by a single MCAT. Each zone has full control of its administrative domain and can operate independently from other zones. A federation of MCAT zones allows the sharing of data and resources between zones in the federation. Users in one zone with proper permission, can access data and resources in another zone. Special attention was kept to make sure that security compromise in one zone will not significantly affect the security of another zone. http://www.sdsc.edu/srb/matrix/wiki2/index.php/Zones gives more detailed information on the zone design.


Master/Slave MCAT

A SRB federation (zone) can be configured to run with a single Master MCAT plus zero or more Slave MCATs.

The purpose of the Slave MCAT is to improve responsiveness across a Wide-Area-Network. SRB functions that read from the MCAT, and do not update, can then query the Slave MCAT, which can be local. The decisions on which SRB calls go to a Slave (if defined) or Master MCAT is done by some of the SRB code which "knows" which calls are read only. It is up to the administrator to configure Slave MCATs to be local. http://www.sdsc.edu/srb/matrix/wiki2/index.php/Master_Slave_MCAT gives more detailed information on the Master/Slave MCAT design.

TOP


The SRB agent design details

As described above, the SRB is designed based on the traditional client/server model. Client applications are provided with a set of simple API to communicate with the SRB servers. The main daemon, the SRB master, is responsible for the simple tasks of listening for incoming connections, and spawning a SRB agent to handle each connection once the client is authenticated. The SRB agents are responsible for receiving and servicing all subsequent client requests.

Image:img004.GIF

Figure 4 gives a simplified view of the SRB agent design.

At the top is the "dispatcher" module, which listens for incoming client requests and dispatches the requests to the proper request handlers. The "dispatcher" is also responsible for returning the results to clients.

Clients are provided with two sets of API, the high-level and low-level API. The high-level API handles data sets that use the MCAT server for meta data management, while the low-level API handles data sets that do not use the MCAT. When using the high-level API to create a data set, the data set is automatically registered in MCAT and the MCAT keeps all relevant information associated with the data set until the data set is deleted. On the other hand, data sets created using the low-level API are not registered in MCAT. When accessing a data set using the low-level API, a client needs to provide all required information such as the physical location (host address and full path name) and the storage system type of the data set. Detailed descriptions of the client API are given in [#Appendix A Appendix A].

The high-level request handler of the SRB agent handles all high-level client requests, and the low-level request handler handles low-level client requests.

Through the high-level API, a client can access the following type of services provided by the SRB agents:

1) Create and remove data sets.

2) Perform I/O operations on data sets.

3) Manipulate and query meta data associated with the data sets and collections.

4) Manipulate and query meta data associated with other objects such as users, user-groups and resources managed by the SRB.

All these operations require the high-level request handler to interact with the MCAT service to manipulate and query the meta data stored in MCAT. In addition, for requests that perform operations on data set (1) and 2)), low-level requests will be generated and dispatched to the low-level request handler for further processing.

A set of library calls is used by the high-level request handler to interact with the MCAT service. These calls allows the SRB agent to register, unregister and modify meta data associated with data sets, users and storage resources, and to query the meta data. Basically, these routines translate easy-to-understood input parameters into complex SQL queries and send them to the MCAT server. Upon receiving the query results from the MCAT server, these routines parse the query results into simple forms before returning them to the caller.

The type of MCAT call made by the SRB agent depends on the type of client call the SRB agent is trying to handle. For example, to handle a "create data set" call, a query is sent to the MCAT server on the requested storage resource. If the client is allowed to create data set in this storage resource, it returns the physical location (host name, directory path) and storage system type of the requested storage system to the caller. Using the returned directory path, a unique physical path name (e.g., UNIX or HPSS path name) is generated. A low-level "create" call is generated and dispatched, passing on the host name, path name and storage system type parameters to the low-level request handler. If the low-level create call is successful, a "register data set" call is made to the MCAT server, passing on the data set name, the collection in which to put the data set, the storage resource, the physical path name and the user name. The MCAT registration routine does a few sanity checks (such as the permission to create data sets in the specified collection, duplicate data set name, etc). If the registration is successful, a set of meta data associated with the newly created data set is added to the MCAT catalog. Finally, the SRB agent returns a handle to the client. The client can then use this handle for subsequent read/write calls. If the data set register call is unsuccessful, a low-level unlink call is dispatched and an error is returned to the client.

The low-level request handler handles low-level client requests dispatched directly from the request dispatch, or high-level client requests passed indirectly through the high-level request handler module. This module performs the basic I/O operations on behalf of the client on the various storage systems managed by the SRB agent. The common parameters passed to this module are the resource location (host name), the storage system type (UNIX, HPSS, DB2, etc) and the physical path name (UNIX path name). If the requested storage system is not local, it dispatches a remote call to the appropriate SRB agent. Otherwise, it calls its low-level drivers to carry out the I/O operation.

Two types of drivers, the file-type and the DB-type, are supported. The file-type drivers provide UNIX-like file I/O interfaces to the file system of the storage systems, i.e., UNIX, UniTree and HPSS. Other non-file system type drivers, such as FTP and HTTP, are also grouped under this category because the similarities in the I/O interface. I/O operations supported by the file-type drivers are create, open, close, unlink, read, write, sync, seek, stat, chmod, mkdir, opendir, closedir and readdir.

The DB-type drivers provide I/O interface to DB large objects stored in DBMS, i.e., DB2, Oracle and Illustra. . I/O operations supported by the DB-type drivers are create, open, close, unlink, read, write and seek.

TOP


The SRB container design

Due to the relative high overhead of creating/opening files in archival storage systems such as HPSS, they are not suitable for storing large number of small files typically found in digital library systems. The container concept was specifically designed to circumvent this type of limitation. The idea is through the use of containers, many small files can be aggregated in the cache system before storage in the archival storage system. The metadata for translating an inContainer object to file path, offset and size are kept in the MCAT and the SRB I/O drivers have been adapted to handle inContainer objects.

A caching system has been designed to handle container I/O. All container writes are done to the cache copy first. When a client is done filling up the container, a syncContainer call can be made to write the entire container copy to the archival storage system. For reading an object that has been stored in a container (inContainer object), if the container is not already on cache, the SRB server will stage the entire container to the cache first. Reading is done only on the cache copy. The metadata for translating an inContainer object to file path, offset and size are kept in the MCAT and the SRB I/O drivers have been adapted to handle inContainer objects.

The current version of the SRB server has been designed to handle multiple archival and cache storage resources. Multiple storage resources for containers was implemented through the use of logical resource. At the time of container creation, a logical resource is associated with the container being created through the srbContainerCreate() API.

Logical resource is an abstraction that contains one or more physical resources. Physical resource is a SRB abstraction that describe the underlying storage resources that can be accessed by the current SRB server configuration. In MCAT, a set of attributes is associated with a physical resource. Among other things, these attributes identify the location of the resource (host address and the path of the SRB vault), the type of resource (UNIX file system, HPSS file system and DB large objects), and the resource class (cache or permanent, primary or normal), etc.

The logical resource used to create a container should contain both classes of physical resources: "permanent/archival" (e.g., HPSS) and "cache" (e.g., UNIX disk file system) resources. A "cache" copy can be purged while a "permanent" copy is not purgable. Multiple resources for each resource class are allowed in which case, one and only one must be designated as "primary" for each resource class.

All container I/Os are done only to the copy on the "cache" type resource. If a "cache" copy does not exist (purged), the SRB server will attempt to stage a copy from the primary permanent resource to the primary cache resource first. If the primary permanent copy is not available (system or device down), other permanent copies will be used. Similarly, if the primary cache is not available, other cache resources belonging to the same logical resource will be used. It should be noted that the staging is done for the whole container rather than individual container segments. Once a "cache" copy is found or staged, read/write I/O can be carried out on objects contained in the container. If the cache copy has been modified, the srbSyncContainer() call or the Ssyncont command can be used to synchronize the cache copy to the archival resource.

The following software has been developed to support containers:

API - Five new APIs, srbContainerCreate(), srbRmContainer(), srbGetContainerInfo(), srbSyncContainer() and srbReplContainer() have been created to create, remove, query container info, synchronize and replicate(or stage) copies of a container.

Command line utilities - Five command line utilities: Smkcont, Srmcont, Ssyncont, Slscont and Sreplcont to create, remove, sync, list meta-data and replicate (stage) copies of a container. The Slscont command allows the listing of all containers owned or accessible by a user as well as listing all inContainer objects stored in a given container.

TOP

Installing and configuring the SRB servers

The SRB architecture is a federation of servers running on various hosts. The installation of a SRB federation includes:

1) Installation of the MCAT

2) Installation and configuration of Non-MES (MCAT Enabled Server) servers.

3) Installation and configuration of a MES server.

Installation of the MCAT



The following describes how to install an SRB Metadata Catalog (MCAT), i.e. an MCAT-enabled SRB server.

See Databases for background information on the MCAT, how it interacts with database systems, and how to set up the environment for the SRB server to interact with the MCAT database.

If you are using Postgres on Linux, Mac OS X, or Solaris, this entire procedure can be done for you automatically via the install.pl perl script. We recommend you use install.pl, especially if you want to do a basic stand-alone installation. There are many steps, and this script simplifies the procedure considerably. See the beginning of the script for documentation; you will need to set a few parameters by editing the script.

For SRB 3.0 (SRB Zones, the Federated MCAT), some additional steps are needed to set up local and remote zones, and syncronize MCAT information between them. These are described in the Zones writeup (see especially the last section 3.0). The SRB current install.pl script will handle this setup for you, if you'd like to set up a couple MCATs/Zones for testing and experimentation. We recommend this as a safe and efficient way to understand how Zones work. See Zones Testing.

In the following, we assume that there is a database called MCAT installed on either DB2, ORACLE, PostgreSQL, or Sybase (see Build and the mk.config file for more information). Also, the user-id installing SRB and MCAT must have privileges for the creation of objects in the database. (See Databases for how to check this. For Postgres also see below and the Postgres installation documentation.)

The database should have roll-forward-recovery set and backups done so as to enable recovery from crashes. If you give another name for the database, please refer to section on MCAT CONFIGURATION.

Starting with SRB 2.0.0, Postgres (PostgreSQL) can now be used to host an MCAT database. Postgres is open source and is relatively easy to install and operate, so for simple tests and experimentation with the SRB, it may be preferred (unless you have ready access to a managed Oracle or DB2 system).

If you are building on Mac OS X, you may need to alias gmake to make in your .cshrc file:

 alias gmake make

since the SRB makefiles run 'gmake'. On the Macs, make is gmake.

Postgres notes

If you are using older versions of Postgres, such as version 7.2.2 or 7.2.3, ODBC is included, but more recent versions such as 7.3.3 (spring 2003) have a separate release of ODBC, such as psqlodbc-07.03.0100.

For this separate psqlodbc release, copy the unpacked files into your postgres source directory: src/interfaces/odbc. You'll need to mkdir the odbc part.

When you configure Postgres, include --enable-odbc and probably --without-readline, for example:

% ./configure --prefix=/scratch/slocal/pgsql --enable-odbc --without-readline

--without-readline is needed for the separate ODBC if you don't have the readline package installed (whatever that is).

You will need to copy Postgres include files iodbc.h isql.h isqlext.h into the Postgres install include directory before building the srb/mcat software, for example: % cd postgresql-7.3.3/src/interfaces/odbc % cp iodbc.h isql.h isqlext.h /scratch/slocal/pgsql/include/

  1. Begin separate-ODBC additional steps (recent Postgres)

For the separate ODBC versions, in addition to configure, gmake, and gmake install at the top level, you will also need to configure and gmake after you cd to src/interfaces/odbc. For example: % cd src/interfaces/odbc % ./configure --prefix=/Users/schroeder/srb/postgres % gmake

For Linux, you should include --enable-static on the configure line to create static libraries. For example: % ./configure --prefix=/Users/schroeder/srb/postgres --enable-static

On the Mac OS X, this doesn't seem to work right so you need to create the static libraries by hand: % cd src/interfaces/odbc % cp psqlodbc.lo psqlodbc.o # avoid missing _globals % libtool -o libpsqlodbc.a *.o % cp libpsqlodbc.a /Users/schroede/postgres/lib % cd /Users/schroede/postgres/lib % ranlib libpsqlodbc.a

The Linux, the ODBC build has a different (odd) problem, in that it creates a file psqlodbc.a instead of libpsqlodbc.a. The workaround here is to just copy it after you install it. % cp psqlodbc.a libpsqlodbc.a

  1. End separate-ODBC

Before you run srb/mcat software that accesses Postgres (either interactively while installing or when the srb server is started), you need to adjust PATH to include the postgres bin directory, adjust LD_LIBRARY_PATH to include the postgres library directory, and perhaps define a PGDATA environment variable to refer to the postgres directory. (PGDATA is part of Postgres; if defined you don't need to specify the data directory on command lines.) It is convenient to put this in a file and source it. For example:

set path=($path /scratch/slocal/pgsql/bin) setenv LD_LIBRARY_PATH /scratch/slocal/pgsql/lib setenv PGDATA /scratch/slocal/pgsql/data

You also need a .odbc.ini file in the home directory with a few settings including the name of the host running the postgres server, the database name, and username. For example:

% more ~/.odbc.ini
[PostgreSQL]
Debug=0
CommLog=1
Servername=testbrick.sdsc.edu
Database=mcat
Username=schroede

Before starting up postmaster (the postgres server), modify data/pg_hba.conf (for example /scratch/slocal/pgsql/data/pg_hba.conf) to allow the local host to have tcp/ip access to the database. Use nslookup to find the IP address of your SRB host and add it to the file. The line is host all xxx.yyy.zzz.aaa 255.255.255.255 trust For example, for a host with IP 132.249.20.58: host all 132.249.20.58 255.255.255.255 trust For newer versions for Postgres (7.3.3) there is another field so now the form is: host all all xxx.yyy.zzz.aaa 255.255.255.255 trust See the lines at the end of the pg_hba.conf file.

When you run postmaster (the postgres server), include the -i option to enable tcp/ip, for example: % ./postmaster -D /scratch/slocal/pgsql/data -i

    or

% ./pg_ctl start -o '-i' -l /tmp/mypgsql.log

The MdasConfig needs to have postgres as the MDASDBTYPE and PostgreSQL as the MDASDBNAME (the actual database name that you chose (createdb Name) is in the .odbc.ini file). You can use MdasConfig.psg as a template. An example MdasConfig for Postgres is: % more MdasConfig MDASDBTYPE postgres MDASDBNAME PostgreSQL MDASINSERTSFILE /scratch/slocal/srb2_0f/data/mdas_inserts METADATA_FKREL_FILE metadata.fkrel DB2USER schroede DB2LOGFILE /scratch/slocal/Install_f/data/pgsql.log DBHOME /scratch/slocal/pgsql/data


All 


In the following discussion SRBDIR stands for the directory in which the SRB software package has been unpacked for building (i.e. up one level from where this file is).


1) Before building and installing MCAT software, you should build the client and server software at the top level (SRBDIR). See the README.build document for a description of this general SRB build procedure. As part of that SRB build, if you are running an MCAT-enabled SRB, you will select the MCAT DBMS system (oracle, DB2, or postgres) and version. You will also need to enable Javagui (build the Java components) and specify the jdk home (see README.MCAT.ADMIN).

(On Mac OS X, the Javagui currently does not build due to differences in how java is installed there, so skip the --enable--javagui and --enable-jdkhome and see a later section on the command-line admin.)

For example (for Oracle): ./configure --enable-oraver=815 --enable-oraconn=TNS --enable-oramcat --enable-mcatver=20 --enable-orahome=/usr/local/apps/oracle --enable-javagui --enable-jdkhome=/usr/local/apps/jdk1.4.1

Or for Postgres: ./configure --enable-psgmcat --enable-psghome=/scratch/slocal/pgsql --enable-installdir=/scratch/slocal/SRB_Install --enable-javagui --enable-jdkhome=/usr/local/apps/jdk1.4.1

followed by: gmake

Or ('gmake clean' and 'gmake' or 'gmake all')


2) 'cd data' (SRBDIR/data, not SRBDIR/MCAT/data) and update the following files:

      hostConfig
      mcatHost
      MdasConfig

There are comments within these describing the needed fields.

See a later section of this document for more on MdasConfig.

(In 2.0.0 metadata.fkrel and MdasConfig in SRBDIR/MCAT/data/ no longer need to be checked; they will always match what is in SRBDIR/data because configure creates the SRBDIR/MCAT/data files as softlinks into SRBDIR/data.)

3) cd ../MCAT/data (i.e. SRBDIR/MCAT/data)

There are scripts and input files for the supported DBMSs: Oracle (*.ora), DB2 (*.db2), and Postgres (*.psg). In the example below, we use ora but substitute the DB type you are using.

Oracle: sqlplus user/pass@DBName @catalog.install.ora >& myinstall.results.ora (This runs sqlplus taking the input commands from catalog.install.ora)

Or Postgres: psql mcatTest < catalog.install.psg > & myinstall.results.psg

diff myinstall.results.ora install.results.ora

  or

diff myinstall.results.psg install.results.psg Make sure the differences were explainable.

(There are scripts, such as installmcat.ora, that performs step 3 and some of 4 but we recommend doing them as separate steps as described here.)


4) cd .. (to be in SRBDIR/MCAT)

gmake (to build the MCAT utilities and test programs)

cd bin (i.e SRBDIR/MCAT/bin; or else add this bin directory to your path) ../data/test.catalog >& ../data/mytest.results.ora

cd ../data diff mytest.results.ora test.results.ora Make sure the differences are explainable.

There is no test.results.psg file as Postgres results should be the same as Oracle. So: diff mytest.results.psg test.results.ora

Error messages in the results file are normal parts of this test. In fact, the tests are designed to see that correct errors are returned when appropriate. So you can't just look through the results file for errors, but instead must compare it with the provided reference file.

To convert error codes to descriptions, use Serror. For example, to explain error code -3219:

  % ../../utilities/bin/Serror -3219
  DATA_SUBCOLLECTION_NOT_UNIQUE: DATA_SUBCOLLECTION_NOT_UNIQUE

So this means that, most likely, one is attempting to insert an item into the database that already exists.

If the first command fails (test_srb_mdas_create_user), and all commands fail, then there is probably a basic communication problem with the DBMS. Most likely, the MdasConfig file has a problem; recheck it to make sure it is configured for your database, your user id for the database, etc. Recheck the above setup descriptions and README.databases.

Note that you cannot run this script twice and get correct results (unless all commands fail), as some items (such as users) are created and not deleted. If there are problems at this point, you may want to remove the database and reinstall it when you are ready to try again (see uninstalling the MCAT database, below). For Postgres it is easiest to just drop the database and create it again (dropdb and createdb).

Also, you may notice 14 or so "unexpected EOF" debug messages in the DBMS server logs, but these are normal and can be ignored. The test programs, in some cases, exit without closing the connection


5) Create a privileged SRB account using the pre-defined 'srb' user:

Choose a domain name for your SRB system (YOURDOMAIN below), an Admin user name for your domain (YOUR_ADMIN_NAME), and a password for the admin account (A_PASSWD).

Then set up environment variables with for the user 'srb': % setenv srbUser srb % setenv srbAuth CANDO % setenv mdasDomainName sdsc

And create your domain: % bin/ingestToken Domain YOURDOMAIN gen-lvl4 (gen-lvl4 is a pre-defined token that can be used as the parent.)

And then create the admin user account: % bin/ingestUser YOUR_ADMIN_NAME A_PASSWD YOURDOMAIN sysadmin

And unset the environment variables: % unsetenv srbUser srbAuth mdasDomainName

(It would also be possible to create this account with the Srb Admin Tool, but the above is easier, since you would then have to start the SRB server, and then restart it after creating and changing to the admin user.)


6) Update the SRB user's authentication settings.

Using the username and password for the sysadmin account you have just created in step 5, update the ~/.srb/.MdasEnv and ~/.srb/.MdasAuth files for the Unix user under which the SRB server will run.


7) Install and run the SRB server

See the README.INSTALL file, but basically:

cd back to the top level of your source tree (SRBDIR) and: gmake install

cd to the install directory/bin. Then:

 runsrb

(If you are running the srb on an alternative port, configure now updates the srb.h include file to set it, so nothing else is needed.)


8) Use the SRB Admin Tool to complete the setup

Once the SRB server starts up OK, use the Srb Admin Tool (SAT) to change the password of the user 'srb' (the bootstrap admin user). After that, use the SAT to create resources, add users and groups, etc. See readme.dir/README.MCAT.ADMIN.

8b) If you have trouble with Java (OS X, for example), you can use the command line tools for admin (like ingestToken ingestUser above). Creating a local resource so you can Sput, it is fairly straight forward.

You'll need to create a storage directory, into which the data will be stored. We used /scratch/slocal/srb/srbVault/ in the following example.

First you need to create a "location", which describes your server that will handle the resource:

 % bin/ingestLocation 'YOURHOST' 'YOURHOST.INTERET.DOMAIN:NULL:NULL' 'level4' YOUR_ADMIN_NAME YOURDOMAIN

for example:

 % bin/ingestLocation 'miner' 'miner.sdsc.edu:NULL:NULL' 'level4' srb sdsc

Then you create the resource:

 % bin/ingestResource 'resource name' 'unix file system' 'YOURHOST' '/fullpath/?USER.?DOMAIN/?SPLITPATH/?PATH?DATANAME.?RANDOM.?TIMESEC' permanent 0

for example:

 % bin/ingestResource 'ux-miner' 'unix file system' 'miner' '/scratch/slocal/srb/srbVault/?USER.?DOMAIN/?SPLITPATH/?PATH?DATANAME.?RANDOM.?TIMESEC' permanent 0

The above Variables are used by the SRB when creating files in the storage directory, and the above example is a standard way to do it. Sput a file and explore the storage directory to see how it works.


Uninstalling the MCAT database.


Normally, you would never want to uninstall the MCAT database; particularly once users, resources and data are making use of the system. But for testing, one may want to. This is essentially the reverse of step 3. Instead of installing the MCAT tables into the database we are removing them.

uninstallmcat.ora is an example. The key line is to run sqlplus taking the input from data/catalog.cleanup.ora. The catalog.cleanup.ora contains a list of sql commands to remove the MCAT tables. So to remove MCAT, run that command line with the catalog.cleanup.ora (or .db2 or .psg) file for your database system. For example:

sqlplus dbadminuser/passwd @data/catalog.cleanup.ora

For Postgres it is easiest to just drop the database and create it again (dropdb and createdb).



MCAT CONFIGURATION (MdasConfig file)

The file MdasConfig holds all the system-level details tailoring each sites MCAT installation. The file contains the following information:

	MDASDBTYPE        <database-type>
	MDASSCHEMENAME    <schema-name> 
	MDASDBNAME        <database-name>
	MDASINSERTSFILE   <insertion-log-file-name>
	METADATA_FKREL_FILE <schema-semantics-file>
	DB2INSTANCE          <database-instance-name>
	DB2PASSWORD       <owner-password>
	DB2USER           <owner-name>
	DB2LOGFILE        <log-file-name>
	DBHOME            <database-home-directory>

<database-type> can be either db2, oracle, or postgres

<schema-name> If your Oracle or db2 schema is "MCAT" then the value should be 'MCAT.'. Note that a period (.) is needed at the end of the value. Sometimes this is the same as the schema administrator name '<srbadminuser-name>.'. This line ('MDASSCHEMENAME <schema-name>') is not used for postgres but then appears in the .odbc.ini file (see above).

<database-name> mcat or other name that you want to give the catalog database.

<database-instance-name> instance name if needed can be null if not needed

<owner-name> <srbadminuser-name> creating and using MCAT <owner-password> password of above. A few systems allow null passsword and use Unix-authentication. if password is given here make sure that the file is not readable by others.

<database-home-directory> home directory of the database

<schema-semantics-file> set to 'metadata.fkrel'

<insertion-log-file-name> file for logging insertions and updates. can be /dev/null

<log-file-name> file for logging database error messages. Recommened to be in $(SRBDIR)/MCAT/data for the MCAT installation copy and $(SRBDIR)/data for the SRB system copy.

For both DB2, Oracle, and Postgres, the user and passwords are in the fields: DB2PASSWORD and DB2USER. DB2INSTANCE is just for DB2 databases.

See Databases for Oracle and Postgres examples.

TOP

Building the SRB software

If you want to build a complete, stand-alone, SRB system you will need to build an MCAT-enabled server and install the MCAT. Starting with SRB 2.0.0, this is somewhat easier now (although still involved) and can be done using Postgres as the DMBS (which can be freely downloaded and installed). But you should start with the MCAT Install document which includes step by step procedures. It will refer you back here for some of the steps.

Starting with SRB 2.1.1 we now have an install script, install.pl, that can do a complete Postgres, MCAT and SRB installation. See README.MCAT.INSTALL. With SRB 3.0.0, this script can run on Solaris too, as well as the original Linux and Mac OS X.

The SEA authentication system is no longer supported; for the secure authentication use either the Encrypt1 form of MDAS_AUTH authentication, or GSI. For data encryption, see the new data Data Encryption Compression feature.

1) Build configuration Recent versions use the configure script to configure the build. Most of the configurable parameters for building the SRB server and client can be configured using the "./configure" script. Run "./configure --help" for more information.

All configurable parameters for building the SRB server and the client library, including those set by the configure script, are defined in the mk/mk.config.in file. (The configure script automatically generates a third file, mk/mk.config, using mk/mk.config.in as a template.)

Those parameters that cannot be modified via the configure script (because flags for those parameters have not yet been implemented) are set by directly editing the mk/mk.config.in file prior to running "./configure". Comments in the mk/mk.config.in file make it clear whether or not a particular parameter can be set through the configure script, and if so, how to do so.

NOTE: The configure script does a number of self tests before the configuration is carried out. One of the tests it does is the compiler test which it assumes "gcc" as the default compiler. If "gcc" is not installed or if the test of "gcc" failed (which happened on an AIX platform), the configure script should be re-run with the env variable CC set to cc or other compilers. This will override the default in the compiler test.

If the configure script still failed, do the following:

  1. cd SRB2_0_0rel
  2. ./config.rescue
  3. edit the mk/mk.config file

2) Configure examples

  1. Non-MCAT-enabled server and client, type in 'configure'. This will configure the mk.config file to build a non-MCAT enabled server and client with the default settings.
  2. Non-MCAT-enabled server and client with java enabled, type in 'configure --enable-javagui=yes --enable-jdkhome=/usr/local/apps/jdk1.4.1'where /usr/local/apps/jdk1.4.1 is the directory where the JAVA JDK 1.4.1 is installed.
  3. MCAT-enabled server with Oracle 8.1.5 MCAT, type in 'configure --enable-oramcat --enable-oraver=815 --enable-orahome=/usr/local/apps/oracle/product/8.1.7'where /usr/local/apps/oracle/product/8.1.7 is the Oracle home directory.

2) Parameters in the mk/mk.config file

The SRB architecture supports multiple SRB servers running on various hosts. Each SRB server may be built with different options, as set by the configure script and/or defined in the mk/mk.config.in file. For example, the SRB server on host A may include the driver for accessing HPSS and the SRB server on host B may include the driver for accessing the Lobj stored in DB2, etc.

The parameters are self-explanatory through the comments given in this file. Some of the more important parameters are discussed below:

installDir - The absolute path of the SRB install directory.

PORTNAME - The OS platform of this SRB port. Currently, the SRB software runs on 8 platforms. i.e., valid PORTNAME's are : PORTNAME_solaris, PORTNAME_sunos, PORTNAME_linux, PORTNAME_osx, PORTNAME_aix, PORTNAME_alpha, PORTNAME_c90 and PORTNAME_sgi.

SRB_LARGEFILE64 - defines whether the 64 bit file size is supported by the underlining driver of this SRB server. Current, 64 bit file size is supported by the PORTNAME_solaris, PORTNAME_aix, PORTNAME_linux and PORTNAME_c90 platforms.

ORAMCAT - defines that this SRB server being built is MDAS enabled and the MCAT is stored in Oracle DBMS. Normally, only one SRB server is MDAS enabled.

DB2MCAT - defines that this SRB server being built is MDAS enabled and the MCAT is stored in Oracle DBMS. Normally, only one SRB server is MDAS enabled. NOTE : Both ORAMCAT and DB2MCAT cannot be defined at the same time.

ADDR_64BIT - defines whether to compile for 64 bits address. This option has only been tested for the the Solaris and Linux platforms.

PARA_OPR - defines whether this SRB server support parallel operation API.

MDAS_AUTH - defines whether the plain text and encrypted password MDAS authorization scheme will be supported. If used, the user/passwd pair registered with the MDAS catalog will be used to authenticate a user. Comment it out if the SRB server does not support MDAS authorization. NOTE : A server can be built to support either MDAS_AUTH (plain or encrypted ('ENCRYPT1')) or GSI_AUTH, or both.

GSI_AUTH - defines whether the GSI authentication scheme is supported. This is set when --enable-gsi-auth is included on the configure line. NOTE : A server can be built to support either MDAS_AUTH or GSI_AUTH, or both.

LIB_GSI_AUTH - Set by configure when --enable-gsi-auth is included (i.e. GSI_AUTH is defined). LIB_GSI_AUTH specifies where the GSI client libraries are located. The optional configure --enable-globus-location=path can also be used specify the parent location of the GSI libaries, and will cause LIB_GSI_AUTH to be adjusted.

JAVA_GUI and javaDir - JAVA_GUI defines whether the srbBrowser should be built. javaDir specifies the directory where the JDK software is installed. (e.g. /usr/local/apps/Java). See README.srbBrowser for more details.

3) "cd" to the main SRB directory and type in "gmake clean" and then "gmake" to make the SRB software. The Makefile contains various options to make and clean all or a subset of the build.

  • gmake --- build all.
  • gmake clean --- clean all.
  • gmake srb --- build only the SRB server and client.
  • gmake clean_srb --- clean only the SRB server and client.
  • gmake util --- build only the utilities (S commands). See README.utilities for more details.
  • gmake clean_util --- clean only the utilities.
  • gmake browser - build only the java srbBrowser GUI. See README.srbBrowser for more details.
  • gmake clean_browser - clean only the java srbBrowser.

4) (Optional) Type in "gmake install" to install the software in the $(installDir) directory. This procedure installs the following modules in the $(installDir) directory:

  • bin/runsrb - The script that starts the SRB
  • bin/srbMaster2_0_0 - The frontend server.
  • bin/srbServer - The backend server (forked by the srbMaster1_0 for each client connection).
  • bin/libSrbClient.a - The client library.
  • data/hostAuthConfig - The optional (needed only if HOST_BASED_AUTH in the mk.config file is set) host based authorization configuration file.
  • data/mcatHost - This file identifies the host on which the MCAT enabled SRB server is running.
  • data/hostConfig - This is the optional SRB host configuration file. It is only needed when when you want to add aliases to your local hostName.
  • data/hpssCosConfig - This is the optional HPSS Class of Services configuration file. It is only needed if HPSS in the mk.config file is set.
  • data/hpssNodceAuth - The file contains authentication info for non-dce HPSS. It is only needed if the HPSS and NO_DCE flags in the mk.config file are set.
  • data/MdasConfig - The MDAS configuration file.
  • data/metadata.fkrel - This file defines the foreign key relationship between the MDAS catalog tables and is used internally by the SRB for query generation. This file should not be changed between releases.
  • data/LobjConfig - The database configuration file for the DB Large Object driver. Basically, it contains the userID and password for accessing each database server.

TOP

Server Configuration

Overview

SRB server configuration

The SRB architecture is a federation of servers running on various hosts. The installation of a SRB federation includes:

1) Installation of the MCAT - see this link MCAT_Install for details.

2) Installation and configuration of Non-MES (Non-MCAT Enabled Server).

3) Installation and configuration of a MES server.

This note describes the configuration and startng of the SRB servers (MES and Non-MES) after the software has been built as described in this link Build.

Quick setup of a Non-MES (non-MCAT Enabled Server)

This section provides short notes on quick setup and startup of the SRB server. For more detail descriptions of setup, please read the section on #Setup of a MES server.

There is a script included in the release, installServer.pl, which can be used to help install a Non-MES. Please refer to the comments at the top of that for more information.

Setup for servers with UNIX driver only:

1) Build the Non-MES server software according to this link Build.

2) Configure the SRB user environment for the "server user" with the ~/.srb/.MdasEnv and ~/.srb/.MdasAuth files as described in User_Environment.

3) Configure the data/mcatHost file which specifies the address of the MCAT enabled server. See #Setup of a MES server for details.

4) Configure the data/hostConfig file. Normally the configuration of this file is not needed unless the host has a complicated network alias scheme and the SRB server may not be able to get all the aliases associated with the host. When a SRB server is started, it outputs a line in the data/srbLog file specifying all the host name and aliases understood by the server. This information should be checked. See #Setup of a MES server for details.


5) To run the SRB server, type in:

   cd bin
   runsrb

This should start the SRB server. If it is not running, the data/log/srbLog.mmddyy file should give hint on why it is not running.

6) Check the data/log/srbLog.mmddyy file - The first line should contain something like:

   LocalHostName:  orion, localhost, orion.sdsc.edu  .....
   followed by:
   Local storage vault conf:
   resource: demoResc, storSysType: 0, vaultPath: /data/mwan/test/Vault
   resource: demoResc1, storSysType: 0, vaultPath: /data/mwan/test/Vault
   Local Zone :
   ZoneName = sdscbrick8  HostName = srbbrick8  PortNum = 6834
   Remote Zone :
   ZoneName = sdscdemo  HostName = srb.sdsc.edu  PortNum = 6668
   NOTICE:Feb 28 16:26:44: srbMaster version SRB-3.4.0&G is up.


The LocalHostName list shows all the addresses of your local host recognized by the SRB server. It uses the gethostbyname() call to get the local addresses. There are situations where the local host address registered with MCAT may not show up with this call. Then, the "Local storage vault conf" will be empty. In this case, use the data/hostConfig file to include the MCAT registered host address to your local addresses.

Also verify the ZoneName and PortNum.

Setup of a MES server

1) Build the MES server software according to this link Build.

2) Configure the SRB user environment for the "server user" with the ~/.srb/.MdasEnv and ~/.srb/.MdasAuth files as described in User_Environment.

3) Setting up the SRB configuration files - Modifications to the following configuration files are needed for each SRB server installation:

data/mcatHost - This file identifies the host on which the MCAT enabled server is running. This file should contain the following lines:

   line 1 - The host on which the MCAT enabled server is running
   line 2 - The authentication scheme to use when connecting to the MCAT 
            enabled server. Valid schemes are : ENCRYPT1 and GSI_AUTH. 
            The scheme will default to ENCRYPT1 if this line is not present. 
   line 3 - If GSI_AUTH is chosen, this line represents the 
            "Distinguish Name" of the MCAT enabled server user.

data/MdasConfig - The MDAS configuration file. This file is required only by the MES server. This file contains basic informations required for SRB to interface with the MCAT database. Configurable parameters include SRB's database userid, password and log file location.

data/hostConfig - This is the optional SRB host configuration file. It is only needed when when you want to add aliases to your local hostName.

bin/runsrb - This is the startup script for the SRB server. Modifications of some env variables defined in this file may be required. The parameters are self-explanatory through the comments given in this file.

Many of the user environment parameters (srbPort, AUTH_SCHEME, mcatZone, etc) set in the .MdasEnv file (see 2)) can be overridden by setting these parameters in the runsrb script. This may be necessary if the same UNIX account is used to run several SRB servers each with different srbPort and/or AUTH_SCHEME, etc.

Some of the more important parameters are:

   srbPort - defines the port number to use in this zone. It overrides
   the value defined in the .MdasEnv file.
   MaxThread - defines the maximum number of threads to use for parallel
   transfer. The default is 4.
   X509_USER_KEY and X509_USER_CERT specify where the SRB server's 
   GSI key and certificate are located. They are only relevant if
   the server supports GSI authentication.
   GENERATE_GUID - specifies whether a GlobalIdentifier will be
   generated and stored as part of a srbObject registration.
   GftpSeverDn - Valid only if GRID_FTP driver is configured in this 
   server.  This defines the Distinguish Name of the user for the 
   GridFTP server.
   PRE_SPAWN_CNT defines the number of srbServer to prespawn. 
   The purpose of prespawning server is to improve interactivity. 
   SingleSvrPort - configuring SRB servers to use a single port 
   for firewall consideration
   commPortNumStart - Specifies the first allowable port number for SRB
   client/server. This overrides the value specified with the 
   --enable-commstart option of "configure". A zero value disable 
   restriction on port number.
   commPortNumCount - Specifies the number of allowable ports for SRB
   client/server.
   ALLOW_NON_CHECKING_REGISTERY_OF_DATA is an environment variable that
   specifies whether to allow users to register data without checking
   if SRB is allowed to access it. Note that this can be a security hole.
   ServerTimeOut - This allows the srbServer process to timeout and exit if
   it has not received new commands from the connecting client/server for
   the number of seconds specified by ServerTimeOut. 
   logfileInt - The interval in days for switching to a new logfile. The
   default interval is 5 days. i.e., a new logfile is generated every 5 days.


data/data/srbLog.mmddyy - This is the log file of the SRB server. A new log file with file name srbLog.mmddyy (mm = month, dd = day and yy = year) will be created every 5 days by default. It is a good place to check for problems.


5) SRB maintenance mode - The SRB servers can be put into maintenance mode by creating a file named "srb.allow" in the "data" directory. If the "srb.allow" file is empty, the SRB server will not accept any additional connection from anyone. Existing connections are allowed to continue with their works. The "srb.allow" file can be edited as text file to add users to be allowed to make connections during maintenance mode. The format is userName@domainName for each user.

Each individual SRB server can be put into maintenance mode independently from the other SRB servers. If the MCAT server is put into maintenance mode, the whole federation within the zone is effectively put into maintenance mode.





'Bold textItalic text''Italic textBold text=== Setup of server with HPSS driver ===

1) Build the SRB server with the HPSS flag set in the mk/mk.config file. In addition, the HPSS driver can use either the normally used DCE authentication or the NON-DCE implementation developed by Mike Gleicher. Please contact Mike at mkg@san.rr.com for informations regarding the licensing of the Non-DCE HPSS client/server software. If the NON-DCE authentication is used, the NO_DCE flag in mk/mk.config should be set before the build. If it is not set, DCE authentication is assumed.


2) Configure the data/hpssCosConfig file as described in the comment fields in the file. This file configures the HPSS Class Of Service (COS). Basically, this file contains a table with two columns - "COS ID" and "Maximum file size in KBytes". It is used by the HPSS driver for choosing the appropriate COS based on the "dataSize" input parameter of the srbObjCreate() call.


3) Set the variable hpssCOSId in the bin/runsrb to the default class id to be used for creating hpss files.

4) If your installation's hpss library was built with no default servers or location server, you may have to set env variables such as HPSS_LS_NAME for version 4, or HPSS_HPNS_NAME and HPSS_BFS_NAME for version 3 in the bin/runsrb script. Please check with your HPSS administrator regarding these parameters.

5) HPSS authentication configuration

i) For NON-DCE authentication, configure the data/hpssNodceAuth file as described in the comment fields in the file. Basically, this file should contain the HPSS userID/password pair of the SRB user (the userID under which the SRB server runs). . NOTE: The NON-DCE authentication works only with the Non-DCE HPSS client/server implemented by Mike Gleicher.

ii) For DCE authentication, create a DCE keytab file and change the srbKeytabName parameter in bin/runsrb to point to this file path and the srbPrincipalName to the UserId associated with this keytab file.

HPSS setup note from a user

The following gives the install note made by Jean-Yves Nief of IN2P3 for a NON-DCE HPSS server:

1) in mk/mk.config, set the following parameters:

   HPSS=2
   NO_DCE=1

within the sequence: ifdef NO_DCE, set:

 HPSS_LIB_DIR=/opt/hpss/lib 
 HPSS_HDR_DIR=/opt/hpss/include (whenever the HPSS lib and include paths)

and type in "gmake" to make to SRB code.

2) Install for example in "/var/hpss/etc/ndcl.keyconfig" the file containing the non DCE key config file of the HPSS core server you are using. Then set the following env variables

   setenv HPSS_REUSE_CONNECTIONS 1    # not mandatory
   setenv HPSS_DEBUGPATH stderr
   setenv HPSS_DEBUG 0     # certainly interesting to put it at 1 

First time in order to track down possible problems related to HPSS, set:

   setenv HPSS_NDCG_SERVERS cchpstd1  # the name of your core HPSS server
   setenv HPSS_NDCL_KEY_CONFIG_FILE /var/hpss/etc/ndcl.keyconfig.test  
   # file path where the key config file ==> mandatory for authentication.

3) Configure the data/hpssNodceAuth and data/hpssCosConfig files as described above.

Setup of server with GridFTP driver

1) Build the SRB server with the following configurations:

a) Edit the mk/mk.config file and set:

   GRID_FTP = 6

b) Enable GSI authentication with the normal configure options (e.g., configure --enable-gsi-auth --enable-globus-location=<path>)

c) Type in "gmake" to make to SRB code.

2) Use the MCAT admin tool to create a special "Location" for the GridFTP resource. This "Location" is unlike others because more informations are need (The address, port number and Distinguish Name of the GridFTP server) for GridFTP. The Format of the netprefix (Host Address in the Java admin tool) which normally contains the host address of the SRB resource, has been modified as follows:

   SrbResAddr:GridFTPSvrAddr[:GridFTPSvrPortNum][GridFTPSvrDn]


where,

   SrbResAddr = The normal host address of the SRB resource
   GridFTPSvrAddr = The host address of the GridFTP server (required
   for GridFTP driver)      
   GridFTPSvrPortNum = The port number of the GridFTP server
   (Optional).
   GridFTPSvrDn = The Distinguish Name of the GridFTP server user.
   (Optional. If it is not specified, the server will use the
   GftpSeverDn value specified in the "runsrb" script). The 
   advantage of specifying the Dn in the "Location" is the
   SRB server may need to interact with more that one GridFTP servers
   each having a different Dn and using a single Dn specified by
   GftpSeverDn in the "runsrb" script will not be able to support
   such usage.

An example of a netprefix for GridFTP:

   srbbrick7.sdsc.edu:srbbrick3.sdsc.edu:8888:
   /C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb

3) Use the MCAT admin tool to create a "Physical Resource" for the GridFTP resource with the following configurations;

   Location - The special location created above.
   Resource Type - gridFTP (you may need to use the "Add New Token"
   function of the MCAT admin tool to create this Resource Type).

4) The server configured to support GSI authentication (see README.gsi.htm).

5) File ownership in gridFTP resources - Normally, files stored in SRB are physically stored in the resource vault path directory and owned by the SRB server ("server owned"). Users do not usually access these files directly using APIs and commands native to the file system but access them indirectly through the SRB server using SRB APIs and utilities. However, some users have expressed interest in a "user owned" environment where files stored in SRB are physically owned by the users themselves. But currently, SRB servers are not designed to run with special privileges (such as root or chown privileges for UNIX FS type resources) which makes "user owned" environment not feasibly.

However, gridFTP servers are normally run with root privileges and we can use the gridFTP resources to create a "user owned" environment. The current implementation is if a file is stored using the resource's vault path, it defaults to the usual "server owned" environment where the server's GSI credential is used for gridFTP authorization. However, if a file is stored outside of the resource's vault path, it will be in a "user owned" environment where the user's proxy certificate is used for gridFTP authorization. In this case, the SRB user must use the GSI delegation authentication scheme (by setting AUTH_SCHEME to "GSI_DELEGATE").

Starting the SRB server

The servers in the SRB environment must be started in the following order :

       i) The MCAT DBMS server.
       ii) The MCAT enabled SRB server.
       iii) The rest of the SRB servers.

The script bin/runsrb is used to start the SRB server. To start the SRB server, type in :

       . cd bin
       . runsrb


To shutdown the SRB server, type in:

       . cd bin
       . killsrb

SRB maintenance mode

The SRB servers can be put into maintenance mode by creating a file named "srb.allow" in the "data" directory. If the "srb.allow" file is empty, the SRB server will not accept any additional connection from anyone. Existing connections are allowed to continue with their works. The "srb.allow" file can be edited as text file to add users to be allowed to make connections during maintenance mode. The format is userName@domainName for each user.

Each individual SRB server can be put into maintenance mode independently from the other SRB servers. If the MCAT server is put into maintenance mode, the whole federation within the zone is effectively put into maintenance mode.


Configuring server timeout

The server timeout feature allows the srbServer process to timeout and exit if it has not received new commands from the connecting client/server for the predefined period of time. The timeout allows the system to reclaim resources if the client has been idle for a period of time. The timeout is enabled by setting the ServerTimeOut environment value in the "runsrb" script. The ServerTimeOut value must not be too small especially on a MCAT enabled server because the connecting server may be busy transferring a very large file. Internally ServerTimeOut defaults to 3600 sec if it is set to less than this value. Setting ServerTimeOut also trigger a fixed 3600 sec timeout for the connection between the MCAT enabled server and the MCAT database. This allows the MCAT database to reduce the number of active connections. The connection to the MCAT database is re-established when a new request is received.

TOP

Scommands - command line client utilities

Introduction

Scommands are command line utilities that run in the Unix, Windows or Mac OS command shells. Most Scommand names are preceded by an "S". These are Unix-like commands for accessing SRB data and metadata.


Build

The Scommands can be built as a part of the complete SRB Build via the Server Download or as a client-only build http://www.sdsc.edu/srb/tarfiles/SRB3_4_2client.tar which is unencrypted and contains a subset of the SRB3.4.2 release. See the README file for build instructions. After the build, the Scommand executables can be found in the utilities/bin directory. There are also older releases available at http://www.sdsc.edu/srb/tarfiles/SRB3_4_1client.tar and http://www.sdsc.edu/srb/tarfiles/SRB3_4_0client.tar

For Mac OS X, we have pre-build s-commands for download:

 http://www.sdsc.edu/srb/tarfiles/SRB3_5_0clientOSX.tar.gz
 and the previous
 http://www.sdsc.edu/srb/tarfiles/SRB3_4_2clientOSX.tar.gz

For Windows platform, we have pre-built s-commands for download.

 http://www.sdsc.edu/srb/tarfiles/SRBWindowsScommands_331.zip
 http://www.sdsc.edu/srb/tarfiles/SRBWindowsScommands_342.zip
 http://www.sdsc.edu/srb/tarfiles/SRBWindowsScommands_341.zip
 http://www.sdsc.edu/srb/tarfiles/SRBWindowsScommands_340.zip
 http://www.sdsc.edu/srb/tarfiles/SRBWindowsScommands_350.zip

Scommands Usage Help

For more detailed information on individual commands, read the Scommand Manpages online or in the $installDir/utilities/man directory of the software release.


Note files must be installed in c:\srb for windows as should the MdasEnv files (no loading).

Getting started

Setting up the client user environment

Before running any Scommands, the client's User Environment must be set up using the instrustions given in - User Environment.

Simple Examples of Scommands

The following is an example of an SRB session. Standard Unix commands (ls, pwd, cd, etc.) will still work for your local machine during all SRB sessions.

% Sinit [start a new SRB session]

% Smkdir newDir [make an SRB directory "newDir"]

% SgetColl -ahc newDir [display information about "newDir"]

 --------------------------- RESULTS ------------------------------
 user_name :user
 access_constraint :all
 data_grp_name :/home/user.sdsc/newDir
 -----------------------------------------------------------------

% Scd newDir [change to the SRB "newDir" directory]

% Sput file_for_SRB . [put a local file "file_for_SRB" in "newDir"]

% Sls -l [list SRB "newDir" directory contents]

 /home/user.sdsc/newDir:
   user    0 sdsc-unix-mi           12 2002-04-19-15.37  file_for_SRB

% Scd .. [move up one SRB directory]

% Scp newDir/file_for_SRB . [copy "file_for_SRB" to the new directory]

% Spwd [print SRB working directory]

 /home/user.sdsc

% Sls [list SRB home directory contents]

 /home/user.sdsc:
   file_for_SRB
   C-/home/user.sdsc/newDir

% SgetD file_for_SRB [display information about "file_for_SRB"]

 --------------------------- RESULTS ------------------------------
 data_name :file_for_SRB
 data_typ_name :generic
 path_name :/misc/srb/srb/SRBVault/user.sdsc/48/17/file_for_srb
 data_grp_name :/home/user.sdsc
 data_size :12
 phy_rsrc_name :sdsc-unix-misc
 container_name :
 offset :0
 -----------------------------------------------------------------

% Sget file_for_SRB . [write "file_for_SRB" to local machine]

% Sexit [end SRB session] .

Advanced Users

The document gives a quick tutorial of using some of the advanced features of 
Scommand for data movement.

Last updated: March 10, 2005


Table of contents
1) Sequential and Parallel Sput/Sget
2) Parallel I/O for third party transfer operations
3) Bulk Operations
4) Checksuming in SRB
5) Srsync
6) Using replication to backup SRB data
7) Using Container
8) Ssh - The SRB shell
9) Hints for writing scripts that contain Scommands.


1) Sequential and Parallel Sput/Sget
. Default is sequential. e.g.,

> Sput -v lfile
LOCAL:lfile->SRB:lfile | 84.315 MB | 8.706 MB/s | 9.68 s | 2005.02.16 16:52:24 

. "-M" option - Parallel I/O mode for large file transfer. e.g.,

> Sput -vM lfile
LOCAL:lfile->SRB:lfile | 84.315 MB | 13.264 MB/s | 6.36 s | 2005.02.16 16:55:20 | 3 thr

Note the extra "| 3 thr" which means 3 parallel threads were used.

. -m parallel I/O mode - this mode is not recommended (and may be removed at
   some time) because of firewall issues.  -m is server-initiated parallel 
   I/O and -M is client-initiated parallel I/O.

. The -N option can be used to specify the number of threads to use. e.g.,

> Sput -vMN4 lfile
LOCAL:lfile->SRB:lfile | 84.315 MB | 14.042 MB/s | 6.00 s | 2005.03.01 12:03:35 | 4 thr
 
. The options for Sget are the same as those for Sput.

2) Parallel I/O for third party transfer operations such as Sreplicate, Scp, 
Sphymove, Sbackup, SsyncD and Srsync. 

The server will decide which transfer mode to use depending on the
size of the file being transfered. If the file size is larger than 30
MB (a server configuration parameter - SizePerThread), parallel I/O
will be used. Otherwise, Sequential I/O will be used.
 
3) Bulk Operations 

. For transferring large number of small files ( < 1 MB) efficiently.

. Implemented in Sput, Sget, Sphymove and Scp.

. -b option to specify bulk operation. 

. Bulk Sput - this example bulk uploads recursively local directory testdir
to SRB: 

> Sput -bS demoResc testdir testdir1  

. Bulk Sput into a container - this example bulk uploads recursively
local directory testdir into container "cont1":

> Smkcont -S demoResc cont1
> Sput -bc cont1 testdir testdir2

. Bulk Sget - these 2 examples bulk download the two collections just
uploaded:

> Sget -b testdir1
> Sget -b testdir2	# note that the specification of container is not needed

. Bulk Scp - this example bulk copies the collection testdir1 to testdir3:

> Scp -b -S demoResc1 testdir1 testdir3 

. Bulk Sphymove - this example moves all files in the testdir collection 
from the "demoResc" resource to the "demoResc1".

> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc             3000977 2005-03-04-09.14   Sls
  fedsrbbrick8  0 demoResc             2984132 2005-03-04-09.14   foo1

> Sphymove -b -s demoResc -S demoResc1 testdir

> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc1             3000977 2005-03-04-09.14   Sls
  fedsrbbrick8  0 demoResc1             2984132 2005-03-04-09.14   foo1

. Bulk Sphymove into a container - this example moves all files in the testdir 
collection into container cont1:
> Sphymove -bc cont1 testdir
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc             IC  3000977 2005-03-04-13.49   Sls
  fedsrbbrick8  0 demoResc             IC  2984132 2005-03-04-13.49   foo1

The Sphymove command may be a useful tool for the SRB administrator to
move users files from resources to resources.

4) Checksuming in SRB 

. Implemented in both the SRB client and server.

. The checksum parameter is a MCAT metadata associated with each SRB data file. 

. Implemented in Sput, Sget, Schksum and Srsync.

. Uses the SYS V checksuming algorithm.

. Sput -k - uploads a file and registers the checksum value associated
with the file. The checksum value is computed in Sput on the client
side. The following example recursively uploads a local directory
testdir and registers the checksum values.

> Sput -kvr testdir
LOCAL:testdir/foo1->SRB:foo1&REG_CHKSUM=35186 | 3.001 MB | 5.194 MB/s | 0.58 s | 2005.02.16 17:05:37  | chksum=35186
LOCAL:testdir/foo2->SRB:foo2&REG_CHKSUM=56600 | 2.984 MB | 3.856 MB/s | 0.77 s | 2005.02.16 17:05:39  | chksum=56600

. Sput -K - checksum verification. e.g., the following command
recursively uploads a local directory testdir; the server then
computes the checksum by reading back the file just uploaded and
verifies that with the checksum value computed locally.

> Sput -Kvr testdir
LOCAL:testdir/Sls->SRB:Sls&VERIFY_CHKSUM=42018 | 3.001 MB | 5.170 MB/s | 0.58 s | 2005.03.04 16:58:12  | chksum=42018
LOCAL:testdir/foo1->SRB:foo1&VERIFY_CHKSUM=55539 | 2.984 MB | 3.121 MB/s | 0.96 s | 2005.03.04 16:58:13  | chksum=55539

. Sget -k - Download a SRB file and verify the checksum value by
comparing the registered value with the checksum of the local file
just downloaded. e.g.,

> Sget -kvr testdir testdir1
SRB:Sls->LOCAL:testdir1/testdir/Sls | 3.001 MB | 0.734 MB/s | 4.09 s | 2005.03.04 16:59:20  | chksum=42018
SRB:foo1->LOCAL:testdir1/testdir/foo1 | 2.984 MB | 0.810 MB/s | 3.68 s | 2005.03.04 16:59:24  | chksum=55539

. Schksum can be used to list the registered checksum value. e.g.,

> Schksum -lr testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir
    Sls                         42018   
    foo1                        55539   

. Schksum can also be used to compute and register checksum values if
it does not already exist. e.g.,

> Sput -r testdir testdir1	# did not request checksum
> Schksum -lr testdir1		# list the checksum
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1
    Sls                         0       
    foo1                        0       

> Schksum -vr testdir1		# compute the checksum if it does not exist 

/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1
    Sls                         42018      new_chksum
    foo1                        55539      new_chksum

> Schksum -fvr testdir1		# force computation and registration of checksum
				# even though they exist
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1
    Sls                         42018   
    foo1                        55539   

. Srsync - The Srsync command uses the checksum value to sync
data. This command will be discussed in detail in 5).

5) Srsync - Data synchronization between client's local files and SRB
files, and between SRB files. If a good copy already exists in the target, no file transfer will be done.
 
. Three modes of synchronization:
 
	a) From local files to SRB files. e.g.,

	    > Srsync -Mvr testdir s:testdir1

	Note that the target argument is prepended with "s:" indicating the
	target is an SRB file/collection. The "M" option specifies that in
	case an upload is needed, parallel mode I/O will be done.  The "v" is
        for verbose.
	
	b) From SRB files to local files. e.g.,

	    > Srsync -Mvr s:testdir1 testdir

	c) SRB files to SRB files. e.g.,

	    > Srsync -vr s:testdir1 s:testdir2

	Note that the "M" option is not needed because this involves server to
	server third party operations where the servers will decide the mode
	of data transfer.

. Synchronization between source and target is based on comparing the
checksum values or the files sizes. The checksum comparison is the
default method, and the much faster but less accurate files size
comparison method can be specified with the "-s" option. e.g.,

Srsync -mvrs s:testdir1 testdir

6) Using replication to backup SRB data 

. Commands that involves replica - Sreplicate, Sbkupsrb, Ssyncd, Sget, Sput
and Scp.
 
. Each SRB file can have multiple physical copies (or replica) and
these copies can be stored in different resources.

. Sreplicate - make a replica. e.g., the following example makes replica in 
resource demoResc1:

> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-04-16.58 % Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-04-16.58 % foo1
> Sreplicate -rS demoResc1 testdir
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-04-16.58 % Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-09.38 % Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-04-16.58 % foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-08-09.38 % foo1

The second column on the output is the replica number. Another copy will be
made if Sreplicate is run again. e.g.,

> Sreplicate -rS demoResc1 testdir
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-04-16.58 % Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-09.38 % Sls
  fedsrbbrick8  2 demoResc1                3000977 2005-03-08-09.41 % Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-04-16.58 % foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-08-09.38 % foo1
  fedsrbbrick8  2 demoResc1                2984132 2005-03-08-09.41 % foo1

. Sbkupsrb - Make a backup replica of a file to the specified "Backup
Resource".
It differs from Sreplicate in that if a good copy already exists in
the "Backup Resource", nothing will be done, while Sreplicate will
keep on making new replica. Example:

> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-10.25   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-10.25   foo1

> Sbkupsrb -r -S demoResc1 testdir
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-10.25   Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-10.30   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-10.25   foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-08-10.30   foo1
> Sbkupsrb -r -S demoResc1 testdir
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-10.25   Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-10.30   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-10.25   foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-08-10.30   foo1

Note that the second Sbkupsrb does nothing because a good copy already
exists in demoResc1.

. Ssyncd - Synchronize all copies of a file with the last modified copy. e.g.:

> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-10.37   Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-10.38   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-10.37   foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-08-10.38   foo1

The following command overwrites replica 0 of foo1:

> Sput -n 0 -S demoResc foo2 testdir/foo1
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-10.37   Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-10.38   Sls
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-10.42 % foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-08-10.38   foo1

Note that the size of foo1 for n=0 changed to 2988703 and the "%" in the 
sixth column indicates that it is the most recently modified copy.  

> Ssyncd -r testdir
>  Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-10.37   Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-08-10.38   Sls
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-10.42 % foo1
  fedsrbbrick8  1 demoResc1                2988703 2005-03-08-10.46 % foo1

Note that the copy of foo1 in resource demoResc1 has been updated.

Ssyncd can also be used to broadcast a copy to all the resources in a 
logical resource using the -a option. e.g.,

> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc2                3000977 2005-03-14-17.12 % Sls
  fedsrbbrick8  0 demoResc2                2984132 2005-03-14-17.12 % foo1
  fedsrbbrick8  0 demoResc2                2988703 2005-03-14-17.12 % foo2
> Ssyncd -r -aS demoLResc testdir
> Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc2                3000977 2005-03-14-17.12 % Sls
  fedsrbbrick8  1 demoResc1                3000977 2005-03-14-17.14 % Sls
  fedsrbbrick8  2 demoResc                 3000977 2005-03-14-17.14 % Sls
  fedsrbbrick8  0 demoResc2                2984132 2005-03-14-17.12 % foo1
  fedsrbbrick8  1 demoResc1                2984132 2005-03-14-17.14 % foo1
  fedsrbbrick8  2 demoResc                 2984132 2005-03-14-17.14 % foo1
  fedsrbbrick8  0 demoResc2                2988703 2005-03-14-17.12 % foo2
  fedsrbbrick8  1 demoResc1                2988703 2005-03-14-17.14 % foo2
  fedsrbbrick8  2 demoResc                 2988703 2005-03-14-17.14 % foo2

Note that the logical resource contains physical resources demoResc and demoResc1.
The Ssyncd -a command makes a copy in each of these two resources.
 
. Sget/Sput - The "-n replNum" option is used to specified the replica for
which the operation is to be performed. e.g.,

> Sput -n 0 -S demoResc foo2 testdir/foo1
> Sget -n 1 testdir/foo1

. For Sget, if there are more that one copy of a file and the "-n
replNum" option is not used, the server will pick the copy to be
downloaded based on the following rules:

i) Always pick copy stored in cache/temporary class first and
archival/permanent class last.

ii) within the same class, pick the copy that is stored on the same host
as the host where is client is connected to first.

iii) If i) and ii) are the same, then pick one randomly.
  

7) Using Containers

. Container - Physical grouping of small files so that they can be
more efficiently stored on tapes.

. Commands that involves containers - Smkcont, Srmcont, Slscont, Ssyncont, 
Sreplcont, Sput, Sget, Scp and Sphymove

. Smkcont - Make a container. e.g.,

> Smkcont -S demoResc cont1
> Slscont -l cont1
  Container Name                                                          Resource         Size        MaxSize     Count  
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc         0           209715200   0      

. Put files into container. e.g.,

> Sput -r -c cont1 testdir testdir1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  0 demoResc             IC  3000977 2005-03-08-11.30   Sls
  fedsrbbrick8  0 demoResc             IC  2984132 2005-03-08-11.30   foo1

Note that the 4th column "IC" indicates the file is stored in a
container.
 
. Slscont - shows information about a container. e.g.,

> Slscont -l cont1
  Container Name                                                          Resource         Size        MaxSize     Count  
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc         5985109     209715200   2      

Note that "Count" is 2 meaning there are 2 files stored in container
"cont1".

> Slscont -F cont1 	# list all files stored in container "cont1"
Containers:
/sdscbrick8/container/fedsrbbrick8.sdsc/cont1
  InContainer Objects                                                                                   Offset      Size      
  /sdscbrick8/home/fedsrbbrick8.sdsc/testdir1/Sls                                                       0           3000977   
  /sdscbrick8/home/fedsrbbrick8.sdsc/testdir1/foo1                                                      3000977     2984132   

. Srmcont - Remove a container. e.g.,

> Srmcont cont1
Srmcont: Error removing container '/sdscbrick8/container/fedsrbbrick8.sdsc/cont1': -3295
CONTAINER_NOT_EMPTY: CONTAINER_NOT_EMPTY

Note that the container is not empty and hence the error message. The "-f" option
will force the removal of all files stored in the container first and then
remove the container. e.g.,

> Srmcont -f cont1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:

. Ssyncont - Synchronization of container copies. Typically, containers are
created using a logical resource which contains two or more physical resources.
In this example, our logical resource "demoLResc" contains two resources:
"demoResc1" which is a "cache" class resource and "demoResc" which is a 
"permanent" class resource. e.g.,

> SgetR demoLResc
Querying Zone=sdscbrick8
--------------------------- RESULTS ------------------------------
rsrc_name: demoLResc
netprefix: srbbrick8.sdsc.edu:NULL.NULL
rsrc_typ_name: logical
default_path: 
phy_default_path: /data/mwan/test/Vault/?USER.?DOMAIN/?SPLITPATH/?PATH?DATANAME.?RANDOM.?TIMESEC
phy_rsrc_name: demoResc1
rsrc_typ_name: unix file system
rsrc_class_name: cache
user_name: fedsrbbrick8
domain_desc: sdsc
zone_id: sdscbrick8
-----------------------------------------------------------------
rsrc_name: demoLResc
netprefix: srbbrick8.sdsc.edu:NULL.NULL
rsrc_typ_name: logical
default_path: 
phy_default_path: /data/mwan/test/Vault/?USER.?DOMAIN/?SPLITPATH/?PATH?DATANAME.?RANDOM.?TIMESEC
phy_rsrc_name: demoResc
rsrc_typ_name: unix file system
rsrc_class_name: permanent
user_name: fedsrbbrick8
domain_desc: sdsc
zone_id: sdscbrick8
-----------------------------------------------------------------
 
When a file is uploaded into a container, it will be put into the container copy
on the "cache" resource first. Ssyncont synchronizes the "permanent" copy of the  
container with the "cache" copy. e.g.,

> Smkcont -S demoLResc cont1
> Sput -r -c cont1 testdir testdir1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  0 demoResc1            IC  3000977 2005-03-08-12.12   Sls
  fedsrbbrick8  0 demoResc1            IC  2984132 2005-03-08-12.12   foo1


Note that the copies are in "demoResc1" which is a "cache" class resource.

> Ssyncont cont1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  0 demoResc1            IC  3000977 2005-03-08-12.12   Sls
  fedsrbbrick8  1 demoResc             IC  3000977 2005-03-08-12.12   Sls
  fedsrbbrick8  0 demoResc1            IC  2984132 2005-03-08-12.12   foo1
  fedsrbbrick8  1 demoResc             IC  2984132 2005-03-08-12.12   foo1
> Slscont -l cont1
  Container Name                                                          Resource         Size        MaxSize     Count  
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc         5985109     209715200   2      
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc1        5985109     209715200   2      

Note that we now have two copies of the container.

> Sput -c cont1 foo2 testdir1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  0 demoResc1            IC  3000977 2005-03-08-12.12   Sls
  fedsrbbrick8  1 demoResc             IC  3000977 2005-03-08-12.12   Sls
  fedsrbbrick8  0 demoResc1            IC  2984132 2005-03-08-12.12   foo1
  fedsrbbrick8  1 demoResc             IC  2984132 2005-03-08-12.12   foo1
  fedsrbbrick8  0 demoResc1            IC  2988703 2005-03-08-12.17   foo2

Note that foo2 is now only in the "demoResc1" which is a "cache" class resource.

> Ssyncont -d cont1	# "-d" means delete the 'cache" copy after sync 
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  1 demoResc             IC  3000977 2005-03-08-12.12   Sls
  fedsrbbrick8  1 demoResc             IC  2984132 2005-03-08-12.12   foo1
  fedsrbbrick8  1 demoResc             IC  2988703 2005-03-08-12.17   foo2
> Slscont -l cont1
  Container Name                                                          Resource         Size        MaxSize     Count  
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc         8973812     209715200   3      

Note that after "Ssyncont -d", there is only one container copy.

. Family of containers - When a container is full, the SRB server automatically 
renames the full container "myContainer" to "myContainer.NNNN" where NNNN is a 
random integer number. The server then creates an  empty  container  with  the
original container name "myContainer". In effect, this scheme creates a family of 
containers and the container never appears to run out of space from a client
standpoint. The "-a" option is used indicate that the operation is for the 
whole family. e.g.,

> Smkcont -S demoLResc -s 4000000 cont1 # container size is 4 MB so it will overflow
> Sput -r -c cont1 testdir testdir1
> Slscont -la cont1	# "-a" to indicate listing the whole family 
  Container Name                                                          Resource         Size        MaxSize     Count  
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc1        2988703     4000000     1      
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1.1110315260                demoResc1        5985109     4000000     2      
 
Note the first container is full and has been renamed to "cont1.1110315260"

> Ssyncont -a cont1 	# sync the family of container
> Slscont -la cont1
  Container Name                                                          Resource         Size        MaxSize     Count  
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc         2988703     4000000     1      
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1                           demoResc1        2988703     4000000     1      
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1.1110315260                demoResc         5985109     4000000     2      
  /sdscbrick8/container/fedsrbbrick8.sdsc/cont1.1110315260                demoResc1        5985109     4000000     2      

> Srmcont -af cont1	# Remove all containers in the family

. Bulk operations with containers - containers also work with bulk operations.

Example - Bulk Sput into a container:
> Smkcont cont1
> Sput -bc cont1 testdir testdir1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  0 demoResc             IC  3000977 2005-03-08-13.07   Sls
  fedsrbbrick8  0 demoResc             IC  2984132 2005-03-08-13.07   foo1
  fedsrbbrick8  0 demoResc             IC  2988703 2005-03-08-13.07   foo2

Example - Bulk Phymove into a container
> Sput -b testdir testdir2
> Sls -l testdir2
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir2:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-13.10   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-13.10   foo1
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-13.10   foo2

> Sphymove -bc cont1 testdir2	# bulk move files in testdir2 into container "cont1"
> Sls -l testdir2
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir2:
  fedsrbbrick8  0 demoResc             IC  3000977 2005-03-08-13.10   Sls
  fedsrbbrick8  0 demoResc             IC  2984132 2005-03-08-13.10   foo1
  fedsrbbrick8  0 demoResc             IC  2988703 2005-03-08-13.10   foo2

. Associating a container with a collection - The "Smkdir -c" command can
be used to associate a container with the collection being made. Then all
files placed in this collection will be put into the associated container.
e.g.,

> Smkcont cont1
> Smkdir -c cont1 testdir1
> Sput foo1 testdir1
> Sls -l testdir1
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir1:
  fedsrbbrick8  0 demoResc             IC  2984132 2005-03-14-17.32   foo1
> Slscont -F cont1
Containers:
/sdscbrick8/container/fedsrbbrick8.sdsc/cont1
  InContainer Objects                                                                                   Offset      Size      
  /sdscbrick8/home/fedsrbbrick8.sdsc/testdir1/foo1                                                      0           2984132   

Note that the new file "foo1" is now stored in container "cont1".

8) Ssh - The SRB shell. Similar to UNIX shells, Ssh executes commands (both normal 
UNIX commands and S-commands) read from a terminal or a file. The advantage of
using Ssh instead of the normal sequence of Sinit->Scommands...->Sexit is
the Ssh maintains the connection to the SRB server until the Ssh shell is
terminated which greatly reduces the connection overhead and improves interactivity.
In the normal Sinit->Scommands...->Sexit sequence, a new connection to the SRB 
server is made for each individual Scommand.  

. Sinit and Sexit is NOT needed when running Ssh.
. The Ssh can be run interactively as a standalone shell. e.g.,
> Ssh		# type in Ssh to start the shell. A "Ssh >" prompt will be displayed
Ssh > Sput -b testdir	# start type in Scommands
Ssh > Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-15.26   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-15.26   foo1
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-15.26   foo2
Ssh > Sls -l testdir > foo
Ssh > more foo
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-15.26   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-15.26   foo1
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-15.26   foo2
Ssh > rm foo
Ssh > exit

The drawback of the standalone Ssh is it does not support some of the UNIX shell
specific commands and properties. e.g.,
Ssh > setenv foo foo1
sh: setenv: command not found

. Using Ssh to run a UNIX shell - To get around the above problem, use the Ssh to 
run a UNIX shell to create an SRB enabled interactive shell. e.g.,

> Ssh -c bash
bash-2.05a$   # Now we are in the bash interactive shell. This bash shell 
	      # inherited a connection to the SRB server from Ssh 
bash-2.05a$ Sput -b testdir
bash-2.05a$ Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-15.48   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-15.48   foo1
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-15.48   foo2
bash-2.05a$ !Sls	# it understands the bash special char '!'
Sls -l testdir
/sdscbrick8/home/fedsrbbrick8.sdsc/testdir:
  fedsrbbrick8  0 demoResc                 3000977 2005-03-08-15.48   Sls
  fedsrbbrick8  0 demoResc                 2984132 2005-03-08-15.48   foo1
  fedsrbbrick8  0 demoResc                 2988703 2005-03-08-15.48   foo2
bash-2.05a$ Sget -r testdir1
bash-2.05a$ ls -l testdir1	# It can do normal UNIX command
total 8768
-rw-r-----    1 mwan     sys200    3000977 Mar  8 15:53 Sls
-rw-r-----    1 mwan     sys200    2984132 Mar  8 15:53 foo1
-rw-r-----    1 mwan     sys200    2988703 Mar  8 15:53 foo2
bash-2.05a$ foo=foo1	# execute commands specific to bash
bash-2.05a$ export foo
bash-2.05a$ exit	# exit the srb enabled bash shell

. The SRB enabled UNIX shell can only be created for shells that are
based on the Bourne Shell, e.g., sh, bash, ksh and perl. Shells that
are based on the C shell such as csh and tcsh will NOT work because
these shells do not allow the inheritance of file descriptors
(connection to the SRB server) from their parents.

. Using Ssh to run script, e.g.,
> Ssh -c myShellScript

where myShellScript is a shell script written in Bourne Shell based shells. 
myShellScript should not contain any Sinit nor Sexit statements. 

9) Hints for writing scripts that contain Scommands. There are a couple ways
to do it:

    i) Ssh -c myShellScript
    This will use a single connection to the SRB server to execute all
    Scommands in the script. 

    ii) Sinit->Scommands...->Sexit sequence in the shell script.  
    This will work most of the time but with some combinations of OS and
    UNIX shell, the Scommand may be confused about the session ID. In this
    case, do the following:

        a) cd to ~/.srb and type in "cp .MdasEnv .MdasEnv.script"

	b) setenv mdasEnvFile ~/.srb/.MdasEnv.script

TOP

The SRB client API

'

Overview

Clients are provided with a set of API for sending requests and receiving response to/from the SRB servers.

A majority of the API performs some kind of operations on data sets. These APIs can be subdivided into 2 classes: high-level APIs and low-level APIs. The high-level APIs handle requests from clients for operations on data sets and resources that are in their logical represenatations. A typical high-level API handler queries the MCAT to convert the logical representations to their physical representations and then calls the low-level APIs to perform low level functions. In another words, high-level APIs handle requests with inputs in their logical representations while the low-level APIs handle physical representations type requests.


A normal client will only use the high-level APIs to interact with the SRB while the low-level APIs are mostly used for server to server type requests. In fact, most low level APIs are privileged operations. Only the high-level APIs will be described in this document.


The rest of the API handles client/server connections, interacts with MCAT for information query and meta data creation and update.

TOP


Description of API

A brief description of the SRB client API is given below.


1) Client/server connection API ? This set of API handles opening and closing connection to a SRB server.

srbConnect - Initiate a connection to a SRB server. This call is replacing clConnect.

clConnect - Initiate a connection to a SRB server. This call is being replaced by srbConnect.

tiUserConnect - Initiate a connection to a SRB server by a TICKET USER. Normal authentication will be bypassed for a TICKET USER.

clFinish - Close the current connection and free the srbConn data structure.

clErrorMessage - Return the ErrorMessage of a connection. This is the error message returned by the SRB server from the current client call.


2) High-Level API:

srbObjOpen - Open a SRB data set.

srbObjOpenWithTicket - Open a SRB data set using a ticket.

srbObjCreate - Create a SRB data set.

srbObjClose - Close an opened data set.

srbObjUnlink - Unlink a data set.

srbObjRead - Read a block of data from a data set into buffer.

srbObjWrite - Write content of buffer to a data set.

srbObjSeek - Change the current read or write location on an data set.

srbObjSync - Sync an opened SRB object.

srbObjStat - Stat an SRB object.

srbObjStat64 - Stat an SRB object and put results in a stat64 struct.

srbObjGetdents - The SRB equivalent of the UNIX getdents call.

srbObjGetdents64 - The SRB equivalent of the UNIX getdents64 call.

srbObjProxyOpr - Proxy Operation request.

srbObjProxyFunct - Proxy Function request.

srbObjReplicate - Make a copy of a data set.

srbObjMove - Move a data set from one resource to another.

srbCreateCollect - Create a SRB collection.

srbListCollect - List a SRB collection.

srbModifyCollect - Modify a SRB collection.

srbIssueTicket - Issue a ticket.

srbRemoveTicket - Cancel a ticket.

srbRemoveTicketWithZone - Cancel a ticket.

srbContainerCreate - Create a container.

srbGetContainerInfo Get container info.

srbRmContainer - Remove a container.

srbSyncContainer - Synchronize copies of a container.

srbContainerOpen - Open a container for ingestion.

srbContainerClose - Close an opened container.

srbReplContainer - Replicate/stage a container.

srbObjGet - Download a file from SRB to local file system using parallel I/O.

srbObjGetC - The client initiated version of srbObjGet..

srbObjPut - Upload a file from local file system to SRB using parallel I/O.

srbObjPutC - The client initiated version of srbObjPut.

srbObjCopy - Copy a data set from one SRB path to another SRB path..

srbSyncData - Synchronize all copies (replica) of an SRB object with the most recently modified version

srbBulkRegister - Register inContainer object in bulk.

srbExecCommand - Remote execution of command.

srbExecCommandC - Remote execution of command - client initiated.

srbExecFunction - Proxy Operation that exec a server builtin function.

srbBulkLoad - Bulk upload a set of data files.

srbBulkUnload - Bulk download of non-container files in a collection recursively.

srbBulkUnloadC - The client initiated socket conection version of Bulk download of non-container files in a collection recursively.

srbBulkCopy - Bulk copy recursively from a collection to another.

srbObjChksum - Checksum a SRB data file.

srbGetFsFreeSpace - get the Vault File System free space.

srbObjFStage - Stage a SRB data file. Only applicable to samfs resource

srbBackupData - Backup a data object - Make a replica to the backup resource.

srbGetMcatZone - Get the Mcat Zone of this user

srbGetSvrVersion - Get the Server version

srb_perror - Emits an SRB error message string corresponding to the input error code.



2) API that interact with MCAT ? This set of API interacts with MCAT for information query and meta data creation and update.

srbGetDatasetInfo - Get some default Info on a SRB data set.

srbGetDataDirInfo - Get arbitrary meta data info on users, data sets or resources by querying the MCAT catalog.

srbGetDataDirInfoWithZone - Get arbitrary meta data info on users, data sets or resources by querying the MCAT catalog with an additional zone input.

srbGenQuery - This is a more compact form of srbGetDataDirInfo. Instead of using fixed array of selval and qval, it uses the genQuery struct.


srbRegisterDataset - Register a SRB data object.

srbUnregisterDataset - Unregister a SRB data object.

srbSetAuditTrail - Setting and Unsetting Audit Trail.

srbModifyDataset - Modify a SRB data set

srbChkMdasAuth - Authenticate a userName/passwd.

srbChkMdasSysAuth - Authenticate a userName/passwd for sys admin access.

srbRegisterUserGrp - Register a user group

srbRegisterUser - Register a user.

srbModifyUser - Modify a user info.

srbModifyExtMetaData - Modify an extensible metadata table in MCAT.

srbGetPrivUsers - Read the privileged users list.

srbGetMoreRows - Get more rows of result from a srbGetDatasetInfo, srbGetDataDirInfo, srbListCollect or srbGetPrivUsers call.

srbGenGetMoreRows - This is a more compact form of srbGetMoreRows.

freeSqlResult - Free any memory associated with a SQL result struct including the SQL result struct itself.

clearSqlResult - Clear the memory associated with a SQL result struct, but the SQL result struct is not freed.

printSqlResult - Print the content of a SQL result struct.

srbRegisterLocation - Register/create a new location.

srbIngestToken - Ingest/create a new Token of type ResourceType, DataType,UserType, Domain, or Action.

srbRegisterResource - Register/create a new resource (physical or compound).

srbRegisterLogicalResource - Register/create a new logical resource.

srbRegisterReplicateResourceInfo - Add another physical resource to a logical or compound resource.

srbDeleteValue - Delete/remove a value of type location, user, or resource (other types are implemented but untested).

srbSetupSessionPublicKey - Get the MCAT-enabled server's public key in preparation for transferring encryptioned information.

srbSetupSession - set up a session (for encryption) with the MCAT-enabled server.


TOP

Setting up Client Environment

'

Building and installing the SRB client library

This section describes the steps for building and installing the SRB client library ? libSrbClient.a.

1) Build and install the SRB client/server software described in http://www.sdsc.edu/srb/matrix/wiki2/index.php/Installation

2) Type in "gmake install" to install the software in the $(installDir) directory. This procedure installs the following module in the $(installDir) directory:

bin/libSrbClient.a - The client library.

TOP


Setting up the client user environment

(This use to be the readme.dir/README.utilities document.)


Before a SRB client (Scommands or C client API or Jargon) can connect to the SRB server, the SRB user environment must be setup. There are two ways to accomplish this:

1) Have an environment file called .MdasEnv.

This file is located in the .srb sub-directory in your home directory. (Windows has difficulty creating a file or directory beginning with "." so the Windows Scommands look for the file in your home directory\srb. i.e. C:\Documents and Settings\myUserName\srb. The Jargon API will look for either directory. This file contains information for initializing the SRB client environment.) (A template of the MdasEnv file) This is a text file that contains lines of parameter/value pair where each values are in quote. The parameters are given in the following:

i) mdasCollectionHome - This is equivalent to the user's home directory (collection) and is created by the MCAT administrator at the time of the user registration.
ii) mdasDomainHome - This is the domain name associated with the user. This domain name is assigned to the user at the time of the user registration.
iii) srbUser - This is the user name of the user. This name is assigned to the user at the time of the user registration.
iv) srbHost - This is the default host address of the SRB server when a SRB client initiates a SRB connection. This parameter can be overridden at run time by setting the environment variable 'srbHost'.
v) srbPort (optional) - This is the port number of the SRB server.
If 'srbPort' is not specified, it will default to port number 5544.
vi) AUTH_SCHEME (optional) - This parameter defines the authentication scheme to be used. Valid input values are:
'ENCRYPT1' - A password scheme with the passwords encrypted going from clients to servers.
'GSI_AUTH' - Use the GSI authentication scheme. If this option is chosen, an additional parameter SERVER_DN given below is required.
'GSI_DELEGATE' - Use the GSI Delegation (proxy) certificate for authentication. The advantage is that this certificate can be passed from server to server whereby the user's identity continues to be maintained across servers and across zones. This scheme solves the cross zone authentication issues. A slight drawback is that the overhead is somewhat higher than the normal 'GSI_AUTH' scheme. If this option is chosen, an additional parameter SERVER_DN given below is required.
'GSI_SECURE_COMM' - Use the GSI authentication scheme and use the GSI I/O library for all socket communication between client and server. If this option is chosen, an additional parameter SERVER_DN given below is required. If this parameter is not defined, the 'ENCRYPT1' Authentication scheme will be used.
vii) SERVER_DN(optional) - This is the "Distinguish Name" of the user running the SRB server. Value for this parameter can be obtained from the SRB administrator. This input is meaningful only if the authentication scheme 'GSI_AUTH' or 'GSI_SECURE_COMM' is chosen.
viii) defaultResource (optional but recommended) - The reosurce to use if a user does not specify a resource in some Scommands (Sput, Scp, Sreplicate, etc).
ix) mcatZone (optional but recommended) - The Zone associated with this user. If this parameter is not given, the SRB server will query the MCAT for it. But this will add to the overhead.


2) Use the UNIX environment variables to specify these parameters.

This method of specification overrides those specified using the .MdasEnv file. Environment variables recognized by the Scommands are:

i) srbUser - The client user
ii) mdasDomainName - The domain of the client user
iii) srbHost - The hostname of the SRB-server
iv) srbPort - The port number of the SRB-server (optional)
v) mdasResourceName or defaultResource - The default resource
vi) mdasCollectionName - The current working collection.
vii) srbAuth - The password.
viii) AUTH_SCHEME - The authentication scheme to be used.
ix) SERVER_DN - The distinguished name of the server user (valid only for GSI authentication).
x) mcatZone (optional but recommended) - The Zone associated with this user.


Other relevant environment variables :

i) mdasEnvFile - Instead of the .srb/.MdasEnv file, this env variable specifies the alternative path for this file.
ii) mdasAuthFile - Instead of the .srb/.MdasAuth file, this env variable specifies the alternative path for this file.


Setting up user authentication

The SRB software can be built to handle ENCRYPT1 by default or to include GSI_AUTH if the --enable-gsi-auth option of the "configure" command is used when configuring the software for build.

At runtime, a client may choose an authentication scheme to use by defining the AUTH_SCHEME parameter in the .MdasEnv file. Each authentication scheme requires a different setup by the user.

Each authentication scheme requires a different setup by the user.

i) ENCRYPT1 setup - This is a plain text password setup where the password is generated at the time of user registration with the MCAT catalog. Upon receiving this password from the MCAT administrator, a user should place it in a file named ~/.srb/.MdasAuth which will be used by the SRB library to authenticate the user.
ii) GSI_AUTH setup - Please read GSI and follow the HTTP links for obtaining certificate/key and GSI server environment setup suggested by in the document. Once the certification/key has been obtained and converted to PEM format for GSI use, look into the cert pem file with a text editor and locate the user "Distinguish Name" string. The string representing the "Distinguish Name" should look like the following:
          subject=/C=US/O=NPACI/OU=SDSC/UID=srb/CN=Storage Resource Broker/Email=srb@sdsc.edu

Copy this string and register it as the "Distinguish Name" of the SRB server user. Supply this string to all clients for use as the SERVER_DN input in the clients' ~/.srb/.MdasEnv files.


Every time, you use the Client Utilities, from a new shell run 'Sinit' to initialize the environment and to check connection with SRBMaster. Running Sinit in the middle of a session will reset the client environmental file variables.


Use Sexit when finished with using Scommands. Use Serror to find the meaning of error numbers generated when running Scommands. The SRB_SRC env variable which defines the SRB source directory must be defined to be the 'src' sub-directory in the SRB's directory.

See also


[[#SRB - The Storage Resource Broker | TOP]

Locations of the SRB client library and header files

The SRB client library is installed in $(installDir)/bin/libSrbClient.a.

A client program should include the header file "srbClient.h" and the compilation header file search directive should include the following directories:

$(buildDir)/src/include

$(buildDir)/src/catalog/include


The Makefile in the test/examples directory gives an example of a makefile for making SRB client code.


TOP


Client Programming

Examples on the use of SRB client APIs can be found in the test/examples/src directory of the source distribution.We will go through some of these examples in this chapter.


Connecting and disconnecting to the SRB Server

The sconnect.c file in the examples directory gives an example of a client connecting and disconnecting to the SRB Server. The client uses the srbConnect () call to connect to the SRB server. Prototype of this call is:

srbConn * srbconnect (char* srbHost, char* srbPort, char* srbAuth, char *userName, char *domainName, char *authScheme, char *serverDn)

srbHost - the host address of the SRB server. srbPort - the port number of he SRB server. srbAuth - the password if ENCRYPT1 authentication scheme is used. userName - the user name of the client domainName - the domain name of the client. authScheme - the authentication scheme to use - "ENCRYPT1" or "GSI_AUTH" serverDn - the Distinguish Name of the server user.

A NULL value can be used for any one of these inputs in which case, the values will be taken from the .MdasEnv file.

If this call is successful, a non-null handle will be returned which can be used for all subsequent API calls. Upon completion, the clFinish call should be used to close the connection to the SRB server.

The following is some codes taken from the sconnect.c file:

   srbConn *conn;
   srbResult *res;


   conn = srbConnect (HOST, PORT, SRB_AUTH, USER, DOMAIN, AUTH_SCHEME, 
     SERVER_DN);
   if (clStatus(conn) != CLI_CONNECTION_OK) {
       fprintf(stderr,"Connection to srbMaster failed.\n");
       fprintf(stderr,"%s",clErrorMessage(conn));
       srb_perror (2, clStatus(conn), "", SRB_RCMD_ACTION|SRB_LONG_MSG);
       clFinish (conn);
       exit (1);
   }
   printf ("Connected to srbServer successfully\n");
   clFinish(conn);


TOP


Error handling

The SRB uses two methods to return error conditions to the client. The first is the commonly used method of returning a value with the client API. In general, for integer type return values, a negative returned value implies error. For string type returns, a NULL returned value implies error. The error code for integer type returns are given in include/srb_error.h . Typically, these are error codes that are internally generated by the SRB and MCAT. Given the integer error code, an error message string can be generated using the srb_perror() API or the Serror utility.

The SRB server can also return a more informative error conditions in the form of a text string, but the client must use the clErrorMessage(conn) call to retrieve the message. Typically, it contains a short message describing the error condition and often times, the error codes from the underlying storage system (e.g., HPSS, UniTree, etc) is also included.

TOP


Client programming examples




The following files in the test/examples/src directory give some examples of using the SRB client API:

sconnect.c - connect to a SRB server.

sput.c - upload a file to SRB.

sget.c - download a file from SRB.

sreplicate.c - replicate a SRB file.

sgetcopy.c - download a specific replica from SRB.

sstat.c - Get the stat (similar to the fstat() call) of a SRB file.

sgetdents.c - This the the SRB equivalent of the getdents() call in UNIX.

squery.c - query the MCAT metadata.


The following are two examples of using the SRB client API.


1) Uploading a file to SRB - sput.c

   /* Need to connect to SRB first */
   conn = srbConnect (HOST, PORT, SRB_AUTH, USER, DOMAIN, AUTH_SCHEME, 
     SERVER_DN);
   if (clStatus(conn) != CLI_CONNECTION_OK) {
       fprintf(stderr,"Connection to srbMaster failed.\n");
       fprintf(stderr,"%s",clErrorMessage(conn));
       srb_perror (2, clStatus(conn), "", SRB_RCMD_ACTION|SRB_LONG_MSG);
       clFinish (conn);
       free (buf);
       exit (1);
   }
   /* Create a srb file. need to specify the fileName - argv[2] */
   out_fd = srbObjCreate (conn, MDAS_CATALOG, argv[2],
     DATATYPE, RESOURCE ,COLLECTION, NULL, -1);
   if (out_fd < 0) { /* error */
       fprintf(stderr, "can't create srb file \"%s\", status = %d\n", argv[2], 
       out_fd);
       fprintf(stderr,"%s",clErrorMessage(conn));
       clFinish (conn);
       free (buf);
       exit (1);
   }
   /* Open a local file with filename = argv[1]. */
   in_fd = open (argv[1], O_RDONLY, 0);
   if (in_fd < 0) { /* error */
       fprintf(stderr, "can't open file\"%s\"\n", argv[1]);
       clFinish (conn);
       free (buf);
       exit (1);
   }
   /* Read from the local file and write to the just created srb file */
   total = 0;
   while ((bytesRead = read(in_fd, buf, BUFSIZE)) > 0) {
       /* Write to the data object */
       bytesWritten = srbObjWrite(conn, out_fd, buf, bytesRead);
       if (bytesWritten < bytesRead) {
          fprintf(stderr, "Error: Read %d bytes, Wrote %d bytes.\n ",
          bytesRead, bytesWritten);
           clFinish (conn);
           free (buf);
           exit (1);
       }
       total += bytesWritten;
   }
   /* remember to close the opened files */
   srbObjClose (conn, out_fd);     /* close the SRB file */
   close (in_fd);       /* close the local file */

2) MCAT query example - squery.c


The srbGetDataDirInfo API:

int srbGetDataDirInfo(srbConn* conn, int catType, char qval[][MAX_TOKEN], int *selval, mdasC_sql_result_struct *myresult, int rowsWanted)

is a general purpose routine used by clients to query the MCAT. The schema of the MCAT is quite complicated containing hundreds of tables and attributes. The srbGetDataDirInfo() API provides a simple way for clients to query the MCAT without having to know the MCAT schema by providing a single table view to the MCAT. From a client standpoint, the MCAT contains of a single table with hundreds of attributes. The query inputs are the qval[][] and selval[] arrays and the result of the query is returned in the myresult struct containing rows of attribute values.

Both selval[] and qval[][] are arrays of size MAX_DCS_NUM and are indexed by values given in src/catalog/include/mdasC_db2_externs.h under the heading DCS-ATTRIBUTE-INDEX DEFINES. Each attribute is represented by an element in the array. The selval[] array specifies a list of attrbutes to select in the query and qval[][] specifies a lists of conditions (predicates) in the query. e.g., selval[USER_NAME] = 1 means the "user_name" attribute is to be retrieved.

In squery.c:

   char qval[MAX_DCS_NUM][MAX_TOKEN];
   int selval[MAX_DCS_NUM];


   memset (qval, 0 sizeof (qval));
   memset (selval, 0, sizeof (selval);



   /* set up the query: Given the Collection, select the path name, 
    * resource name, the network address and the resource type.
    */
   sprintf(qval[DATA_GRP_NAME]," = '%s'",argv[1]);
   selval[DATA_NAME] = 1;
   selval[SIZE] = 1;
   selval[PHY_RSRC_NAME] = 1;
   selval[DATA_OWNER] = 1;
   selval[REPL_TIMESTAMP] = 1;
   /* Send the query request */
   if ((retVal = srbGetDataDirInfo(conn, MDAS_CATALOG, qval, selval,
    &myresult, ROWS_WANTED)) < 0) {
       fprintf(stderr, "can't srbGetDataDirInfo. status = %d.\n", retVal);
       clFinish (conn);
       exit (1);
   }
   /* Dump the result */
   printSqlResult (&myresult);
   clearSqlResult (&myresult); /* clear the content of myresult */



TOP


The JAVA based srbBrowser


The SRB Browser is a Java client program. The GUI of the browser was implemented using Java but most SRB specific functions were implemented with existing C library call through the JNI interface. This was a prototype implementation at one point, is fairly basic, but is used by a few users.


This is a short note on how to build and use the srbBrowser software.

This software must be built using JDK 1.4.1 or later. It uses the SRB client library native codes to communication with the SRB server.

Building the JAVA based srbBrowser

To build the srbBrowser software:

1) configuring the build - type in:

   . cd SRB3.4.0  (the SRB main dir)
   . configure --enable-javagui=yes --enable-jdkhome=/usr/local/apps/jdk1.4.1

where /usr/local/apps/jdk1.4.1 is the directory where the JAVA JDK 1.4.1 is installed. This directory should be changed based on your site installation. This will set the parameters "JAVA_GUI and "javaDir" in the mk/mk.config file. If the configure script failed, these parameters can be set manually by editing the mk/mk.config file.

2) You must first Build the SRB client/server software before the srbBrowser can be built.


3) To make the srbBrowser, type in:

   cd java
   gmake clean
   gmake


Running the JAVA based srbBrowser

To run the srbBrowser:

   setenv DISPLAY to your workstation.
   include the JAVA JDK bin path to you default path. e.g.,
   set path = ( $path /usr/local/apps/jdk1.4.1/bin)
   cd SRB3.4.0/java
   type in "bin/srbBrowser". 


A srbBrowser GUI should come up on your screen.

A quick rundown some of the functions .

1) The "Default" pull down menu allow a client to set the "default storage resource", "default transfer mode" the "default container" and the "default comment". The "default resource" will be used for all subsequent dataset creation until it is changed. The "default transfer mode" (serial/parallel/bulk) will be used for all subsequent import/export operations. The "default container" will be use to store data created by operations originated from the "Container" menu. The "default comment" can be used for creating new comments for datasets. These default settings are displayed in the metadata panel.

2) The "Operations" menu shows the type of operations that can be performed depending what was selected. The following operations have been implemented:

   For most operations, the "shift" key or the "control" key can be
   used to select multiple files/collections for the subsequent operations.
   "New" - make a new  "Collection" 
   "Replicate" - Replicate a dataset or collection (recursively). 
   Two choices are allowed: replicate to the default storage resource 
   or choose a resource for replication.
   Copy a dataset or collection - The following steps should be taken:
     1) Select a dataset or a collection to be copied (source).
     2) Select the "Copy" menu from the "Operation" pulldown menu.
     3) Select the destination dataset or collection.
     4) Select the "Paste" menu from the "Operation" pulldown menu.

o "Delete" - Delete a dataset or a collection recursively.

o Import a file or directory from a local file system. The following steps should be taken: 1) Select the destination dataset or collection. 2) Select the "Import from Local FS" menu from the "Operation" pulldown menu. 3) A file chooser dialog box should appear. Browse through the local FS and selection a file or a directory to be imported from (source). Click on the "Open" button of the file chooser dialog box.

       o Export a dataset or collection to a local file system.  
         The following steps should be taken:
           1) Select the source dataset or collection to be exported.
           2) Select the "Export to Local FS" menu from the "Operation" 
           pulldown menu.
           3) A file chooser dialog box should appear. Browse through the
           local FS and selection a file or a directory to export to
           (destination). Click on the "Open" button of the file chooser 
           dialog box.

o "Register" and "Unregister" local files and directory. These functions may be recursive. Comparing to the "Import" and "Export" functions, registering a local file does not involve

         the the copying of the file physically. Rather, the file is
         registered with MCAT and hence becomes visible to the SRB world.

For "Register" to work, your local file must be accessible by one of the SRB server in the federation. The following three steps must be taken to register a local file:

1) Set the default resource (through the "Default" menu) to the resource controlled by the SRB server that can access your local file.

2) Select a collection where you want this registered dataset or collection to reside.

3) Select the "Register Local FS" menu item from the "Operation" pull down menu and then selection the local file or directory you wish to register.

o "Rename" - Rename a dataset.

o "Display" - allows graphical (jpeg, gif, etc) and text (C, html, etc) files to be displayed in the srbBrowser window.

         This can also be achieved by double clicking the file displayed 

in the metadata window.

3) The "Container" menu shows the type of "container" operations that can be performed. The following operations have been implemented:

       o "New Container" - make a new "Container"

o "Delete Container" - delete a "Container"

o "Sync Container" - Synchronize the "permanent" copy of the container with the "cache" copy.

o "Copy" and "Paste to Container" operations are the same as the "Copy" and "Paste" operations in the "Operation" menu except all objects created by the "Paste to Container" operation are stored in the "default container".

o "Import to Container" operation is the same as the "Import from Local FS" operation originated from the "Operations" menu except all objects created by the "Import" operation are

         stored in the "default container".

4) The "Metadata" menu allows the manipulation and querying of system level metadata. Manipulation of two attributes - "Access Control" and "Comments" has been implemented thus far.

o The "Set Access Cntl" menu can be used to grant access permission of a dataset or a collection to individual users or group of users. Recursive operation is allowed for setting the access permission of a collection.

o The "List Access Cntl" menu can be used to list the access permission of datasets and collections.

o The "New Comment" menu can be used to attach a new "comment"

         to a dataset. A new comment can be created completely new or

by adding to the default comment.

o The "Modify Comment" menu can be used to modify an existing comment. Modification can be done by replacing the entire comment or appending to the current comment. The menu also allows the deletion of a comment.

o The "Query Comment" menu can be used to query datasets based

         on the content of these comments. If a collection is selected

for querying, the query will be done for all descendants of the collection. If a dataset is selected for querying, the content the comment for this dataset will be listed.

5) The "View" menu has only one sub-menu - "Refresh". The srbBrowser always try to update the tree structure and the metadata displayed in the window whenever changes were made by the srbBrowser. The "Refresh" function allowed the display to be refreshed after changes were made external to the srbBrowser.


TOP