Server Configuration

From SRB

SRB server configuration

The SRB architecture is a federation of servers running on various hosts. The installation of a SRB federation includes:

1) Installation of the MCAT - see this link MCAT_Install for details.

2) Installation and configuration of Non-MES (Non-MCAT Enabled Server).

3) Installation and configuration of a MES server.

This note describes the configuration and startng of the SRB servers (MES and Non-MES) after the software has been built as described in this link Build.

Contents

Quick setup of a Non-MES (non-MCAT Enabled Server)

This section provides short notes on quick setup and startup of the SRB server. For more detail descriptions of setup, please read the section on #Setup of a MES server.

There is a script included in the release, installServer.pl, which can be used to help install a Non-MES. Please refer to the comments at the top of that for more information.

Setup for servers with UNIX driver only:

1) Build the Non-MES server software according to this link Build.

2) Configure the SRB user environment for the "server user" with the ~/.srb/.MdasEnv and ~/.srb/.MdasAuth files as described in User_Environment.

3) Configure the data/mcatHost file which specifies the address of the MCAT enabled server. See #Setup of a MES server for details.

4) Configure the data/hostConfig file. Normally the configuration of this file is not needed unless the host has a complicated network alias scheme and the SRB server may not be able to get all the aliases associated with the host. When a SRB server is started, it outputs a line in the data/srbLog file specifying all the host name and aliases understood by the server. This information should be checked. See #Setup of a MES server for details.


5) To run the SRB server, type in:

   cd bin
   runsrb

This should start the SRB server. If it is not running, the data/log/srbLog.mmddyy file should give hint on why it is not running.

6) Check the data/log/srbLog.mmddyy file - The first line should contain something like:

   LocalHostName:  orion, localhost, orion.sdsc.edu  .....
   followed by:
   Local storage vault conf:
   resource: demoResc, storSysType: 0, vaultPath: /data/mwan/test/Vault
   resource: demoResc1, storSysType: 0, vaultPath: /data/mwan/test/Vault
   Local Zone :
   ZoneName = sdscbrick8  HostName = srbbrick8  PortNum = 6834
   Remote Zone :
   ZoneName = sdscdemo  HostName = srb.sdsc.edu  PortNum = 6668
   NOTICE:Feb 28 16:26:44: srbMaster version SRB-3.4.0&G is up.


The LocalHostName list shows all the addresses of your local host recognized by the SRB server. It uses the gethostbyname() call to get the local addresses. There are situations where the local host address registered with MCAT may not show up with this call. Then, the "Local storage vault conf" will be empty. In this case, use the data/hostConfig file to include the MCAT registered host address to your local addresses.

Also verify the ZoneName and PortNum.

Setup of a MES server

1) Build the MES server software according to this link Build.

2) Configure the SRB user environment for the "server user" with the ~/.srb/.MdasEnv and ~/.srb/.MdasAuth files as described in User_Environment.

3) Setting up the SRB configuration files - Modifications to the following configuration files are needed for each SRB server installation:

data/mcatHost - This file identifies the host on which the MCAT enabled server is running. This file should contain the following lines:

   line 1 - The host on which the MCAT enabled server is running
   line 2 - The authentication scheme to use when connecting to the MCAT 
            enabled server. Valid schemes are : ENCRYPT1 and GSI_AUTH. 
            The scheme will default to ENCRYPT1 if this line is not present. 
   line 3 - If GSI_AUTH is chosen, this line represents the 
            "Distinguish Name" of the MCAT enabled server user.

data/MdasConfig - The MDAS configuration file. This file is required only by the MES server. This file contains basic informations required for SRB to interface with the MCAT database. Configurable parameters include SRB's database userid, password and log file location.

data/hostConfig - This is the optional SRB host configuration file. It is only needed when when you want to add aliases to your local hostName.

bin/runsrb - This is the startup script for the SRB server. Modifications of some env variables defined in this file may be required. The parameters are self-explanatory through the comments given in this file.

Many of the user environment parameters (srbPort, AUTH_SCHEME, mcatZone, etc) set in the .MdasEnv file (see 2)) can be overridden by setting these parameters in the runsrb script. This may be necessary if the same UNIX account is used to run several SRB servers each with different srbPort and/or AUTH_SCHEME, etc.

Some of the more important parameters are:

   srbPort - defines the port number to use in this zone. It overrides
   the value defined in the .MdasEnv file.
   MaxThread - defines the maximum number of threads to use for parallel
   transfer. The default is 4.
   X509_USER_KEY and X509_USER_CERT specify where the SRB server's 
   GSI key and certificate are located. They are only relevant if
   the server supports GSI authentication.
   GENERATE_GUID - specifies whether a GlobalIdentifier will be
   generated and stored as part of a srbObject registration.
   GftpSeverDn - Valid only if GRID_FTP driver is configured in this 
   server.  This defines the Distinguish Name of the user for the 
   GridFTP server.
   PRE_SPAWN_CNT defines the number of srbServer to prespawn. 
   The purpose of prespawning server is to improve interactivity. 
   SingleSvrPort - configuring SRB servers to use a single port 
   for firewall consideration
   commPortNumStart - Specifies the first allowable port number for SRB
   client/server. This overrides the value specified with the 
   --enable-commstart option of "configure". A zero value disable 
   restriction on port number.
   commPortNumCount - Specifies the number of allowable ports for SRB
   client/server.
   ALLOW_NON_CHECKING_REGISTERY_OF_DATA is an environment variable that
   specifies whether to allow users to register data without checking
   if SRB is allowed to access it. Note that this can be a security hole.
   ServerTimeOut - This allows the srbServer process to timeout and exit if
   it has not received new commands from the connecting client/server for
   the number of seconds specified by ServerTimeOut. 
   logfileInt - The interval in days for switching to a new logfile. The
   default interval is 5 days. i.e., a new logfile is generated every 5 days.


data/data/srbLog.mmddyy - This is the log file of the SRB server. A new log file with file name srbLog.mmddyy (mm = month, dd = day and yy = year) will be created every 5 days by default. It is a good place to check for problems.


5) SRB maintenance mode - The SRB servers can be put into maintenance mode by creating a file named "srb.allow" in the "data" directory. If the "srb.allow" file is empty, the SRB server will not accept any additional connection from anyone. Existing connections are allowed to continue with their works. The "srb.allow" file can be edited as text file to add users to be allowed to make connections during maintenance mode. The format is userName@domainName for each user.

Each individual SRB server can be put into maintenance mode independently from the other SRB servers. If the MCAT server is put into maintenance mode, the whole federation within the zone is effectively put into maintenance mode.





'Bold textItalic text''Italic textBold text=== Setup of server with HPSS driver ===

1) Build the SRB server with the HPSS flag set in the mk/mk.config file. In addition, the HPSS driver can use either the normally used DCE authentication or the NON-DCE implementation developed by Mike Gleicher. Please contact Mike at mkg@san.rr.com for informations regarding the licensing of the Non-DCE HPSS client/server software. If the NON-DCE authentication is used, the NO_DCE flag in mk/mk.config should be set before the build. If it is not set, DCE authentication is assumed.


2) Configure the data/hpssCosConfig file as described in the comment fields in the file. This file configures the HPSS Class Of Service (COS). Basically, this file contains a table with two columns - "COS ID" and "Maximum file size in KBytes". It is used by the HPSS driver for choosing the appropriate COS based on the "dataSize" input parameter of the srbObjCreate() call.


3) Set the variable hpssCOSId in the bin/runsrb to the default class id to be used for creating hpss files.

4) If your installation's hpss library was built with no default servers or location server, you may have to set env variables such as HPSS_LS_NAME for version 4, or HPSS_HPNS_NAME and HPSS_BFS_NAME for version 3 in the bin/runsrb script. Please check with your HPSS administrator regarding these parameters.

5) HPSS authentication configuration

i) For NON-DCE authentication, configure the data/hpssNodceAuth file as described in the comment fields in the file. Basically, this file should contain the HPSS userID/password pair of the SRB user (the userID under which the SRB server runs). . NOTE: The NON-DCE authentication works only with the Non-DCE HPSS client/server implemented by Mike Gleicher.

ii) For DCE authentication, create a DCE keytab file and change the srbKeytabName parameter in bin/runsrb to point to this file path and the srbPrincipalName to the UserId associated with this keytab file.

HPSS setup note from a user

The following gives the install note made by Jean-Yves Nief of IN2P3 for a NON-DCE HPSS server:

1) in mk/mk.config, set the following parameters:

   HPSS=2
   NO_DCE=1

within the sequence: ifdef NO_DCE, set:

 HPSS_LIB_DIR=/opt/hpss/lib 
 HPSS_HDR_DIR=/opt/hpss/include (whenever the HPSS lib and include paths)

and type in "gmake" to make to SRB code.

2) Install for example in "/var/hpss/etc/ndcl.keyconfig" the file containing the non DCE key config file of the HPSS core server you are using. Then set the following env variables

   setenv HPSS_REUSE_CONNECTIONS 1    # not mandatory
   setenv HPSS_DEBUGPATH stderr
   setenv HPSS_DEBUG 0     # certainly interesting to put it at 1 

First time in order to track down possible problems related to HPSS, set:

   setenv HPSS_NDCG_SERVERS cchpstd1  # the name of your core HPSS server
   setenv HPSS_NDCL_KEY_CONFIG_FILE /var/hpss/etc/ndcl.keyconfig.test  
   # file path where the key config file ==> mandatory for authentication.

3) Configure the data/hpssNodceAuth and data/hpssCosConfig files as described above.

Setup of server with GridFTP driver

1) Build the SRB server with the following configurations:

a) Edit the mk/mk.config file and set:

   GRID_FTP = 6

b) Enable GSI authentication with the normal configure options (e.g., configure --enable-gsi-auth --enable-globus-location=<path>)

c) Type in "gmake" to make to SRB code.

2) Use the MCAT admin tool to create a special "Location" for the GridFTP resource. This "Location" is unlike others because more informations are need (The address, port number and Distinguish Name of the GridFTP server) for GridFTP. The Format of the netprefix (Host Address in the Java admin tool) which normally contains the host address of the SRB resource, has been modified as follows:

   SrbResAddr:GridFTPSvrAddr[:GridFTPSvrPortNum][GridFTPSvrDn]


where,

   SrbResAddr = The normal host address of the SRB resource
   GridFTPSvrAddr = The host address of the GridFTP server (required
   for GridFTP driver)      
   GridFTPSvrPortNum = The port number of the GridFTP server
   (Optional).
   GridFTPSvrDn = The Distinguish Name of the GridFTP server user.
   (Optional. If it is not specified, the server will use the
   GftpSeverDn value specified in the "runsrb" script). The 
   advantage of specifying the Dn in the "Location" is the
   SRB server may need to interact with more that one GridFTP servers
   each having a different Dn and using a single Dn specified by
   GftpSeverDn in the "runsrb" script will not be able to support
   such usage.

An example of a netprefix for GridFTP:

   srbbrick7.sdsc.edu:srbbrick3.sdsc.edu:8888:
   /C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb

3) Use the MCAT admin tool to create a "Physical Resource" for the GridFTP resource with the following configurations;

   Location - The special location created above.
   Resource Type - gridFTP (you may need to use the "Add New Token"
   function of the MCAT admin tool to create this Resource Type).

4) The server configured to support GSI authentication (see README.gsi.htm).

5) File ownership in gridFTP resources - Normally, files stored in SRB are physically stored in the resource vault path directory and owned by the SRB server ("server owned"). Users do not usually access these files directly using APIs and commands native to the file system but access them indirectly through the SRB server using SRB APIs and utilities. However, some users have expressed interest in a "user owned" environment where files stored in SRB are physically owned by the users themselves. But currently, SRB servers are not designed to run with special privileges (such as root or chown privileges for UNIX FS type resources) which makes "user owned" environment not feasibly.

However, gridFTP servers are normally run with root privileges and we can use the gridFTP resources to create a "user owned" environment. The current implementation is if a file is stored using the resource's vault path, it defaults to the usual "server owned" environment where the server's GSI credential is used for gridFTP authorization. However, if a file is stored outside of the resource's vault path, it will be in a "user owned" environment where the user's proxy certificate is used for gridFTP authorization. In this case, the SRB user must use the GSI delegation authentication scheme (by setting AUTH_SCHEME to "GSI_DELEGATE").

Starting the SRB server

The servers in the SRB environment must be started in the following order :

       i) The MCAT DBMS server.
       ii) The MCAT enabled SRB server.
       iii) The rest of the SRB servers.

The script bin/runsrb is used to start the SRB server. To start the SRB server, type in :

       . cd bin
       . runsrb


To shutdown the SRB server, type in:

       . cd bin
       . killsrb

SRB maintenance mode

The SRB servers can be put into maintenance mode by creating a file named "srb.allow" in the "data" directory. If the "srb.allow" file is empty, the SRB server will not accept any additional connection from anyone. Existing connections are allowed to continue with their works. The "srb.allow" file can be edited as text file to add users to be allowed to make connections during maintenance mode. The format is userName@domainName for each user.

Each individual SRB server can be put into maintenance mode independently from the other SRB servers. If the MCAT server is put into maintenance mode, the whole federation within the zone is effectively put into maintenance mode.


Configuring server timeout

The server timeout feature allows the srbServer process to timeout and exit if it has not received new commands from the connecting client/server for the predefined period of time. The timeout allows the system to reclaim resources if the client has been idle for a period of time. The timeout is enabled by setting the ServerTimeOut environment value in the "runsrb" script. The ServerTimeOut value must not be too small especially on a MCAT enabled server because the connecting server may be busy transferring a very large file. Internally ServerTimeOut defaults to 3600 sec if it is set to less than this value. Setting ServerTimeOut also trigger a fixed 3600 sec timeout for the connection between the MCAT enabled server and the MCAT database. This allows the MCAT database to reduce the number of active connections. The connection to the MCAT database is re-established when a new request is received.