SRM-SRB

From SRB

Academia Sinica (ASGC) is planning to develop an SRM interface to SRB and has asked the SRB team to help them formulate a plan and schedule for this. This page is being used to coordinate and plan that project. Participants include Wei-Long, Eric Yen, and Ethan Lin at Academia Sinica (ASGC), Taiwan, Abhishek Singh Rana at UCSD, and Reagan Moore and Wayne Schroeder of the SRB team (SDSC/UCSD).

Team members, please add to this document, and modify it, as you see fit. This is a draft document for comment and change. This is a community-editable wiki system, and may be a good tool to help us organize: a common document that we call all view and edit. The Wiki system maintains an edit log too, and provides a easy way to see what has changed when and by who. But if you'd rather use some different method, that would be fine too, and we can remove this.

Academia Sinica (ASGC) is interested in developing an SRM (Storage Resource Manager) interface to SRB. They are currently using Castor for LCG (it's gLite 3.0 now) and gLite will be the major middleware for the e-Science infrastructure for Taiwan and probably for most of the Asia Pacific Countries. SRM is the standard interface between gLite and the storage system, so they'd like to develop a SRM-SRB interface for gLite to access data in some of their existing SRB collections. They have also deployed DPM (Disk Pool Manager).

gLite 3.0 includes SRM 2, so I believe the desired functionality is SRM 2. (Correct?)


The following is a draft list of the major steps, as we currently see them. Once these are decided, we can try to estimate the times required for each which would then become the draft schedule:


Contents

Develop Use-Case Document(s)

This will help all of us better understand the particular functionality that is needed. Do you want to write SRB files as well as read them? Do you want to stage files, the way an achive would (i.e. move them off of SRB tape systems to SRB disk)?

It may be that taking an SRM that normally talks to a local file system, and modifying it to interface to SRB, will provide all that is needed. Or it might be better to start with one that talks to an archival storage system.

Use case 1: Port DPM on top of the SRB. The SRB would be treated as an archive. The port would allow a user to stage files to the disk file system where DPM is running from a distributed SRB collection. DPM would manage the disk file system storage space, guaranteeing enough room for the file staging. The SRB would support access to tape archives and other distributed storage system for long term storage.

Use case 2: Port SRM as an interface on top of the SRB. In this case, the SRM interface would cause the SRB to stage a file to a particular disk cache that is controlled by the SRB. Again, the SRM interface would guarantee space for the file staging, manage the locks, etc. The difference with the first scenario is that the SRB would still manage the properties of the staged file.

Create a list of SRM implementations that could be used by ASGC as the code base, Java or 'C'

  • CERN CASTOR SRM - SRM interface implementation for CASTOR MSS - C.
  • DESY/FNAL dCache SRM - SRM interface implementation for Enstore MSS and Disk (dCache with NFS-like database-based virtual FS) - Java.
  • LCG DPM - SRM interface implementation for Disk (FS?) - C.
  • LBNL DRM - SRM interface implementation for Disk (NFS) - Java.
  • JLab SRM - SRM interface implementation for JASMine MSS.

Evaluate how authentication works in both SRM and SRB

The SRM servers run as root and use GSI to authenticate to backend storage systems. SRB can use GSI too so this interface may be fairly straight-forward. But I'm concerned that there may be some details of how these all work that will make it more difficult. Therefore, I think it would be worthwhile to understand this in a fair amount of detail.

Evaluate each of the SRM implementations and select for testing

Based on all of the above, acquire the code for one, two, or a few implementations, and study them. Evaluate which one are the best fit for what you are trying to do.

Modify, build, test, and debug this SRM

The idea is to become familiar with how the SRM system operates and how to test it. If I were doing it, I would need to spend quite a bit of time on this. ASGC may already be sufficiently familiar with SRM but a fairly deep understanding will be needed to complete this project.


Convert the SRM to make SRB calls in place of the Unix I/O or Archive calls, simplifying or ignoring authentitation initially.

My guess is that authentication will require special attention, so you'd probably be better off getting things to first work without it.

Add in the authentication (GSI) and get that to work properly.