Disclaimer: "The views and conclusion contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Advanced Projects Research Agency or the US Government".

The National Archives and Records Administration (NARA) has funded a supplement to the Distributed Object Computation Testbed (DOCT) to examine application of supercomputing technology to the formation of a persistent digital archive. The DOCT project provides hardware and software systems on which to demonstrate advanced supercomputing technology that manipulates data on the scale required by NARA. The challenges facing NARA can only be solved through use of advanced technology:

• By 1999, demonstrate ability to accession one million electronic records per year with a total data archive size measured in terabytes.
• By 2000, demonstrate how to scale the system to support accession of 40 million records per year.
• Demonstrate the ability to migrate any component of the archive to new technology to guarantee persistence over periods of hundreds of years.
• Demonstrate the ability to identify and present any document stored within the archive.

As technology evolves, every component of the persistent archive will have to change to avoid obsolescence and loss of functionality. The activities in this project are categorized under the following three major task headings:

• Task I: Data ingestion. The project will analyze the requirements for production of Archival Information Packages and recommend technologies and procedures for ingest of collections of as many as several million physical files. Relevant Ingest functions include receiving and validating submission information packages; generating an Archival Information Package (AIP) which complies with the internal archive data model and documentation standards; extracting descriptive information (DI) from the AIPs for inclusion in the archive database; and coordinating updates to Archival Storage and Data Management. The internal archives standards implemented in the AIPs should refer to NARA's established standards for electronic records; however, the continued viability of these internal standards is an issue for evaluation in this project. The project should evaluate technical strategies for the production of AIPs and demonstrate one or more these strategies. Among other factors, evaluation of technical strategies should consider the likelihood and difficulty of downstream migration of AIPs as the technologies on which they depend become obsolete. The Ingest process should take into account the necessity to identify specific software dependencies of files in all cases where specific software may be required either to retrieve our display any record. In most cases, software dependency can be identified by specifying the format of the file (e.g., SGML, plain ASCII, JPG, TIFF, 'pdf', etc).
• Task II: Persistent storage. The project will identify and evaluate options for long-term preservation of AIPs according to archival standards. Relevant Archival Storage functions include receiving AIPs from ingest and adding them to permanent storage; managing the storage hierarchy; performing routine and special error checking; providing disaster recovery capabilities; and providing AIPs for access and dissemination to fulfill dissemination requests. While this project will not address requirements and issues related to specific customer request for access to AIPs or dissemination of AIPs; nevertheless, basic requirements for the archival storage systems include the need to be able to locate, retrieve and output any record. Similarly, the project need not specifically address refreshing the media on which archives holdings are stored; however, recommendations concerning ingest, archival storage and data management should recognize that periodic media refreshment will be necessary in archival storage. Additionally, the project should address requirements for retrieval and rendering of records which are stored in software dependent formats.
• Task III: Metadata archiving. Relevant Data Management functions include administering the archive database functions (maintaining schema and view definitions and referential integrity); performing database updates (loading new descriptive information or archive administrative data); performing queries on the data management data to generate result sets; producing reports from these result sets and activating requests triggered by time or data management updates. The research should specifically address the necessity to identify each record as a persistent object, including identifying variable attributes and methods that need not be maintained throughout the life of the object.

For each task, the goal is to identify the appropriate information management strategies that are needed to assure the ability to persistently store, manage, and retrieve electronic records. The strategies will be evaluated in terms of commercially available technologies. Prototypes will be assembled to demonstrate the ability to accession at least 1 million records per year, and manage multiple terabytes of data. The major challenge will be to show that the prototype can be designed such that any component is replaceable with new technology, and that any new mechanisms for indexing or retrieving the electronic records can be accommodated.

The architecture that is needed to support the persistent storage of archived records is represented in Figure 1, categorized by the levels of organization used to manage data. For each level, languages can be specified to specify the information management capabilities. The associated data flow and data control mechanisms can then be implemented, using the proposed formatting standards. The goal of the persistent storage project is to identify where current technology is adequate, and where new technology is needed to build persistent archives. Standards are needed to explicitly define interactions at each level of this hierarchy to make it possible to migrate the underlying hardware and software systems onto new technology.

Figure 1. Information Management Architecture for a Persistent Archive

The tasks listed in this report are defined in a Research and Development Plan and Schedule, accessible at the URL:

http://www.sdsc.edu/DOCT/Publications/nara-rdps.doc

The software infrastructure is being assembled through the integration of digital library, information discovery, data handling, data processing, and archival storage systems. Many of these systems are already integrated as part of the DOCT testbed. When possible, commercial systems will be identified that can support the persistent data archive

• The archival storage system will be based on the High Performance Storage System (HPSS) currently in use at SDSC.
• The data processing software will be locally written at SDSC and executed on parallel compute platforms. Packages available from commercial vendors will be tested for feasibility of use.
• The database technology used to store the collections will be based on the IBM Universal Database (UDB). This product has been integrated with HPSS and is expected to meet the data ingestion goals.
• The data handling system will be based on the Storage Resource Broker that was developed as part of the Distributed Object Computation Testbed. Comparable commercial products will be evaluated.
• The data presentation environment will be based on a vendor-neutral, language independent collaboration environment that is being developed as part of the National Partnership for Advanced Computational Infrastructure (NPACI).
• The information discovery system will be initially based on the Metadata Catalog developed as part of the DOCT project. With funding from NPACI, this system is being upgraded to support the extensible schema that are also required by the NARA project. Comparable commercial products will be evaluated.
• The digital library services used to categorize electronic records will be based on commercial products. We will also evaluate automated generation of metadata from XML document DTDs using technology developed under NPACI.

Management will be achieved through development of a Research and Development Plan and Schedule. The tasks specified by this plan will be developed in collaboration with the National Archives.

Demonstrations of the technology will be arranged with the National Archives as results are achieved. The first demonstration is expected in February, 1999, in which an ingestion rate of 1 million records per year will be shown.

SDSC will demonstrate data ingestion for several different document collections, each with a large number of documents, and quantify the issues involved in ingesting these collections into a long-term archival storage system. The collections considered include; electronic mail, word processing documents, a database consisting of service tickets, a patent document collection, and selected heterogeneous documents acquired from Web sites.

For each corpus, the ingestion tasks include:

1. forming the corpus for the test demonstrations.
2. ingesting the data in "raw" form into the archive. The separation of data load from the data accession steps will allow the construction of scalable systems.
3. validating the "raw" input data. This will require processing the loaded data on a parallel computer. A typical approach will be to compare each loaded document against a DTD.
4. creating an error database, if necessary. This will require developing appropriate metadata to describe the degraded data, such that retrieval of specific records will still be possible.
5. classifying each validated document. This will require consensus on a persistent archive schema. We propose integrating Dublin Core provenance information with the SDSC IVcore image metadata, and additional metadata needed to migrate the data management software forward in time. To ensure the ability of the system to evolve, extensible schema will be used.
6. indexing, cataloging, and archiving validated documents. This will require encapsulating each electronic record as an Archival Information Package. We will explore the use of XML for annotating each record with the required metadata, such as access control information.
7. performing a price/performance analysis for the hardware, software, and labor involved, including time taken.

Where applicable, alternate technologies will be studied to support ingestion of different types of document collections. A possible technology that will be considered for data accessioning is the IBM research prototype, Grand Central Station, which has the capability to parse the structure of various types of documents and generate indexes based on the known structure.

The data ingestion tasks are:

• I.A – Ingestion of Usenet news posting corpus, comprising about 1 million postings
• I.B – Ingestion of word processing document corpus, representing electronic records that can be obtained from Web sites.
• I.C – Ingestion of heterogeneous data collections, to demonstrate that structured documents can be archived, retrieved, and presented in their original form.

• I.D – Migration of the USPTO patent digital library to the XML formatting standard to quantify the technology needed to migrate collections to new standards.

This task involves technology evaluations of persistent storage systems to verify that such systems are capable of providing the performance that will be necessary for long term archiving of very large numbers of electronic records. In addition, the task also addresses issues in dealing with technology evolution over the long term.

The tasks related to persistent storage are:

• II.A – Performance modeling of the archival storage system to demonstrate the ability to scale the system to very large sizes and accession loads.
• II.B – Migration strategies for handling technology evolution, including migration to new media, to new software products, and to new presentation systems.

This task involves studying various issues related to handling changes in metadata over a period of time. Even if there is no technology evolution over a period of time, one can always expect the metadata related to individual documents, or to the corpus as a whole, to evolve over time. Long term archiving must account for such evolution by archiving metadata definitions, along with the metadata itself.

The tasks related to metadata management are:

• III.A – Schema evolution, with support for publication of the multiple versions of extensible schema, and support for retrieval against older versions of schema.

• III.B – Ontology evolution, how to deal with indexing and searching collections when the terminology and meanings of terms has changed over time.

The DOCT testbed has been upgraded to provide access to the latest version of HPSS (version 3.2), along with the most recent version of the associated IBM Universal Database. The combined product is in test, with the expectation that the accession of the 1 million records can be managed by UDB while the archive holds the electronic records.

A two-stage process is being used with initially the loading of electronic records, followed by their categorization and storage into the combined UDB/HPSS product. Migration of electronic records to new media will be managed using HPSS technology. Migration of electronic records to new formatting or presentation standards will require processing of records. This will be coordinated through use of the SRB data handling system.

We are maintaining a Research and Development Plan & Schedule (RDPS) document for this project. The RDPS is a detailed, comprehensive overview of all NARA tasks, including all deliverables (reports and demonstrations). This document is the authoritative source for our project goals, tasks, deliverables, schedule, and expected work loads. The RDPS is available via the World Wide Web at the URL:

http://www.sdsc.edu/DOCT/Publications/nara-rdps.doc

Also note that many of the concepts that are being explored within this project are also supported through funding provided by other federal agencies. These efforts are documented under the URL:

http://www.npaci.edu/DICE/