Building the NPACI Grid:
Integrating the
Human Infrastructure

Pacific Rim Group Evolves
into International Model
of Collaboration

by Cassie Ferguson

When several hospitals in Taiwan were quarantined during the SARS (Severe Acute Respiratory Syndrome) epidemic, doctors and other medical professionals were suddenly cut off from the rest of the world. Quarantined physicians could no longer seek help from specialists at other institutions, and hospital staffs and patients were unable to see their families. On May 13, 2003, the World Health Organization reported that the viral respiratory illness had infected 7,548 people worldwide, killing 573.

Early on May 15, 2003, Fang-Pang Lin of Taiwan’s National Center for High-performance Computing (NCHC) sent a request for immediate technical assistance to Peter Arzberger, chair of the Pacific Rim Application and Grid Middleware Assembly (PRAGMA) Steering Committee, and several other members of that committee. Teri Simas, PRAGMA program manager, forwarded the message to the rest of the PRAGMA members, an international group of researchers whose communications are usually focused on the establishment and promotion of grids and applications. From there, the request spread to the Global Grid Forum and the Access Grid Community.

Almost immediately, offers to assist poured in from around the world, with volunteers ready to provide gear, remote expertise, and Chinese-speaking support staff. In less than 12 hours, Access Grid technology was deployed from the U.S. to Taiwan.

On the evening of May 17, an emergency video teleconference was held to review the logistics for implementation of the Access Grid network within three Taiwanese hospitals. The list was later expanded to seven sites. The goal was to set up a network-based collaboration environment using the Access Grid, which would extend standard video- and teleconferencing and allow physicians to share detailed x-ray images, patient data and other information in online meetings among several sites. The Access Grid would also host private virtual rooms for patients or hospital staff to visit with family members.

By May 20, Access Grid nodes had been delivered to San-Chung and Chang-Gung Hospitals. On May 21, they had been installed. The network connection between Jen-Ai Hospital and NCHC was successfully tested on May 22, and on May 29, nodes were established in Jen-Ai Hospital and in Taiwan’s Center for Disease Control. Chung-Hwa Telecom deployed a dedicated backbone network with a one Gbps bandwidth.

“Thanks to PRAGMA, an alliance was been formed,” said Lin, director of the NCHC’s grid computing division. “NCHC had a responsibility to assist in handling this arduous task, and with assistance offered from the international grid community, we contributed to the nationwide call to assist in fighting the disease, relieving the epidemic, and ultimately saving many lives.”

PROMOTING THE GRID

While a group of computational scientists may seem like unlikely candidates for a team combating a deadly epidemic, it may be even more unusual that those researchers, from institutions worldwide, united without regard to political or scientific boundaries to achieve a common goal in record time. Arzberger, director of Life Sciences Initiatives at the University of California, San Diego (UCSD), said that kind of effort represents the essence of PRAGMA (www.pragma-grid.net), which includes members from 18 institutions and organizations across the Pacific Rim.

PRAGMA was founded as an open organization with the goals of establishing sustained collaborations and advancing the use of the computational grid among a community of investigators at leading research institutions across the Pacific Rim. According to Arzberger, “PRAGMA was founded on the premise that the conduct of science is global and more examples arise that point to the challenges that must be faced internationally; that the grid promises to revolutionize science as much as networking has done to our daily activities; and that the grid is still too difficult to use by most researchers.”

In March 2002, PRAGMA held its inaugural workshop in San Diego, establishing the initial governance and community of the effort along with opening discussion about which projects to address. The meeting chair, Philip Papadopoulos, who is director of SDSC’s Grid and Cluster Computing program and co-PI on the National Science Foundation (NSF) PRAGMA awards, brought up the central issues of the conference: to explore the resources, software, and policy related to a pan-Pacific grid and what can be done to make the grid useful for applications.

“Collaboration is the key to success. With PRAGMA, we are trying to build a community that focuses on the needs of the applications,” Papadopoulos said. As part of the effort to promote the international construction and use of the grid, PRAGMA recognizes that no one institution or economic entity has all of the talent or all of the resources to do this.

At the conclusion of the workshop, Arzberger wrote a note to thank the participants, who had actively participated in presentations and working group planning sessions. “There is a Chinese saying that happiness is something to do, someone to love, something to hope for. With a broad interpretation, we should feel happy about what we have accomplished in a short period of time. We have taken the first step, and we will be challenging each other to take subsequent steps to create sustainable collaborations.”

HEADLINES AND SCIENCE

During that first workshop, Satoshi Matsuoka of the Global Scientific Information and Computing Center at the Tokyo Institute of Technology in Japan referred to building a Pacific Rim research grid in terms of the movie Field of Dreams, and said, “If we build it, the scientists will come.”

Less than two years later, scientists have arrived, with PRAGMA’s computing infrastructure having caught up to its social infrastructure. Its resources have been used in research that cuts across scientific fields: chemistry, biology, physics, and medicine. PRAGMA has acted as the catalyst for scientific and computing accomplishments, which have not only caught the attention of the research community, but also made headlines worldwide. In recognition of the success of the initial PRAGMA effort, in the summer of 2003 the NSF awarded the program $1.2 million over three years.

Jysoo Lee, deputy chair of the PRAGMA Steering Committee and head of the Supercomputing Research Department at the Korea Institute of Science and Technology Information (KISTI), said, “PRAGMA focuses on real applications that are critically dependent on grid technology. It complements other projects in the Asia Pacific grid community.”

Among the success stories: In November 2002, researchers at the National Institute of Advanced Industrial Science and Technology (AIST) set an intercontinental data transfer record, pushing data from a high-energy physics experiment at an average speed of 741 Megabits per second over 10,000 kilometers from the U.S. to Japan. The record was set in the course of participating in the “Bandwidth Challenge,” a networking contest held as part of SC2002, the annual conference of high performance computing, held that year in Baltimore.

High-energy physics experimental data generated by Monte Carlo simulation was distributed to seven clusters (190 PCs) via a number of high-speed networks, including two governed by PRAGMA member TransPAC. By conducting parallel data transmissions between the clusters and sharing the data between high-speed networks, the researchers were able to boost the overall data transfer rate. Additionally, it was the first time that clusters in the U.S. and Japan had been integrated and a single application used to send multiple terabytes of data via multiple TCP streams across the Pacific Ocean.

“An important thing to note about this ‘speed record’ is that it used the network and attached clusters in a manner very similar to the way scientists will actually use these resources,” according to TransPAC. “[It] is an example of how grid computing and high performance networking will be intimate partners in the future of scientific computation.”

The telescience application, demonstrated at iGRID2002 is a model PRAGMA collaboration. In controlling a microscope in Osaka from UCSD via portals and the use of new networking technology of IPv6, PRAGMA extended a long standing collaboration between these two institutions by involving NCHC and its expertise in visualization. Grid Datafarm and Data Grid middleware were involved in the demonstration at iGRID2002, which took place in Amsterdam. The demonstraton involved machines on three continents and real-time control of an instrument, distributed tomographic reconstructions of specimens under the microscope in Osaka using machines at the three participating institutions, and visualization of the output. Each participant contributed to this collaboration and benefited from the expertise from their collaboration.

NCHC is applying its experiences in the field of ecology and biodiversity for researchers at the Taiwan Forestry Research Institute’s Fushan Research Station, to control sensors at Fushan and to share data with their counterparts worldwide.

PRAGMA has also provided a means for ecology and biodiversity researchers at the Fushan Research Station to share data with their counterparts worldwide. In early 2003, the Taipei Times chronicled the addition of an Ecology-Grid system (Eco-Grid) to the Taiwan Ecological Research Network (TERN). Fushan, one of five sites in TERN, is famed for its pristine rainforest and contains more than one-third of Taiwan’s plant and animal species.

the Taiwan Eco-Grid, which includes remote high-resolution cameras, sound recorders, and wireless sensors, will capture information on animal activity and plant growth, and will be a key to long-term monitoring of soil erosion and channel sedimentation. The data, which include observations and research results, are fed to a system set up by PRAGMA member NCHC under the National Science Council, saving researchers the time and expense of trekking to remote sites in the 2,700-acre preserve.

With the help of PRAGMA, the Taiwan Eco-Grid has been connected, both in terms of a computing grid and science collaborators, to the international ecology research community. According to Arzberger, this will allow researchers to perform comparative studies between sites that would not have previously been possible.

Chemists, computational chemists, and computer scientists with PRAGMA members Monash University, Kasetsart University, Cray Inc., AIST, and SDSC, along with the Victorian Partnership for Advanced computing, recently collaborated on a ground-breaking demonstration at the fourth PRAGMA workshop, held in Melbourne, Australia, in conjunction with the 2003 International Conference on Computational Science.

As reported in GridToday, the researchers used a tool called Nimrod/G to distribute the task of processing a quantum chemical modeling code, GAMESS, to six clusters in three countries. Running the code overnight, the researchers calculated a “pseudo-potential,” which can ultimately be used to model large molecular systems such as proteins.

“These results allow us to explore many more parameter combinations than we had been able to do in the past,” said Wibke Sudholt, a postdoctoral researcher at SDSC. “Future applications of this technology could range from understanding complex biochemical reactions, to designing new drugs, to basic structure-function relationships in materials. This international example of collaboration has opened up a new field of inquiry.”

PRAGMA members are also participants in the Encyclopedia of Life (EOL), an ambitious project to catalog the complete complement of proteins from every living species in a flexible, powerful reference system available via the Web. The project draws on the skills of experts in biology, data and knowledge systems, and grid computing. It also uses some of the world’s most powerful computational resources, potentially up to 2,416 processors when counting clusters at the three PRAGMA EOL sites, including NPACI’s Blue Horizon and the TeraGrid.

“The genome projects have led to a lot of new questions, most importantly, ‘How can we best use this sequence information?’” said Philip Bourne, SDSC’s director of Integrative Biosciences and a UCSD professor of pharmacology. “The Encyclopedia of Life is part of the answer to those questions. The EOL will permit comparative proteomics—comparison of proteins within and between species. This will lead to identification of new functions for these proteins and, in the best case, highlight potential new drug targets. At the very least we will have an encyclopedic reference of existing proteins that will educate a broad community in the role of these proteins in living systems.”

In February 2003, the Bioinformatics Institute of Singapore became the first international site to use protein annotation software developed at SDSC to completely process the genome of the Tiger Pufferfish, or Takifugu rubripes. Since then, the Tokyo Institute of Technology has joined the project. As of September 2003, the EOL contained the annotated proteomes of 140 species out of a potential 1,000 for which full or partial genomes are publicly available.

INTERNATIONAL MODEL

Such science vignettes, along with the dramatic story of the SARS Grid, are an indication of the potential created by PRAGMA. “The grid will allow researchers to focus on collaborative technologies and efforts, which must be developed in an international context,” said William Chang, program manager in the NSF’s Office of International Science and Engineering. “The nature of the PRAGMA project, its openness, broad geographical scope, focus on standard technologies and software, should allow the lessons learned within the targeted scientific communities to be transferred to other research users of grid technologies and infrastructure, as well as across national and political boundaries.”

PRAGMA communicates and disseminates the results of its efforts though activities that include a series of member workshops hosted by participant sites. The first four meetings were held in San Diego, hosted by SDSC and UCSD; the meeting in Seoul was hosted by KISTI; the meeting in Fukuoka was hosted by AIST in conjunction with the Asia-Pacific Advanced Network; and the Melbourne meeting was hosted by Monash University and the Australia Partnership for Advanced Computing. Future meetings are planned in Hsinchu, Taiwan, in October 2003 to be hosted by NCHC, and in Beijing in May 2004 to be hosted by the Computer Network Information Center of the Chinese Academy of Sciences.

Even at the first meeting, it became clear that if PRAGMA were to be long-lived, it would be necessary to lay out a model for interaction. Over the first 10 months, PRAGMA evolved operating principles and procedures that guide decision-making processes and the commitments of its members, as well as growth of PRAGMA by the addition of new members. At the fourth workshop, several new members joined PRAGMA, notably Academia Sinica Computing Centre and Kasetsart University. Subsequently, a partnership between the Asia-Pacific Advanced Network and PRAGMA was formalized, drawing upon the mutual interests of both groups in networking and grid, as well as overlapping interest in the area of natural resources. Finally, PRAGMA extended its model of international research collaborations to include industry. In September 2003, PRAGMA initiated an Industrial Affiliate program to encourage mutually beneficial collaborations between industry and the research community, with Cray Inc., as its first member.

“Cray is interested in understanding the needs of the emerging grid community and the applications for this new technology,” said Richard Russell, Cray’s vice president of Asia-Pacific Sales. “We feel that our alliance with PRAGMA will help us to better understand the grid-technology needs of our customers. It also presents the opportunity to demonstrate how Cray systems can advance scientific progress, while actively sponsoring development of international applications and collaborations.”

Having set a new standard in international collaborations, PRAGMA may evolve into a model for efforts beyond grids and science. Arzberger said, “We hope that PRAGMA will inspire other international collaborations and promote new means to nurture, sustain, and explain those collaborations in order that we as a global society can address critical global issues and improve economic growth, quality of life, and the health of our planet.”