Skip to content


METASYSTEMS | Contents | Next

A Worldwide Virtual Computer for an Advancing Legion of Applications

Andrew Grimshaw,
University of Virginia
Metasystems thrust area leader

The Legion Team
University of Virginia

Network Weather Service
Molecular Science
Biomolecular Structure and Energetics
Biological Data Representation and Query
Earth Systems Science
Multi-scale, Multi-resolution Modeling

A s computers become faster and more powerful, it becomes possible to study more complex problems by running larger simulations. Understanding the shape of the body's proteins, the structure of the brain, the properties of new materials, and the behavior of the environment are just a few questions for which scientists are seeking answers. However, scientists' ability to devise larger and more complex simulations can still exceed the power of many supercomputers and the talents of the best programmers. To harness more and larger computers while making it easier on programmers, the Legion project at the University of Virginia is working to turn networks of powerful computers into a worldwide virtual computer.

In the same way that a desktop computer has an operating system to manage the flow of data between the processor, memory, disks, and network connections, Legion integrates the operating systems of individual computers to create an operating system for the Internet. An architecture that creates such a virtual computer is also called a "metasystem," and Legion is one of the projects of NPACI's Metasystems thrust area. Andrew Grimshaw of the University of Virginia, leader of the Legion project, is also the Metasystems thrust area leader.





Since the first release of Legion in November 1997, the Legion team has continued to improve the system's stability and capabilities. "In the past year, we've done a lot to make the system more robust," said Fritz Knabe, senior research scientist with the Legion team. "Many of the changes aren't visible to users, except in the sense that they see a more polished environment when using Legion."

Some technical advancements to Legion in the past year include an improved batch queuing environment, more robust security and Kerberos authentication, a completed Interface Definition Language compiler for CORBA, and extensive development of a Java-based graphical interface that will be released soon. More on the Legion architecture can be found in the July-September 1998 enVision.

To further solidify the underlying components of Legion, Grimshaw and colleagues at Virginia have started a spin-off company, Applied Metacomputing, to commercialize Legion products and deploy Legion. The company, still in the early stages, is initially focusing on Web infrastructure uses of the Legion architecture.

In another step designed to ease the user experience, the Legion team has also established a Customer Service group, in which many Legion team members participate. "The Customer Service group gives us an organized approach to helping users," Knabe said. "The group provides a central location to receive help requests and distribute answers, which makes it easier for users to get up to speed. We also create accounts and provide other administrative assistance so that users can focus on getting science done."

The primary testbed for is the Centurion cluster at Virginia. Initially, Centurion comprised 64 nodes with Digital Alpha processors. In the past year, Centurion's second phase has been installed with funding from the Office of Naval Research. Today, Centurion comprises 128 nodes with 533-MHz Alpha processors and 128 nodes with two 400-MHz Pentium II processors. Altogether, Centurion has a peak performance of more than 200 gigaflops, 64 gigabytes of memory, and more than 1,700 gigabytes of disk storage.

Legion has also been installed on a larger-scale testbed that spans Virginia, SDSC, the University of Michigan, the National Center for Supercomputing Applications, and on occasion UC Berkeley. In addition to Centurion, Legion runs on Sun Enterprise Servers, the IBM SP and the CRAY T90 at SDSC, and the IBM SP at Michigan.

Top | Contents | Next

A. Grimshaw, A. Ferrari, F. Knabe, M. Humphrey. "Wide-Area Computing: Resource Sharing on a Large Scale." IEEE Computer, 32(5): 29-37, May 1999.

Figure 1. Protein Structure Scanning Grid
Figure 1. Protein Structure Scanning Grid
Russ Altman's group at Stanford is using Legion to scan protein structures for active sites of interest. The grid shows points to be sampled, while blue circles show radial bins evaluated around each sample point.



"Centurion offers substantial computing power for users," Knabe said. "The main carrot we have in attracting Legion users is that if you're willing to roll with the punches of an experimental testbed, we'll give you significant computing time." Users interested in Legion and the Centurion testbed should contact the Legion team.

Currently, Legion and Centurion are being used by researchers from a dozen different institutions, including Harvard Medical School, the Naval Oceanographic Office, the Naval Research Laboratory, NASA's Ames Research Center, and Oregon State University. NPACI partners and others from the University of Virginia, Stanford Medical Informatics, SDSC, UC San Diego, UCLA, and The Scripps Research Institute are also mustering the Centurion testbed in NPACI projects.

NASA researchers, for example, are performing fluid dynamics simulations to design aircraft wing flaps. Other projects include a Navy shallow water ocean simulator, surgical planning simulations, large-scale gene sequence comparisons, quantum chemistry calculations, and biosphere modeling.

In particular, a number of NPACI partners are using Legion to make advances in both computer and computational science. At UCLA, for example, C. Roberto Mechoso and Anthony Drummond of the Department of Atmospheric Sciences are using Legion to couple multi-scale and multi-resolution global circulation models as part of an Earth Systems Science project.

At UC San Diego, Francine Berman, professor in the Computer Science and Engineering Department, has worked with Legion on an NPACI Metasystems collaboration that is also part of a Programming Environments and Training (PET) project of the Department of Defense High-Performance Computing Modernization Program.

"We have been investigating an application-level scheduler (AppLeS) for a magnetohydrodynamics code targeted to Legion," Berman said. "The scheduler partitions the application based on Network Weather Service forecasts of deliverable resource performance." Experiments on the Centurion cluster have demonstrated that runs using AppLeS techniques show dramatically improved performance--an average of 2.5 times better performance than runs scheduled using static techniques. (For more on AppLeS and the Network Weather Service, see pp. 10–11.)

Top | Contents | Next

Figure 2. Protein Scanning Results Figure 2. Protein Scanning Results
Figure 2. Protein Scanning Results
Legion will let Altman's group scan many proteins simultaneously. Left: Protein 1HCK has one ATP-binding site; red dots show the ATP-binding site predicted by the Altman's group scanning method. Right: Protein 1JAP has two calcium and two zinc binding sites. The three red dots show the predicted Ca+ binding sites.


Two projects in the NPACI Molecular Science thrust area are taking advantage of Legion to expand the capabilities of their applications. Charles Brooks at The Scripps Research Institute leads a Molecular Science project on Biomolecular Structure and Energetics in which the CHARMM and AMBER molecular dynamics programs are being tested on Centurion.

CHARMM and AMBER both have efficient message passing implementations for parallel calculations on single machines. But neither has been adapted to run on clusters of processors and machines, exactly the task for which Legion was designed. Legion may enable Brooks and other biologists to accomplish large-scale computations in a relatively short time.

"One of my primary scientific interests involves characterizing the physical mechanisms by which proteins fold," Brooks said. " Studying how a protein folds has the potential to significantly improve biologists' understanding of how a protein sequence relates to its structure and function."

To address these questions, Brooks computes the folding free-energy landscape of proteins using atomic-detail models, including solvent. However, calculating the folding landscape for a single protein can take a month on 512 processors of a CRAY T3E. "We have Legion benchmarks using CHARMM for tightly coupled applications on the Centurion clusters but have not yet moved to use the larger NPACI-wide testbed," Brooks said. "We are encouraged by the potential and anticipate this marriage will be fruitful for both the scientific and computer science communities."

As part of a Molecular Science project on Biological Data Representation and Query, Russ Altman and colleagues at the Stanford University School of Medicine are taking codes developed for serial processors and using Legion to apply them simultaneously to many identical problems. In particular, the problem involves spawning many smaller jobs from a server and compiling results (Figures 1 and 2).

"We have code that scans a Protein Data Bank (PDB) file and suggests where active sites of interest might be located," said Altman, who also leads the Molecular Science thrust area. "Finding the interesting functional regions of 3-D structures--which may bind calcium or ATP or perform some enzymatic reaction--is not easy because of the molecules' complexity, but is important for understanding which regions of the proteins to focus attention upon." Liping Wei developed the original algorithm along with Jeffrey Chang, Allison Waugh, and Altman. Catherine Ying, with Glenn Williams, is porting the code to Legion.

Altman's group has validated the code on test sets and is considering using the code on the more than 9,000 structures in the PDB. An even more ambitious goal is to scan every time slice of a molecular dynamics run, which can have millions of time slices. This would allow biologists to determine if certain sites are forming and then unforming during normal molecular motion, or if they are stable over time. A metasystem like Legion will give Altman's group easy access to expandable computing power with a minimum of coding, administration, and bookkeeping.

"We recently scanned more than 1,600 structures on a four-processor SUN Enterprise 450 machine in 12 hours," Altman said. "A system like Legion could potentially do these scans while a biologist waits, so that biologists can perform such studies easily and reproducibly."--DH

Top | Contents | Next
Top | Contents | Next