Skip to content


    Skip to content

     

    TERASCALE HORIZONS | Contents | Next

    Expanding the Horizon for Academic Scientific Computing

    PROJECT LEADER
    Wayne Pfeiffer
    Jay Boisseau
    Giri Chukkapalli
    SDSC

    S ince the National Science Foundation supercomputing centers, including SDSC, were created 15 years ago, the face of academic scientific computing has changed exponentially, propelled by the scientific mysteries being pursued. Quests to understand the human genome, map the human brain, and chart the global circulation of the world's oceans, among others, have taxed the limits of computing power. For years, a machine capable of performing a trillion calculations per second--one teraflops, the type of power required for the most sophisticated and complex of simulations--was an elusive dream. In January 2000, however, that dream became tangible in Blue Horizon, a teraflops-capable IBM RS/6000 SP system delivered to SDSC and now in production. The machine is the most powerful available for academic research in the United States, and one of the 10 most powerful computers in the world.

    FOUR TRUCKS, TWO ELEVATORS

    GALACTIC MPIRE AND DECRYPTION

    TERASCALE SCIENCE

    ON THE HORIZON

    At the dedication ceremony for Blue Horizon held in February, IBM Vice President for Strategic Alliances and Pervasive Computing Lou Bifano said, "By linking the computing power of the RS/6000 SP with the leading researchers of our time, we bring together the best of human intelligence and technology."

    Blue Horizon's performance is generated by 1,152 Power3 processors, each with a clock rate of 222 MHz. Just one of these processors is more powerful than the first supercomputer installed at SDSC in 1985, a four-processor Cray X-MP system. The Power3 processors are organized into 144 eight-processor SMP High Nodes with 4 GB of memory each, giving the system a total of 576 GB of memory. Associated disk can store 5.1 terabytes of data.

    Top | Contents | Next

    NPACI's Blue Horizon NPACI's Blue Horizon IBM RS/6000 SP at SDSC

    FOUR TRUCKS, TWO ELEVATORS

    Space constraints on the machine room floor had to be resolved before the more than 40 towers of the teraflops system--which occupy about 1,500-square feet--could be installed. To clear room, the previous IBM SP system was relocated to another part of the floor. Soon, it will be split and redistributed to the University of Texas at Austin and the University of Michigan, where the processors will once again become available as NPACI resources. The processors sent to Michigan will be integrated with a similar system that already serves NPACI users.

    Four tractor-trailers transported the towers to SDSC on Thanksgiving weekend 1999. During installation, their weight overtaxed the capacity of the building's freight elevator, necessitating a re-route through the basement and up the passenger elevator. That elevator, too, was no match for the computer's heft, and was knocked out of service. However, all of the towers were in place by the end of the installation weekend.

    SDSC and IBM collaborated throughout the period leading up to the center's December acceptance of Blue Horizon. Together, they discovered and resolved a problem in the mapping of memory to cache that led to excessive variation in program run-times. This problem had not manifested itself on smaller SP installations because the probability of its occurring in a given node was exceedingly small. Working together, SDSC and IBM improved the cache management for large systems using the new Power3-based SMP nodes. This operating system patch will be included in the April release of AIX, IBM's 64-bit version of UNIX that runs on the machine.

    "This is a tribute to the teamwork between IBM and SDSC staff who worked throughout the holidays to install the system," said NPACI and SDSC Director Sid Karin on the machine's official acceptance. "IBM's attention and commitment to problem resolution let us to make the machine available to researchers tackling large scientific problems early in 2000."

    Top | Contents | Next

    GALACTIC MPIRE AND DECRYPTION

    SDSC researchers used the teraflops system even before its arrival at the center. As part of a collaboration with the Hayden Planetarium at the American Museum of Natural History, Blue Horizon was used to generate a scientifically accurate fly-through of the Orion Nebula.

    The Orion Nebula sequence--three minutes of high-resolution animation displayed by seven independent projectors--was produced by the MPIRE Galaxy Renderer software package running on 800 of Blue Horizon's processors for 12 hours, generating more than 28,000 separate images that filled more than 100 gigabytes of digital storage. SDSC's Greg Johnson traveled to IBM's facility in Poughkeepsie, New York, in November to generate the visualizations on the machine while it was still being tested.

    The machine was also subjected to rigorous acceptance tests once it arrived at SDSC. As part of these tests, the performance of four applications on both the teraflops SP and the older SP was compared for various numbers of processors ranging from one to 128. The applications included AMBER for molecular dynamics; GAMESS for quantum chemistry; and PARTREE and SCF for astrophysics. In all cases, these applications ran faster on Blue Horizon by factors that varied from 1.11 to 1.92.

    One application, SCF, was also run on all 1,152 processors and showed good scaling over a large processor range. For a fixed run-time, it was possible to solve a problem more than 700 times faster on 1,152 processors than on a single processor.

    On another test, run over four consecutive days in December, the teraflops system's 144 compute nodes were in use more than 86 percent of the time, peaking at 95.7 percent usage on Christmas Day.

    "Many of the processors weren't being used while we were running the diagnostics and benchmark programs," said SDSC's Jeff Makey. "The RC5-64 Challenge software let us test the idle processors at the same time." RC5-64 is a code-breaking challenge organized by distributed.net. The IBM RS/6000 SP placed among the top five daily participants for a week and held the top position for several days.

    Top | Contents | Next

    TERASCALE SCIENCE

    Early use of Blue Horizon--covered in more detail in articles in this issue of enVision--has substantiated NPACI's hopes for a teraflops-capacity machine. "The faster machine will enable us to make breakthroughs in many different scientific areas," Karin said.

    The Scientific Computing group at SDSC has ensured that early users are ready for the new architecture of Blue Horizon. Attendees at an August 1999 Parallel Computing Workshop sponsored by the group became the first to gain hands-on experience with a prototype SP system and beta-version hardware and software that were provided to the center by IBM. Other training sessions have been held, and documentation is available on NPACI's HotPage. The machine allows symmetric multi-processing within a node and message passing among nodes, introducing a hybrid programming paradigm. OpenMP, or Pthreads, will be used to program for the machine's shared-memory nature, and distributed memory programming is achieved using the Message Passing Interface library.

    While in early production status, Blue Horizon was put through a series of "big science" runs to demonstrate the machine's capabilities. Users with scalable, parallelized code were selected to run problems requiring the processing power of a teraflops system. All 1,152 processors could be dedicated to these codes, generating significant results.

    "The early users have been extremely helpful in testing the system and enabling us to measure the performance of world-class scientific applications," said Jay Boisseau, SDSC associate director for Scientific Computing. "We have worked closely with them to help port and optimize their codes for the new system. Most importantly, they have run some leading-edge scientific simulations, so we were able to contribute to scientific advancements in QCD, astrophysics, quantum chemistry, neuroscience, and ocean modeling."

    Peter Hauschildt, an astrophysicist from the University of Georgia, completed a big science run of PHOENIX, a program to model the atmospheres and synthesize the spectroscopic features of a wide variety of astronomical objects (see page 8). "The thermodynamic state and opacities of the cool gas present in the atmospheres of brown dwarfs and extrasolar giant planets is of fundamental importance for the study of the structure and evolution of this class of astronomical objects," said Hauschildt about the performance of PHOENIX on Blue Horizon. "Our run on Blue Horizon allowed us to calculate the most detailed and accurate tables of such opacities, for subsequent simulations of their evolution and 3-D hydrodynamical simulations of the formation of convection zones in their atmosphere. These calculations are extremely compute and I/O intensive. Only with a machine such as Blue Horizon could we have included all the detailed microphysics that we needed to consider."

    Scientific Computing group member Giri Chukkapalli--who managed the flow of Big Science runs through Blue Horizon, working one-on-one with the researchers and their collaborators--appreciated the opportunity to directly impact scientific investigations. "Pushing big science runs through the teraflops machine has been a unique challenge," he said. "We have learned a lot about the machine that will help us be better prepared for and more responsive to the needs of production users."

    Top | Contents | Next

    ON THE HORIZON

    Blue Horizon is now a regular allocated resource within NPACI, and training on the architecture and its programming paradigm will continue at SDSC and at partner institutions. A session will be held in April at the University of Texas at Austin and will be followed by a training institute at SDSC later in 2000.

    Discussion of the innovations that Blue Horizon may make possible frequently took center stage at the NPACI All-Hands Meeting 2000 held in February at SDSC. As the alpha projects gain momentum and other activities within the thrust areas mature, the computing capability of Blue Horizon will be tested by applications in neuroscience, biology, the physical sciences, and astronomy, among others.

    "Scientists are modern-day explorers," Karin said. "And like all explorers, we're interested in what lies just beyond the Horizon." --AF *

    Top | Contents | Next