|
MESH
REFINEMENT
CODE
DETAILS
FACING
THE MUSIC
 lassical
music lovers have always appreciated orchestral conductors with
great dynamic range. Arturo Toscanini, Fritz Reiner, and Bruno
Walter were renowned for making symphony orchestras both thunder
and whisper. Walter in particular won a reputation for the ability
to serve up the tinkle of a triangle and an actual cannon in Tchaikovsky’s
1812 Festival Overture. From pianissimo to fortissimo, Bruno Walter
explored the very limits of audibility. And if there is a Bruno
Walter of cosmology, it is surely Michael L. Norman, a physics
professor at UC San Diego.
 |
|
|
|
Figure 1:
Very First Protostellar Object
|
Norman’s cosmological
structure code, Enzo, can span 12 orders of magnitude in space and
time. It can pursue the slightest gravitational perturbation of
a nearly uniform primal gas—the tinkling triangle of astrophysics—all
the way to the cannon-roaring condensations of gas, crushed into
volumes so much tinier than the initial volume that the ratio can
only be expressed in scientific notation: 10-30. Enzo can follow
the story from the consequences of the Big Bang to the coalescence
of the first star in exquisite detail; indeed, in “quadruple
precision,” in a manner faithful to the best idea cosmologists
have of the initial physics and chemistry of the process.
With former graduate students Greg Bryan (now a lecturer at Oxford
University) and Tom Abel (now a visiting scientist at Cambridge
University), Norman has submitted a paper on the most recent Enzo
calculations as an entry for this year’s Gordon Bell Award.
The prize, named after a pioneer in high-performance computing,
is given annually at the Supercomputing conference for the best
performance improvement in a parallel computing application.
Norman’s group also has written another paper focused on its
results, which will appear in Science in the coming months.
The physics of present-day star formation is complicated, because
the interstellar medium is itself composed of the remnants of previous
generations of stars, including not only hydrogen and helium but
also heavier elements, in abundances important enough to affect
the star-formation process.
“In contrast,” said Norman, “the formation of the
first star takes place in a much simpler environment: the gas is
only hydrogen and helium, and the initial conditions can be precisely
specified by cosmological models. It’s a clean initial-value
problem—and it’s the starting point for the formation
of all other structure in the universe, from galaxies to superclusters.”
Using the NPACI Blue Horizon supercomputer at SDSC, Bryan, Abel,
and Norman simulated the condensation of the universe’s first
star (Figure 1).
They began with what most astronomers believe made up the material
composition of the universe: 10 percent ordinary “baryonic”
matter—composed of protons, neutrons, and electrons; and 90
percent cold dark matter, which is exotic, nonbaryonic material.
Cold dark matter is a generic name for species of weakly interacting
particles that are cold—with negligible velocity dispersion—at
the era when the universe first became matter dominated. The properties
of cold dark matter allow tiny pre-existing fluctuations (the tinkling
triangles of astrophysics, if you will) to grow on all scales without
hindrance.
MESH
REFINEMENT
While the composition
of cold dark matter is unknown, its mass can be estimated and
its role in gravitational condensation can be calculated. The
salient feature of cold dark matter is the power spectrum of its
density fluctuations—there are more flutes than cannons—suggesting
that cosmic structure is formed “bottom-up,” by the
gravitational amplification of initially small fluctuations.
So the computational problem becomes: how do you follow a reasonably
large sample of the infant universe as its hydrodynamic perturbations
lead to a collapsing protogalactic object (millions of solar masses)
and, within that, to protostellar (hundreds of solar masses) clouds?
More specifically, how do cosmologists follow all that with the
correct chemistry and thermodynamics in a properly expanding cosmological
space-time continuum?
The gravitational problem alone has been attacked by N-body simulation,
but Enzo is much more ambitious, just as a great Romantic symphony
is more complex than a baroque country dance. Enzo’s time-dependent
calculation is carried out over the full three dimensions on a
structured, adaptive grid hierarchy that follows the collapsing
protogalaxy and subsequent protostellar cloud to near-stellar
density, starting from primordial fluctuations a few million years
after the Big Bang. Enzo thus combines a number of computational
modules. There is a hydrodynamic solver for the primordial gas,
an N-body (particle) solver for the collisionless cold dark matter,
a special solver specifically for the gravitational field, and
a 12-species chemical reaction solver for the primordial gas chemistry.
“We can’t do all this on a uniform mesh,” said
Norman, “because there is no computer large enough to contain
all the spatiotemporal scales we must follow.” Instead, the
researchers use structured adaptive mesh refinement, a way of
adding octaves to the cosmic range. While solving the equations
on a uniform grid, the code follows the quality of the solution
and, when necessary, adds an additional fine mesh over any region
that requires enhanced resolution. The finer mesh obtains its
boundary conditions from the coarse mesh. The finer grid is also
used to improve the solution on its parent. As the evolution continues,
the finer mesh may need to be moved, resized, or removed. Even
finer meshes may be required, producing a tree structure that
may continue to any depth.
Enzo spawns a new mesh when any cell accumulates enough mass that
refinement is needed to preserve a given mass resolution in the
solution, or when a minimum length criterion to resolve perturbations
is exceeded (Figure 2).
 |
|
Figure 2:
Another Fine Mesh
(©
2001 Tom Abel; used by permission)
|
Norman pointed out
that a fluctuation containing the mass of the Milky Way galaxy
will collapse by a factor of about 1,000 before it comes into
dynamical equilibrium. A code to follow such a collapse would
need to have a spatial dynamic range of 105. Resolving the formation
of individual stars—even very large ones—within a galaxy-full
of gas would require even more resolution, a spatial dynamic range
on the order of 1020. The work just completed is at a spatial
dynamic range of 1012—roughly the ratio of the diameter of
the Earth to that of a human cell.
CODE
DETAILS
Enzo is implemented
in C++, an object-oriented high-level language, with some compute-intensive
kernels in Fortran 77. The object-oriented approach provides two
benefits: encapsulation (a single mesh is the basic “object,”
or building block) and extensibility (new physics may be added
easily at all levels).
“The hard part in running Enzo on a variety of platforms
is parallelization and load balancing,” said Robert Harkness,
a computational astrophysicist at SDSC who is working with Norman
to prepare the Enzo code for runs across the TeraGrid. This is
the orchestral equivalent of deciding how many violins, violas,
cellos, and bass viols will make up the string section. “We
are using a version running under the Message-Passing Interface,
which allows us to exploit the object-oriented design by distributing
objects over the processors, rather than attempting to distribute
only the smallest grids themselves,” said Harkness. “The
small subgrids are generally numerous, and many may be quite short-lived,
enabling them to be created and removed on a single processor.”
Other innovations include the creation of “sterile objects”
that contain information about the sizes and locations of grids,
but without the solutions. Each processor can hold the entire
hierarchy of grids as sterile objects as it works the solution
on any grid or grids that are local to the processor, which reduces
communications traffic. Ultimately, however, the process of obtaining
boundary values for the root (largest) grid is nonlocal; the sterile
objects enable the code to “pipeline” the global communications
so that they are received first where the solution can be examined
first.
Accurate description of the positions of grids and particles within
the problem domain requires the code to work at “extended
precision” (double, where a single-precision word is 64 bits
long; quadruple on 32-bit words). “Fortunately,” Harkness
said, “Blue Horizon supplies 128-bit arithmetic.” The
researchers have developed methods of restricting the 128-bit
work to only those parts of the code that require it, only when
necessary.
FACING
THE MUSIC
While Enzo in its present
form is challenging enough for current platforms, the possibility
of distributing it across the TeraGrid beckons—and together
with that, the necessity to increase the dynamic range of the
code. At present, it begins with what might be called a statistically
significant fraction of the early universe—a large volume
of isotropic cold dark matter and gas—and proceeds to follow
the physics down to protogalactic and protostellar objects.
A key test of the model physics will come when the dynamic range
is extended to the level of the first star itself, as its gravitational
collapse overcomes its radiative expansion, enabling the fusion
of hydrogen into helium with the release of energy in its core.
It is the progress of this reaction that leads to the fusion of
helium into nuclei of heavier elements that eventually influence
the formation of next-generation stars.
“It seems that it will be possible, using this method, to
extend the dynamic range on both ends of the spectrum,” Norman
said. In orchestral terms, the thunder will be more thunderous
and the whispers will be all but inaudible—and the supercomputer
platforms will be, collectively, the universe’s largest,
noisiest concert hall. —MM 
|
PROJECT LEADER
Michael L. Norman
UCSD
PARTICIPANTS
Tom Abel
Harvard-Smithsonian Center for Astrophysics,
Cambridge University
Greg Bryan
MIT, Oxford University
|
SDSC - UC San Diego, MC 0505 |
9500 Gilman Drive |
La Jolla, CA 92093-0505
