Title: High-Frequency Simulations of Global Seismic Wave Propagation Using SPECFEM3D_GLOBE

Authors: L. Carrington, D.Komatitsch, M. Laurenzano, M. Tikir, D. Miche,N. Le Goff, A. Snavely, J. Tromp

Abstract: SPECFEM3D_GLOBE is a spectral-element application enabling the simulation of global seismic wave propagation in 3D anelastic, anisotropic, rotating and self-gravitating Earth models at unprecedented resolution. A fundamental challenge in global seismology is to model the propagation of waves with periods between 1 and 2 seconds, the highest frequency signals that can propagate clear across the Earth. These waves help reveal the 3D structure of the Earth's deep interior and can be compared to seismographic recordings. We broke the 2 second barrier using the 62K processor Ranger system at TACC. Indeed we broke the barrier using just half of Ranger, by reaching a period of 1.84 seconds with sustained 28.7 Tflops on 32K processors. We obtained similar results on the XT4 Franklin system at NERSC and the XT4 Kraken system at University of Tennessee Knoxville, while a similar run on the 28K processor Jaguar system at ORNL, which has more memory per processor, sustained 35.7 Tflops (a higher flops rate) with a 1.94 shortest period. For the final run we obtained access to the ORNL Petaflop System, a new very large XT5 just coming online, and achieved 1.72 shortest period and 161 Tflops using 149,784 cores. With this landmark calculation we have enabled a powerful new tool for seismic wave simulation, one that operates in the same frequency regimes as nature; in seismology there is no need to pursue periods much smaller because higher frequency signals do not propagate across the entire globe. We employed performance modeling methods to identify performance bottlenecks and worked through issues of parallel I/O and scalability. Improved mesh design and numbering results in excellent load balancing and few cache misses. The primary achievements are not just the scalability and high teraflops number, but a historic step towards understanding the physics and chemistry of the Earth's inte-rior at unprecedented resolution.

Reference: @inproceedings{carrington08specfem, Author = { L. Carrington and D.Komatitsch and M. Laurenzano and M. Tikir and D. Miche and N. Le Goff and A. Snavely and J. Tromp}, Booktitle = {Supercomputing 2008}, Title = {High-Frequency Simulations of Global Seismic Wave Propagation Using SPECFEM3D_GLOBE}, Year = {2008}}