Barcelona Supercomputing Center (BSC) and the National Center for Supercomputing Applications (NCSA) have efficiently solved large complex problems using parallel direct solvers on one of the most powerful supercomputers in the world. The ill-conditioned finite element systems arise from electromagnetic modeling problems in geophysical exploration studied under the Aurora project that is sponsored by Repsol and whose goal is joint inversion on HPC platforms to meet the most recent requirements of the industry.
Vladimir Puzyrev from the Computer Applications in Science and Engineering (CASE) department and his collaborators from NCSA conducted performance and scalability tests on the NCSA’s Blue Waters petascale computing system. A typical code takes several days to factorize the largest complex matrix that was solved in only 30 seconds using the IBM WSMP solver on 65,536 cores of Blue Waters. This unprecedented level of parallel scalability of direct solvers is demonstrated for the first time.
Up to 80-90 percent of the total computational time of three-dimensional geophysical inversion is taken by solving thousands of large sparse systems of linear equations. By using high-quality unstructured meshes, the electromagnetic field behavior can be modeled with extreme fidelity. However, finite element discretization of large real-world problems leads to extremely ill-conditioned sparse linear systems. Direct linear solvers have proven their robustness and efficiency for multisource problems when the expensive matrix factorization needs to be performed only once, and then multiple solutions are rapidly obtained by relatively low cost back solve operations.
Until recently, the limitations of CPU computing power and memory requirements have made the use of direct solvers impractical in three-dimensional problems. The scalability study in the paper recently published in Computers & Geosciences credibly shows that direct solvers can be efficiently used in large HPC simulations, enabling extremely realistic and robust three-dimensional modeling. This could lead to dramatic advances in understanding the properties of the subsurface and be a breakthrough in geophysical exploration for the foreseeable future.
The paper:
http://www.sciencedirect.com/science/article/pii/S0098300416300164