Enhancing speed and scalability of the ParFlow simulation code

Abstract

Regional hydrology studies are often supported by high-resolution simulations of subsurface flow that require expensive and extensive computations. Efficient usage of the latest high performance parallel computing systems becomes a necessity. The simulation software ParFlow has been demonstrated to meet this requirement and shown to have excellent solver scalability for up to 16,384 processes. In the present work, we show that the code requires further enhancements in order to fully take advantage of current petascale machines. We identify ParFlow’s way of parallelization of the computational mesh as a central bottleneck. We propose to reorganize this subsystem using fast mesh partition algorithms provided by the parallel adaptive mesh refinement library p4est. We realize this in a minimally invasive manner by modifying selected parts of the code to reinterpret the existing mesh data structures. We evaluate the scaling performance of the modified version of ParFlow, demonstrating good weak and strong scaling up to 458k cores of the Juqueen supercomputer, and test an example application at large scale.

Mathematics Subject Classification (2010)

Preview

Notes

Acknowledgments

The development of this work is made possible via the financial support by the collaborative research initiative SFB/TR32 “Patterns in Soil-Vegetation-Atmosphere Systems: Monitoring, Modeling, and Data Assimilation,” project D8, funded by the Deutsche Forschungsgemeinschaft (DFG). Authors B. and F. gratefully acknowledge additional travel support by the Bonn Hausdorff Centre for Mathematics (HCM) also funded by the DFG.

We also would like to thank the Gauss Centre for Supercomputing (GCS) for providing computing time through the John Von Neumann Institute for Computing (NIC) on the GCS share of the supercomputer Juqueen at the Jülich Supercomputing Centre (JSC). GCS is the alliance of the three national supercomputing centers HLRS (Universität Stuttgart), JSC (Forschungszentrum Jülich), and LRZ (Bayerische Akademie der Wissenschaften), funded by the German Federal Ministry of Education and Research (BMBF) and the German State Ministries for Research of Baden-Württemberg (MWK), Bayern (StMWFK), and Nordrhein-Westfalen (MIWF).

Our contributions to the ParFlow code and the scripts defining the test configurations for the numerical experiments exposed in this work are available as open source at https://github.com/parflow.