Service

ParFlow+p4est

An open-source parallel watershed flow model

Scientific area

High resolution parallel simulation of variably saturated flow.

Short description

ParFlow is an integrated hydrology modeling software that simulates saturated and variably saturated flow in heterogeneous porous media. The code employs a discretization of the three dimensional Richards equation based on cell centered finite differences. The nonlinearity of the resulting equations is addressed with a globalized Newton's method. The sparse system of algebraic equations arising in each Newton step is solved numerically with a multigrid preconditioned Krylov iteration. ParFlow has been demonstrated previously to have excellent solver scalability up to 16,384 processes.

As part of our work in the DFG SFB/TR32 project D8, we have identified ParFlow's way of parallelization of the computational mesh as the main limiting factor in further extending its scalability. Hence, we have reorganized the mesh subsystem to use fast partitioning algorithms provided by the parallel adaptive mesh refinement software library p4est, which is known for its modularity and proven scalability.

We have created a modified version of ParFlow that is coupled to p4est to use it as its parallel mesh manager, currently focusing on uniform meshes. Our approach is minimally invasive and preserves most of ParFlow's data structures, the configuration system and the setup and solver pipeline. We have executed the modified version on up to 458,752 MPI processes of Juqueen, demonstrating good parallel scalability and significantly reduced setup time.

Saturation field from an infiltritation experiment in the presence of macroporosity and layered soil horizons. The heterogeneous permeability parameter is simulated to model vertical and lateral preferential flow paths, respectively.

Scalability

458,752 cores on BlueGene/Q (JUQUEEN)

Strong scaling timing results of ParFlow+p4est on JUQUEEN for different problem sizes. The test case solves a 3D regular topography problem in which lateral flow is driven by slopes based on sine and cosine functions.