How to make GROUND coding compatible with Parallel PHOENICS

Discussion

The domain decomposition in Parallel PHOENICS is usually handled automatically.
The directions chosen will depend on the grid size, solver settings and number
of cells used. The decomposition will be made in such a way as to minimise the
data exchange between processors.

Each subdomain is assigned to a different parallel processor, and each subdomain
is enlarged at each adjoining end by the addition of two columns, rows or slabs (planes) of cells.
These additional 'overlap' or 'halo' cells are used to store field values
from the adjacent subdomains so as to facilitate data exchange between subdomains
during the parallel computation.

Loops in which indices are restricted to the current cell and its' immediate neighbours will work without any change.

The ground coding of loops will require special attention if:

summations are made, whether over the domain or over patches;

references are made to non-neighbour indices.

The reason is that Parallel PHOENICS interprets each loop as referring to the
current subdomain, including 'halo' cells. Such coding is likely to occur in Group 19
or Group 11.

Example 1

This is an example of a GROUND file which includes coding in Group 19 Section 6 which
sums up the gas volume over the whole field within the main IZ loop of
PHOENICS. Ground coding in other groups and sections is not shown because it did not
require modification for use with Parallel PHOENICS.

Head of GROUND file

SUBROUTINE GROUND
INCLUDE '/phoenics/d_includ/farray'
INCLUDE '/phoenics/d_includ/satear'
INCLUDE '/phoenics/d_includ/grdloc'
INCLUDE '/phoenics/d_includ/satgrd'
INCLUDE '/phoenics/d_includ/grdear'
INCLUDE '/phoenics/d_includ/grdbfc'
INCLUDE '/phoenics/d_includ/parear'
LOGICAL INPARDOM

The foregoing coding includes the use of various parallel utilities such as

INPARDOM(ix,iy,iz) - this returns .FALSE. if the cell
(ix,iy,iz) is one of the 'halo' cells, and .true. otherwise.

GLSUM(real) - sums a single real value over all processors
and sends the global sum back to all processors.

NPROC - is the number of processors - always 1 for sequential.

MYID - is the number of the current processor.
The master processor has the ID number 0. In sequential runs this is always 0.

Example 2

This example shows how to sum up the sources, in this case of mass, over all patches
to find the total mass inflow.

The problem lies in that on each processor the sequence of patches may be different, as
not all patches will exist on all processors. However, when summing across processors the
loop indices must be in step, otherwise the summation will fail and the run will stall.

In the example code, the following variables and routines are used:

NPROC - the number of processors. Will be > 0 for a parallel run.

GD_NUMPAT - this is the total number of patches in the global domain. NUMPAT
is the number of patches on the current processor, or in a sequential run.

GD_INDPAT(iglob,1) - this returns the local patch index for the global
patch iglob. If the value returned is < 0, the patch does not exist on the current processor.

PGETCV(iglob,ivar,coef,val) - this returns the COefficient and VALue for variable
ivar for global patch iglob. If the coefficient is returned as -999.0, no COVAL exists for this variable
at this patch.

GLSUM(real) - which sums a single real value over all processors
and sends the global sum back to all processors.

MYID - is the number of the current processor.
The master processor has the ID number 0. In sequential runs this is always 0.

Head of GROUND file

SUBROUTINE GROUND
INCLUDE '/phoenics/d_includ/farray'
INCLUDE '/phoeclos/d_includ/d_earth/parvar'
INCLUDE '/phoenics/d_includ/satear'
INCLUDE '/phoenics/d_includ/grdloc'
INCLUDE '/phoenics/d_includ/satgrd'
INCLUDE '/phoenics/d_includ/grdear'
INCLUDE '/phoenics/d_includ/grdbfc'
INCLUDE '/phoenics/d_includ/parear'

PGETCV(iglob,ivar,coef,val) - this returns the COefficient and VALue for variable
ivar for global patch iglob. If the coefficient is returned as -999.0, no COVAL exists for this variable
at this patch.

The following routines are useful for distributing data from the master processor
to all the slave processors, often after having been read at the start of the run.