Before calling configure, first customize the
default configuration. To customize the default configuration,
modify env_conf.xml and env_mach_pes.xmlbefore invoking configure. The env_build.xml
and env_run.xml files can also be changed at this step.

env_mach_pes.xml contains variables that determine the layout
of the components across the hardware processors. Those variables
specify the number of processors for each component and determine
the layout of components across the processors used.
See env_mach_pes.xml variables for a
summary of all env_mach_pes.xml variables.

Optimizing the throughput or efficiency of a CESM
experiment often involves customizing the processor (PE) layout for
load balancing.
The component PE layout is set in env_mach_pes.xml.

CESM1 has significant flexibility with respect to the layout of
components across different hardware processors. In general, its
CESM components -- atm, lnd, ocn, ice, glc, and cpl -- can run on
overlapping or mutually unique processors. Each component is
associated with a unique MPI communicator. In addition, the driver runs
on the union of all processors and controls the sequencing and
hardware partitioning. The processor layout for each component is
specified in the env_mach_pes.xml file via three settings: the
number of MPI tasks, the number of OpenMP threads per task, and the
root MPI processor number from the global set.

the atmosphere component will run on 64 hardware processors using 16
MPI tasks and 4 threads per task starting at global MPI task 32.
There are NTASKS, NTHRDS, and ROOTPE input variables for every
component in env_mach_pes.xml. There are some important things
to note.

NTASKS must be greater or equal to 1 (one) even for inactive (stub) components.

NTHRDS must be greater or equal to 1 (one). If NTHRDS is set to 1, this
generally means threading parallelization will be off for that component.
NTHRDS should never be set to zero.

The total number of hardware processors allocated to a component is
NTASKS * NTHRDS.

The coupler processor inputs specify the pes used by coupler
computation such as mapping, merging, diagnostics, and flux
calculation. This is distinct from the driver which always
automatically runs on the union
of all processors to manage model concurrency and sequencing.

The root processor is set relative to the MPI global communicator, not the
hardware processors counts. An example of this is below.

The layout of components on processors has no impact on the science.
The scientific sequencing is hardwired into the driver. Changing
processor layouts does not change intrinsic coupling lags or coupling
sequencing. ONE IMPORTANT POINT is that for a fully
active configuration, the atmosphere component is hardwired in the driver to
never run concurrently with the land or ice component. Performance
improvements associated with processor layout concurrency is therefore
constrained in this case such that there is never a performance reason
not to overlap the atmosphere component with the land and ice components.
Beyond that constraint, the land, ice, coupler and ocean models can
run concurrently, and the ocean model can also run concurrently with the
atmosphere model.

If all components have identical NTASKS, NTHRDS, and ROOTPE
set, all components will run sequentially on the same hardware
processors.

The root processor is set relative to the MPI global
communicator, not the hardware processor counts.
For instance, in the following example:

the atmosphere and ocean are running concurrently, each on 64
processors with the atmosphere running on MPI tasks 0-15 and the ocean
running on MPI tasks 16-79. The first 16 tasks are each threaded 4
ways for the atmosphere. The batch submission script ($CASE.$MACH.run)
should automatically request 128 hardware processors, and the first 16
MPI tasks will be laid out on the first 64 hardware processors with a
stride of 4. The next 64 MPI tasks will be laid out on the second set
of 64 hardware processors.

If you set ROOTPE_OCN=64 in the preceding example, then a total of
176 processors would have been requested and the atmosphere would
have been laid out on the first 64 hardware processors in 16x4
fashion, and the ocean model would have been laid out
on hardware processors 113-176. Hardware processors 65-112 would
have been allocated but completely idle.

The case initialization type is set in env_conf.xml. A CESM
run can be initialized in one of three ways; startup, branch, or
hybrid. The variable $RUN_TYPE determines the
initialization type and is set to "startup" by default when
create_newcase is invoked. This setting is only important for
the initial run of a production run when the $CONTINUE_RUN
variable is set to FALSE. After the initial run, the $CONTINUE_RUN
variable is set to TRUE, and the model restarts exactly using input
files in a case, date, and bit-for-bit continuous fashion.

Start date for the run in yyyy-mm-dd format.
This is only used for startup or hybrid runs.

RUN_REFCASE

Reference case for hybrid or branch runs.

RUN_REFDATE

Reference date in yyyy-mm-dd format for hybrid or branch runs.

This is a detailed description of the different ways that
CESM initialization runs.

startup

In a startup run (the default), all components are
initialized using baseline states. These baseline states are set
independently by each component and can include the use of restart
files, initial files, external observed data files, or internal
initialization (i.e., a "cold start"). In a startup run, the coupler
sends the start date to the components at initialization. In addition,
the coupler does not need an input data file. In a startup initialization, the
ocean model does not start until the second ocean coupling (normally
the second day).

branch

In a branch run, all components are initialized using a
consistent set of restart files from a previous run (determined by
the $RUN_REFCASE and $RUN_REFDATE variables in env_conf.xml).
The case name is generally changed for a branch run, although it does
not have to be. In a branch run, setting $RUN_STARTDATE
in env_conf.xml is ignored because the model components obtain the
start date from their restart datasets. Therefore, the start date
cannot be changed for a branch run. This is the same mechanism that is
used for performing a restart run (where $CONTINUE_RUN
is set to TRUE in the env_run.xml file).

Branch runs are typically used when sensitivity or parameter
studies are required, or when settings for history file output streams
need to be modified while still maintaining bit-for-bit
reproducibility. Under this scenario, the new case is able to produce
an exact bit-for-bit restart in the same manner as a continuation run
if no source code or component namelist inputs
are modified. All models use restart files to perform this type of
run. $RUN_REFCASE and $RUN_REFDATE are
required for branch runs.

To set up a branch run, locate the restart tar file or restart
directory for $RUN_REFCASE and $RUN_REFDATE
from a previous run, then place those
files in the $RUNDIR directory. See setting up a branch run for an
example.

hybrid

A hybrid run indicates that CESM will be initialized more like
a startup, but will use initialization datasets from a previous
case. This is somewhat analogous to a branch run with relaxed
restart constraints. A hybrid run allows users to bring together
combinations of initial/restart files from a previous case
(specified by $RUN_REFCASE) at a given model output
date (specified by $RUN_REFDATE). Unlike a branch
run, the starting date of a hybrid run (specified by
$RUN_STARTDATE) can be modified relative to the
reference case. In a hybrid run, the model does not continue in a
bit-for-bit fashion with respect to the reference case. The resulting
climate, however, should be continuous provided that no model source
code or namelists are changed in the hybrid run.
In a hybrid initialization, the
ocean model does not start until the second ocean coupling (normally
the second day), and the coupler does a "cold start" without a restart file.