Flight-Checks (Pre and Post)

Overview

Flight checks, AKA preflights and postflights, new in Qube 6.6, are prog=
rams and/or scripts that are run on the worker just before and after t=
he actual job or agenda starts.

The exit codes of flight checks determine the processing and exit status=
of their respective job instances or agendas.

System-wide =
vs Job-specific

System-wide flight checks are installed by the site administrator into&n=
bsp;predefined locations on the worker systems, and are run for every job. =
By default, system-wide flight checks are installed in one of four sub=
directories under $QBDIR/flightCheck on the=
worker. More precisely:

$QBDIR/flightCheck/instance/pre (instance-level preflights)

$QBDIR/flightCheck/instance/post (instance-level postflights)

$QBDIR/flightCheck/agenda/pre (agenda-level preflights)

$QBDIR/flightCheck/agenda/post (agenda-level postflights)<=
/li>

where $QBDIR is the system's Qube install loc=
ation, which is, by default, "/usr/local/pfx/qube" on Linux,=
"/Applications/pfx/qube" on Mac OS X, and "C:\Program =
Files\Pfx\Qube" on Windows.

The worker parameter worker_flight_check_path may be set in qb.co=
nf (or qbwrk.conf), to override the default "$QBDIR/flightCheck&q=
uot;

Any file with a ".txt" extension in those subdirectories are i=
gnored, but every other file will be attempted to run at their respect=
ive timings. It is expected that those files are all proper ex=
ecutables for the platforms on which they are intended to run=
-- otherwise, they will fail and thus jobs will fail.

Individual Jobs may also specify their own flight checks at submission. =
For example, the "qbsub=
" command now supports options such as "-preflights", "=
-postflights", "-agenda_postflights" and &=
quot;-agenda_postflights" that may be used to specify a comma-=
separated list of full-paths to flight c=
heck executables on a per-submission basis. See the "qbsub" documentation for =
details. The Qube API's job submit routine also may be used to=
specify flight checks when submitting jobs. These API jo=
b attribute names are "preflights" and "postflights" fo=
r instance-level, and "agenda_preflights" and "agenda_postfl=
ights" for agenda-level flight checks.

As with the system-wide flight checks (see: Universal Callbacks), these job-specific ones al=
so need to be proper executables for the target platform.

Multiple flight checks may be installed or specified for a single j=
ob. Job-specific flight checks are run before the system-wide ones. No=
te that if any flight check program fails (i.e., returns non-zero), all sub=
sequent flight checks for that job instance or agenda are not run.

The postflight checks are still run if an instance or agenda-item fails.=

Instance-level vs Agenda-level flight checks=
and the effect of their exit codes

Instance-level preflights and postflights are run on the worker just bef=
ore and after job instances are processed, respectively. Similarly, agenda-=
level preflights and postflights are run just before and after agendas are =
processed, respectively.

A non-zero exit code returned from a flight check will abort the process=
ing of any additional flight checks for the respective job instance or agen=
da, and also affect their exit status. More specifically:

If an instance-level preflight exits non-zero, any additional pref=
lights for the job instance are skipped, the execution of the job instance =
is canceled, and the job instance is reported to the supervisor as "fa=
iled".

If an agenda-level preflight program exits non-zero, any addtional pref=
lights for the agenda are skipped, the agenda will be unprocessed, and repo=
rted as "failed". The job instance will move onto processing the =
next agenda.

If an instance-level postflight exits non-zero, any additional pos=
tflights for the job instance are skipped, and the job instance is reported=
to be "failed".

If an agenda-level postflight exits non-zero, any additional postflight=
s for the agenda are skipped, and the agenda is reported as "failed&qu=
ot;. The job instance will move onto processing the next agenda.

Qube-specific environment variables set in the f=
lightCheck environments

Variables in both instance-level and agenda-level fligh=
tChecks:

QB_JOBSLOTS (how many job slots got allocated, can var=
y when using the "host.processors=3DN+" reservation)

QB_SUPERVISOR

Additional variables in agenda-level flightChecks:

QB_FRAME_END

QB_FRAME_NUMBER

QB_FRAME_PADDING

QB_FRAME_RANGE

QB_FRAME_START

QB_FRAME_STEP

QB_PADDED_FRAME_END

QB_PADDED_FRAME_NUMBER

QB_PADDED_FRAME_START

QB_PADDED_FRAME_STEP

A=
ddtional variables in postFlight checks

QB_INSTANCE_STATUS

QB_FRAME_STATUS

QB_SYSTEM_EXIT_CODE (only set by cmdline and cmdrange =
jobs)

In postflight scripts, access the status of the last-processed&nbsp=
;frame or instance via the QB_FRAME_STATUS and&n=
bsp;QB_INSTANCE_STATUS environment variables, respect=
ively. These will be set to strings such as "complete" and "=
failed".