Often when writing code for an IOC there is no obvious task under which to execute. A good example is completion code
for an asynchronous device support module. EPICS supplies the callback tasks for such code.

If an IOC tasks ``crashes" there is normally no one monitoring the vxWorks shell to detect the problem. EPICS provides a
task watchdog task which periodically checks the state of other tasks. If it finds that a monitored task has terminated or
suspended it issues an error message and can also call other routines which can take additional actions. For example a
subroutine record can arrange to be put into alarm if a monitored task crashes.

Since IOCs normally run autonomously, i.e. no one is monitoring the vxWorks shell, IOC code that issues printf calls
generates errors messages that are never seen. In addition the vxWorks implementation of fprintf requires much more
stack space then printf calls. Another problem with vxWorks is the logMsg facility. logMsg generates messages at
higher priority then all other tasks except the shell. EPICS solves all of these problems via an error message handling
facility. Code can call any of the routines errMessage, errPrintf, or errlogPrintf. Any of these result in the
error message being generated by a separate low priority task. The calling task has to wait until the message is handled but
other tasks are not delayed. In addition the message can be sent to a system wide error message file.

16.2 General Purpose Callback Tasks

EPICS provides three sets of general purpose IOC callback tasks. The only difference between the task sets is their scheduling
priority: low, medium or high. The low priority tasks runs at a priority just higher than Channel Access, the medium priority tasks at a
priority about equal to the median of the periodic scan tasks, and the high priority tasks at a priority higher than the event scan task. The
callback tasks are available for any software component that needs a task under which to run some job either immediately
or after some delay. Jobs can also be cancelled during their delay period. The callback tasks register themselves with
the task watchdog (described below). They are created with a generous amount of stack space and can thus be used for
invoking record processing. For example the I/O event scanner uses the general purpose callback tasks.

The number of general purpose threads per priority level is configurable.
On SMP systems with multi-core CPUs, the throughput can be improved and the latency (time between job scheduling and processing) can be lowered by running multiple parallel callback tasks, which the OS scheduler may assign to different CPU cores. Parallel callback tasks must be explicitly enabled (see 16.2.5 below), as this feature is disabled by default for compatibility reasons.

The following steps must be taken in order to use the general purpose callback tasks:

Include callback definitions:

#include <callback.h>

Provide storage for a structure that is a private structure for the callback tasks:

CALLBACK mycallback;

It is permissible for this to be part of a larger structure, e.g.

struct {
...
CALLBACK mycallback;
...
} ...

Make calls (in most cases these are actually macros) to initialize the fields in the CALLBACK:

callbackSetCallback(CALLBACKFUNC func, CALLBACK *pcb);

This defines the callback routine to be executed. The first argument is the address of a function that will be given the address of the CALLBACK and returns void. The second argument is the address of the CALLBACK structure.

callbackSetPriority(int, CALLBACK *pcb);

The first argument is the priority, which can have one of the values: priorityLow, priorityMedium, or
priorityHigh. These values are defined in callback.h. The second argument is again the address of the
CALLBACK structure.

callbackSetUser(void *, CALLBACK *pcb);

This call is used to save a pointer value that can be retrieved again using the macro:

callbackGetUser(void *,CALLBACK *pcb);

If your callback function exists to process a single record inside calls to dbScanLock/dbScanUnlock, you can use this shortcut which provides the callback routine for you and sets the other two parameters at the same time
(the user parameter here is a pointer to the record instance):

callbackSetProcess(CALLBACK *pcb, int prio, void *prec);

Whenever a callback request is desired just call one of the following:

Both can be called from interrupt level code. The callback routine is passed a single argument, which is the same argument that was passed to callbackRequest, i.e., the address of the CALLBACK structure. The second routine is a shortcut for calling both callbackSetProcess and callbackRequest. Both return zero in case of success, or an error code (see below).

The following delayed versions wait for the given time before queueing the callback routine for the relevant thread set to execute.

Both callbackRequest and callbackRequestProcessCallback may be called from interrupt context. Both return zero for success, or one of the following error codes: S_db_notInit for a NULL callback pointer, S_db_badChoice for an illegal priority value, or S_db_bufFull when the associated queue is full.

The delayed versions of the callbackRequest routines wait the given time before queueing the callback.

The callback requests put the requests for each callback priority into a separate ring buffer. These buffers can by default
hold up to 2000 requests. This limit can be changed by calling callbackSetQueueSize before iocInit in the
startup file. The syntax is:

int callbackSetQueueSize(int size)

16.2.5 Parallel Callback Tasks

To enable multiple parallel callback tasks, and set the number of tasks to be started for each priority level, call
callbackParallelThreads before iocInit in the startup file. The syntax is:

int callbackParallelThreads(int count, const char *prio)

The count argument is the number of tasks to start, with 0 indicating to use the default (number of CPUs), and negative numbers indicating to use the number of CPUs minus the specified amount.

The prio argument specifies the priority level, with "" (empty string), "*", or NULL indicating to apply the definition to all priority levels.

The default value is stored in the variable callbackParallelThreadsDefault (initialized to the number of CPUs), which can be changed using the iocShell's var command.

16.3 Task Watchdog

EPICS provides a task that acts as a watchdog for other tasks. Any task can request to be watched, and most of the IOC
tasks do this. A status monitoring subsystem in the IOC can register to be notified about any changes that occur. The
watchdog task runs periodically and checks each task in its task list. If any task is suspended, an error message is
displayed and any notifications made. The task watchdog provides the following features:

Include module:

#include <taskwd.h>

Request by a task to be monitored:

taskwdInsert (epicsThreadId tid, TASKWDFUNC callback, VOID *usr);

This adds the task with the specified tid to the list of tasks to be watched, and makes any requested notifications
that a new task has been registered. If tid is given as zero, the epicsThreadId of the calling thread is used
instead. If callback is not NULL and the task later becomes suspended, the callback routine will be called with
the single argument usr.

Remove task from list:

taskwdRemove(epicsThreadId tid);

This routine must be called before the monitored task exits. It makes any requested notifications and removes the
task from the list of tasks being watched. If tid is given as zero, the epicsThreadId of the calling thread is
used instead.

This call provides a set of callbacks for the task watchdog to call when a task is registered or removed or when any
task gets suspended. The usr pointer given at registration is passed to the callback routine along with the tid of
the thread the notification is about. In many cases the insert and remove callbacks will be called from the
context of the thread itself, although this is not guaranteed (the registration could be made by a parent thread for
instance). The notify callback also indicates whether the task went into or out of suspension; it is called in both
cases, unlike the callbacks registered with taskwdInsert and taskwdAnyInsert.

Rescind notification request:

taskwdMonitorDel(const taskwdMonitor *funcs, void *usr);

This call removes a previously registered notification. Both funcs and usr must match the values given to
taskwdMonitorAdd when originally registered.

Print a report:

taskwdShow(int level);

If level is zero, the number of tasks and monitors registered is displayed. For higher values the registered task
names and their current states are also shown in tabular form.

The following routines are provided for backwards compatibility purposes, but are now deprecated:

taskwdAnyInsert(void *key, TASKWDANYFUNC callback, VOID *usr);

The callback routine will be called whenever any of the tasks being monitored by the task watchdog become
suspended. key must have a unique value because the task watchdog system uses this value to determine which
entry to remove when taskwdAnyRemove is called.