The Eventer

The eventer is designed to perform micro tasks without the overhead of a
context switch. The underlying goal is to support millions of "seemingly"
concurrent heavy tasks by modifying the tasks to be reactive to state changes,
make small, non-blocking progress, and yielding control back to the event loop.

Not all work can be done in a non-blocking fashion (e.g. disk reads/writes,
and intense computational work). For this, the eventer provides work queues
that allow for blocking operations.

Events (eventer_t) have a callback and a closure at their heart. The
rest of the fields dictate when, why and possibly where the callback will be
invoked with the provided closure.

Event types are dictated by the mask set in the eventer_t object.
There are four basic event types available for use:

File Descriptor Activity

File descriptor based events ("fd events") can react to any of three conditions (by bitwise OR):

EVENTER_READ : the availability of data to read

EVENTER_WRITE : the availability of buffer space so a write may succeed

EVENTER_EXCEPTION : an error condition has occured on the file descriptor

Note that under most circumstances, the file descriptor should be a socket. In
typical POSIX systems, these fd events don't fire as expected on files.

The return value of callbacks for fd events represents the new
mask that should be used (for subsequent callback invocations). If 0 is
returned, the event will be removed from the system and eventer_free will
be invoked.

Operations on the file descriptor

While the fd field of the eventer_t is a normal file descriptor and normal
POSIX operations can be performed on it (such as read(2), write(2), etc.). These
operations are all abstracted away behind the eventer_read, eventer_write, etc.
convenience functions.

The opset can be changed to support SSL operations and, in such operations, some
non-blocking operations can require some non-obvious events to make progress.
Namely, an SSL read may fail with EAGAIN, but require an EVENTER_WRITE event
to continue due to a renegotiation.

Asynchronous Events

Asynchronous events within libmtev run within the context of an eventer_jobq_t.
Programmatically, or via configuration, these job queues can be created. The
context of the job queue can control how many threads are running and how many
events may be queued using an advisory limit. Job queue concurrency can be
controlled at run-time.

Job queues have several attributes related to concurrency: min, floor, desired, and max.

The min and the max describe "functional correctness" constraints on the jobq. For example,
if it would "malfunction" if more than one job ran concurrently, then the max should be set
to one. Setting the concurrency (or desired concurrency) for a jobq sets a target thread
count to be achieved if there is work to be done. Floor sets a minimum thread count to
maintain even if there is no work to be done. Min and max should not be changed at run-time
and provide the domain boundaries for desired.

A jobq thread B will invoke the callback with mask = EVENTER_ASYNCH_WORK if whence is set and in the future.

Thread B will invoke the callback with mask = EVENTER_ASYNCH_CLEANUP

The event will return to thread A and callback will be invoked with mask = EVENTER_ASYNCH

Choosing Threads

By default new events are created on the "current" event loop thread. This
has the effect of causing all new connections from a listener to gang on a
single event loop thread. If an event is added from a non-event-loop thread,
it will be assigned to thread 1.

The thr_owner field of the eventer_t structure describes which event
loop thread owns the event. This can be changed using the eventer_choose_owner_pool
and eventer_choose_owner functions.

To take an event and move it to a random thread within its current eventer pool:

move_event2.c (snippet)

To switch an event from one thread to another, simply reassign the
thr_owner and then return immediately with the desired mask. The
eventer will reschedule the event on the requested thread. Be careful
to not ping-pong back and forth without making proress!