The thread pool attribute (pool_attr) controls
various aspects of the thread pool, such as which functions
get called when a new thread is started or dies, the total
number of worker threads, the minimum number, and so on.

The functions that you fill into the above structure can be taken from
the dispatch layer (dispatch_block(), ...),
the resmgr layer (resmgr_block(), ...) or they can be of your own making.
If you're not using the resmgr layer functions,
then you'll have to define THREAD_POOL_PARAM_T to
some sort of context structure for the library to pass between the various functions.
By default, it's defined as a resmgr_context_t but since
this sample is using the dispatch layer,
we needed it to be adispatch_context_t.
We defined it prior to doing the includes above since the header files refer to it.
THREAD_POOL_PARAM_T

Part of the above structure contains information telling the
resource manager library how you want it to handle multiple
threads (if at all). During development, you should design
your resource manager with multiple threads in mind. But
during testing, you'll most likely have only one thread
running (to simplify debugging). Later, after you've ensured
that the base functionality of your resource manager is
stable, you may wish to “turn on” multiple
threads and revisit the debug cycle.

The following members control the number of threads that are running:

lo_water

Minimum number of blocked threads.

increment

Number of thread to create at a time to achieve lo_water.

hi_water

Maximum number of blocked threads.

maximum

Total number of threads created at any time.

The important parameters specify the maximum thread count and the increment.
The value for maximum should ensure that there's always a thread in a RECEIVE-blocked state.
If you're at the number of maximum threads,
then your clients will block until a free thread is ready to receive data.
The value you specify for increment will cut down on the number of times your driver needs to create threads.
It's probably wise to err on the side of creating more threads and
leaving them around rather than have them being created/destroyed all the time.

You determine the number of threads you want to be
RECEIVE-blocked on the MsgReceive() at any time
by filling in the lo_water parameter.

If you ever have fewer than lo_water threads
RECEIVE-blocked, the increment parameter
specifies how many threads should be created at once, so
that at least lo_water number of threads are once
again RECEIVE-blocked.

Once the threads are done their processing, they will return
to the block function. The hi_water variable
specifies an upper limit to the number of threads that are
RECEIVE-blocked. Once this limit is reached, the threads
will destroy themselves to ensure that no more than
hi_water number of threads are RECEIVE-blocked.

To prevent the number of threads from increasing without
bounds, the maximum parameter limits the absolute
maximum number of threads that will ever run simultaneously.

When threads are created by the resource manager library,
they'll have a stack size as specified by the
thread_stack_size parameter. If you want to
specify stack size or priority, fill in pool_attr.attr
with a proper pthread_attr_t pointer.

The thread_pool_attr_t structure contains
pointers to several functions:

block_func()

Called by the worker thread when it needs to block waiting for some message.

handler_func()

Called by the thread when it has unblocked because it received a message.
This function processes the message.

context_alloc()

Called when a new thread is created.
Returns a context that this thread uses to do its work.

context_free()

Free the context when the worker thread exits.

unblock_func()

Called by the library to shutdown the thread pool or change the number of running threads.

Start the thread pool.
This function may or may not return, depending on the flags passed to thread_pool_create().

thread_pool_destroy()

Destroy a thread pool.

thread_pool_control()

Control the number of threads.

In the example provided in the
multithreaded resource managers
section, thread_pool_start(tpp) never returns because we
set the POOL_FLAG_EXIT_SELF bit.
Also, the POOL_FLAG_USE_SELF flag itself never returns,
but the current thread becomes part of the thread pool.

If no flags are passed (i.e. 0 instead of any flags),
the function returns after the thread pool is created.