Chapter 13 STREAMS Multiplex Drivers

This chapter describes how STREAMS multiplexing configurations are created and
also discusses multiplexing drivers. A STREAMS multiplexer is
a driver with multiple streams connected to it. The primary function of the multiplexing
driver is to switch messages among the connected streams. Multiplexer configurations
are created from the user level by system calls.

STREAMS Multiplexers

STREAMS-related system calls are used to set up the “plumbing,”
or stream interconnections, for multiplexing drivers. The subset of these calls that
allows a user to connect (and disconnect) streams below a driver is referred to as multiplexing. This type of connection is referred to as a one-to-M, or
lower, multiplexer configuration. This configuration must always contain a multiplexing
driver, which is recognized by STREAMS as having special characteristics.

Multiple streams can be connected above a driver by open(2) calls. This accommodates the loop-around
driver and the driver that handled multiple minor devices in Chapter 9, STREAMS Drivers. There is no difference between the connections
to these drivers. Only the functions performed by the driver are different. In the
multiplexing case, the driver routes data between multiple streams. In the device
driver case, the driver routes data between user processes and associated physical
ports. Multiplexing with streams connected above is referred to as an N-to-1, or upper,
multiplexer. STREAMS does not provide any facilities beyond open and close to connect
or disconnect upper streams for multiplexing.

From the driver's perspective, upper and lower configurations differ only in
the way they are initially connected to the driver. The implementation requirements
are the same: route the data and handle flow control. All multiplexer drivers require
special developer-provided software to perform the multiplexing data routing and to
handle flow control. STREAMS does not directly support flow control among multiplexed
streams. M-to-N multiplexing configurations are implemented by using both of these
mechanisms in a driver.

As discussed in Chapter 9, STREAMS Drivers, the multiple streams that represent minor devices are actually distinct
streams in which the driver keeps track of each stream attached to it. The STREAMS
subsystem does not recognize any relationship between the streams. The same is true
for STREAMS multiplexers of any configuration. The multiplexed streams are distinct
and the driver must be implemented to do most of the work.

In addition to upper and lower multiplexers, more complex configurations can
be created by connecting streams containing multiplexers to other multiplexer drivers.
With such a diversity of needs for multiplexers, providing general-purpose multiplexer
drivers is not possible. Rather, STREAMS provides a general purpose multiplexing facility.
The facility enables you to set up the intermodule or driver plumbing to create multiplexer
configurations of generally unlimited interconnection.

Building a Multiplexer

The example in this section builds a protocol multiplexer
with the multiplexing configuration shown in Figure 13–1. To free users from the need to know about the underlying
protocol structure, a user-level daemon process is built to maintain the multiplexing
configuration. Users can then access the transport protocol directly by opening the
transport protocol (TP) driver device node.

An internetworking protocol driver (IP) routes data from a single upper stream
to one of two lower streams. This driver supports two STREAMS connections beneath
it. These connections are to two distinct networks; one for the IEEE 802.3 standard
through the 802.3 driver, and another to the IEEE 802.4 standard through the 802.4
driver. The TP driver multiplexes upper streams over a single stream to the IP driver.

Figure 13–1 Protocol Multiplexer

Example 13–1 shows how this
daemon process sets up the protocol multiplexer. The necessary declarations and initialization
for the daemon program follow.

This multilevel multiplexed stream configuration is built from the bottom up.
The example begins by first constructing the IP multiplexer. This multiplexing device
driver is treated like any other software driver. It owns a node in the Solaris file
system and is opened just like any other STREAMS device driver.

The first step is to open the multiplexing driver and the 802.4 driver, thus
creating separate streams above each driver as shown in Figure 13–2. The stream to the 802.4 driver may now be connected below
the multiplexing IP driver using the I_LINKioctl(2).

I_LINK takes two file descriptors as arguments. The first
file descriptor, fd_ip, is the stream connected to the multiplexing
driver, and the second file descriptor, fd_802_4, is the stream
to be connected below the multiplexer. The complete stream to the 802.4 driver is
connected below the IP driver. The stream head's queues of the 802.4 driver are used
by the IP driver to manage the lower half of the multiplexer.

I_LINK returns an integer value, muxid, which is used
by the multiplexing driver to identify the stream just connected below it. muxid is ignored in the example, but it is useful for dismantling a multiplexer
or routing data through the multiplexer. Its significance is discussed in Dismantling a Multiplexer.

The following sequence of system calls continues building the Internetworking
Protocol multiplexer (IP):

The stream above the multiplexing driver
used to establish the lower connections is the controlling stream and has special
significance when dismantling the multiplexing configuration. This is illustrated
in Dismantling a Multiplexer. The stream
referenced by fd_ip is the controlling stream for the IP multiplexer.

The order in which the streams in the multiplexing configuration are opened
is unimportant. If intermediate modules in the stream are necessary between the IP
driver and media drivers, these modules must be added to the streams associated with
the media drivers (using I_PUSH) before the media drivers are attached
below the multiplexer.

The number of streams that can be linked to a multiplexer is restricted by the
design of the particular multiplexer. The manual page describing each driver describes
such restrictions (see SunOS Reference Manual, Intro(7)). However, only one I_LINK operation is allowed for each lower stream; a single stream cannot be linked
below two multiplexers simultaneously.

Continuing with the example, the IP driver is now linked below the transport
protocol (TP) multiplexing driver. As seen in Figure 13–1, only one link is supported below the transport driver.
This link is formed by the following sequence of system calls:

Because the controlling stream of the IP multiplexer has been linked below the
TP multiplexer, the controlling stream for the new multilevel multiplexer configuration
is the stream above the TP multiplexer.

At this point, the file descriptors associated with the lower drivers can be
closed without affecting the operation of the multiplexer. If these file descriptors
are not closed, all subsequent read(2), write(2), ioctl(2), poll(2), getmsg(2),
and putmsg(2) calls issued
to them fail. That is because I_LINK associates the stream head
of each linked stream with the multiplexer, so the user may not access that stream
directly for the duration of the link.

The transport driver supports several simultaneous streams. These streams are
multiplexed over the single stream connected to the IP multiplexer. The mechanism
for establishing multiple streams above the transport multiplexer is actually a by-product
of the way in which streams are created between a user process and a driver. By opening
different minor devices of a STREAMS driver, separate streams will be connected to
that driver. The driver must be designed with the intelligence to route data from
the single lower stream to the appropriate upper stream.

The daemon process maintains the multiplexed stream configuration through an
open stream (the controlling stream) to the transport driver. Meanwhile, other users
can access the services of the transport protocol by opening new streams to the transport
driver; they are freed from the need for any unnecessary knowledge of the underlying
protocol configurations and subnetworks that support the transport service.

Multilevel multiplexing configurations should be assembled
from the bottom up. That is because the passing of ioctl(2) through
the multiplexer is determined by the nature of the multiplexing driver and cannot
generally be relied on.

Dismantling a Multiplexer

Streams connected to a multiplexing driver from above with open(2), can be dismantled by closing each stream with close(2). The mechanism for dismantling streams that have
been linked below a multiplexing driver is less obvious, and is described in Disconnecting Lower Streams.

I_UNLINKioctl(2) disconnects each multiplexer link
below a multiplexing driver individually. This command has the form:

ioctl(fd, I_UNLINK, muxid);

where fd is a
file descriptor associated with a stream connected to the multiplexing driver from
above, and muxid is the identifier that was returned by I_LINK when a driver was linked below the multiplexer. Each lower driver
may be disconnected individually in this way, or a special muxid value
of MUXID_ALL can be used to disconnect all drivers from the multiplexer
simultaneously.

In the multiplexing daemon program, the multiplexer is never explicitly dismantled.
That is because all links associated with a multiplexing driver are automatically
dismantled when the controlling stream associated with that multiplexer is closed.
Because the controlling stream is open to a driver, only the final call of close for that stream will close it. In this case, the daemon is the only
process that has opened the controlling stream, so the multiplexing configuration
will be dismantled when the daemon exits.

For the automatic dismantling mechanism to work in the multilevel, multiplexed
stream configuration, the controlling stream for each multiplexer at each level must
be linked under the next higher-level multiplexer. In the example, the controlling
stream for the IP driver was linked under the TP driver. This resulted in a single
controlling stream for the full, multilevel configuration. Because the multiplexing
program relied on closing the controlling stream to dismantle the multiplexed stream
configuration instead of using explicit I_UNLINK calls, the muxid values returned by I_LINK could be ignored.

An important side effect of automatic dismantling on the close is that a process
cannot build a multiplexing configuration with I_LINK and then
exit. exit(2) closes all files
associated with the process, including the controlling stream. To keep the configuration
intact, the process must exist for the life of that multiplexer. That is the motivation
for implementing the multiplexer as a daemon processs, see Multiplexing Driver Example.

If the process uses persistent links through I_PLINKioctl(2), the multiplexer configuration remains
intact after the process exits. These links are described in Persistent Links.

Routing Data Through a Multiplexer

As demonstrated, STREAMS provides a mechanism
for building multiplexed stream configurations. However, the criteria by which a multiplexer
routes data are driver dependent. For example, the protocol multiplexer might use
address information found in a protocol header to determine the subnetwork over which
data should be routed. You must define its routing criteria.

One routing option available to the multiplexer is to use the muxid value to determine the stream to which data is routed (remember that each
multiplexer link has a muxid). I_LINK passes
the muxid value to the driver and returns this value to the user.
The driver can therefore specify that the muxid value accompany
data routed through it. For example, if a multiplexer routed data from a single upper
stream to one of several lower streams (as did the IP driver), the multiplexer can
require the user to insert the muxid of the desired lower stream
into the first four bytes of each message passed to it. The driver can then match
the muxid in each message with the muxid of
each lower stream, and route the data accordingly.

Connecting And Disconnecting Lower Streams

Multiple streams are created above a driver/multiplexer by use of the
open system call on either different minor device, or on a cloneable device file.
Note that any driver that handles more than one minor device is considered an upper
multiplexer.

To connect streams below a multiplexer requires additional software in the multiplexer.
The main difference between STREAMS lower multiplexers and STREAMS device drivers
is that multiplexers are pseudo-devices and multiplexers have two additional qinit structures, pointed to by fields in streamtab(9S):
the lower half read-side qinit(9S) and
the lower half write-side qinit(9S).

The multiplexer is divided into two parts: the lower half and the upper half.
The multiplexer queue structures allocated when the multiplexer was opened use the
usual qinit entries from the multiplexer's streamtab(9S). This is the same as any open of the STREAMS
device. When a lower stream is linked beneath the multiplexer, the qinit structures at the stream head are substituted by the lower half qinit(9S) structures identified in the streamstab for the multiplexers. Once the linkage is made, the multiplexer
switches messages between upper and lower streams. When messages reach the top of
the lower stream, they are handled by put and service routines specified in the bottom half of the multiplexer.

Connecting Lower Streams

A lower multiplexer is connected as follows:
the initial open to a multiplexing driver creates a stream, as
in any other driver. open uses the st_rdinit and st_wrinit elements of the streamtab structure
to initialize the driver queues.. At this point, the only distinguishing characteristics
of this stream are non-NULL entries in the streamtab(9S)st_muxrinit and st_muxwinit fields.

These fields are ignored by open. Any other stream subsequently
opened to this driver will have the same streamtab and thereby
the same mux fields.

Next, another file is opened to create a (soon-to-be) lower stream. The driver
for the lower stream is typically a device driver This stream has no distinguishing
characteristics. It can include any driver compatible with the multiplexer. Any modules
required on the lower stream must be pushed onto it now.

Next, this lower stream is connected below the multiplexing driver with an I_LINKioctl(2) (see streamio(7I)). The stream head points to the stream head
routines as its procedures (through its queue). An I_LINK to the
upper stream, referencing the lower stream, causes STREAMS to modify the contents
of the stream head's queues in the lower stream. The pointers to the stream head routines,
and other values, in the stream head's queues are replaced with those contained in
the mux fields of the multiplexing driver's streamtab. Changing the stream head routines on the lower stream means
that all subsequent messages sent upstream by the lower stream's driver are passed
to the put procedure designated in st_muxrinit,
the multiplexing driver. The I_LINK also establishes this upper
stream as the control stream for this lower stream. STREAMS remembers the relationship
between these two streams until the upper stream is closed or the lower stream is
unlinked.

Finally, the stream head sends an M_IOCTL message with ioc_cmd set to I_LINK to the multiplexing driver. The M_DATA part of the M_IOCTL contains a linkblk(9S) structure. The multiplexing driver stores information
from the linkblk(9S) structure
in private storage and returns an M_IOCACK acknowledgement. l_index is returned to the process requesting the I_LINK.
This value is used later by the process to disconnect the stream.

An I_LINK is required for each lower stream connected to
the driver. Additional upper streams can be connected to the multiplexing driver by
open calls. Any message type can be sent from a lower stream to user processes along
any of the upper streams. The upper streams provide the only interface between the
user processes and the multiplexer.

No direct data structure linkage is established for the linked streams. The
read queue's q_next is NULL and the write queue's q_next points to the first entity on the lower stream. Messages flowing upstream
from a lower driver (a device driver or another multiplexer) will enter the multiplexing
driver put procedure with the queue represented in l_qbot as the queue_t for the put procedure. The multiplexing driver has to route the messages to the appropriate
upper (or lower) stream. Similarly, a message coming downstream from user space on
any upper stream has to be processed and routed, if required, by the driver.

Note –

It is the responsibility of the driver to handle routing of messages between
the upper and lower streams, or between any lateral stream that is part of the multiplexer.
This operation is not handled by the STREAMS framework.

In general, multiplexing drivers should be
implemented so that new streams can be dynamically connected to (and existing streams
disconnected from) the driver without interfering with its ongoing operation. The
number of streams that can be connected to a multiplexer is implementation dependent.

Disconnecting Lower Streams

Dismantling a lower multiplexer
is accomplished by disconnecting (unlinking) the lower streams. Unlinking can be initiated
in three ways:

As in the link, an unlink sends a linkblk(9S) structure
to the driver in an M_IOCTL message. The I_UNLINK call,
which unlinks a single stream, uses the l_index value returned
in the I_LINK to specify the lower stream to be unlinked. The latter
two calls must designate a file corresponding to a control stream, which causes all
the lower streams that were previously linked by this control stream to be unlinked.
However, the driver sees a series of individual unlinks.

If no open references exist for a lower stream, a subsequent unlink will automatically
close the stream. Otherwise, the lower stream must be closed by close(2) following the unlink. STREAMS automatically dismantles
all cascaded multiplexers (below other multiplexing streams) if their controlling
stream is closed. An I_UNLINK leaves lower, cascaded multiplexing
streams intact unless the stream file descriptor was previously closed.

Multiplexer Construction Example

This section describes an example of multiplexer construction
and usage. Multiple upper and lower streams interface to the multiplexer driver.

The Ethernet, LAPB, and IEEE 802.2 device drivers terminate
links to other nodes. The multiplexer driver is an Internet Protocol (IP) multiplexer
that switches data among the various nodes or sends data upstream to users in the
system. The net modules typically provide a convergence function that matches the
multiplexer driver and device driver interface.

Streams A, B, and C are opened by the process, and modules are pushed as needed.
Two upper streams are opened to the IP multiplexer. The rightmost stream represents
multiple streams, each connected to a process using the network. The stream second
from the right provides a direct path to the multiplexer for supervisory functions.
The control stream, leading to a process, sets up and supervises this configuration.
It is always directly connected to the IP driver. Although not shown, modules can
be pushed on the control stream.

After the streams are opened, the supervisory process typically transfers routing
information to the IP drivers (and any other multiplexers above the IP), and initializes
the links. As each link becomes operational, its stream is connected below the IP
driver. If a more complex multiplexing configuration is required, the IP multiplexer
stream with all its connected links can be connected below another multiplexer driver.

Multiplexing Driver Example

This section contains an example of a multiplexing driver
that implements an N-to-1 configuration. This configuration might be used for terminal
windows, where each transmission to or from the terminal identifies the window. This
resembles a typical device driver, with two differences: the device-handling functions
are performed by a separate driver, connected as a lower stream, and the device information
(that is, relevant user process) is contained in the input data rather than in an
interrupt call.

Each upper stream is created by open(2).
A single lower stream is opened and then it is linked by use of the multiplexing facility.
This lower stream might connect to the TTY driver. The implementation of this example
is a foundation for an M-to-N multiplexer.

As in the loop-around driver (Chapter 9, STREAMS Drivers), flow control requires the use of standard and special code
because physical connectivity among the streams is broken at the driver. Different
approaches are used for flow control on the lower stream, for messages coming upstream
from the device driver, and on the upper streams, for messages coming downstream from
the user processes.

Note –

The code presented here for the multiplexing driver represents a single-threaded,
uniprocessor implementation. See Chapter 12, Multithreaded STREAMS for details on multiprocessor and multithreading issues
such as locking for data corruption and to prevent race conditions.

The four streamtab entries correspond to the upper
read, upper write, lower read, and lower write qinit structures.
The multiplexing qinit structures replace those in each lower stream
head (in this case there is only one) after the I_LINK has concluded
successfully. In a multiplexing configuration, the processing performed by the multiplexing
driver can be partitioned between the upper and lower queues. There must be an upper-stream
write put procedure and lower-stream read put procedure.
If the queue procedures of the opposite upper/lower queue are not needed, the queue
can be skipped, and the message put to the following queue.

In the example, the upper read-side procedures are not used. The lower-stream
read queue put procedure transfers the message directly to the
read queue upstream from the multiplexer. There is no lower write put procedure
because the upper write put procedure directly feeds the lower
write queue downstream from the multiplexer.

The driver uses a private data structure, mux. mux_mux[dev] points
back to the opened upper read queue. This is used to route messages coming upstream
from the driver to the appropriate upper queue. It is also used to find a free major
or minor device for a CLONEOPEN driver open case.

muxopen checks for a clone or ordinary open call. It initializes q_ptr to point at the mux_mux[] structure.

The core multiplexer processing is as follows: downstream data written to an
upper stream is queued on the corresponding upper write message queue if the lower
stream is flow controlled. This allows flow control to propagate toward the stream
head for each upper stream. A lower write service procedure, rather than a write put procedure, is used so that flow control, coming up from the driver below,
may be handled.

On the lower read side, data coming up the lower stream are passed to the lower
read put procedure. The procedure routes the data to an upper stream
based on the first byte of the message. This byte holds the minor device number of
an upper stream. The put procedure handles flow control by testing
the upper stream at the first upper read queue beyond the driver.

First, there is a check to enforce that the stream
associated with minor device 0 will be the single, controlling stream. The ioctls are only accepted on this stream. As described previously, a controlling
stream is the one that issues the I_LINK. There should be only
a single control stream. I_LINK and I_UNLINK include
a linkblk structure containing the following fields:

l_qtop is the upper write queue from which the ioctl(2) comes. It always equals q for an I_LINK, and NULL for I_PLINK.

l_qbot is the new lower write queue. It is the former
stream head write queue and is where the multiplexer gets and puts its data.

l_index is a unique (system-wide) identifier for
the link. It can be used for routing or during selective unlinks. Since the example
only supports a single link, l_index is not used.

For I_LINK, l_qbot is saved in muxbot and a positive acknowledgement is generated. From this point on,
until an I_UNLINK occurs, data from upper queues will be routed
through muxbot. Note that when an I_LINK, is
received, the lower stream has already been connected. This enables the driver to
send messages downstream to perform any initialization functions. Returning an M_IOCNAK message (negative acknowledgement) in response to an I_LINK causes the lower stream to be disconnected.

The I_UNLINK handling code nulls
out muxbot and generates a positive acknowledgement. A negative
acknowledgement should not be returned to an I_UNLINK. The stream
head ensures that the lower stream is connected to a multiplexer before sending an I_UNLINKM_IOCTL.

Drivers can handle
the persistent link requests I_PLINK and I_PUNLINKioctl(2) in the same manner, except that l_qtop in the linkblk structure passed to the put routine is NULL instead of identifying the controlling stream.

muxuwput handles M_FLUSH messages as a
normal driver does, except that there are no messages queued on the upper read queue,
so there is no need to call flushq if FLUSHR is
set.

M_DATA messages are not placed on the lower write message
queue. They are queued on the upper write message queue. When flow control subsides
on the lower stream, the lower service procedure, muxlwsrv, is
scheduled to start output. This is similar to starting output on a device driver.

Upper Write service Procedure
Sample

The following
example shows the code for the upper multiplexer write service procedure:

muxlrput receives messages from the linked stream. In this
case, it is acting as a stream head and handles M_FLUSH messages.
The code is the reverse of a driver, handling M_FLUSH messages
from upstream. There is no need to flush the read queue because no data is ever placed
in it.

muxlrput also handles M_ERROR and M_HANGUP messages. If one is received, it locks up the upper streams by
setting muxerr.

M_DATA messages are routed by checking the first data byte
of the message. This byte contains the minor device of the upper stream. Several checks
examine whether:

The device is in range

The upper stream is open

The upper stream is full

This multiplexer
does not support flow control on the read side; it is merely a router. If the message
passes all checks, it is put to the proper upper queue. Otherwise, the message is
discarded.

The upper stream close routine clears the mux entry
so this queue will no longer be found. Outstanding bufcalls are
not cleared.

Persistent Links

Keeping a process running
merely to hold the multiplexer configuration together is not always desirable, so, “free
standing” links below a multiplexer are needed. A persistent link is such a
link. It is similar to a STREAMS multiplexer link except that a process is not needed
to hold the links together. After the multiplexer has been set up, the process may
close all file descriptors and exit, and the multiplexer remains intact.

With I_LINK and I_UNLINKioctl(2) the file descriptor associated with the stream
above the multiplexer used to set up the lower multiplexer connections must remain
open for the duration of the configuration. Closing the file descriptor associated
with the controlling stream dismantles the whole multiplexing configuration.

Two ioctl(2)s, I_PLINK and I_PUNLINK, are used to create and remove persistent links that are associated
with the stream above the multiplexer. close(2) and I_UNLINK are not able to disconnect the persistent links (see strconf(1) and strchg(1)).

The format of I_PLINK is:

ioctl(fd0, I_PLINK, fd1)

The first file descriptor, fd0, must reference the stream
connected to the multiplexing driver and the second file descriptor, fd1, must reference the stream to be connected below the multiplexer. The persistent
link can be created as follows:

The persistent link can still exist even
if the file descriptor associated with the upper stream to the multiplexing driver
is closed. The I_PLINKioctl(2) returns
an integer value, muxid, that can be used for dismantling the multiplexing
configuration. If the process that created the persistent link still exists, it may
pass the muxid value to some other process to dismantle the link,
if the dismantling is desired, or it can leave the muxid value
in a file so that other processes may find it later.

Several users can open the MUX driver and send data to Driver1 since the persistent
link to Driver1 remains intact.

The I_PUNLINKioctl(2) is used to dismantle the persistent link. Its
format is:

ioctl(fd0, I_PUNLINK, muxid)

where fd0 is the file descriptor associated with stream connected
to the multiplexing driver from above. The muxid is returned by
the I_PLINKioctl(2) for
the stream that was connected below the multiplexer. I_PUNLINK removes
the persistent link between the multiplexer referenced by fd0 and
the stream to the driver designated by the muxid. Each of the bottom
persistent links can be disconnected individually. An I_PUNLINKioctl(2) with the muxid value
of MUXID_ALL will remove all persistent links below the multiplexing
driver referenced by fd0.

The following code example shows how to dismantle the previously given configuration:

Do not use the I_PLINK and I_PUNLINKioctls with I_LINK and I_UNLINK. Any
attempt to unlink a regular link with I_PUNLINK or to unlink a
persistent link with the I_UNLINKioctl(2) causes
the errno value of EINVAL to be returned.

Because multilevel multiplexing configurations are allowed in STREAMS, persistent
links could exist below a multiplexer whose stream is connected to the above multiplexer
by regular links. Closing the file descriptor associated with the controlling stream
will remove the regular link but not the persistent links below it. On the other hand,
regular links are allowed to exist below a multiplexer whose stream is connected to
the above multiplexer with persistent links. In this case, the regular links will
be removed if the persistent link above is removed and no other references to the
lower streams exist.

The construction of cycles is not allowed when creating links. A cycle could
be constructed by:

Creating a persistent link of multiplexer 2 below multiplexer 1

Closing the controlling file descriptor associated with the multiplexer
2

Reopening the file descriptor again

Linking the multiplexer 1 below the multiplexer 2

This is not allowed. The operating system prevents a multiplexer configuration
from containing a cycle to ensure that messages cannot be routed infinitely, which
would create an infinite loop or overflow the kernel stack.

Design Guidelines

The following are general multiplexer design
guidelines:

The upper half of the multiplexer acts like the end of the upper stream.
The lower half of the multiplexer acts like the head of the lower stream. Service
procedures are used for flow control.

Message routing is based on multiplexer-specific criteria.

When one stream is being fed by many streams, flow control may have
to take place. Then all feeding streams on the other end of the multiplexer have to
be enabled when the flow control is relieved.

When one stream is feeding many streams, flow control may also have
to take place. Be careful not to starve other streams when one becomes flow controlled.

Upper and lower multiplexers share the same perimeter type and concurrency
level. (See MT STREAMS Perimeters for
information about perimeters.)