in.mpathd

- IP multipathing daemon

Synopsis

/usr/lib/inet/in.mpathd

Description

The in.mpathd daemon performs failure and repair detection for IP interfaces that
have been placed into an IPMP group (or optionally, for all IP
interfaces on the system). It also controls which IP interfaces in an
IPMP group are “active” (being used by the system to send or receive
IP data traffic) in a manner that is consistent with the administrator's
configured policy. Functioning interfaces that are not active are referred to as
alternate interfaces.

The in.mpathd daemon can detect IP interface failure and repair through three
methods: by monitoring the IFF_RUNNING flag for each IP interface (link-state based
failure detection), by sending and receiving ICMP probes on each IP
interface (ICMP probe-based failure detection), or by transitive probing. Failure detection by
transitive probing is done by sending and receiving ICMP probes on active
interfaces, while determining the liveness of alternate interfaces through their ability to
send and receive link-layer (that is, transitive) probes to an active interface.

By default, only link-state based failure detection is enabled. This requires the
driver to support link-status notification. ICMP probe-based failure detection must be enabled
through the configuration of one or more test addresses, which are described
below. Transitive probing can be enabled by modifying the value of the SMF
property (shown below) to true:

svc:/network/ipmp/config/transitive-probing

See EXAMPLES for more information on how to modify the value of
the transitive-probing property.

Both ICMP and transitive probe-based failure detection methods test the entire IP
interface send and receive path. The ipmpstat(1M) utility can be used to check
which failure detection methods are enabled.

If transitive-probing is set to true, and no test addresses are configured
for a given IPMP group, then transitive probing will be used. If
it is set to false (default value), then transitive probing will not
be used under any circumstance.

If only link-state based failure detection is enabled, then the health of
the interface is determined solely from the state of the IFF_RUNNING flag.
If probes have been enabled, the interface is considered failed
if either link-state or probes indicate a failure, and repaired
once the failure detection method has indicated the failure has been corrected.
Although all interfaces in a group need not be configured with the
same failure detection methods, transitive probing will be disabled on any interface
of a group that has at least one IP test address.

As mentioned above, to perform ICMP probe-based failure detection, in.mpathd requires a
test address on each IP interface for the purpose of sending and
receiving probes. Each address must be marked NOFAILOVER (see ifconfig(1M)) and in.mpathd
will be limited to probing targets on the same subnet. Each address
may be configured statically or acquired by means of DHCP. To find targets,
in.mpathd first consults the routing table for routes on the same subnet,
and uses the specified next-hop. If no routes match, it sends all-hosts
ICMP probes and selects a subset of the systems that respond. Thus,
for probe-based failure detection to operate, there must be at least one neighbor
on each subnet that responds to ICMP echo request probes. The
ipmpstat(1M) utility can be used to display both the current probe target
information and the status of sent probes.

Both IPv4 and IPv6 are supported. If an IP interface is plumbed
for IPv4 and an IPv4 test address is configured then in.mpathd will
start sending ICMPv4 probes over that IP interface. Similarly, if an IP
interface is plumbed for IPv6 and an IPv6 test address is configured, then
in.mpathd will start sending ICMPv6 probes over that IP interface. However, note
that in.mpathd will ignore IPv6 test addresses that are not link-local. If
both IPv4 and IPv6 are plumbed, it is sufficient to configure only
one of the two, that is, either an IPv4 test address or an
IPv6 test address. If both IPv4 and IPv6 test addresses are configured,
in.mpathd probes using both ICMPv4 and ICMPv6.

As mentioned above, in.mpathd also controls which IP interfaces in an IPMP
group are “active” (used by the system to send and receive IP
data traffic). Specifically, in.mpathd tracks the administrative configuration of each IPMP group
and attempts to keep the number of active IP interfaces in each group
consistent with that configuration. Therefore, if an active IP interface fails, in.mpathd
will activate an INACTIVE interface in the group, provided one exists (it
will prefer INACTIVE interfaces that are also marked STANDBY). Likewise, if an IP
interface repairs and the resulting repair leaves the IPMP group with more
active interfaces than the administrative configuration specifies, in.mpathd will deactivate one of
the interfaces (preferably one marked STANDBY), except when the FAILBACK variable is used,
as described below. Similar adjustments will be made by in.mpathd when offlining
IP interfaces (for instance, in response to if_mpadm(1M)).

The in.mpathd daemon accesses three variable values in /etc/default/mpathd: FAILURE_DETECTION_TIME, FAILBACK and
TRACK_INTERFACES_ONLY_WITH_GROUPS.

The FAILURE_DETECTION_TIME variable specifies the probe-based failure detection time. The shorter the
failure detection time, the more probe traffic. The default value of FAILURE_DETECTION_TIME
is 10 seconds. This means that IP interface failure will be detected by
in.mpathd within 10 seconds. The IP interface repair detection time is always
twice the value of FAILURE_DETECTION_TIME. Note that failures and repairs detected by
link-based failure detection are acted on immediately, though in.mpathd may ignore link state
changes if it suspects that the link state is flapping due to
defective hardware; see DIAGNOSTICS.

By default, in.mpathd limits failure and repair detection to IP interfaces that
are configured as part of a named IPMP group. Setting TRACK_INTERFACES_ONLY_WITH_GROUPS to
no enables failure and repair detection on all IP interfaces, even if they
are not part of a named IPMP group. IP interfaces that are
tracked but not part of a named IPMP group are considered to
be part of the “anonymous” IPMP group. In addition to having no
name, this IPMP group is special in that its IP interfaces are
not equivalent and thus cannot take over for one another in the event
of an IP interface failure. That is, the anonymous IPMP group can
only be used for failure and repair detection, and provides no high-availability
or load-spreading.

As described above, when in.mpathd detects that an IP interface has repaired,
it activates it so that it will again be used to send
and receive IP data traffic. However, if FAILBACK is set to no,
then the IP interface will only be activated if no other active IP
interfaces in the group remain. However, the interface may subsequently be activated
if another IP interface in the group fails.

SMF Management

The in.mpathd daemon service is managed by the service management facility,
smf(5), under the service identifier:

svc:/network/ipmp:default

Administrative actions on this service, such as enabling, disabling, or requesting restart,
can be performed using svcadm(1M).

Examples

Example 1 Enabling Fault Detection by Transitive Probing

The following example shows the sequence of SMF commands used to enable
fault detection by transitive probing.

See Also

Diagnostics

IP interface interface_name has a hardware address which is not unique in
group group_name; offlining

Description: For probe-based failure detection, load-spreading, and other code IPMP features to work
properly, each IP interface in an IPMP group must have a unique
hardware address. If this requirement is not met, in.mpathd will automatically offline all
but one of the IP interfaces with duplicate hardware addresses.

IP interface interface_name now has a unique hardware address in group group_name;
onlining

Description: The previously-detected duplicate hardware address is now unique, and therefore in.mpathd has
brought interface_name back online.

Description: For in.mpathd to perform probe-based failure detection, each test address in the
group must be unique.

No test address configured on interface interface_name disabling probe-based failure detection on
it

Description: For in.mpathd to perform probe-based failure detection on an IP interface, it
must be configured with a test address: IPv4, IPv6, or both.

IP interface_name in group group_name is not plumbed for IPv[4|6], affecting IPv[4|6]
connectivity

Description: All IP interfaces in a multipathing group must be homogeneously plumbed. For
example, if one IP interface is plumbed for IPv4, then all IP
interfaces in the group must be plumbed for IPv4, or IPv4 packets
will not be able to be reliably sent and received. The STREAMS modules
pushed on all IP interfaces must also be identical.

The link has come up on interface_name more than 2 times in
the last minute; disabling repair until it stabilizes.

Description: To limit the impact of interfaces with intermittent hardware (such as a
bad cable), in.mpathd will not consider an IP interface with a frequently
changing link state as repaired until the link state stabilizes.

Invalid failure detection time of time, assuming default 10000 ms

Description: An invalid value was encountered for FAILURE_DETECTION_TIME in the /etc/default/mpathd file.

Too small failure detection time of time, assuming minimum of 100 ms

Description: The minimum value that can be specified for FAILURE_DETECTION_TIME is currently 100
milliseconds.

Invalid value for FAILBACK value

Description: Valid values for the boolean variable FAILBACK are yes or no.

Invalid value for TRACK_INTERFACES_ONLY_WITH_GROUPS value

Description: Valid values for the boolean variable TRACK_INTERFACES_ONLY_WITH_GROUPS are yes or no.

Cannot meet requested failure detection time of time ms on (inet[6] interface_name)
new failure detection time for group group_name is time ms

Description: The round trip time for ICMP probes is higher than necessary to
maintain the current failure detection time. The network is probably congested or
the probe targets are loaded. in.mpathd automatically increases the failure detection time to
whatever it can achieve under these conditions.

Improved failure detection time time ms on (inet[6] interface_name) for group group_name

Description: The round trip time for ICMP probes has now decreased and in.mpathd
has lowered the failure detection time correspondingly.

IP interface failure detected on interface_name

Description: in.mpathd has detected a failure on interface_name, and has set the IFF_FAILED
flag on interface_name, ensuring that it will not be used for IP
data traffic.

IP interface repair detected on interface_name

Description: in.mpathd has detected a repair on interface_name, and has cleared the IFF_FAILED
flag. Depending on the administrative configuration, the interface_name may again be used for
IP data traffic.

All IP interfaces in group group are now unusable

Description: in.mpathd has determined that none of the IP interfaces in group can
be used for IP data traffic, breaking network connectivity for the group.

At least 1 IP interface (interface_name) in group group is now usable

Description: in.mpathd has determined that at least one of the IP interfaces in
group can again be used for IP data traffic, restoring network connectivity
for the group.

The link has gone down on interface_name

Description: in.mpathd has detected that the IFF_RUNNING flag for interface_name has been cleared,
indicating that the link has gone down.

The link has come up on interface_name

Description: in.mpathd has detected that the IFF_RUNNING flag for interface_name has been set,
indicating that the link has come up.