Multipathing is controlled by the in.mpathd
daemon which detects failures

The configuration file is /etc/default/mpathd:

# cat mpathd
#
#pragma ident "@(#)mpathd.dfl 1.2 00/07/17 SMI"
#
# Time taken by mpathd to detect a NIC failure in ms. The minimum time
# that can be specified is 100 ms.
#
FAILURE_DETECTION_TIME=10000
#
# Failback is enabled by default. To disable failback turn off this option
#
FAILBACK=yes
#
# By default only interfaces configured as part of multipathing groups
# are tracked. Turn off this option to track all network interfaces
# on the system
#
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

You can configure interfaces by direct editing of hostname.interface
configuration files or by using command line utilities. For example:

Configure the second interface as part of the the same
interface multipath group as in step 2

Configure a test IP address for the second interface.

Start the in.mpathd IPMP process to monitor the interfaces.

Check the interface configuration.

Grouping Physical Interfaces. In configuration of multipath interfaces
you need to use the ifconfig command to configure groups. This command
uses a new group parameter that requires a group name. The ifconfig
command places both the IPv4 and IPv6 instances of the interface in that group.
The group parameter has the following syntax:

ifconfig interface-name group group-name

Create IPv4 Test Addresses. The in.mpathd multipathing daemon
requires a test IP address for detecting failures and repairs. You must use
a routeable address for this IP address. The subnet prefix of the address must
be known to any routers present on the link. You use the ifconfig command's
new -failover option to configure a test address. Use the following
syntax to configure a test address:

To verify the system’s failover configuration, or to change the operational status
of IPMP interfaces, use the if_mpadm

Introduction

Network interfaces are exposed to failure because they connect to network cables
and hardware components in the form of switches or hubs. Failure of any of these
interfaces results in network failure, even if the network interface card (NIC)
that is in place does not fail. Sun offers two features that address customer network
bandwidth demands: IPMP and Sun Trunking software.

IPMP enables multiple interfaces with different IP addresses on the same subnet
to be connected to the same network segment. If any one of these interfaces fail,
current network connections through that interface will be migrated to another interface
automatically: physical interface on failed adapter will became logical on another
(working) LAN adapter. This is a pretty neat idea.

In general, multipathing is a method of using redundancy and automatic
fail-over that provides at least two physical paths to a target resource. It allow
recovery from a single network path failure by re-routing data traffic. That is
important for network storage. Varian of multipathing called trunking also
allows for the parallel transmission of data, which can result in faster throughput
and increased scalability.

Solaris 10 supports Solaris network multipathing via:

IPMP load spreading—outgoing network traffic is able to utilize several
network interfaces

network trunking — multiple physical network interfaces are treated
as one; combining these interfaces, which is accomplished in the TCP/IP stack,
allows link aggregation and availability

IPMP in Solaris has the following features/capabilities:

Eliminates a single network adapter as a single point-of-failure in
these cases:

Enables interfaces to failover within approximately 10 seconds when using
the default configuration. Can be configured by adjusting the parameters in
the /etc/default/mpathd file.

Can be configured for both IPv4 and IPv6.

Allows interfaces to be configured as Standby Interfaces.
These types of interfaces are only used for failover and are not used for outbound
load spreading, unless they are explicitly chosen by an application.

The Sun multipathing software (MPxIO) allows the merging of multiple SCSI layer
paths, such as an iSCSI device exposing the same LUN via two different iSCSI target
names (or the same name with different target portal group tags). In addition, one
FC path and one iSCSI path can be merged by MPxIO if the target device supports
these options. Additional functionality, such as iSCSI Multiple Connections / Session
(MC/S), might be supported in future Solaris releases.

Multiple physical connections between a host and a target provides two major
benefits: availability and link aggregation.

When multiple physical connections exist between host and storage, if one connection
is lost, then I/O can be rerouted through other paths. To maximize availability
and eliminate single points of failure, the following components in a storage architecture
must be made redundant:

IPMP is a product that was first included with the Solaris 8. IPMP provides enhanced
network availability. IPMP enables the Solaris to recover from network path failures.
IPMP also provides increased throughput by spreading the outbound load across interfaces
when multiple network adapters are connected to the same IP link; for example, to
the same Ethernet switch.

If a failure occurs in the network link and an alternate adapter is configured,
the IP address fails over. The network access changes automatically from the failed
adapter to the alternate adapter, allowing uninterrupted access to the network.
When there are multiple network adapters that are connected to the same IP link,
increased throughput can be achieved by spreading the outbound load across multiple
network interfaces.

IPMP has the following features:

Eliminates a single network adapter as a single point-of-failure in
these cases:

Enables interfaces to failover within approximately 10 seconds when
using the default configuration.

Can be configured by adjusting the parameters in the /etc/default/mpathd
file.

Can be configured for both IPv4 and IPv6.

Allows interfaces to be configured as Standby Interfaces. These types
of interfaces are only used for failover and are not used for outbound
load spreading, unless they are explicitly chosen by an application.

IPMP Requirements

Unique media access control (MAC) addresses must be configured on each
network interface (the default configuration for old Sun network adapters
has all network interfaces on a specific server using the same MAC address).
IPMP requires that all LAN adapters have unique MAC addresses. Switched configurations
use MAC addresses when making network decisions. Therefore for old Sun hardware,
you must change the system’s default configuration to ensure that each LAN adapter
has a unique MAC address to avoid a MAC address conflict.

Multiple network adapter interfaces must be connected on each subnet.
You can configure IPMP with a single network interface to take advantage of
network failure detection. To use the full benefit of IPMP, make sure that two
or more network interfaces are connected to the same subnet.

A network adapter group name must be assigned to IPMP interfaces.
Interfaces that are to be deployed as multipath interfaces must belong to a
multipath group. The in.mpathd multipath process uses the multipath group.
Use a meaningful name that does not include spaces when you choose a group name.
The multipath name is local to the system and is not used across the network.

A test address is assigned to an interface. The multipath process
uses test addresses, which must be routable addresses, to monitor the status
of each individual interface. Use the test addresses to detect failure and recovery
of an interface. These addresses are deprecated at configuration time to make
sure that they cannot be used to pass network traffic from other applications.

Additional hosts must exist on the same subnet. The test interfaces
use ICMP echo request, reply, or both to hosts that they reach by addressing
the 224.0.0.1 multicast group or the default router, as listed in the /etc/defaultrouter
file.

Interface Failure Detection and Repair

The in.mpathd process can detect both the failure and the repair of an
interface by:

Sending ICMP echo requests to the special host (for example default
router) and monitoring responses through the interface test address.

Monitoring the internal IFF_RUNNING flag on the interface

An interface has failed if either of these two detection methods indicates a
failure. An interface is considered repaired only if both methods report that the
interface is operational and can send and receive packets through the interface.

To detect the failure or repair of interfaces that belong to the multipath group,
the in.mpathd process sends ICMP echo requests from the logical IPMP interfaces
to targets connected to the local network. The in.mpathd process determines
which targets to probe dynamically. If five consecutive probes do not receive replies,
the interface is considered failed.

Adjust the failure detection time by editing the FAILURE_DETECTION_TIME
variable from the default value of 10,000 milliseconds (10 seconds) in the /etc/default/mpathd
file.

When responses to the ICMP echo requests are not received and a specific time
period has elapsed, the physical interface is considered failed. The IP address
that is associated with the failed address is moved to a new
logical interface associated with another physical interface in the same IPMP group.
Communications that were taking place continue to function as though the original
interface is still working properly.

ICMP echo requests are still attempted through the failed NIC to detect if a
physical interface is repaired.

If all the NICs or targets appear to fail at the same time, this is a group failure,
and no failover is performed. The in.mpathd process flushes all of the current
targets and attempts to discover new targets. Because in.mpathd process dynamically
determines what targets to probe, you cannot configure the targets. Routers connected
to the link are chosen as targets for probing. If no routers exist on the link,
arbitrary hosts on the link are chosen by sending a multicast packet to the “all
hosts” multicast address. When you configure IPMP, be sure to have at least one
additional system on the network.

You can configure multipathing by changing configuration files and rebooting,
or you can work at the command line to avoid rebooting the system.

Configuring Multipathing Using Configuration Files

This example shows IPMP configuration on an existing configured qfe0 interface
and on an existing but unconfiguredqfe1 interface on the myhost (192.168.1.1)
system. The multipath group is called mpgrp-one. The test addresses are:

192.168.1.50 for the qfe0 interface

192.168.1.51 for the qfe1 interface

The data address for the qfe0 interface remains 192.168.1.1, and the data address
for the qfe1 interface is 192.168.1.45.

To configure IPMP, complete the following steps, which are described in greater
detail in the next sections.

1. Verify the Solaris OE release. The /etc/release file contains information
about the installed version of the Solaris. It should be 9 or higher.

The preceding output indicates that the system is still in its default mode and
uses the same MAC address for each interface. This is indicated by the setting
of the local-mac-address? variable to false. You now use
the eeprom utility to change the local-mac-address? variable to true:

# eeprom local-mac-address?=true

Verify that the local-mac-address? variable is set to true:

# eeprom local-mac-address?

Note – Depending on the combination of your system’s firmware and hardware
architecture, you must either plumb the interface or reboot the system to enable
unique MAC address assignment after changing the eeprom variable.

3. Define data and test IP addresses. You can add the data and test IP
addresses to the /etc/inet/hosts file for the sake of clarity. After editing
the /etc/inet/hosts file, use the tail utility to view the new information:

addif myhost-test0 Creates the next unused logical interface,
and assigns it the IP address associated with the myhost-test0 name
in /etc/hosts. deprecated Marks the address as a deprecated address.
Addresses that are marked as deprecated are not used as source addresses for
outbound packets unless either there are no other addresses available on this
interface or the application is bound to this address explicitly.

The output from the ifconfig -a command shows DEPRECATED as one of the
flags associated with this interface.

-failover Marks the address as a non-failover address. Addresses that are
marked in this way do not fail
over when the interface fails. The output from the ifconfig -a command shows
NOFAILOVERas
one of the flags associated with this interface.

Create the /etc/hostname.qfe1 file to contain contents similar to the following:

The interface’s index number is 3, the same as the physical interface that
supports this logical interface.

The qfe0:1 interface MAC address is not shown because logical interfaces
use the same MAC address as the physical interface that supports the logical
interface.

The DEPRECATED and NOFAILOVER flags indicate that the interface is not to
be used by any application (other than the in.mpathd process), and the
interface must not be failed if a communication failure occurs.

The RUNNING flag is also monitored by the in.mpathd process to ensure
that communications are functioning as expected. The system remains available
to users if either of the multipath network interfaces fail or become unusable
for any reason.

You must know the state of the system if you need to restore it. Before making
any changes to the system, view the system’s interface configuration by performing
the command:

The system remains available to users if either of the multipath network interfaces
fail or become unusable for any reason. As a precaution, reboot the system when
it is convenient to verify that multipathing is properly configured during system
boot.

Command Line Utilities-based Configuration

Configure the qfe0 Interface as Part of a Multipath Group.

To configure the qfe0 interface as part of a multipath group, specify the
name of the group, mpgrp-one, of which the qfe0 interface will be a part:

Viewing Multipath Operation

To verify the system’s failover configuration, or to change the operational status
of IPMP interfaces, use the if_mpadm utility. You can use this utility to
take an interface offline (detach), by forcing a failover, and verifying that an
alternate interface takes over as expected. If configuration errors occur, they
appear at this stage. Also, use the if_mpadm utility to reattach a detached
interface.

The detached interface is assigned an IP address of 0.0.0.0, and a new
logical interface, qfe1:2, is automatically created over the functional qfe1 physical
interface. The new logical interface has the IP address that was assigned to the
physical qfe0 interface while it was working. To reattach an offline interface,
perform the command:

The qfe0 interface is reassigned its original IP address, and the qfe1:2 logical
interface is automatically removed.

Troubleshooting a Multipath Network Configuration

Incorrectly configured network interfaces might not properly fail over
when connectivity to an interface fails for any reason. It is important to thoroughly
test your network interface after you configure IPMP.

Carefully read messages in the /var/adm/messages file or in the console
window to take the proper troubleshooting steps when you configure and test the
IPMP. For example:

# Nov 17 23:19:51 myhost in.mpathd[475]: Failures cannot be detected
on qfe0 as no IFF_NOFAILOVER address is available

The message indicates that the in.mpathd process with a process number
of 475 senses that IPMP is not properly configured. To investigate further, use
the ifconfig command:

The output indicates that the configuration process is not complete. Recall that
IPMP requires a test address on a logical interface for each physical interface.
After defining a test interface with the ifconfig command, the following
message appears:

Created new logical interface qfe0:1
Setting netmask of qfe0:1 to 255.255.255.0
# Nov 17 23:17:41 myhost in.mpathd[475]: Failure detection restored
on qfe0 as an IFF_NOFAILOVER address is available

The in.mpathd process reports that it can now perform failure detection.
Be aware that more than one interface is required to provide effective failover.
To view the interface configuration, use the ifconfig command:

FAIR USE NOTICE This site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
in our efforts to advance understanding of environmental, political,
human rights, economic, democracy, scientific, and social justice
issues, etc. We believe this constitutes a 'fair use' of any such
copyrighted material as provided for in section 107 of the US Copyright
Law. In accordance with Title 17 U.S.C. Section 107, the material on
this site is distributed without profit exclusivly for research and educational purposes. If you wish to use
copyrighted material from this site for purposes of your own that go
beyond 'fair use', you must obtain permission from the copyright owner.

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no
less then 90 days. Multiple types of probes increase this period.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be
tracked by Google please disable Javascript for this site. This site is perfectly usable without
Javascript.

Original materials copyright belong
to respective owners. Quotes are made for educational purposes only
in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development
of this site and speed up access. In case softpanorama.org is down currently there are
two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with.We do not warrant the correctness
of the information provided or its fitness for any purpose.