Link aggregation is a great redundancy solution for IBM i partitions
that have multiple physical Ethernet ports available, all of which can be
connected to a single Ethernet switch that has support for link aggregation. However,
as some have commented on those articles, link aggregation does not work with
virtual Ethernet ports that are connected to the internal Ethernet switch of
the Power Hypervisor. This means that it is not useful to partitions that
connect to the network via a Shared
Ethernet Adapter (SEA) in a Virtual I/O Server (VIOS)
partition or via an Ethernet
layer-2 bridge provided by an IBM i partition (which was also added in Technology
Refresh 3).

SEA Failover

One solution is to use a pair of VIOS partitions, each with an SEA
bridged to the physical network, and to have the Shared
Ethernet Adapters configured for failover, which provides redundancy without any
additional configuration in the bridged partitions.

Virtual IP Addressing

Bridged IBM i partitions can get Ethernet redundancy from the existing
Virtual IP Address (VIPA) support provided by IBM i. Specifically, I will
describe solutions based on VIPA
Proxy ARP.

Unlike other IP interfaces, a VIPA is not bound to a line description. Instead,
the user chooses a list of existing IP interfaces that will serve as proxies to
connect the VIPA to the network; these are called its preferred interfaces. They
are kept in priority order, and the VIPA will be serviced by the first active
interface in the list.

Physical Interface With Virtual Backup

Given this, we can build a setup that will default to using a physical
Ethernet port but will use a virtual Ethernet port instead if the physical
port becomes unavailable. For this example, we assume that the IP addresses
below are all available and appropriate to the system's network, the physical
Ethernet port is managed by line description PHYSETH, the virtual Ethernet port
is managed by line description VIRTETH, and that VIRTETH is bridged to the
physical network by a VIOS SEA or IBM i layer-2 bridging. We create an IP
interface on each port, then create a VIPA, preferring the physical interface
over the virtual Ethernet interface.

Once all three interfaces are started, the VIPA should be accessible
from the network via PHYSETH; if PHYSETH fails, then VIRTETH will take over.

When each partition has a physical interface available, this can be a
great solution. Each partition has control over its own physical Ethernet
resource, and a single bridging support partition can provide a backup for all
of the other partitions simultaneously.

For direct use, this requires that the first line description in the
preferred interface list will fail if it can't access the network (for example,
if the cable is pulled or the Ethernet adapter or switch fails for some
reason). This is true for IBM i physical Ethernet ports (and for aggregate line
descriptions) but not true for bridged virtual Ethernet ports (which are not
connected directly to the network). Thus, this solution requires changes if we
want to support failover between two virtual Ethernet ports.

Virtual Interface With Virtual Backup

This can be done by writing a program that does some level of direct
health check for each preferred interface and rearranges the preferred
interfaces so that the first is always working. Attached below is an example CL
script that monitors each preferred interface to see that it can ping a
specific remote IP address, and changes the priority order if the first
interface becomes unavailable while the second is available.
Download 121912 monitor

Given this, we can fail over between two virtual Ethernet ports. This
enables more complicated configurations that provide Ethernet redundancy for
the clients without setting up any failover between bridging partitions.

For example, consider a system with several IBM i partitions, two of
which have physical Ethernet resources. In each of those partitions, we
configure a layer-2 bridge: the first bridge through one Power Hypervisor
virtual LAN (VLAN), and the second through a different VLAN. Then, in each of
the other IBM i partitions, we configure two virtual Ethernet interfaces and
VIPA as above, with a virtual-to-virtual failover script. In half of those
partitions, we choose the first bridge as primary; in the other half, we make
the second bridge primary.

In this system, all of the physical Ethernet resources are being used,
and the physical network is unaware of anything in the configuration. If one
bridge stops working for any reason, the half of the partitions that were using
it will start using the other bridge. This can provide acceptable service on systems
with several IBM i partitions but not a lot of available physical Ethernet
resources, at the cost of some additional configuration work.

In short, IBM i and Power Systems network and virtualization
technologies give users the tools they need to meet their performance and
redundancy targets. We are always looking for ways to improve our networking
options as well.

January 24, 2012

This week’s blog was written by Colin DeVilbiss, an IBM software engineer. He works mostly on communications device drivers for IBM i in the development lab in Rochester, Minn. Thanks, Colin.

In 7.1 Technology Refresh 3, IBM i added the capability to aggregate multiple Ethernet ports in one Ethernet line description. An aggregated line description offers two significant benefits over a single-resource line description: improved reliability and parallel throughput.

An aggregate line description stays available as long as at least one of its aggregated Ethernet ports is working; for example, if four Ethernet ports are aggregated in one line description, the line description will remain active even if three of those Ethernet ports lose their link, and all of the workloads will continue running.

In addition, workloads with multiple parallel TCP connections or with multiple remote clients will be able to take advantage of having multiple physical connections to the same IP address, increasing throughput with each available resource. For example, if four Ethernet ports are aggregated, each running at one gigabit per second, the aggregate could support up to four gigabits of throughput across multiple connections.

New Capabilities

Link aggregation gives IBM i administrators a new tool to improve their network configurations, taking advantage of underutilized resources and adapting to meet new demands in the business.

For example, if a partition has a two-port Ethernet adapter, but IBM i is only using one port, it's straightforward to aggregate the two ports together and have additional protection against a failure in a cable, switch port, or port on the adapter, while potentially doubling throughput on some workloads, potentially with no additional hardware cost.

Likewise, if an IP interface is getting close to saturating its capacity, link aggregation lets the administrator scale the underlying line description's capacity incrementally, without changing technology for the individual links. That is, if one gigabit per second isn't enough, link aggregation can provide two gigabits per second of potential throughput without needing to jump to 10-gigabit-per-second infrastructure.

Creating an Aggregate Line Description

The main requirements for creating an aggregate line description are:

An IBM i partition at 7.1 with Technology Refresh 3 and its PTF group loaded

Two or more gigabit-capable physical Ethernet resources in the partition

A switch that supports link aggregation (sometimes under the names “trunking” or “teaming”)

At the switch, the corresponding ports must be configured into a static aggregate; switches vary, and each manufacturer has its own configuration commands and interfaces, so check there.

Assuming that the desired Ethernet resources are CMN02 and CMN03, creating the aggregate line description is as simple as:

using static assignment of resources (the only supported policy in TR3)

assigning outgoing frames to ports based on the source and destination IP addresses and TCP ports (which is likely to provide the best “spreading” of traffic across the ports)

aggregating CMN02 and CMN03

Varying the line description on will bring all of the ports active, and DSPLIND can be used to see the status of each port. TCP/IP can use the line description just like any other Ethernet line description.

In short, Ethernet Link Aggregation lets 7.1 TR3 users make better use of their Ethernet resources, potentially improving the reliability and performance of their network workloads without purchasing any additional hardware.

IBM Systems Magazine is a trademark of International Business Machines Corporation. The editorial content of IBM Systems Magazine is placed on this website by MSP TechMedia under license from International Business Machines Corporation.