Multilink PPP over ATM and Multilink PPP over Frame Relay (MLPoATM and
MLPoFR) was introduced in Cisco IOS® Software Release 12.1(5)T. This feature is
targeted at customers with Frame Relay/ATM interworking (FR/ATM IW) that deploy
Voice over IP (VoIP) across medium to low speed WAN links. Prior to this
feature, there was no common Layer 2 fragmentation scheme that was supported by
Cisco IOS on both ATM and Frame Relay customers with FR/ATM IW were forced to
do Layer 3 fragmentation.

This document is intended for networking personnel involved in the
design and deployment of VoIP networks that involve MLPoATM and Frame Relay
networks. Cisco recommends that you have knowledge of these topics:

Frame Relay

ATM

PPP

MLP

Frame Relay/ATM interworking

Voice Quality of Service (QoS) Configuration

This document is not intended to provide technology training on these
subjects. A list of reference materials is included at the end of this
document. Cisco recommends that you review and understand these documents prior
to reading this document:

The case study included in this document
is based on a production network that uses these software and hardware
versions:

The core Cisco 3660 routers run Cisco IOS Software Release
12.2(5.8)T. The requirement for cRTP over ATM dictates the use of Cisco IOS
Software Release 12.2T. This known issue is resolved in Cisco IOS Software
Release 12.2(8)T1:

The branch Cisco 2620 routers are in the process of being upgraded
from Cisco IOS Software Release 12.2(3) to one 2.2(6a). The Cisco 2620s also
act as branch H.323 gateways. The upgrade is triggered by a gateway-related
issue. As far as the WAN and QoS features are concerned, Cisco IOS Software
Release 12.2(3) does not exhibit any significant
issues.

When you design ATM and Frame Relay networks with MLP, you must
understand the data link overhead. Overhead influences the amount of bandwidth
consumed by each VoIP call. It also helps determine the optimum MLP fragment
size. It is critical to optimize the fragment size to fit an integral number of
ATM cells, especially on slow speed PVCs where a significant amount of
bandwidth is wasted on cell padding. The data link overhead on MLP over Frame
Relay and ATM PVCs depends on these factors:

The mode of operation of the FRF.8 IW device (transparent or
translational).

The direction of the traffic (ATM to Frame Relay or Frame Relay to
ATM).

The PVC leg. Overhead is different on the ATM and Frame Relay legs of
the PVC.

The traffic type. Data packets have an MLP header; VoIP packets do
not.

This table shows the data link overhead for a data packet. It details
the number of bytes in the various Frame Relay, ATM, LLC, PPP, and MLP headers
for all permutations of the FRF.8 mode of operation, traffic direction, and PVC
leg.

The translational PVC is the easiest to comprehend as the overhead is
the same in both directions. This is because the FRF.8 device translates
between the MLPoATM and MLPoFR formats. As a result, the frame format is MLPoFR
on the Frame Relay leg in both directions. The format on the ATM leg is MLPoATM
in both directions.

The transparent PVC is slightly messier because the overhead differs in
the two directions. This complexity arises because the Frame Relay router sends
packets in the MLPoFR format. This format is carried across by the IW device
onto the ATM side. Similarly, the ATM router sends packets in the MLPoATM
format. This format is carried across by the IW device onto the Frame Relay
side. Therefore, the result is different frame formats in the two directions on
each leg.

In comparison, the overhead on an end-to-end Frame Relay PVC that uses
FRF.12 is 11 bytes. Therefore, on an end-to-end Frame Relay link, FRF.12 is a
more efficient choice for link fragmentation and interleaving (LFI) than MLP.
On end-to-end ATM virtual circuits (VCs), MLP is the only choice since there is
no standards-based fragmentation available. However, end-to-end ATM VCs are
medium to high speed. Therefore, LFI is not required. The exception to this
rule is slow speed ATM VCs over digital subscriber line (DSL).

The PPP ID is present in the first MLP fragment only. Therefore, the
overhead in the first fragment is always two bytes more than in subsequent
fragments.

The table here shows the data link overhead for a VoIP packet. It
details the number of bytes in the various Frame Relay, ATM, LLC, and PPP
headers for all permutations of the FRF.8 mode of operation, traffic direction,
and PVC leg. The main difference between a data and a VoIP packet is that VoIP
packets are sent as PPP packets and not as MLP packets. All other aspects are
identical to the data scenario.

FRF.8 Mode

Transparent

Translation

Frame Relay to Frame Relay

Traffic direction

Frame Relay to ATM

ATM to Frame Relay

Frame Relay to ATM

ATM to Frame Relay

Frame Relay or ATM leg of PVC

Frame Relay

ATM

ATM

Frame Relay

Frame Relay

ATM

ATM

Frame Relay

Frame Flag (0x7e)

1

0

0

1

1

0

0

1

1

Frame Relay Header

2

0

0

2

2

0

0

2

2

LLC DSAP/SSAP (0xfefe)

0

0

2

2

0

2

2

0

0

LLC Control (0x03)

1

1

1

1

1

1

1

1

1

NLPID (0xcf for PPP)

1

1

1

1

1

1

1

1

1

PPP ID

2

2

2

2

2

2

2

2

0

Payload (IP+User Datagram Protocol
(UDP)+RTP+Voice)

0

0

0

0

0

0

0

0

0

AAL5

0

8

8

0

0

8

8

0

0

FCS

2

0

0

2

2

0

0

2

2

Total Overhead (bytes)

9

12

14

11

9

14

14

9

7

In comparison, the data link overhead for a VoIP packet on an
end-to-end Frame Relay PVC is shown in the far right column.

When you provision bandwidth for VoIP, the data link overhead must be
included in the bandwidth calculations. This table shows the per call bandwidth
requirements for the various flavors of VoIP. It is based on the
characteristics of the PVC. The calculations in this table assume a best-case
scenario for cRTP (for example, no UDP checksum and no transmission errors.)
Headers are then consistently compressed from 40 bytes to 2 bytes.

FRF.8 Mode

Transparent

Translation

Frame Relay to Frame Relay

Traffic direction

Frame Relay to ATM

ATM to Frame Relay

Frame Relay to ATM

ATM to Frame Relay

Frame Relay or ATM leg of PVC

Frame Relay

ATM

ATM

Frame Relay

Frame Relay

ATM

ATM

Frame Relay

G.729 - 20 ms Samples - No cRTP

27.6

42.4

42.4

28.4

27.6

42.4

42.4

27.6

26.8

G.729 - 20 ms Samples - cRTP

12.4

21.2

21.2

13.2

12.4

21.2

21.2

12.4

11.6

G.729 - 30 ms Samples - No cRTP

20.9

28.0

28.0

21.4

20.9

28.0

28.0

20.9

20.3

G.729 - 30 ms Samples - cRTP

10.8

14.0

14.0

11.4

10.8

14.0

14.0

10.8

10.3

G.711 - 20 ms Samples - No cRTP

83.6

106.0

106.0

84.4

83.6

106.0

106.0

83.6

82.8

G.711 - 20 ms Samples - cRTP

68.4

84.8

84.8

69.2

68.4

84.8

84.8

68.4

67.6

G.711 - 30 ms Samples - No cRTP

76.3

97.9

97.9

76.8

76.3

97.9

97.9

76.3

75.8

G.711 - 30 ms Samples - cRTP

66.3

84.0

84.0

66.8

66.3

84.0

84.0

66.3

65.7

Because the overhead varies on the different legs of the PVC, it is
necessary to design for the worst-case scenario. For example, consider G.729
with 20 millisecond (msec) sampling and cRTP across a transparent PVC. The
bandwidth requirements for this scenario on the Frame Relay leg is 12.4 Kbps in
one direction and 13.2 Kbps in the other. Provisioning needs to be done with
the assumption of 13.2 Kbps per call.

In comparison, the VoIP bandwidth requirement on an end-to-end Frame
Relay PVC is also shown in the far right column of the above table. The
additional overhead of PPP compared to native Frame Relay encapsulation results
in an extra bandwidth consumption between 0.5 Kbps and 0.8 Kbps per call.
Therefore, Frame Relay encapsulation with FRF.12 makes more sense than MLP on
an end-to-end Frame Relay VC.

The reason to use MLP on a Frame Relay/ATM PVC is to allow for large
data packets to be fragmented into smaller chunks. The router then expedites
VoIP packets by interleaving them in between the data fragments. In Cisco IOS,
the fragmentation size is not configured directly. Instead, the desired delay
is configured with the help of the ppp multilink
fragment-delay command. Cisco IOS then uses this formula
to calculate the appropriate fragment size:

fragment size = delay x bandwidth/8

When you do MLP across ATM, the fragment size needs to be optimized so
that it fits into an integral number of cells. If this optimization is not
done, the bandwidth required can almost double due to padding. For example, if
each fragment is 49 bytes long, two ATM cells are required to carry each
fragment. Therefore, 106 bytes are used to carry a payload of only 49 bytes.

Cisco IOS automatically optimizes the fragment size to use an integral
number of ATM cells when it performs MLPoATM and MLPoFR. Cisco IOS
automatically rounds up the calculated fragment size to the next integral
number of ATM cells. No new CLI commands are added. Cisco IOS performs this
optimization on both the Frame Relay and ATM ends of a MLPoFR/ATM PVC. It is
possible that an MLP PVC can be end-to-end Frame Relay. In this case,
optimizing it for ATM is not required. However, Cisco IOS optimizes this
scenario for ATM since it has no way to detect whether the remote end is ATM or
Frame Relay.

Note: Due to rounding, the delay that results can be slightly higher than
that configured. This rounding error is more significant on low-speed PVCs.

You can also configure optimization manually. Since the fragment size
cannot be specified directly in Cisco IOS, calculate the optimal fragment size
and convert it into a delay. When you calculate the fragment size, adjust for
the data link overhead, as the MLP code assumes that every link is High-Level
Data Link Control (HDLC) and has 2 bytes of data link overhead. In addition to
the HDLC data link overhead, the MLP code includes in its calculations the 8
bytes made up of the MLP ID, the MLP sequence number, and the PPP ID as
outlined in the first table above.

Use this procedure in order to calculate the delay to be configured in
Cisco IOS:

Calculate the theoretical fragment size based on the desired delay
and the bandwidth of the PVC:

fragment = bandwidth * delay / 8

Make sure that the fragment is a multiple of 48 bytes, so that it
fits into an integral number of ATM cells.

The formula to calculate the cell aligned fragment size is:

fragment_aligned = (int(fragment/48)+1)*48

Adjust for the data link overhead that is not taken into account by
MLP.

As seen earlier, this overhead differs based on the PVC
characteristics. Consider the ATM side of the PVC as this is the side for which
you optimize. This table shows the number of bytes of data link overhead on the
ATM side.

FRF.8 Mode

Transparent

Translation

Traffic Direction

Frame Relay to ATM

ATM to Frame Relay

Frame Relay to ATM

ATM to Frame Relay

Frame Relay or ATM leg of PVC

ATM

ATM

ATM

ATM

LLC DSAP/SSAP (0xfefe)

0

2

2

2

LLC Control (0x03)

1

1

1

1

NLPID (0xcf for PPP)

1

1

1

1

AAL5

8

8

8

8

Non MLP overhead (bytes)

10

12

12

12

To arrive at the fragment size on which MLP bases its calculations,
subtract the data link overhead from the desired cell-aligned fragment size.
Add back 2 bytes in order to compensate for the HDLC encapsulation that MLP
always assumes.

The formula to calculate the fragment size that the MLP code targets
is:

fragment_mlp = fragment_aligned - datalink_overhead + 2

Convert the fragment size that results into the corresponding delay
with this formula:

delay = fragment_mlp/bandwidth x 8bits/byte

In most cases, the delay calculated is not an integral number of
milliseconds. Since Cisco IOS only accepts an integer value, you must round
down. Therefore, the delay value you configure in Cisco IOS with the help of
the ppp multilink fragment-delay command is:

fragment_delay = int(fragment_mlp/bandwidth x 8bits/byte)

In order to compensate for the above rounding error, fudge the
bandwidth used by MLP to convert from delay to fragment. The fudged bandwidth
that you configure in the Cisco IOS with the help of the
bandwidth command is:

bandwidth = fragment_mlp/fragment_delay * 8

This table shows the optimal values of ppp multilink
fragment-delay and bandwidth for the
most common PVC speeds. A target delay of 10 msec is assumed. In order to
simplify the table, the calculations do not differentiate between transparent
and translational PVC, or between traffic directions. The maximum difference in
data link overhead is only 2 bytes. Therefore, the penalty for designing for
the worst case of 12 bytes is small. Also shown in the table is the "real"
delay that is encountered due to the fact that you increase the fragment size
so that fragments fit into an integral number of cells.

Special consideration is given to traffic shaping and policing in a
Frame Relay/ATM IW environment. The issues in the Frame Relay to ATM direction
are different to the issues in the ATM to Frame Relay direction. Therefore,
each topic is described separately.

The main issue in the Frame Relay to ATM direction is the potential
expansion in bandwidth consumption when going from frame to cell. For example,
a 49 byte frame on the Frame Relay side consumes two cells, or 106 bytes, on
the ATM side. Therefore, it cannot be assumed that the sustainable cell rate
(SCR) equals the committed information rate (CIR). The worst case scenario
occurs when each Frame Relay frame contains 1 byte of payload. Each of these
bytes consumes a full 53 byte cell on the ATM side. As an example of this
concept, this extreme and unrealistic scenario dictates an SCR that is 53 times
the CIR. Two more realistic examples are:

A G.729 VoIP packet is 60 bytes long, or 69 bytes (when MLP and Frame
Relay overhead are included). On the ATM leg, it consumes two cells or 106
bytes. Therefore, if all traffic carried is VoIP, then an appropriate mapping
is SCR = 106/69 = 1.5 x CIR.

A Telnet packet that carries a single keystroke contains 40 bytes of
TCP/IP header and 1 byte of data. On the Frame Relay side, this totals 56
bytes, including overhead. However, on the ATM side this packet expands to two
cells. In this case, SCR = 106/56 = 1.9 x
CIR.

Appendix A of the ATM Forum standard, BISDN Inter Carrier
Interface (B-ICI) Specification Version 2.0, describes two methods
of mapping between SCR and CIR. While both provide a scientific way to derive
SCR from CIR, neither one is any more accurate than the data to which they are
applied. One of the numbers required by the formulas is the typical frame size,
a number that is hard to determine in a real network. Also, a number that can
potentially change as new applications are rolled out on an existing ATM/Frame
Relay network. The best recommendation to solve this problem is to work closely
with your carrier since their policing policy will be of critical importance.
With the assistance of the carrier, this fail-safe approach is possible:

Frame Relay to ATM Direction - In the Frame Relay to
ATM direction, the carrier needs to police traffic inbound at the Frame Relay
ingress point only. For instance, on the Frame Relay switch connected to the
Frame Relay attached customer premises equipment (CPE) device, the carrier
polices traffic according to the contracted CIR. This policing policy is
illustrated in the figure here. No further policing should happen once traffic
is allowed into the network at the ingress point. The implication of this
policing policy is that data received on the Frame Relay side is allowed to
expand and consume whatever bandwidth is required to allow for cell tax, AAL
overhead, and padding in order to carry it across the ATM leg of the network.
In most cases, the ATM bandwidth required is less than twice the Frame Relay
bandwidth used.

ATM to Frame Relay Direction - In the ATM to Frame
Relay direction, the opposite is experienced. Bandwidth usage is reduced when
going from ATM to frame as cell tax, AAL overhead, and when padding is removed.
However, because the potential transmit rate of the ATM side is much higher
than that of the Frame Relay link, setting up the traffic shaping correctly on
the ATM router is critical for the integrity of voice. If the shaping is too
loose, then the ATM router feeds data at a rate faster than the physical speed
of the Frame Relay link at the other end. As a result, packets start queuing up
on the FRF.8 switch. At some point, packets start to drop . and since the
ATM/Frame Relay networks do not distinguish between voice and data, VoIP
packets are also dropped.

At the same time, if shaping is too restrictive, then throughput
suffers. Because of ATM cell tax, AAL overhead and padding is removed by the
FRF.8 device. It does not consume bandwidth on the Frame Relay link. Therefore,
you can oversubscribe the ATM side of the PVC slightly. The amount of padding
and AAL overhead varies depending on average frame sizes and how aggressive the
fragmentation is. Because you cannot accurately qualify this overhead, you are
better off not trying to optimize for it. On the other hand, cell tax is
completely deterministic. It is 5 bytes for every 48 bytes of payload. You can
optimize for cell tax by setting the shaping target on the ATM router to 53/48
x SCR. The policing on the carrier side must be set to allow for this slight
oversubscription.

MLPoATM/Frame Relay is currently only tested and supported with a
service-policy attached to either the virtual-template or dialer-interfaces.
Omitting the service-policy can cause the feature to not work. One example of
this is documented in Cisco bug ID
CSCdu19313
(registered customers only)
.

MLPoATM/Frame Relay clones two virtual-access interfaces for each
PVC. One of these represents the PPP link. The other represents the MLP bundle.
The show
ppp multilink command is used to tell which is which.
Multiple PPPoFR links per bundle is not supported. Putting two PPPOFR circuits
into one bundle traffic will not be load balanced well across the circuits,
since the PPPOFR driver does not provide the flow control signaling that real
serial drivers do. MLPPP load balancing over ATM or frame relay might show
noticeably less effectiveness than the same load balancing over physical
interface.

Each PVC is associated with four different interfaces, namely the
physical interface, the sub-interface, and two virtual-access interfaces. Only
the MLP virtual-access interface has fancy queuing enabled. The other three
interfaces must have first in, first out (FIFO) queuing.

When you apply a service-policy
command to a virtual-template, Cisco IOS displays this normal warning
message:

These messages are normal. The first message advises that a
service-policy is not supported on the PPP virtual-access interface. The second
message confirms that the service-policy has taken effect on the MLP bundle
virtual-access interface. This fact is verified with the help of the
show
interfaces virtual-access, show
queue and show
policy-map interface commands to check the queuing
mechanism.

PPP authentication is not strictly required since a PVC is like a
leased-line. However, PPP authentication is handy as the show ppp
multilink command is then used to determine the name of the
router at the other end of the PVC.

Frame Relay traffic shaping must be enabled for MLPoFR PVCs on the
Frame Relay router.

Cisco IOS Software Release 12.2 initially supported a maximum of
twenty-five virtual templates per router. This limitation can impact the scale
of an ATM distribution router if every PVC is required to have a unique IP
address. The workaround for affected versions is to use IP unnumbered or to use
dialer interfaces instead of virtual templates. In Cisco IOS Software Release
12.2(8)T, the support is increased to 200 virtual templates.

Some service providers do not yet support PPP translation on their
FRF.8 devices. Whenever this limitation is in place, the providers must
configure their PVCs for transparent mode.

Most examples in the Cisco IOS documentation show configurations that
include a Frame Relay or ATM sub-interface. This sub-interface serves no
purpose. The virtual template should just be attached to the physical
interface. The result is a more compact and manageable configuration. This is
especially important if there are a large number of PVCs.

Use the show ppp multilink command as a
fool-proof way to determine if there are any cell/frame drops on the carrier
side. The only acceptable fragment loss is one caused by cyclic redundancy
check (CRC) errors.

Although MLPoATM/Frame Relay was introduced in Cisco IOS Software
Release 12.1(5)T, bugs in this and subsequent releases dictate that care be
taken when you select the Cisco IOS Software Release. Cisco recommends to use
the latest maintenance release of the Cisco IOS 12.2 mainstream. However, other
VoIP feature requirements can dictate the use of a different Cisco IOS Software
Release, such as 12.2(2)XT if Survivable Remote Site Telephony (SRST) is
required. This table lists some of the known issues. When you select Cisco IOS,
each Cisco bug ID should be evaluated against the chosen
IOS.

This section describes one of the first customer deployments of the
MLPoATM/Frame Relay feature. The customer is referred to by the fictitious name
XYZ Ltd. In 2001, XYZ Ltd replaced their PBXs with an IP Telephony solution. As
part of this project, a new IP network was built. and Frame Relay/ATM
interworking was chosen for the WAN. The carrier that provides the WAN service
uses Newbridge switches to deliver the ATM and Frame Relay services.

The XYZ Ltd network is a hub and spoke network that connects twenty-six
branches with two core sites. Each of the core sites is served by an E3 ATM
attached Cisco 3660 router. Eighteen of the twenty-six branches are medium
sized. They have dual Frame Relay PVCs that connect back to the 3660s at the
two core sites through ATM/Frame Relay IW. The remaining eight branches are
very small. They connect back to the closest medium sized branch through a
single Frame Relay PVC. All branch routers are Cisco 2620. An end-to-end ATM
PVC connects the two 3660 routers at the two hub sites. This figure illustrates
the topology.

This table shows the Frame Relay access speeds and CIR. The CIR varies
from 32 kbps to 256 kbps. Also shown in the table is the maximum number of
simultaneous VoIP calls carried by each PVC.

Site

Remote Site

Access (kbps)

CIR (kbps)

Number of Calls

Branch 1

Core 1

320

192

6

Branch 2

Branch 22

128

64

2.0

Branch 3

Core 1

576

256

8.0

Branch 4

Branch 16

64

32

2.0

Branch 5

Core 1

128

64

2.0

Branch 6

Branch 3

64

32

2.0

Branch 7

Core 1

512

256

8.0

Branch 8

Core 1

512

256

8.0

Branch 9

Branch 13

128

256

2.0

Branch 10

Core 1

256

128

4.0

Branch 11

Core 2

128

96

2.0

Branch 12

Core 1

128

64

2.0

Branch 13

Core 1

768

256

12.0

Branch 14

Core 1

192

96

4.0

Branch 15

Core 1

192

96

4.0

Branch 16

Core 1

448

192

8.0

Branch 17

Branch 13

128

64

2.0

Branch 18

Core 1

128

96

2.0

Branch 19

Branch 16

128

64

2.0

Branch 20

Core 1

64

32

2.0

Core 2

Core 1

34000

256

12.0

Branch 21

Branch 13

128

64

2.0

Branch 22

Core 1

384

192

6.0

Branch 23

Core 1

512

256

8.0

Branch 24

Core 1

192

96

2.0

Branch 25

Core 1

128

96

4.0

Branch 26

Branch 1

64

32

2.0

Frame Relay traffic shaping is performed by the branch routers.
Bursting beyond CIR is permitted. Cisco IOS traffic shaping is set to shape to
the access speed, with mincir equal to CIR. If backward explicit congestion
notifications (BECNs) are received from the carrier, then the router throttles
back to mincir. This approach is against Cisco recommendations when doing VoIP
over Frame Relay. Voice is already in trouble by the time the router responded
to congestion notifications. However, in this case the carrier believes that
its network is sufficiently over provisioned to never drop any frames or cells,
and hence, BECNs should never be seen.

ATM traffic shaping is set to shape at the frame access speed at the
remote end, plus cell tax. For instance, if the access speed is 512 Kbps, then
SCR is set to 512x53/48 = 565 Kbps. This shaping rate is used to maximize
throughput. This is safe because cell tax is stripped at the IW point. The
policing on the carrier side is configured generously so that the slight
oversubscription is allowed.

cRTP is used across the WAN. The hot spot as far as CPU is concerned is
the Cisco 3660 distribution router at core site 1. By adding the numbers in the
above table, it is determined that the theoretical maximum number of VoIP calls
that traverse this router is 102 calls. Performance numbers from this graph
indicate that the Cisco 3660 CPU load for 100 cRTP sessions (which are fast
switched) is approximately 50 percent.

Virtual templates are used with IP numbered WAN links. One virtual
template is required per PVC which is possible since the total number of PVCs
that terminate on each Cisco 3660 is less than twenty-five.