The Effect of AS Path Prepending

Suppose you multihome to two ISPs that are similar: they interconnect at mostly the same NAPs and Internet Exchanges, and they peer with mostly the same networks. Under these circumstances, other networks see two similar paths for the routes you announce. Figure 6-4 shows an example of this.

Figure 6-4. Multihoming to similar ISPs

If AS E (the example multihomed network, AS 60055) wants to receive more traffic over ISP A and thus makes the AS path longer over ISP B, the majority of traffic will flow over the ISP A, which now has the shorter path. In a situation with two similar ISPs, AS path prepending gives them only three choices:

The default traffic distribution, which may or may not be balanced

Longer path over ISP A: majority of traffic comes in over ISP B

Longer path over ISP B: majority of traffic comes in over ISP A

Table 6-1 shows which route is preferred in the situation shown in Figure 6-4 without path prepending, with prepending the path to ISP A, and with prepending the path to ISP B.

Table 6-1: Prepended paths over similar ISPs

AS X

AS Y

AS Z

Traffic distribution

Prepend to A

AEEBE

AEEBE

AEEBE

ISP A: 15%
ISP B: 85%

No prepending

AEBE

AE
BE

AEBE

ISP A: 40%
ISP B: 60%

Prepend to B

AE
BEE

AE
BEE

AE
BEE

ISP A: 90%
ISP B: 10%

For the purposes of calculating the traffic distribution, it's assumed that A always handles 15% of the traffic, B always 10%, and ASes X, Y, and Z are all the source of 25% of incoming traffic. ASes with "even" letters (X, Z) prefer to send traffic over ISP B when the paths are of equal length; "odd" ASes (Y) prefer ISP A in this example. The preferred path is listed in bold type in the table.

When the two ISPs are not as similar, increasing the length of the AS path has a more gradual effect, because the paths over ISPs A and B aren't the same for all networks. Figure 6-5 shows multihoming to dissimilar ISPs.

Figure 6-5. Multihoming to dissimilar ISPs

In this example, ISP B is a much smaller ISP that doesn't peer with networks X and Y, but rather buys transit service from AS C to reach those networks. Networks V and W don't peer directly with ISP A, so even if the path over ISP B becomes a lot longer, they'll still prefer to send traffic to AS E over ISP B. Because this is a peering link, sending the traffic over this route is cheaper. Network Z, on the other hand, will immediately route traffic over ISP A when the path over ISP B is prepended, because the connections to both A and B are peering links. Table 6-2 shows the possible traffic distribution using prepending.

Table 6-2: Prepended paths over dissimilar ISPs

AS C

AS V

AS W

AS X

AS Y

AS Z

Traffic distribution

2 to ISP A

BE

BE

BE
CBE

AEEECBE

AEEECBE

AEEEBE

A: 15%
B: 85%

1 to ISP A

BE

BE

BE
CBE

AEECBE

AEE
CBE

AEEBE

A: 35%
B: 65%

No prepending

BE

BE

BE
CBE

AE
CBE

AE
CBE

AEBE

A: 55%
B: 45%

1 to ISP B

BEE

BEE

BEE
CBEE

AE
CBEE

AE
CBEE

AE
BEE

A: 75%
B: 25%

The traffic distribution in this example is 15% from ISP A, 5% from ISP B and ASes V and W, 10% from AS C, and 20% from ASes X, Y, and Z.

TIP: All else being equal, it's a good idea to select dissimilar ISPs, for instance, one tier-1 ISP that peers with all the other large networks, and one tier-2 ISP that peers with many small networks. This way, you have a wide range of traffic engineering options.

Setting Outbound Communities

In many cases you'll want to prepend the path for certain upstream networks or peers of a transit ISP and not for others. For instance, if two of your ISPs have a transit network in common, you might want to have one ISP announce a prepended path to this transit network without changing the path that other transit networks and peers see over that ISP. To avoid spending a lot of time implementing this type of policy upon customer request, many ISPs provide their customers (and sometimes their peers) with a list of communities that trigger actions such as path prepending and setting the Local Preference. This can then be done for each route individually.

Well-Known Communities

Communities were introduced in BGP by RFC 1997. This RFC also defines the three well-known communities listed in Table 6-3.

Table 6-3: Well-known communities

Well-known community

Action

no-export (0xFFFFFF01)

Don't advertise this route to eBGP peers.

no-advertise (0xFFFFFF02)

Don't advertise this route to any peers, iBGP, or eBGP.

no-export-subconfed (0xFFFFFF03)

Advertise this route to iBGP peers with the same AS number, but not to other confederation members.

These communities can be useful under certain circumstances, for example, if an ISP wants to set the no-export community on routes it sends to a customer to make sure the customer doesn't accidentally announce the ISP's routes to another ISP. If the customer provides transit services to customers of his own, however, they won't receive the route unless the original customer of the ISP removes the no-advertise community.

WARNING: Setting the no-export, no-advertise, or no-export-subconfed communities can have the (possibly unwanted) side effect that no routes are announced, even if there are other routes that would otherwise be eligible for announcement.

For instance, if you set the no-advertise community on routes announced to ISP B, other customers of ISP B won't see these routes because they aren't advertised. This is as intended. But routes with the same NLRI that ISP B has learned from ISP A will not be advertised either, because ISP B considers the directly received routes with the no-advertise community best, and only the best route is eligible for further announcement over BGP.

Common Community Actions

ISPs accepting communities provide their customers with a list of communities they use and what action is taken for each community. It's possible to set several communities for a single route, but the results may be unpredictable if your ISP doesn't expect this. Many networks list the communities they accept in their AUT-NUM object in the Routing Registry they use. Table 6-4 shows a fairly typical list of communities an ISP might accept.

Table 6-4: An example of communities an ISP accepts

Community

Action

50066:70

Set Local Preference to 70.

50066:90

Set Local Preference to 90.

50066:110

Set Local Preference to 110.

50066:5010

Announce this route for transit to ISP C.

50066:5020

Don't announce this route to transit ISP or peer C.

50066:5040

Prepend AS path to C once.

50066:5041

Prepend AS path to C twice.

50066:5042

Prepend AS path to C three times.

50066:10040

Prepend AS path once at interconnect point I.

50066:10041

Prepend AS path twice at interconnect point I.

50066:10042

Prepend AS path three times at interconnect point I.

Some ISPs require you to set a community indicating a route should be announced for transit, 50066:5010 in this example. This isn't common: most networks do the opposite and allow you to set a community indicating the route shouldn't be announced to transit networks. Be sure to notice the subtle difference between not announcing for transit and not announcing to transit networks: in the first case, the potential transit network still receives the announcement, but the route is treated as a peering route and not announced to peers and upstream networks of the transit network. In the latter case, the transit network doesn't get to hear the route at all.

Influencing the Local Preference in Upstream ASes

The MED is specifically intended to inform an upstream ISP that one connection should be preferred over another, but today this is often done with communities. Using communities instead of the MED may have benefits internal to the ISP network. For example, the ISP is then free to use the MED for another purpose, as we did for outbound traffic engineering in the beginning of this chapter. And, unlike the MED, using a community to set the Local Preference inside an ISP network also makes it possible to use a link to an ISP as a backup for a link to another ISP. When the Local Preference is set sufficiently low for the intended backup connection, the ISP it connects to will completely ignore the route and always send traffic over the other ISP as long as there is a route present over this ISP. This can't be accomplished with the MED; the AS path length overrides it, the MED isn't communicated from AS to AS, and by default, the MED is looked at only when two connections terminate at the same AS.

The impact of changing the Local Preference depends on the Local Preference values an ISP uses for routes learned from transit, peers, and customers. In this example, that would be 80 for transit, 100 for peers, and 120 for customers. If you have a fast main connection to this ISP along with a slower backup connection, you'll probably want to set community 50066:110 for routes announced over the backup connection. This makes sure all traffic flows over the main connection and the backup connection is used only when the main connection is unavailable. It's also possible to do this when you connect to two different ISPs: to ISP A with a main connection, and to ISP B with a backup connection. Then you'll want to set community 50066:90 or even 50066:70 so ISP B sends all traffic for you over a peering or transit connection to ISP A.

Example 6-13 shows a configuration for the router terminating the backup connection to ISP B. The main, high-bandwidth connection terminates at another router.

The community 50066:5010 makes this route eligible for announcement as a transit route over AS C, but the community 50066:70 makes sure this route has the lowest possible Local Preference in ISP B's network. Thus, ISP B won't use it as long as there is any other route with the same NLRI (prefix), even if this means routing the traffic over ISP A.

TIP: By default, Cisco routers accept incoming communities but don't transmit them over iBGP or eBGP. The send-community command enables sending communities to a neighbor.

Prepending the AS Path

Some smaller ISPs have path-prepending communities for each peer, but even medium-sized ISPs peer with many networks, so this soon gets out of hand. More often, an ISP has communities to prepend the path to each of its transit ISPs individually, as well as communities to prepend the path for an entire interconnect point. Our example ISP B (AS 50066) accepts path prepending communities for ISP X and interconnect point I.

AS W in Figure 6-5 (shown earlier this chapter) connects both directly to ISP B and also over transit AS C and then ISP B. Supposing the peering link between AS W and ISP B is congested, we'll want incoming traffic from AS W to flow over ISP C. This is accomplished by prepending the path ISP B announces to AS W twice, as is done in the configuration in Example 6-14.

Unfortunately, it's not possible to do something similar for outbound traffic: this will still flow over the congested connection between ISP B and AS C. This isn't the case if it's routed over ISP A and not over ISP B, of course, as is done in Example 6-4 earlier this chapter (in a previous article). Also, setting community 50066:10041 means the path is prepended twice towards all peers at this interconnect point. This may be undesirable, for instance if AS Z connects to ISP B over the same interconnect point as AS W. AS Z now sees the path C B B E over ISP B and the much shorter path AS E over ISP A, so traffic from AS Z will now come in over the connection to ISP A.

In the next installment learn how to announce more specific routes. When prepending the path and setting communities for outbound routes aren't enough to balance incoming traffic, this is the only step left to take. However, because a more specific route always takes precedence over a less specific route, this technique always works.