ISP TECH TALK

by Avi Freedman

MULTI-HOMING
WITH AND WITHOUT
BGP4

In this month's
column we'll explore being "multi-homed" - connected to two or more "upstream" Internet providers.
We'll look at why ISPs want to (most argue should) multi-home, and at some of the advantages and
disadvantages.

We'll also talk about the Border Gateway Protocol Version 4 (BGP4), the magic protocol that runs the
Internet. The myth goes that you must run BGP4 to "multi-home," but this isn't quite true. We'll be talking
quite a bit about BGP4 in future columns - for now, we'll cover just the basic concepts.

Before we can get to the meat of the discussion, though, we have to go over some basic topics.

BEING CONNECTED TO THE INTERNET: PART I

To be connected to the Internet, you need to be able to send data (in the form of IP packets) to every valid
IP address (host) on the Internet. Equally important is that all of the hosts on the Internet know how to send
data to you.

BEING "SINGLE-HOMED"

Figure 1

If you're "single-homed" you are connected to the Internet by one "upstream provider." All of your non-
local IP traffic (traffic destined to the Internet) is going to be sent to that provider, and all of your non-local
IP traffic that comes from the Internet will come in from that provider.

See Figure 1 for a diagram of a single-homed ISP. Jack's ISP has an Ethernet and one Class C's worth of
IP address space. Actually, it's called a /24, or slash 24 - IP Blocks that are 256 addresses in size in CIDR
address space are called /24s. For more information on this, please see April's column.

All of Jack's ISP's machines are numbered in the 207.8.130.0/24. Jack's ISP has a router at .1; servers
from .2 through .6; and a terminal server at .7. Furthermore, Jack's ISP's dial-up users get dynamically-
assigned addresses in the 207.8.130.129 to 207.8.130.159 range, so the terminal server also sometimes
responds to those IP addresses. The router in the network knows about this.

First we'll talk about the internal routing of Jack's ISP's network - and review the basics of IP routing -
and then we'll talk about how the outside world can get data to Jack's ISP.

AN IP ROUTING REFRESHER

Every machine that talks TCP/IP has an "IP routing table." This routing table tells the machine where to
send IP packets. Each IP packet has a source address and a destination address.

When a machine's IP software sees a packet, it has to figure out where to send it. To do this, it examines
the destination address of the packet and matches it against the "best" route in the routing table. To figure
out which route is the best one, the IP software tries to find the "tightest fitting" - or "most specific" route.
(See Figure 2). The "most specific" route is the smallest possible route which "contains" the destination IP
address of the packet in question. The smallest route is the route which represents the fewest number of IP
addresses (i.e. a Class C, or /24, is smaller than a Class B, or /16). The biggest route is the default route -
also called 0.0.0.0.

So let's look inside the router 207.8.130.1. There are four routes in the IP routing table:

("Network Number")

("Specificity")

Specificity

0.0.0.0

0.0.0.0

(/0)

207.106.127.46

Serial0

207.106.127.44

255.255.255.252

(/30)

207.106.127.45

Serial0

207.8.130.0

255.255.255.0

(/24)

207.8.130.1

Ethernet0

207.8.130.128

255.255.255.192

(/27)

207.8.130.7

Ethernet0

Let's say the router gets a packet destined for the web server, 207.8.130.3. It looks at its routing table and
finds 207.8.130.0/24 and 0.0.0.0/0 which match 207.8.130.3. 207.8.130.0/24 is the most specific route, so
it sends the packet out the Ethernet0 interface.

Why did it get this packet? Probably, someone on the Internet is sending a request to the web server and
the packet came in via the serial port. Note that it doesn't matter where the packet came from or
what the source IP address of the packet is. Each packet is routed based only on the destination
address.

Now let's say that the router gets a packet destined for one of the dial-up users, 207.8.130.134. It looks at
its routing table and finds 207.8.130.128/27, 207.8.130.0/24, and 0.0.0.0/0 as matching routes.
207.8.130.128/27 is the most specific route, so it looks at the next-hop of that route, which is 207.8.130.7.
Then it looks and finds that the best route for 207.8.130.7 is 207.8.130.0/24, which has a next-hop of
Ethernet0, so it sends the packet out Ethernet0 to 207.8.130.7.

DEFAULT ROUTING

Each machine on Jack's ISP's network also has a routing table which probably looks like:

Starting IP address Interface ("Network Number")

Network Mask
("Specificity")

CIDR Specificity

Next-Hop

0.0.0.0

0.0.0.0

(/0)

207.106.127.46

Serial0

207.106.127.44

255.255.255.252

(/30)

207.106.127.45

Serial0

207.8.130.0

255.255.255.0

(/24)

207.8.130.X

Ethernet0

207.8.130.128

255.255.255.192

(/27)

207.8.130.7

Ethernet0

Where X is the address of the particular local machine.

Figure 2

Note that you could omit the 207.106.127.44/30 route from the local routing tables, but that would
force all
data destined to the terminal server to "bounce" off of the router at 207.106.127.1. If each local machine
isn't specifically told how to get to the addresses 207.8.130.129 to 207.8.130.159 (the 207.8.130.128/37
route), then packets destined to the local modem users will be sent to the router, which will then send them
to the terminal server. This means that those packets must traverse the Ethernet twice, which chews up
bandwidth on the Ethernet. So it's always better to put all of your local routes into all of your local
machines somehow - but as your network grows you may find that this is a pain to do.

Now, for a local server to get to any other machine or modem user on your network, it will have a
specific
route (some route other than the default route) telling it how to get there (the next-hop). But there won't be
a route in the routing tables on any of your machines for an IP packet destined to a host out on the
Internet - so the route that will "match" packets destined to the Internet is the "default" route - 0.0.0.0/0.

STATIC AND CONNECTED ROUTES

How do the routes get into the IP routing tables? Some are "connected" routes - they are routes
that are associated with the interfaces on the machine. In the case of the router, 207.106.127.44/30 is a
connected route associated with Serial0, which has a local IP address of 207.106.127.45. 207.8.130.0/24 is
a connected route associated with Ethernet0, which has a local IP address of 207.8.130.1.

The other routes in the routing table had to be inserted "statically" as part of the startup procedure of the
router. These "static routes" are "installed" whenever the associated interface is up. 0.0.0.0/0 is a static
route with a next-hop of 207.106.127.46 (the provider's router at the far end of the T-1); and
207.8.130.128/27 is a static route with a next-hop of the terminal server on the local network,
207.8.130.128.

A very important note: Most well-behaved routers will delete any routes associated with interfaces that are
down. For example, the default route 0.0.0.0/0 and the connected route 207.106.127.44/30 will go away if
the connection to the remote provider over the T-1 goes down.

Some of the outbound load-balancing we will talk about later relies on this behavior. But most UNIX
machines - and some routers - don't obey this fundamental rule of routing. If you intend to multi-home
yourself with a particular router, try pulling the serial cord from one interface. Make sure that the routes
associated with that interface go away.

BEING "CONNECTED" TO THE INTERNET: PART II

Figure3

To be connected to the Internet usefully, every machine on your network must have an IP address that is
"reachable" globally. If you have a default route to your router's WAN interface, then it's your provider's
job to figure out how to deliver traffic destined to the world.

But an equally (and arguably more) important job is to announce to all of the key routers on the
Internet that they know how to send packets to you. If even one key router on the Internet doesn't know
how to get to you, you do NOT have global Internet connectivity. A router knows how to get to you
because it has a route in its IP routing table that tells it how to send data to you - it's the same principle as
the IP routing going on in your local network.

ROUTE ANNOUNCEMENTS WITH BGP4:
BEING "SEEN" BY THE INTERNET

The key routers on the Internet are the routers in the "defaultless core." These routers have no default
route. If you want to be reachable by all of the hosts on the Internet, there must be a matching route for
your IP space in all of these routers.

How do those routes get into those key routers? Different providers "peer" with each other. They have
private and public "Exchange Points" where they exchange "route advertisements." These advertisements
are "promises" to carry traffic to various sections of the IP space.

Providers use the BGP4 protocol to advertise routing information to each other. Routers at the "border" of
various networks ("Autonomous Systems" in BGP-speak) exchange routes with each other via "peering
sessions." (See Figures 3 and 4).

Route announcements filter upwards from the edge ISPs, through various transit providers, to the top-level
providers. By the time all of this occurs, all of those key routers know about your routes.

WHAT ROUTES GET ADVERTISED FOR YOU
WHEN YOU'RE SINGLE-HOMED?

If you have your own address space (old Class C blocks) - or more recent CIDR allocations from the
NIC - your pro-vider(s) will advertise those blocks for you as BGP route announcements.
If you are using IP space that you got from your ISP, chances are good that you're part of one of their
"aggregates" (or "IP blocks"). Aggregate routes are larger routes that cover multiple /24s. If you have been
allocated 207.8.130.0/24, a "whois" on 207.8.130.0 will show you who owns the underlying IP block, and
what the specificity is. In the case of 207.8.130.0, a "whois" will show you that it's part of the
207.8.130.0/17 block (half of a Class B in size). If you were to look at the "global routing table" you'd see
that the owner of the 207.8.128.0/17 block, Net Access, advertises 207.8.128.0 "as" a /17 (meaning, with
the specificity of /17).

If you are single-homed, the only way that anyone in the world can send data to you is through your one
provider. So it makes no sense for your provider (Net Access, in this case) to advertise 207.8.130.0/24 as
well as 207.8.128.0! Let's say that Net Access advertised 207.8.128.0/17 and 207.8.130.0/24. The more
specific route for any of your IP addresses would be 207.8.130.0/24, so that is the route that would get
used. If that route were not advertised, however, 207.8.128.0/17 is the route that would get used. The net
effect is the same, and having the more specific route for 207.8.130.0/24 out there is just one more route
for every core router to know about and have to spend memory and CPU time on.

If you're multi-homed, however, at least one of your providers will need to announce a specific route to
match your blocks. More on this shortly.

MORE BGP TERMINOLOGY

As we said, BGP4 as we're looking at it, is a protocol spoken between Autono-mous Systems to advertise
routes. BGP4 is spoken between two routers and routes are exchanged via "peering sessions." Each
Autonomous System has an "Autonomous System Number" (ASN). Net Access's is 4969. UUNET's is
701. There are hundreds of ASNs in use on the Internet.

In addition to the usual information (starting IP address, specificity, next-hop information), each BGP route
has a property called the AS_ PATH.-.the path of Autonomous Systems that the route took from the very
first advertising router. It's just a list of the ASNs in the reverse order of advertisement.

Every time a route is advertised by one router to another over a peering session, the receiving router
"stamps" the incoming route with the ASN of the remote router by prepending the remote ASN to the
beginning of the AS_PATH. When UUNET hears a route from Net Access, for example, it notes that it got
the route from AS 4969 (Net Access's ASN).

Remote routers will generally decide which is the best route of multiple BGP routes by picking the route
with the shortest AS_PATH (that is, the route that has traveled few the fewest providers) to get from start
to finish.

REVISITING THE SINGLE-HOMED PROVIDER

Jack's ISP (the average single-homed provider) will typically have one route announcement "out there" for
each of its routes. This makes sense - you can only get into Jack's ISP's network by one path - sending
data to Net Access.

Jack's ISP's upstream provider must either peer with all of the other key networks on the Internet (so that
routes for Jack's ISP get into all of the default-free core routers), or must get "transit" (be a complete or
partial customer) of some other providers.

For example, AGIS (ASN 4200) will peer with very few other networks, so Net Access (ASN 4969) must
get some Autonomous System who does peer with ASN 4200 to advertise Net Access's routes to AGIS, or
AGIS's routers will not know about Jack's ISP's (Net Access's) routes.

WHY BE MULTI-HOMED?

Additional bandwidth and redundancy are the two major reasons that ISPs multi-home. Multi-homing
(adding another upstream provider) is expensive, costing you at least $1,000 to $2,000 extra per month, so
most providers put it off as long as possible. This is a mistake! In even the smallest areas there is now
generally competition among ISPs, and the ISP with the most reliable network can attract customers
willing to pay for quality and reliability.

If you've got a T-1 to the Internet and it's running at 70 to 80 percent capacity, then you'll start to see
slowness. Or, worse, you could have the dreaded packet loss on your line due to congestion. Your
customers will start leaving you in droves if this happens. Additional bandwidth is a major reason that
many finally break down and multi-home. But consider this: If you are single-homed you are
completely dependent on the up-time and quality of your one upstream provider. Also, you're
completely dependent on your one border router and your one link to that upstream provider. If any of
those components fails you're completely off the Net!

There's only one route out there for you. If your router or T-1 goes down, then your provider's not going
to be able to send data to you. And if your provider becomes disconnected from the Internet or has some
major internal routing problem, then you're also disconnected from some or all of the Internet.

MULTI-HOMING:
ROUTE ADVERTISEMENTS

When you're multi-homed, your routes should be advertised by both providers. If you have two or
more routes out there for each one of your IP blocks, you can sustain a complete loss of a T-1 - or severe
problems with one of your upstream providers - and not have your customers notice a thing.

Figure4

Let's look at Figure 4. In this example, we'll use our friend Jack's ISP, now connected to both Net Access
and to UUNET, and a dial-up user on AOL who wants to query a web server on Jack's ISP's network,
207.8.130.3. AOL's Internet provider is ANS, so AOL sends the packet to somewhere in ANS's network.
ANS sees three BGP route advertisements for 207.8.130.0:

If those routes weren't there, ANS (and thus AOL) would have no connectivity to Jack's ISP's network. If
there was only one route of specificity /24, then there'd only be one "primary" path to Jack's ISP's
network. But since there are two routes for 207.8.130.0/24 (also called two views of this one route), ANS
will choose one of those as the best path and will use it. Incoming load-balancing works on the principle
that roughly half of the networks out there will choose one of multiple paths for any network and that the
rest will pick one of the other paths. We'll go into more detail on this in future columns, and we will
discuss the actual algorithms used by routers and providers to make and tune these decisions.

MULTI-HOMING WITH BGP

If you speak BGP to your providers, then you are the one originating the route advertisements that
put your Autonomous System on the Internet. You have some degree of control over how those routes are
advertised - we'll be talking about methods like AS_PATH padding and changing route specificity in the
future.

And you can also take "full routes" from one or both providers and let your router use BGP "selection
algorithms" to pick the best path for each outgoing packet. This will generally give you a moderate bump
in "quality" of connectivity - perhaps as much as 5 or 10 percent less packet loss to certain destinations.

Also, if you have nothing better to do than tune your connectivity, you can start to fine-tune outgoing data
flow. For example, if you notice that one provider has poor PSI connectivity although it's got the best BGP
routes to PSI, then all of your PSI traffic is sent out to your other provider.

MULTI-HOMING WITHOUT BGP

But you don't need to speak BGP to have your routes be advertised by both providers. They can do it for
you, just as your single upstream provider does it for you when you are single-homed. In this case you are
part of their Autonomous System for routing purposes, and you have to call them on the phone and ask
them to change or add route advertisements for you.

Providers are generally happier to do the work of injecting routes into BGP for you, since it's a bit of work
for them to set up a BGP peering session for you and to carefully filter the routes they hear from you over
that session.

To control outgoing data flow from your network, all you have to do is add another default route - there's
no need to use BGP to take a list of all 45,000 or more routes on the Internet. It's true that you have more
control and flexibility if you have those routes to work with, but the difference is usually not that great
between selecting the best BGP route and round-robin default-routing.

SO WHICH IS BETTER?

You can get 90 percent of the benefit of multi-homing without speaking BGP yourself. But as long as
you're paying for two T-1s, why not take full advantage of them by speaking BGP?

The first problem is that running BGP on your own requires a lot of knowledge (and hopefully experience).
Screw-ups in BGP route advertisements can be felt all over the Internet, and are treated very seriously. If
you don't know what you're doing you'll wind up with a less reliable network instead of a more reliable
one.

The second problem is that it takes a fairly expensive router to take full routes from your providers. Each
route takes a certain amount of CPU and memory, and the current Internet routing "load" of 45,000+ routes
is too much for almost all low-end routers; however, you can take less than full routes - or even just use
BGP to announce your routes and still just default-route for outbound traffic - with a smaller router like a
Cisco 2501.

So the decision is yours. Most pro-viders will let you speak BGP to them, but will be somewhat
unsympathetic if you misconfigure your end and break your connectivity. On the other hand, there are
some advantages to speaking BGP at your end.

For the rest of this column, we'll finish explaining how multi-homing without BGP works.

Next month's column will go into more BGP details and will show you how to safely announce your routes
via BGP with router configurations. This can be done even with a low-end router. We will also discuss
some of the options that you have for managing the incoming routing information which determines how
you send your outbound traffic to the Net.

Now we'll explain both sides of being multi-homed without speaking BGP to your provider. The first is
load-balanced default routing, which is the art of getting data out of your network. The second is how your
providers advertise your routes for you, which controls how other providers hear your routes - and thus
how data flows in to your network.

Keep in mind that these two things (sending data out with default routes and getting others to send data to
you based on route advertisements from your providers) are the key to being connected to the Internet.

DEFAULT ROUTING TO MULTIPLE PROVIDERS

If you go from one provider to two, then you're going to already have a default route installed. You have to
decide how you want to send data out from your network. Once you've decided what you want to do, you
can go about implementing it.

Remember, the default route controls where you send data that is not destined for you local network. So
adding default routes is the way you both increase your outbound bandwidth to the Internet and add the
redundancy that multi-homing provides.

Let's say that Jack's ISP has his original default route installed. In the configuration mode on his Cisco, he
enters:

Now there are two routes (0.0.0.0/0 through 137.39.34.65 with metric 0; and 0.0.0.0/0 through
207.106.127.46 with metric 0) of equal specificity (both are /0 routes) and of equal metric (both have a
metric of 0).

Both routes will now be installed in the Cisco's IP routing table, and the Cisco will now use both T-1s for
outbound data. If one of the serial interaces goes down (because one of the T-1s goes down), one of the
default routes will go away, but one will still be there to make sure your router has a way of sending data
out to the Internet.

Also, this example turns on ip route cache on both interfaces to ensure that the load balancing is
done on a per-connection basis rather than by simply sending every other packet out each
interface. Note that sending the packets out in a round-robin fashion makes TCP/IP unhappy.

MULTI-HOMING TO THE
SAME PROVIDER

Instead of multi-homing to different providers, you can multi-home to the same provider. Net Access, for
example, allows all of their dedicated-line T-1 customers to establish a backup link via the local Frame
Relay "cloud." Customers install a backup default route so they'll only use the Frame Relay path if the
primary T-1 goes down (since they're not paying for additional bandwidth, only redundancy).

Customers who are happy with their upstream providers might want to run multiple links for added
bandwidth or reliability. Sadly, few IP providers are so reliable that customers are tempted to do this, but it
does happen. Also, paranoia dictates that an Act of God can always strike a particular upstream provider or
that provider's POP in a particular city. You almost always get better reliability by multi-homing to
multiple providers.

If you do decide to multi-home to the same provider and run two equal-bandwidth pipes into either the
same router at the provider's side, or routers at the same POP, you can set no ip route-cache on
your serial interfaces and things should work well. Why? Because all of your outgoing packets travel
almost the same path and have almost the same latency this way.

BEING ADVERTISED BY MULTIPLE
PROVIDERS WITHOUT PI-SPACE

We'll use Figure 4 again. Remember April 1997's document on getting provider-independent (PI) space?
The reason it's so important to have "your own" ip space is that without it multi-homing is quite tricky and
requires a lot of cooperation from your original provider.

Let's say that Jack's ISP is using 207.8.130.0/24. Jack's ISP's provider (Net Access) has 207.8.128.0/17.
So Net Access announces only 207.8.128.0/17 to the world. There is no global advertisement for
207.8.130.0/24. In this case, any packet destined to 207.8.128.0/17 will be picked up by the less specific
(more general) route 207.8.128.0/17. We talked about this before, when examining how Jack's ISP's routes
are seen by the outside world when single-homed.

Now Jack's ISP wants to multi-home. So Jack's ISP buys a T-1 from newprovider. Jack's ISP sets up BGP
with both Net Access and newprovider. Suddenly, the world sees two routes for Jack's ISP:

207.106.0.0/16, advertised by Net Access; and 207.8.130.0/24, advertised by newprovider.
Remember, the most specific route always wins, so newprovider will wind up carrying almost all, if not all,
of Jack's ISP's incoming traffic! Certain parts of Net Access's network may actually prefer newprovider's
T-1 to get to Jack's ISP! In fact, this won't happen with Net Access's network, but some providers will
prefer externally-heard more specific routes to get to dual-homed customers.

This state of affairs gets you backup (if newprovider goes down, your data will once again flow
through Net Access based on the aggregate advertisement), but doesn't load-balance. your inbound
traffic. To have incoming load-balancing, you need to advertise two routes of equal specificity.
Even so, inbound load-balancing is tricky and depends on who peers with whom and how different
providers "hear" each other - but unless there are two "competing" equal-specificity routes out there for
you, there's no hope if inbound load-balancing.

The problem is that most largeish providers use something called aggregate-address
statements - and they certainly have some sort of filter to keep the more specific routes floating
around inside of their networks from being advertised to the world. Remember, the world only wants to
hear about 207.8.128.0/17 if the little, more specific, routes inside of 207.8.128.0/17 are not multi-homed.

So what does Net Access have to do? Blow holes in their aggregation statement. One way or another, it's
going to take modifications in Net Access's "border" routers to make incoming load-balancing work
properly for Jack's ISP. Net Access may not want to do this. (In fact, Net Access does do this, but some
providers don't.) Basically, everywhere that Net Access peers with anyone else (and this is usually at least
5 to 10 places), they have to modify their aggregation statements or other filters to "allow" your more
specific route announcement to pass through.

If Net Access blows holes in their aggregation filter using "suppress maps," which we'll talk about in a few
columns, then Net Access can announce:

207.8.128.0/17
207.8.130.0/24

And newprovider will announce:

207.8.130.0/24

Thus, Net Access's other customers who "live" inside of 207.8.128.0/17 will still be advertised, but
207.8.130.0 will now have two advertisements at a specificity of /24, so some load-balancing may happen
for inbound traffic to Jack's ISP.

This is one reason that it's important to choose a primary provider based on how cooperative they'll be
when you want to multi-home.

BEING ADVERTISED WITH PI-SPACE

If you have Provider-Independent (PI) address space, announcing your routes is easier, since you're not a
part of anyone else's aggregate block. Examples include old Class C or Class B blocks that you or your
customers might have, or newer CIDR allocations that you (or, if you sell to ISPs, your customers) might
have.

SUMMARY

We've talked about how the world sees you when you're single-homed and multi-homed, and about what
BGP does. We've also talked about how you can be multi-homed effectively without using BGP.

Next month we'll go into more detail on BGP. We'll go through the details of configuring a router to speak
BGP to multiple providers - and to announce and receive BGP routes.