The Challenge
People tend to underestimate the important of IGP routing features in modern network. So here is a small challenge scenario for you to practice OSPF traffic engineering. Take a look at the diagram below for information on the topology and link bandwidth. You may assume that every router has a loopback interface for network testing and OSPF router-id selection.

There is a large cloud of media servers behind R4, and the users behind R1 need to use full 300Mbps of bandwidth when downloading files off the servers. The network is running single-area OSPF for IP routing. Ensure you can accomplish the above goal without using MPLS Traffic Engineering or Policy Based Routing. You are allowed to create additional logical interfaces, but the routing protocol, OSPF areas, physical links and their characteristics should remain unchanged. Keep the amount of changes to minimum and do not introduce new IP addresses.

The first person to provide a working solution will receive 100 rack rental tokens from our partner company GradedLabs. Please use your valid e-mail address when posting a comment, so we can locate your INE account.

UPD
OK I forgot to rule out the “route-via” option Try solving the task without relying on any “policy-based” routing decisions.

The winner is: Antonie Henning (http://21500.net). Ivan Pepelnjak helped finding a logical “loophole” in my scenario by pointing to the “route-via” option available with GRE tunnels and correctly stating there should be 6 end-to-end tunnels to implement proper load-balancing. Hans Verkerk was close in his idea, but used static routing which was slightly against the rules and not as elegant as Antonie’s solution. Chris Stos-Gale and Nitzan Tzelniker came with the correct solution as well, but Antonie completed the challenge ahead of them. Thanks to everyone for participating in the challenge, it’s been fun!

The Solution:

The problem is that there are three paths with varying minimum bandwidth values (50, 100 and 150, totaling to 300Mbps). Since OSPF does not support unequal-cost load-balancing, it is somewhat challenging to fully use the available bandwidth. There was a lot of ideas posted in the comments, and they mainly fall in three main categories:

1) Modify OSPF costs to create three equal cost paths from R4 to R1. This will result in slow (50Mbps) link oversaturation. Another variation was using three tunnel interfaces between R1/R4 with the same ECMP logic. This results in the same problem.
2) Create six tunnels between R4 and R1 and configure the network so that 3 tunnels go across the fastest path, 2 tunnels take the medium path and one tunnel take the slowest path. This is somewhat similar to MPLS TE. To steer the tunnels you may use either static routes or the “route-via” option (Thanks to Ivan Pepelnjak to pointing me that!!). This solution would work, but violate the “updated” requirement not to use any “policy-based” routing decision, relying purely on OSPF path selection.
3) The solution that I had on mind was splitting the links connecting R4 to it’s neighbors into “sub-channels” proportional to the bandwidth assigned to a given path:

The link labels represent OSPF costs. You only need to split the links at R4, as this is the “source” of the traffic flows. Link splitting could be done in two ways: using logical virtual circuits (e.g. FR PVCs or Ethernet VLANs/VCs) or by using IP tunnels. You will only need to run the IP tunnels between R4 and the directly attached routers, disabling OSPF on the physical link and enabling it on the tunnels. Sample output at R4 for R1′s prefix:

I would like to thank everyone who participated in the “challenge”, I read all your responses but had to stop commenting when I found the right solution. I hope you enjoyed that little scenario as much as I did. Personally, I have some incline toward the “traditional” traffic engineering solutions based on pure IGP metric manipulation. Even though the solution presented does not scale in the real world, where you may resort to a different option (e.g. end-to-end route-via tunnels), it perfectly illustrates the little hacks you can do to a link-state IGP to break the default “ECMP paradigm”.

About Petr Lapukhov, 4xCCIE/CCDE:

Petr Lapukhov's career in IT begain in 1988 with a focus on computer programming, and progressed into networking with his first exposure to Novell NetWare in 1991. Initially involved with Kazan State University's campus network support and UNIX system administration, he went through the path of becoming a networking consultant, taking part in many network deployment projects. Petr currently has over 12 years of experience working in the Cisco networking field, and is the only person in the world to have obtained four CCIEs in under two years, passing each on his first attempt. Petr is an exceptional case in that he has been working with all of the technologies covered in his four CCIE tracks (R&S, Security, SP, and Voice) on a daily basis for many years. When not actively teaching classes, developing self-paced products, studying for the CCDE Practical & the CCIE Storage Lab Exam, and completing his PhD in Applied Mathematics.

39 Responses to “Traffic Engineering Challenge”

Maybe I’m missing something, but I would think that you can build 6 gre tunnels. Leave them all ip unnumbered since the requirement says to not add IPs. For three of the tunnels, the source and destination will make them traverse the 150M link; two will traverse the R1-R2-R3-R4 path; one will traverse the R1-R5-R6 path. This is achieved by setting the source and destination to make them take the right paths. The route are advertised over these paths with a low but equal cost and R1 & R4 are set to have a maximum-paths 6 setting. This should make those routes show up as equal across all six paths and allow 6-way balancing.

We need to set auto-cost reference bandwidth on R1 to 1 so that R1 calculate metric based on hop count to R4.
Then on links beetwen R1-R4 and R5-R4 we change network type to point-to-multipoint . Then change cost on interface R1 to R4 to 3 and cost on interface R5 to R4 to 2.
So metric on R1 to R4 loopback interface would be 4 through R2,R4 and R5.
R1:
auto-cost reference-bandwidth 1
neighbor 10.0.14.4 cost 3
R5:
neighbor 10.0.45.4 cost 2

I’m not sure what do u mean under “users behind R1 need to use full 300Mbps of bandwidth when downloading files off the servers”, but in case I had to route traffic via 1-2-3-4 I would do something like this:

0. auto-cost refference-bandwidth on all routers making cost of any 300mb to 1
1. play with interface costs on R4 making route 1-2-3-4 more preferrable

First, I will modify the ospf cost on different interfaces.
I will put “ip ospf cost 2″ command between R1-R2, R2-R3 and R3-R4, “ip ospf cost 6″ command between R1-R4 and “ip ospf cost 3″ command between R1-R5 and R5-R4.
When you do “show ip route os”, you should see 3 routes for the same destination with a cost equal to 6.

Secondly, I think that you need to put “ip cef distributed” command to activate the load-sharing between theses 3 routes and also you can use full 300Mbps bandwidth (from R1 to R4 : 100Mbps via R2 + 150M via R4 + 50M via R5 = 300M)

I have left the routing protocol intact, but I included several static routes in my solution. Let’s assume subnet 11.11.11.0/24 is terminated on R1 where clients resides which are downloading content from behind R4′s media servers.

Seen from R4, traffic towards subnet where “clients” terminates to R1, must be distributed to R3, R1 and R5 in a 2:3:1 fashion towards 11.11.11.0/24 subnet.

In order to meet the requirement I have created a GRE tunnel interface on R1 and R4 (thanks for the hint!). And after this I created static routes pointing to next-hops and interfaces to fullfill the loadbalancing requirement which was early mentinoned:

on R4 -> R3 -> R2 -> R1 I would set the ospf cost to 10 on all of those links manually using the “ip ospf cost” command under each of the interfaces this would make the metric look like it was 30 on R1

Between R1 and R4 link would set the ip ospf cost on to 30

Between R4 -> R5 -> R1 link I would set up the ip ospf cost of 15 which make the link look like it also had a cost of 30.

Using the metrics R1 will place all three routes in the routing table, since it can equally load balance up to four routes by default and equally share the traffic among the three paths.

Can’t do anything like unequal cost load balancing here because we are not using eigrp which is the only IGP that can support it.

1) Create a tunnel using the loopback IPs of R1 and R4 as the source and destination. Make sure that OSPF is routing across the tunnel between R1 and R4 and use a distribute-list for tunnel recursion issues.
2) Make sure that these end-point IP’s have an equal cost in OSPF across all 3 paths to the tunnel destination from R1 and from R4. (Equal cost load balancing should be already active up to 4 links in OSPF by default).
3) Have each physical interface on R1 and R4 traffic-shape to the lowest bandwidth link for each of the three paths between them.

OSPF won’t do unequal cost load balancing. Therefore, you would need a number of links over the various routes proportionate to the bandwidth share.
In order to do this, create a link for each 50 Mbps. that breaks down as:-
R1-> R2 = 100M Min Bandwidth
1 Physical
1 Tunnel

R1 -> R4 = 150 Mb/s Min Bandwidth
1 Physical
2 Tunnels (use tunnel key to identify which tunnel is which as you have to use the same source / dest addresses)

R1 -> R5 = 50 Mb/s Min Bandwidth
1 Physical

Once you have also entered maximum-paths 6 under the OSPF process, this will cause OSPF to load balance accross 6 links equally causing it to use 300 Mb/s of total bandwidth between R1 and R4.

In order to make the costs equal for all paths, set the ospf costs under each interface so that they are all equal between R1 and R4. In my lab, I set them all to 1, then incremented them to 3 between R1 and R4, and 2 between R1 and R5. This means each path totals up to a cost of 4.

Just create to sub interfaces between R1->R4 (Total 3)
create another sub interface between R1->R2 and R3->R4 (Total 2)
and leave only one interface between R1->R5 and R4->R5
now equalize the metric between R1 and R4 so they will see 6 ECMP links

Since the other two paths result in EQUAL costs, I would say that setting the bandwidth of the link between R1 and R4 to 42.86M will result in a cost of 23.33, equal to other costs and will result in 3 equal cost routes installed in the routing table for R4.

Ivan, I knew you would be the first one to come up with a “real-world” solution, similar to MPLS TE tunnel. I tried to rule out end-to-end tunnels as part of the solution, but to my shame I missed the “route-via” option. However, if that would be a real scenario, I would probably resort to the end-to-end tunnels due to their flexibility.

You are trying to engineer paths from R1 to R4, while the task is looking for optimizing the reverse paths, from R1 to R1. Besides, relying on pure ECMP here will result in slower link overutilization. Maybe I’m missing something else from your solution though.

Using tunnels between R1 and R4 could work, but static routes are not needed. You may simply use the “route-via” option. The more elegant solution would avoid using static routes though. Besides, using just 3 tunnels will result in equal-cost load-balancing and link oversubscription.

Even though you did not post a complete comment, I was amazed by the fact that you provide the exact solution I had on mind! Using logical subinterface has the benefit of solving the “MTU problem” inherent to every tunnel technology. This may not be applicable in a real network, but you can always replace local sub-interface with tunnels connecting two adjacent routers (which brings back the MTU issue though).

I still dont get it, if you want to utilize all 3 links from R1, then all 3 of them should be in the routing table. Why you are saying that creating 3 gre tunnels will over utilize the 50 mbps link ? can you define the traffic flow of the solution ?

If you create 3 tunnels end-to-end between R4 and R1, you will have 3 equal-cost paths in R4. If you try transporting 300Mbps from behind R4, this flow will be split in 3x100Mbps flows equally across the three tunnels. The R4-R3-R2-R1 path is a bottleneck with 50Mbps, which will drop the exceeding 50Mbps of traffic. Thus, your effective utilization will be 250Mbps, with 50Mps underutilization across the fastest link (150Mps).

Dear Petr, thanks for the feedback but i think i am still missing something.
When you split the links from R4 to its neighbors, what R1 will be doing in that case ? will it not be sending the traffic to all 3 links ? how R1 is getting 300mbps throughput in your solution ?

The key thing here is that we are only concerned with the traffic flow FROM R4 TO R1, but not the backwards. There are three paths from R4 to R1, with different minimum bandwidth values: 50M, 100M and 150M. To optimally utilize all three paths we need to load-balance FROM R4 to R1 in proportions 1:2:3. However, OSPF does not support unequal host load-balancing so we “fake” it by using additional equal-cost paths so that paths with more bandwidth receive more traffic than lower-bandwidth paths.

Interesting question This approach does not scale at all plus in reality you dont have “bipolar” scenarios (one source one sink). In real networks there are multiple sources and sinks, described by the network traffic matrix. There are tolls to optimally adjust the IGP metric values to optimize *global* summary link utilization and those are pretty effective, sometimes competing with MPLS TE effectiveness.

Leave a Reply

Currently you have JavaScript disabled. In order to post comments, please make sure JavaScript and Cookies are enabled, and reload the page.Click here for instructions on how to enable JavaScript in your browser.