On February 16, 2009, a widely-distributed Border Gateway Protocol (BGP) route update contained an Autonomous System (AS) path that included approximately 250 entries. As the update propagated through additional Autonomous Systems, each AS prepended its own AS as part of the normal BGP process. When the AS path length exceeded 255 entries, some routers were unable to form correct BGP route updates to their peers, which resulted in corrupted BGP route updates and the subsequent resetting of certain BGP peering sessions. Some resets occurred as a result of a bug in Cisco IOS Software. This event is described in IntelliShield alert 17657.

Devices normally accept BGP update messages and prepend local AS values to a BGP AS path list that is distributed to BGP neighbors. Due to a bug in Cisco IOS Software (CSCsx73770), an error may occur if a Cisco IOS device attempts to send a BGP update message that contains a route with an AS path length greater than 255 to a BGP neighbor. When received by a BGP peer, the invalid BGP update triggers a NOTIFICATION message of “Malformed AS PATH” that is returned to the sender of the incorrectly formatted BGP update message. This scenario causes the BGP session to reset. Information about this Cisco IOS Software bug is available at the following link: CSCsx73770.

Cisco Security Intelligence Operations has identified a method through which administrators can modify device configurations to mitigate the effects of the AS path processing issue. Administrators can limit the amount of AS path segments that are associated with any route by using the bgp maxas-limit feature, which requires the software fix associated with CSCeh13489. All neighbors will be treated uniformly according to the specified policy because this router configuration command is not tied to any specific BGP neighbor. Prior to the functionality change for the bug associated with CSCee30718, the value that can be entered for this argument is a number from 1 to 255.

Because Cisco IOS Software limits the prepending of Autonomous Systems via the route-map function to a total of 10, the most that a Cisco IOS device can add to the AS path-length is 21 AS identifiers, or ten using an ingress route map, ten using an egress route map, and one for normal BGP AS processing. An administrator can set the bgp maxas-limit to 234, which results in a maximum AS path length of 255 AS identifiers that can be sent to the downstream BGP peers without corruption. The bgp maxas-limit command was introduced in Cisco IOS Software Release 12.2 and in 12.0(17)S in the 12.0S train. Using a conservative value of 200 can simplify the configuration and also prevent this condition. Administrators are advised to configure and fully test any limit in a lab environment prior to deployment on a production router.

IntelliShield will continue tracking this issue and the related event. Administrators are advised to continue to monitor edge devices for service failures that may indicate additional outages and employ mitigations where appropriate.

Version 1, February 20, 2009, 2:45 PM: An issue within Cisco IOS Software could cause some devices to fail to process long Autonomous System paths, which may result in performance problems or a denial of service condition.

The security vulnerability applies to the following combinations of products.

Today’s global routing instability trigger (flavor of the month), that of extremely long AS paths, seems to be a bit of a repeat. Some of you may remember this occurring over 5 years ago, and presumably, that same bug in Cisco IOS (CSCdr54230 – inadequate buffer sizing and a knob to limit maximum AS path lengths) is the culprit here as well. This seems to have triggered a great deal of wide-spread routing system instability (and underlying connectivity issues) for a few hours this morning, as illustrated both in the chart below (GMT -7), and by volume of related discussion on a number of operational mailing lists.

In short, AS 47868 (SUPRO-AS) apparently took a notion to prepend it’s own AS number some ~251 times (I say 251 because he would normally put it there at least once when sending to external BGP peers (+1), if you’re counting and get 252) to it’s route advertisements for prefix 94.125.216.0/21, before announcing the route to it’s peers. Some implementations, e.g., Jurassic (> 3 years) versions of Cisco IOS, didn’t allocate enough buffer space for silly long AS paths, and so they blow chunks when they receive the update. However, other vendor implementations and patched versions happily propagate the update, apparently through numerous intermediate ASes, so seemingly random sets of BGP routers in the routing system were taking a dump, or dropping sessions with malformed AS path complaints (which may be a slightly different problem).

The alleged update looked something like this in your average Cisco command line interface (CLI) output, albeit with the leftmost AS numbers varying depending on topological perspective:

I’d highly recommend folk that haven’t patched in a few years do so ASAP (duh), else you are vulnerable to remote targeted attacks, period. Also, I suspect the SUPRO-AS folks could have easily accomplished their traffic engineering goals (assuming that was the intent???) with just a few ASNs prepended, and remind them that many folks auto-suppress routes with AS paths containing more than n ASes, e.g., the knob added to IOS when the IOS buffer sizing issue was fixed.

I’ve seen a few different failure modes reported for this incident, and suspect there may be another issue with storing such large AS paths, or behavior related to sending error notifications and session tear down in other IOS versions or different implementations, in particular, when AS path lengths approach 255 (one byte, indicating number of ASes that can be contained in a single AS_SEQUENCE segment, or dealing with the sum of ASes contained in multiple AS_SET and AS_SEQUENCE segments). I don’t have more details on this at the moment, unfortunately, but will continue digging.

Ohh, and as usual, I’ll note that absent explicit origin AS and path validation in a secure routing protocol, we’ll continue to see problems along these lines.

Some background information on this and a few related problems, for those interested…

In BGP autonomous system (AS) numbers are used to represent unique routing domains, which typically correlate to administrative domains. In order to avoid routing information loops BGP routers, when sending a routing update message to external (not in the local AS) BGP peers, prepend their local AS number to the AS path in the routing update message. While the AS path information primarily serves as a path vector to provide a mechanism for loop detection, it is also considered in BGP’s best path selection algorithm (shorter AS paths preferred over longer ones, hence the common prepending observed in the Internet routing system, namely for traffic engineering purposes), and various routing policies are often applied based on the length and/or contents of the AS path.

The AS path (AS_PATH) attribute is composed one or more TLV-encoded segments, of type AS_SEQUENCE (ordered set of AS numbers the update has traversed) or AS_SET (unordered set of AS numbers the update has traversed – used mostly in proxy aggregation and present on less than one hundred of the ~300k prefixes in the global routing system today). As noted, an AS path can have multiple segments of both types, but when sending an update to an external BGP peer the leftmost segment type needs to be an AS_SEQUENCE, and the local AS needs to be placed in the leftmost position in that AS_SEQUENCE. If the leftmost segment type is not an AS_SEQUENCE, then the local BGP speaker needs to create it and place the local AS in the leftmost position. If it is of type AS_SEQUENCE, then the local BGP speaker simply places the local AS number in the leftmost position. If there’s no AS path data (e.g., the route was originated locally) and the update is being sent to an internal BGP peer, then the local BGP speaker sends an empty (length == 0) AS_SEQUENCE. The length field associated with the AS path segment attribute identifies the number of AS paths contained in a given segment, not the size in octets of the segment.

So, the reason all that’s important is because if a given BGP speaker receives an AS path that has an invalid length, or is in some manner malformed, then the receiving BGP speaker should send a NOTIFICATION message to the peer indicating “Malformed AS_PATH” and tear the BGP session down. Ideally, when malformed AS paths (or other attributes) are generated the BGP sessions would have problems only at the originating AS and the immediately adjacent upstream, and not effect other networks – i.e., implicitly be squelched as close to the source of the problem as possible, and not result in wider instability in the routing system. However, because different routers on the Internet run different versions of BGP routing software, and some vendor implementations are more forgiving than others, what you end up getting is unpredictable propagation of malformed updates (e.g., such as those the Cisco knob noted above introduces), updates that result in BGP routing session tear down multiple AS hops upstream from the originator or source of the malformed update. I should also note that the originator may well not be the source of the malformed update, as any intermediate BGP speaker within a given AS rebuilds the update and could introduce problems.

Furthermore, it’s important to realize that when you tear down a BGP session with a peer, all advertised routes in each direction must be removed, not just the route(s) in the update(s) that triggered the session tear down. After a session is torn down, a clever implementation might opt to keep some amount of state as to exactly which update triggered the NOTIFICATION – if it can glean such information from the received NOTIFICATION (which is usually unlikely), so that when an attempt to re-establish the session occurs – which is usually near immediate, advertisement of that update is suppressed as to avoid triggering another session tear down. However, such specification tweaks by clever implementations are rare in practice, and for good reason. If it were to suppress the update that triggered the session tear down, you’d get non-deterministic reachability for the prefix contained in the update, and that’d be bad. So instead, what usually happens is that the session flaps and “clever” implementations exponentially backoff session re-establishment times until the problem is resolved, with a lot of instability triggered during any intermediate state. This works if it’s between adjacent networks near the origin of the problem, but is really pretty ugly and unacceptable when such malformed updates can be propagated far and wide before session resets are triggered.

A while back, we updated the BGP Confederations RFC 3065 with RFC 5065 to specify that BGP AS confederation segments (AS_CONFED_SET and AS_CONFED_SEQUENCE) must not be sent to peers outside the local BGP confederation, and added error handling procedures that specified if those segment types are received from an external BGP peer, a “Malformed AS_PATH” NOTIFICATION message must be sent, and the session torn down. This was on the heels of an event where an implementation was sending those segments to external BGP peers, and sessions were oscillating as a result. This was between adjacent networks only.

However, another recent BGP incident highlighted similar concerns, that of inclusion of BGP Confederation segments (AS_CONFED_SET and AS_CONFED_SEQUENCE) in AS4_PATH attributes. In short, AS4_PATH attributes are used to “tunnel” 4-octet AS numbers across 2-octet-only ASes. An implementation was including these AS_CONFED_* segments in the AS4_PATH data, they were being tunneled, and it was triggering session resets on remote networks when they were being un-encapsulated.

One might surmise that specially crafted AS_PATHs and update messages might well allow an attacker to launch remote targeted session disruption attacks with a technique such as this…

How to make the lowest price for internet access? For our information, Singapore’s market price for internet broadband for 100mbps is SGD125; it is the same as SGD1.25/Mbps, which 1SGD is equal to IDR700.00.
So how can Singapore get that kind of price? It might be some factors that are:
1. Tier-1 peering agreement affected to Singapore, because of;
2. Singapore is the HUB for them, because of;
3. Singapore gave some benefit to them, because of;
4. There is mutual agreement between worldwide operators and Singapore. That is including for goods and services trading. Singapore provides place and facility for their trading. That will make benefit between Singapore and worldwide companies.