Well hello blog world. Yes it’s been a while since I last posted an update. I meant to update sooner but because I haven’t logged on to WordPress in such a long time I forgot my password. Then the password came to me in a dream. So here I am.

So much has changed in my professional career since I last updated. In fact my studies were interrupted for a while during all these transitions. Let me take a moment and go back in time to catch you guys up:

August 2011

After working 5 years as an IT technician at my current job (nine years total in IT) I finally left the field and moved to a lower paying job. It was a dream come true.

Alright let me set that straight: I took a job as a graveyard NOC technician at large MSO (fancy name for a cable (TV) company). The salary they were offering was over 25% less than what I was currently making. Note that I was pretty much the senior technician at my old job, and I was moving to a huge company where all I was going to do was monitor the network and look for red lights –> green light is good, red is a no no. So the pay wasn’t very good. A 25% pay cut is like $250,000 less than the cool million I used to make (figure is fictitious.. only to illustrate the point.. I really made more. Just kidding).

But here’s the twist, I aced the interview so well that the interviewers were so blown away they had to have me. So they offered to match the previous salary I was making even though I would’ve been overpaid by 25% what they offer. I guess they really needed someone who can tell red light from green light and do it really well. I thought it was really nice of them. Then I found out the company made 1 billion dollars the past year.

Nevertheless I was stoked. I no longer had to deal with Windows 95, no more changing copier papers, no more Exchange 2000. I was now part of an elite task force that watched over the health of the network for the rest of Southern California. But only from 11pm – 8am, Tuesdays through Saturdays.

December 2011

I became a father.

For the 3rd time.

And you know what that does to you study schedule if you’re studying for CCNP… You die!

February 2012

I didn’t actually die. But I finally completed my CCNP in February 2012 – over 4 years after I set out to pursue Cisco networking. The great thing about working at a NOC, on graveyard, is that they actually pay you to study. OK they didn’t pay me to study. But there was ample enough free time that I was able to focus on completing my CCNP. Finally.

It was actually a good timing I started with the older CCNP track (You know, BSCI, BCMSN, ISCW, and ONT). But this was the final year they were offering that track and the very last date I could finish it was July 2012. They have since moved to the current 3-exam format (ROUTE, SWITCH, TSHOOT). At this point I had already completed BSCI and BCMSN and Cisco was willing to honor those exams and only pass TSHOOT to complete CCNP requirements. So I studied and studied for a good 2 months. Basically, reviewing everything on BSCI and BCMSN and doing lots of labs. The exam itself was a joke. By far the easiest I have taken.

November 2012

I finally found a crack in the intricate processes of the recruitment and hiring system of the company. I got myself hired in the IP Routing team as a Network Engineer – the ultimate destination of all human endeavor. Ok maybe not. I did it by sheer persistence and ass-kissing – that’s my crass-way of saying, I worked hard, bid my time, studied, and struck at the right moment. Oh and I also emailed the director and manager of the routing team every chance I got. I wanted to put myself out there and made it known that I existed and I was interested in joining their team. Whenever there was high severity outages where I knew their participation was required, I made sure I was involved in one way or another in triaging or the driving of the resolution. When the opportunity came and a position opened up, the manager of the routing team already knew who I was and all I needed to do was impress him in the interview. The interview process went like this:

Manager: So you’re interested in the Network Engineer position huh?

Me: Yes sir! Very much so. I have been intent on being a part of the team since I started with the company. While I was at the NOC, I have been familiarizing myself with the network and how everything interacts with each other, just trying to get in the mindset of an engineer. And while I was waiting for the opportunity to open, I was making good use of my free time studying to get CCNP certified and position myself as the best candidate for this opportunity. <blah blah, blah, insert LinkedIn blurb>

Manager: Good! When do you want to start?

And that was the extent of the interview.

Present

I have been enjoying every moment as a Network Engineer, finally putting to good use all the knowledge I have learned thus far. One thing about a huge, complex service-provider network is that everything is standardized. In that sense, there is not a lot of room for experimenting and “learning from mistake” experiences. But the good part is that I get to work with current and (more or less) up to date technology. So even though most configuration is already pre-scripted, I still learn a lot of the industry’s best practices and way-of-doing-things. All our major routers are at least at 15.0 code. I get to work with a lot of SP Cisco gear. I work primarily on 7600s and ASR9K but not limited to that. I get to touch some Juniper devices. Occasionally I get to peer into the CRS-3 cli. I have been learning IOS-XR for the past few months and expected to troubleshoot some Junos stuff. And, they now pay me two meeeeelion dollars!!!

Eventually, I will probably leave and move on to more advanced stuff. Or I may stay and move on to more advanced stuff. But regardless, my goal now is to get my CCIE. Well it has been my goal. But I think that my next major milestone is to pass the CCIE written and get neck-deep into CCIE labbing.

So far, in the past 2 months I have been averaging about 2.5 hours of CCIE Written prep. With my busy work and family schedule that is a pretty solid amount of time. But I’m looking to move my schedule around to get a more consistent schedule instead of the sporadic 4 hours one day and none the next two days and 3o minutes one day and 2 hours the next. We’ll see how that works out. Stay tuned… or not.

While going through some BGP examples in Routing TCP/IP Volume II, I came across an issue where RIB failures continue to pop up in the show commands. To illustrate see the following topology I recreated from the book (pg 175, fig 3-6 on Routing TCP/IP II book)

sh ip bgp produces the following output with the “r>” indicating that there is a RIB failure in router Vail.

sh ip bgp rib-failure provides a clue as to what caused the RIB failure

Further troubleshooting shows why

Telluride is advertising prefixes 192.168.1.200/30, 192.168.50.0, and 192.168.75.0 via IBGP towards Vail. Vail, however, is also learning these same routes via OSPF through Aspen. As shown in the sh ip route output for one of the routes above, an OSPF advertisement has an AD of 110, whereas an IBGP advertisement has an AD of 200. Even though the routes are in Vail’s BGP table, the OSPF route is installed in the routing table because of lower administrative distance. BGP then throws a fit because he has the routes in his table and the router ignored him.

This is normal operation for BGP. In fact, even though Vail does not use the route via BGP it is not suppressed or thrown out. If you check the ip routing table in Taos, you’ll see that BGP is sending these same routes from Vail. So at least BGP is happy there

I’m starting to contemplate taking the CCIP track. I’ve been having a lot of fun with MPLS and BGP lately that I have this compelling urge to go all the way with the IP track. The only thing that really holds me back is QoS. LOL! For some weird reason, QoS can’t seem to sit well with me. I don’t think it’s way more complicated than either MPLS or BGP, in terms of understanding the concepts. It’s just that it doesn’t seem to arouse interest in me. But then again, frame relay used to suck for me, but now we’re coo.

So CCIP or not? If I do, it would push back my goal of getting the CCIE R&S. On the upside, I’ll probably know BGP well enough to get it out of the way in my CCIE studies. MPLS doesn’t go much in depth in CCIE R&S but taking the CCIP MPLS exam should give me that much more command of the technology.

So I’m speaking with one of the engineers that will probably be the point person that will implement MPLS for us. I noticed in my studies that nearly all of the configurations required to implement MPLS happen in the Provider space. All these PEs, Ps, LSRs all reside in the service provider from which I have no access to. This particular engineer pretty much confirmed that, yes, the implementation will be “seamless and almost transparent” to the IT staff. To those who are unfamiliar with sales-speak, seamless and transparent basically means it will be installed without adverse affect to the current system (seamless) and almost no change in our network equipments’ configuration (transparent). Well booo.. For once I actually want to get into some trouble mis-configuring things :D. I was joking around with the engineer that maybe they can hire me just for that project and let me do the configurations. I’ll even let my company pay the work I’m doing for them.

Anyway, it’s all good. I’m learning more and more about MPLS and I like it. Currently I’m building a small dynamips lab based on the sample scenarios from the book I’m reading. FYI, I’m reading MPLS Fundamentals by Luc De Ghein should thou wondereth. So far I’m enjoying the book. First time reading the MPLS VPN section got my brain all tied and twisted. But the second time around, concepts are beginning to come together more coherently. I’m also enjoying the fact that I get to lab BGP in the context of applying it to a different technology. Other than labbing BGP, as a technology by itself, and the occasional redistribution between different IGPs, I’m now able to see how it is used in MPLS.

Ok, Ok, I know I said in a previous post that I will be going away from my usual bullet-style blog posts. But I just can’t see any better way to take notes than to do it the way I’ve been doing it for the last couple of years. So here are some notes I’ve compiled over the last few days on MPLS.

MPLS

Stands for Multi-Protocol Label Switching

Uses a new way for routers to forward packets. Instead of the traditional way of forwarding packets based on destination IP address, MPLS routers forward packets using MPLS labels.

Label Switching indicates that the packets are no longer IPv4 packets, IPv6 packets, or even Layer 2 frames when switched. They are instead labeled.

MPLS allows forwarding decisions based on other factors, such as:

Traffic Engineering

QoS requirements

Privacy requirements for multiple customers connected to the same MPLS network (MPLS VPN)

MPLS Header and Label

The MPLS header is a 4-byte (32-bit) field, located immediately before the IP header

The header is also often times referred to as a shim header.

Most will simply refer to the MPLS header as the MPLS label, but in fact the label is actually a 20-bit field in the MPLS header.

The label identifies the portion of a label switched path (LSP). LSP will be discussed later.

The label value can be between 0 and 1,048,575 (or 2^20 – 1)

The MPLS EXP bits is a 3-bit field used for QoS marking. Historically it was called Experimental because the actual used for it was then undetermined.

The bottom-of-stack (S) field (1-bit) is a flag, which when set to 1, means that this is the label immediately preceding the IP header. (see label stacking)

Time-to-Live (TTL) is an 8-bit field used for the same purpose as the IP header’s TTL field.

Label Stacking

Label stacking is the encapsulation of an MPLS packet inside another MPLS packet. In other words, an MPLS header is added “on top of” an existing MPLS header; hence “stacking”.

This is done so that one MPLS LSP can tunnel inside another LSP.

The first label in the stack is called the top label.

The last label is called the bottom label.

The BoS bits for all labels is 0 except the very bottom, which has a value on 1. A value of 1 means that this label is the bottom label and is immediately followed by the IP header.

Encapsulation of Labeled Packet

The label stack sits in front of the Layer 3 packet, before the header of the transported protocol, but after the Layer 2 header.

The MPLS label stack is often called the shim header because of the placement of the packet in the frame.

The Layer 2 encapsulation of the link can almost be any encapsulation that Cisco supports. For example:

PPP

High-Level Data Link Control (HDLC)

Ethernet

The transported protocol could be anything:

IPv4

IPv6

Frame Relay

PPP

HDLC

ATM

Ethernet

Label Switch Router

A Label Switch Router (LSR) is a router that supports MPLS. It is capable of understanding MPLS labels and or receiving and transmitting a labeled packet on data link.

Three kinds of LSRs:

Ingress (edge) LSR – receives unlabeled packet from customers equipment. It inserts a label in front of the packet and send it on a data link.

Intermediate LSR – receives labeled packets, perform an operation on them, switches the packet, and send the packet on the correct data link.

LSRs can do three types of operation:

Pop – removes labels.

Push – adds the label onto the received packet

Swap – the top label of the label stack is replaced with a new label and the packet is switched on the outgoing data link.

Imposing LSR – an LSR that pushes labels onto a packet that was not labeled yet. Done by the ingress LSR.

DisposingLSR – removes all labels form the packet. Done by the egress LSR.

Label Switched Path

It is the path through which a packet takes in the MPLS network (or part of it).

It is a sequence of LSRs that switch a labeled packet through an MPLS network.

The LSP is unidirectional. That means a path to go back requires a complete separate LSP.

Forwarding Equivalance Class

An FEC is a set of packets that a sintgle router forwards:

To the same next hop

Out the same interface

With the same forwarding treatment

All packets belonging to the same FEC have the same label.

Comparison

In a regular IP forwarding process, each time a packet reaches a router, a lookup is performed on the IP destination, the packet’s FEC (that is, the next-hop, outgoing interface, and forwarding treatment) is determined. Using the gathered information the packet is forwarded to the next router. When the packet arrives to that router, this whole process is repeated.

Essentially, the packet’s FEC is determined hop-by-hop at every router that it reaches on its way to the destination.

In an MPLS LSP, when a packet arrives at the first LSR, the FEC is determined once and is not repeated until that packet reaches the end of the LSP – the egress LSR. The intermediate LSRs does not determine a new FEC, as compared to the regular IP forwarding process.

Label Distribution Protocol

For the label distribution in an MPLS LSP to work, each intermediate LSR needs to figure out which outgoing label the incoming label should be swapped.

LDP is the method of choice among the majority of MPLS vendors to distribute labels for IGP prefixes.

LDP is the mechanism that tells the routers which labels to use when forwarding a packet.

For every IGP IP prefix in an LSR’s IP routing table, a local binding is created. A label is binded to an IPv4 prefix. The LSR then distributes this binding to all its LDP neighbors. Once receved by other neighbors, these local bindings become remote bindings. These remote and local bindings are stored in a table called Label Information Base (LIB). The LIB contains all lables known to the LSR

LDP uses a Hello feature to discover LDP neighbors and to determine to what IP address the enusing TCP connection should be made.

Hellos use multicast address 224.0.0.2, using UDP port number 646 for LDP

The Hellos list each LSR’s LDP ID (LID), which consists of a 32-bit dotted-decimal number and a 2-byte label space number (this number has a value of 0 for frame-based MPLS).

After discovering neighbors via an LDP Hello message, LDP neighbors form a TCP connection to each neighbor, again using port 646 (for TDP, port 711 is used).

The unicast address used to form the TCP connections between neighbors must be reachable according to the IP routing table. These addresses can be either the neighbor’s advertised transport address or the address in the LID.

Cisco IOS chooses the IP address in the LPD ID in the following priority (similar to OSPF):

Manually configured address

Highest IP address of a loopback

Highest ID of a non-loopback interface

Label Forwarding Instance Base

While the LIB contains all labels know to the LSR, the LFIB (and the FIB) contains labels only for the currenlty used best LSP segment.

LFIB is the table used to forward labeled packets.

It is populated with the incoming label and outgoing labels for the LSPs

The incoming label is the label from the local binding ont the particular LSR.

The outgoing label is the label from the remote binding chosen by the LSR from all possible remote bindings.

In the following diagrams, observe how LDP advertises bindings betweent the LSRs for the IPv4 prefix 10.0.0.0/8 and how packet switching works when a packet destined to the 10.0.0.0/8 prefix enters the ingress LSR.

In the part of the illustration, each LSR allocates one label per IPv4 prefix. This prefix and its assocated label is the local binding

An LSR chooses the remote binding received from the downstread LSR, which is the next hop in the routing table for the 10.0.0.0 prefix.

The label from the local bindings server as the incoming label and the label from the one remote binding chosen via the routing table serves as the outgoing label.

The bottom of the figure shows a packet entering the Ingress LSR destined to 10.0.0.0/8.

There, a label 129 is imposed on the packet and switched toward the next LSR. The second LSR swaps the incomgin label 129 with 17 and forwards to the next LSR. The incoming label 17 is again swapped with the outgoing label 33.

If I’m inspired, the next post will focus more on the Data Plane and Control Plane of the MPLS forwarding process. And if I’m really brave, I’ll re-read and blog-note the section on MPLS VPN.

This entry is not an authoritative guide. These are merely notes and rehash of the primary text materials and resources that I use. For a thorough guide of the MPLS concepts, consider purchasing MPLS Fundamentals byLuc De Ghein and the MPLS section of CCIE Routing and Switching Certification Guide (4th Edition) by Wendell Odom; as well as following the links on the reference section of this entry.

Recently my interest in Carrier-type technology has been re-invigorated since we decided to order and install MPLS in all our offices. I’ve always been intrigued by MPLS. Several years ago I did a little bit of studying about what MPLS is about and what advantages it offers over traditional IP or Ethernet transport. Although I learned about the gist of MPLS, I didn’t really go in depth to really understand the technology. Mainly because, first, our company did not deploy or make use of the technology, and secondly I was also heavily invested in my CCNP studies.

But now that I and management have been talking to different vendors and carriers about installing MPLS, I have been reading a lot more about the technology in a little bit more depth. It also happens that I am beginning my preparations to take the CCIE R&S written lab as well – and a little portion of MPLS is in the blue print for the exam. So I think this could get a little fun for me.

I have been on a hiatus from blogging since passing my BCMSN (I passed on July 30, 2010), and I have been very liberal in my study schedule. I haven’t had a set or planned pattern to follow and as a result my progress has been very ineffective. I would watch a technology video here and there, take a topic and read/study it here and there, but nothing really learned or stuck in my long term memory.

So as I always do before the start of the New Year, I’m re-affriming my resolution to pass more certifications. Last year I only accomplished 1/3 of my professional-oriented New Year’s resolution; which I consider a failure. So for the coming year, I’ll be doing the same. I haven’t laid out exactly what they would be but it’ll definitely include finally getting the CCNP or at the very least passing the R&S written.

In the last several weeks I have started reviewing my BCMSN stuff. In fact, I’m re-doing my BCMSN study regimen that I set out for myself when I started studying for it. I have just finished going through and labbing fundamental STP/RSTP concepts. Hopefully I can get through the materials faster this time. I have forgotten many of the BSCI concepts so I definitely need to spend more time on that. But as I’m finding out, even though I have more recently passed BCMSN, relative to BSCI, I’m still finding things on switchnig topics that seemed like new concepts for me, even though I know I’ve studied them before. I think it’s just a matter of repetition until these topics become imprinted in my memories.

More than likely, I will probably abandon my old outline-format blog style. At least for the time being. A lot of the things that I go through in my review are concepts that I’ve blogged about before. In fact, I’ve been using my blog as a review material when going back through old subjects. And I’m glad I spent a lot of times blogging my notes. But moving forward, a lot of my blog entries will be updates of what I’m doing. I may blog some lab practices from time to time to help me hammer in some concepts.

Anyway, Merry Christmas and Happy New Year to all my friends out there.

I took my BCMSN exam on Saturday July 24, 2010. As you can tell from the title of the post, I failed. I’m a bit disheartened by the outcome. Although I felt nervous, like I always do, coming into the exam, I was confident I could pass it. I was not over-confident. I just felt that I had enough knowledge to get through.

Notice how the pattern in the scores go from highest to lowest in exactly the same order that Cisco lays out the exam topic. In a way it’s a telling pattern as to how my learning path progressed. In all the study resources I read, the texts were arranged pretty much in the same order as the BCMSN blueprint is layed out. Consequently, I spent a whole lot more time on the topics higher up on the list than I did on topics further down the list. And the scores reflect that. However that is not to say I didn’t feel as prepared on the topics I scored lowest on as I did on the topics I scored the highest. I believe it’s also in the way the exam itself was layed out.

The number one reason I failed the test is time management. No matter how much I read about how you need to manage your time, this always seems to be what gets me. Just to give you an idea of my horrendous management of time:

Out of nearly 60 exam questions, I still had about 15 left by the time the exam expired.

There were 2 sim questions on which I spent nearly 20 minutes working on. It didn’t help that these sim questions came in within the first 15 exam questions.

In each of the sim questions, I spent almost 5 minutes just lingering around checking and double checking that my configurations were correct.

The only positive thing about spending that much time in the configuration sections is that I”m pretty sure I got them both correct or pretty close to being correct. The two sim questions had a lot to do with setting up VLANs, checking spanning tree configuration, and manipulating spanning tree behaviour. And as you can see on the breakdown of the scores, I scored a 100% and 90% on each topic, respectively.

The fact that I spent almost half of my allotted time on the sims and the fact that I still had about 15 questions remaining tells me that had I managed my time better and finished the exam, I might have had enough points to pass. I don’t believe that my scores on wireless and voice is telling of how much I know or don’t know about the topics. Actually, without having finished the exam, I have no way of knowing if I really know enough about those topics or if I should focus more on those topics before my next attempt. My gut tells me to focus more on getting faster on configuration.

Here are some of my thoughts on why I took so long on the sims:

I need to get better at understanding what the question is asking and get down to the requirements of the configuration. I’m always caught off guard by questions that include background scenarios that don’t necessarily pertain to what the problem is asking me to solve.

I tend to linger on one part of the solution over and over trying to make sure that I configured it correctly even after I’ve correctly verified through show commands that the outcome being asked for has been fulfilled. For example, I had a problem where I needed to make a configuration change so that one trunk interface is the preferred path over another. When I was finally able to accomplish the task, I verified it over and over and over again that it was correct.. and there were 3 more tasks waiting to be completed.

I really need to memorize commands in addition to understanding it. A huge part of the problem with the configuration was that I knew what I needed to do but I forget what the exact commands are. Was it spanning-tree vlan root primary or spanning-tree root primary vlan? Was it an interface configuration or a global configuration?

This is my first Cisco exam fail. This is probably not going to be the last – although, ideally, it should be. I always read on many people’s blogs and in forums those comments who have failed an exam, and they always say how passable the exam is and how fair the questions were. And I always thought to myself, how can people fail and exam and afterwards think that it is totally fair and passabel. Well.. ironically, I find myself in the same situation with the same sentiments about the exam. I thought it was totally fair and totally passable.

I would write more about my thoughts on my experience but writing about passing an exam is more fun than writing about failure. So I’m getting back on the horse and ride again. Next test is scheduled July 30th.