Archives

Author: rhuff

I love Cisco Jabber; not just because its been around for awhile or because it’s my preferred instant message client of choice, but because it is a really good, stable and mature product that does it’s job very well. Please take a read of the following feature request I created for the Jabber powers that be. If you agree, please circulate and lets see if we can get some traction on this! Can we please stop the automatic authentication attempts to Webexconnect? Please? Asking for a friend.

Hey there! This will not be a technical information post; though I think, or am at least of the opinion, it is still worth your time to read!

I’m cautious about what I publish on this platform; everyone is busy nowadays and I value the time it takes to really read and digest content, especially content of decent length or substance. I make every effort possible to ensure what I do publish is not only accurate, but useful enough to pay for the reader’s time spent to really read and consume the material and the ideas being communicated.

I am originally from Ohio and I have worked for Cisco partners in the Metro Atlanta area for the last few years. Although I have worked for three different entities since moving to the area, I’ve only ever worked with one team and have survived two acquisitions; impressive, but not unheard of in this industry sector.

As with all things in life though though, the sun will eventually set and rise upon something new. After much debate and personal soul searching this past month, I decided to challenge myself with a new opportunity at ByteWorks! Of particular interest in that decision and what ultimately brings us here together in this post, is introspection and it’s net value.

I think the net value of introspection is not always or necessarily the outcome of the evaluation of one’s own state of being, but rather the ability to be introspective. It requires an elevated and mature emotional state where you are willing to be critical of yourself and, should it be required, admit and accept that you are not right (or as in my case, its called “less right” … lol).

Introspection is a fascinating journey of self discovery that you’ll never reach the end of; no matter how much you know about yourself, you are always ready to teach you something new about yourself.

I spent a decent amount of time being introspective with my decision to challenge myself with a new employer; there are lots of things to consider when making this type of change. I often find that introspection allows you to answer the questions you already knew the answer to but wouldn’t have found any other way. For me, its a way to slow my thought process down and analyze the individual details and run each detail through my mental “decision simulator” and play out how each detail could impact decisions or change the environment of the circumstance. It affords me the chance to evaluate my own decision making process and how that may impact my decisions and circumstances.

I find by following the outcome of this process, despite whether it looks ‘correct’, ‘safe’ or ‘proper’ on the surface ultimately proves to be the best course of action for me in the end. This process has proven itself successful to me time and time again.

Perhaps, Introspection is more spiritual than emotional. Perhaps it’s an opportunity for your id to knock your ego down a few pegs. Personally, I think its very healthy to have these types of internal dialogs and evaluations. I’d find it difficult to think of a reason anyone couldn’t benefit from being introspective.

Give it a try!

Initially, introspection is uncomfortable and perhaps a little awkward. It might seem as if you’re trying to find a “real person” to talk to, much like you would have a conversation with someone in the real world. Just keep trying, it gets easier! Now, I’m not suggesting we all take to the streets talking aloud to ourselves; far from it! Introspective behavior is often an internal conversation or thought evaluation (but it can be out loud); a time where you can ask yourself a question and really evaluate how you would respond and often more importantly, why you would respond that way.

That is what I’ll leave you with in this post, an encouragement to be more introspective!

The Scenario

Customer deployed and managed SD-WAN solution in front of the Edge cluster to the Internet (with two separate transport carriers). I think it was Palos, but we’ll call it a whitebox’ed solution for our purposes

Using MRA and B2B Cisco Expressway configs

UAT for MRA and B2B is accepted and works great

The Problem

A little bit into post go-live the customer applies the zone/search rule config in Cisco Expressway for CMR (basically, SIP URI dialing into WebEx meetings from the video endpoint) and notices that randomly, during a presentation session in the CMR, the BFCP server (AKA, the WebEx meeting) would close the BFCP presentation to the endpoint coming from the customer’s Expressway; all other BFCP clients are still receiving the BFCP presentation. That’s right, it appears that WebEx kicked the BFCP participant coming from the customer’s Edge, but not because the BFCP server closed the session (all other participants remain)! Although it was happening randomly in length of time into the presentation, it would always happen at some point to the endpoint, generally around the 2 minute mark.

The diagnosis

Although random, a consistent’ish length would seem to suggest a timer / re-invite of some flavor, and that would be wrong, as ultimately uncovered. Sparing you all the gory tales of escalation and bus underskirt sliding; the issue was in fact, the SD-WAN solution itself.

The Explanation & The Fix

What happened is that every 120 seconds or so, the BFCP server (WebEx meeting) would send a UDP BFCP packet to all the BFCP presentation subscribers. The customer’s SD-WAN solution was identifying these packets according to the customer (gotta love layer 7 capable firewalls 😊) and queuing them onto a physically different link than which the rest of the BFCP stream was on, thus creating physical asymmetry, delay and latency. In a TCP stream, this would likely be tolerated to a degree as packet loss or delay and/or jitter and would simply re transmit ….. but we are dealing with UDP here, no good!

To resolve, the customer had to classify the traffic and force an active/failover transmission through their SD-WAN solution for that traffic, rather than a “load balance” transmission behavior.

Sleuthing & The Closing

In hind sight, seems simple and makes perfect sense right? However, when your only visibility into the network is the Expressway servers themselves, it can be very challenging to discover because at that point in the topology, everything looks like it is coming from and going to the VIP on the firewall pair. So how do you catch something like this when you can’t see everything? PCAPs. Literally counting f**king packet sequence numbers for 6 hours and identifying a consistent pattern of packets coming out of order and being “lost”.

Hello, I’m back!!

For those of you who still follow me on the Internet or continue to find me in Google -Thank You, I’m back! This time, with a whole new web presence, purpose and content (the site is still under development however)! Several months ago, I took my site down for a variety of reasons, mostly because the site had become “stale” to me, and really, I just wanted to “reboot” it. As life goes though, once it went down, I never found the time to get it back up! Not that I have any more available time now; I have decided to make the time because I have a whole new inspiration for this platform.

Any of you who have followed me for awhile know that I am a Cisco and Apple fanboy thorough and through. However, I have recognized for a bit now that the landscape of the industry is changing and there are new players out there worth mentioning, considering and yes, even using!

Rather than focusing on a vendor-specific narrative, my hope is that you’ll find a more vendor agnostic experience with the content on this platform. I’ll focus more on foundational and open source technology (like SIP, WebRTC, DevOps & Linux … etc) rather than solely how one vendor might implement those technologies in their products. Mastering core foundational technology like SIP or WebRTC will create a much more portable skill set for you, compared to just learning how one vendor uses those technologies within their products.