Month: March 2016

I sat the CWDP exam today. I didn’t really expect to pass, but I felt like I should give it a shot to keep the momentum going, and I was pleasantly surprised to walk away successful!

I was due to renew my CWNA in June, so this is a better refresh than doing the CWNA again. I had to make a conscious effort to step away from Wi-Fi for a while to complete my CCIE (R/S), but I am glad to be back on wireless.

If you’re working on completing the CWDP certification, I highly recommend the CWDP study guide from the PW0-250 version of the exam, from 2011 (Amazon: here) This is the second newest version and includes a much higher level of detail than the current book. Frankly, I didn’t feel like the current study guide covered nearly enough info to feel like I would have earned the cert.

So… onto CWAP! I think CWAP will help fill in a lot of the gaps I feel like I have in my understanding; and I think it will align well with the way I think and troubleshoot. I’m looking forward to spending more time with some PCAPs!

When I started working in mining, all of the mines I worked with were using Cisco APs mounted on the borer to facilitate a network connection to the less mobile equipment. The root of the link is usually the end of an extensible conveyor belt, and the PLCs controlling the belt communicate with the PLCs on the borer. There are things like safety interlocks which shut down the belt if the communication is interrupted. The extensible belt root AP is also uplinked to the rest of the mine network.

The Cisco bridges worked reasonably well at reasonable distances. On the bright side, EIRP was not a concern since we were far enough underground to prevent any sort of radiation into the open atmosphere, so very high gain antennas were standard operating procedure. I tried to find a pic, but just think parabolic dish.
But the borer moves away from the extensible as it does its work. We’re talking thousands of feet, and the cuts that the borer makes don’t stay level over the entire cut – so LoS is not maintained. On one hand, hard rock does an alright job bouncing signal along. On the other, throughput and data rate still faded quickly.

Can’t pull fiber and power in behind a borer – it will need to come back out the way it went in, and moving heavy machinery makes that idea asking for trouble. So ….

Yes, that IS two 1242 APs in a NEMA enclosure with back to back patch antennas, forming a repeater; AND I’m happy to see you!

It worked. No channel reuse (sorry Keith) because it was more important for unskilled miners to be able to move and install them with zero config. The entire set of root, repeaters and end bridge are hard coded to the same frequency.

Oh and if you could see to the ground in that picture, you’d find a car battery and some alligator clips. Fancy.

And yes that’s repeaters plural. The borer just got that far away. Sometimes it goes around a corner and then you need a repeater too.

So it’s not surprising to know that throughput was… not great. When VoIP was first brought out to the borers, we had a problem with the PLC interlock between the borer and the extensible belt dropping anytime someone picked up the phone. Neither the PLC interlock nor the VoIP call required much bandwidth, but there was enough latency or loss to starve the interlock, and VoIP tends to get prioritized by default. We had to do a little QoS reservation for the PLCs to keep them running at the expense of the calls.

So a year or two ago the instrumentation guys on site started playing with these:

This is a Ubiquiti PowerBridge M3. 3.5GHz licensed frequency? No problem – we’re thousands of meters underground! That’s a 20dBi integrated panel antenna, and a 2 SS radio.They use a root AP with a 26dBi antenna, and 12dBi repeaters, if necessary (sorry, I neglected to record the antenna specs on the Cisco gear when we installed them years back).

The results were rather stunning. At distances without repeaters, they can maintain near 100Mbps. With repeaters, they stay stable at distances of six to eight thousand feet without LoS due to elevation change. In the particular set I checked recently, the bridge was operating at the 32.5Mbps data rate (not throughput) over a 5Mhz wide channel, with 25dBm output power on the borer AP (100Mbps data rate was at 20MHz channel widths). The root has a 26dBi antenna, and the repeaters 12dBi. I didn’t have a chance to grab the models and power settings on those though.

Just for fun, here’s an idea of the perspective of the borer AP:

So these are significant (though unquantified) improvements from the 1242 and 1310 Cisco bridges. Had I thought to keep specs from then, perhaps I would have been able to provide more than just anecdotal commentary. But there are a few points that make it clear we’re not making an apples to apples comparison:

3Ghz is not Wi-Fi At a rough level, it also goes farther with less loss.

5Mhz wide channels are also not Wi-Fi. While there are some benefits to narrower channels being a little more robust against interference, interference generally isn’t a concern in this environment

Ubiquiti isn’t exactly known for playing by the rules in general. There are some proprietary enhancements at play – the data sheet for the M3 lists it as a MIMO TDMA protocol device

I’m no Ubiquiti fan. I know their gear is a lower quality hardware, and that they’ve caused some trouble in the unlicensed space. But that’s not the point.

This is a unique environment. Disposable hardware is favoured – service contracts are less useful here, so spares are kept on the shelf for quick deployment, which means low cost is a bonus. Performance is priority, and we’re not constrained by the rules of the surface world.

The engineers and technicians on site like a lot of things about the Ubiquiti bridges, including the simple (yet effective) GUI, and the physical signal strength indicators on the bridge, which simplify aiming.

In the end, I guess it comes back to designing for requirements. Each solution may shine or fail horribly in different scenarios.

I’m not the most experienced Wi-Fi engineer, but I’m learning, and I found this an interesting educational experience. I am curious to hear what the more experienced member of the Wi-Fi community think about Ubiquiti and their place in these less traditional deployments.

I know I’ll continue to see it everywhere. It just happens by accident. They didn’t realize what they were doing.

But they expect me to clean up the mess. Without downtime. Where’s my hammer?

Yet another unintentional asymmetric routing scenario! This one is bugging me a bit. I am in the process of fixing this client’s LAN topology and will remove the asymmetric routing in the process, however, I need to transition services onto a single firewall with as little downtime as possible, without being on site. Not uncommon so far.

Here’s the gist:

Asymmetric routing is ok… on the WAN, on the internet, and in some cases, if you INTEND to do it. Most of the time, it’s just EVIL.

Today, the remote access client tunnel terminates on the branch default gateway (10.1.1.1). I’m reconfiguring the newer firewall (10.1.1.2) to take over that role. The newer firewall was installed to terminate a VPN across the WAN (not internet) circuit to the remote branch. It is now destined to be the sole/main/one ring to rule them all firewall for the single subnet LAN.

Both firewalls are Cisco ASAs. I know that, in order to make this work until I can change the default gateway for the LAN to 10.1.1.2, I will need TCP state bypass on the default gateway (10.1.1.1), right? Right.

Lo’ and behold – TCP state bypass is already configured, for a LOT of traffic – because the remote branch (10.2.2.0/24) is hosting some critical services, and it is also accessible over a different gateway, which is on the same subnet as the default gateway.

Me: “Were you guys aware that you were using TCP state bypass for all traffic to the branch office?”

Client: “Hm, that sounds familiar… where have I seen that before…?”

Right. So the client googled something and figured out how to make it work, but really doesn’t understand what they’re doing (turning off core firewall functionality). But it worked!

I’ve been here before, I’ll be here again.

SO I configure the remote access VPN on the newer firewall (10.1.1.2), without state bypass, and can’t RDP to the DNS server. I expected that. In the diagram, steps 1-3 happen as expected, but step 4 doesn’t happen. Enable the state bypass, problem solved, TCP traffic works. Steps 1-5 work as expected.

Here’s the part that is bugging me though. I’ve made it work with a kludge, knowing full well that I’m going to fix the core problem soon. But I haven’t had time to really figure out what the problem is…

VPN clients couldn’t do name resolution. UDP lookups to port 53 were not getting a response. Since I am working remotely from the site, embedded PCAP is the best tool I have.

PCAP on new ASA (10.1.1.2) inside interface indicates that the DNS query from the VPN client is transmitted on the inside interface to 10.1.1.80. No response is received from the DNS server.

PCAP on default gateway ASA (10.1.1.1) inside interface indicates that the DNS server is, in fact, sending a response to the client at 10.10.10.X, and it arrives INBOUND on the inside interface. An outbound flow is not captured, matching the PCAP on the new ASA.

So looking back at the diagram, steps 1-3 are fine. Step 4 never proceeds.

So… UDP shouldn’t have the same asymmetric routing issue as TCP right (it is TCP state bypass after all)? There is no “state” to check.

Hairpinning is indeed enabled on the default gateway ASA. Informational level logging didn’t indicate anything interesting, but it would still seem that this firewall is dropping the traffic.

I know, I would NEVER leave this in production. But I needed to confirm quickly where the issue lies.

At the moment, the reason for the drop is not coming to me. Let me know in the comments, or on twitter (@bmroute) if you have an idea.

FIX: Credit to my mentor, CCIE Security @rwatt13. Default DNS inspection treats the UDP query/response similarly to a TCP state in that the ASA doesn’t like to see a DNS response without a query first. Disabling DNS guard on the gateway ASA fixed the problem. See some explanation of DNS guard here.

YES, I REMOVED THE STATIC ROUTE FROM THE WINDOWS SERVER. I’m not an animal.

When I started building communications networks 11 years ago, I expected to be sitting at a desk. Or maybe in a server closet. If I was really lucky, I figured I’d be in a data centre, or hanging APs in a medical clinic – and I’ve done all of those! BUT I did NOT expect to be underground, attached via console cable to an AP, mounted onto one of these:

Badass drilling tank

That, is a borer, or miner, or boring machine (not “boring” like “yawn”). It bores into rock, underground, thousands of meters below the surface of the earth. And, yes, it was moving, albeit slowly, whilst I was consoled to that AP, trying to figure out why it couldn’t connect to the root AP back towards the rest of the mine.

My home province of Saskatchewan produces approximately 50% of the world supply of Potash. Potash is a big ingredient in commercial fertilizers – potash helps grow strong food.

This means that potash mining is a big deal here; and since most of it comes from here, it also means that, outside of western Canada, many people have never heard of potash (like perogies and bunny-hugs).

Raw potash ore underfoot

Over the past 8 years, I have been incredibly fortunate to spend a lot of time building communications networks for underground mining operations. I have steel-toed boots, a hard hat with my name on it, several pairs of safety glasses, and as you can see in the title image, the latest in arc-flash fashion.

This is certainly a unique environment – in addition to the underground mine, the sites include large mill facilities which process the raw ore, train yards where processed potash is loaded into rail cars for shipping, and more typical administrative office buildings.

At first glance, you might not notice that this shop is underground

I got involved in mining early into what we now call “the internet of things”. Back then, engineers and instrumentation technologists had been trying to adapt to running their controls (PLCs, HMIs, MPUs – valves, belts, motors) over ethernet. These controls had typically all used proprietary protocols and connections, but the vendors had started to see the benefits to using a standardized technology. When I came along, single VLANs stretched the entire length of the mine underground (100s of kilometres); control systems didn’t quite obey the TCP/IP rules we all know and love; broadcasting constantly like that guy in your office who always talks so loud on the phone that no one else can even read the words on their screens, and these guys were starting to question the impact of running these controls on the same gear as the internet.

Long story short – things are better now! But there are still lots of unique challenges. Mining is a big mix of several networking disciplines:

Routing – No more geographically huge VLANs! Also, VRFs to separate control and traditional traffic on the same physical equipment. Best of all – we actually use dynamic routing now (HALLELUJAH).

Switching – often regular Catalyst along with industrial switches. The industrial switches in particular come in form factors more suited to installation in moving equipment and power sleds. Where cabling can be pulled, there are increasing numbers of access ports everywhere – including riding on that borer.

IE3000 switch in a cabinet mounted on the borer, along with some PLCs

Security – firewalls are big these days as we have begun to heavily protect those controls. Safety is always #1 and controls handle motors, valves, and high voltage electricity.