ARGGHH ARRRRRRRP!!!

I run into this issue several times a year with a certain local DSL/Fiber service provider and ASA firewalls. Sometimes it consistently occurs after an outage, others it is seemingly random. This is the relevant debug ARP input (modified for confidentiality):

?arp-req: generating request for 16.197.160.254 at interface Provider1arp-req: request for 16.197.160.254 still pending?arp-req: generating request for 16.197.160.254 at interface Provider1arp-req: request for 16.197.160.254 still pending?arp-req: generating request for 216.197.160.254 at interface Provider1arp-req: request for 16.197.160.254 still pending

What makes this a REALLY bad thing is that 16.197.160.254 is the ASA’s default gateway. No internet access for you.

Unfortunately, I have little (no) visibility into the provider settings, but I can hazard a guess that there is some sort of spoofing protection in place. Sometimes a reboot of the ISP modem will fix the problem, but often times we have to call the ISP and, while trying to explain ARP to tier 1 support should NOT be too difficult, it is typically an exercise in frustration. Not surprising, often we are asked to plug a laptop directly into the modem with the ASA’s static IP programmed on its NIC, which works, and causes the ISP to cheer “NOT OUR PROBLEM!” Of course, if I plug a laptop directly into the Provider1 interface on the ASA, it gets an ARP reply right away and communication to the “gateway” also works just fine. Ultimately, I have not been able to find an easy, repeatable fix from the side I have control over (the ASA), but sometimes the ISP clears an ARP table to solve the problem.

Something similar occurs from time to time with this ISP with NAT/PAT IP addresses that are NOT the ASA’s interface address. In this case, we can PCAP traffic leaving the ASA with the appropriate IP and MAC towards the ISP, but the ISP will never forward the return traffic to the ASA – the gateway doesn’t create an ARP entry for that IP.

In this case, an easy fix is to temporarily change the ASA interface IP to match the NAT IP. This causes the ASA to generate a gratuitous ARP and suddenly the return traffic gets delivered.

This is certainly different from the first case, where no amount of restarting or GARP seems to convince the ISP gateway to reply to the ASA ARP request for the gateway’s MAC, but I suspect the cause may be related.

I’m currently waiting for the ISP to determine if an engineer who actually knows how to log into the gateway router and look at an ARP table does indeed exist, or whether I’m more likely to watch a unicorn run a red light on my commute home. In the meantime, if anyone can explain what this ISP is doing to cause this behaviour, I’d love to hear it.