Due to the design of the Firewall Services Module's (FWSM) packet
processing, xlates built by incorrectly routed packets can cause traffic
failures for connections through the firewall. In order to select an egress
interface for an inbound packet, the FWSM first checks to see if the
destination IP of the inbound packet matches any existing global IP/Network in
a NAT translation (xlate) for that interface in its xlate table. If a match is
found, the egress interface is simply chosen based on the local interface in
the xlate entry and the firewall does not consult the routing table to make the
egress interface decision.

The default behavior of the FWSM is to build an xlate entry for the
source IP of any permitted packet that is received on one of its interfaces. If
a packet is routed through the network incorrectly (for any number of reasons)
and arrives inbound on the wrong interface of the FWSM, an xlate is built to
reflect this. When this occurs, entries in the xlate table can override entries
in the routing table and cause traffic failures for the affected destinations.

This document describes the symptoms and triggers for this issue, how
to diagnose it, and provides solutions for preventing it from occurring.

At a fundamental level, this issue is caused by an entry in the FWSM's
xlate table that was built by an incorrectly routed packet. Because of the way
the FWSM's packet processing is designed, the firewall checks the xlate table
before it checks the routing table to determine the egress interface. As a
result, if a packet matches an existing xlate the egress interface will be
selected based on that entry, even if the entry conflicts with what is listed
in the routing table. In other words, the xlate table takes precedence over the
routing table.

In order to diagnose this issue, check the output of the
show xlate debug command:

Note: The debug keyword in the show xlate is crucial. Without it, the xlate
entries will not include the interface names that the entry is associated with.

The xlate table shows that there are 3 xlates built for the web server.
The first xlate is built between the inside interface and the
outside interface. The second xlate is built as a hairpinned
or u-turned xlate on the inside interface. The third xlate is
built between the dmz and the outside
interface. The I flag indicates that this is an identity xlate and the IP is
not actually being translated.

The first interface listed in the entry is the "real" or "local"
interface where the IP is supposed to actually exist. The second interface
listed is the "mapped" or "global" interface where the IP is being translated.
Neither of these xlates shown are correct. This is because the web server
(10.30.1.1) actually exists behind the dmz interface. The
third xlate is correct for this network design.

The connection failure occurs because of the first xlate listed in the
table. When the client's TCP SYN packet arrives on the outside interface
destined to 10.30.1.1, the FWSM checks the xlate table and matches the first
entry. This entry indicates that the packet should egress on the
inside interface, which is incorrect, and the packet is
blackholed.

By default, the FWSM will automatically build an identity xlate for any
traffic that does not match an explicitly configured NAT rule. Because of this,
even if a packet erroneously arrives on an incorrect interface, an xlate will
be built. Specifically for this case, packets sourced from 10.30.1.1 arrived
inbound on the inside interface instead of arriving on the
dmz interface as is expected.

The first xlate (inside > outside) was built when
the web server tried to ping a non-existent IP address (10.199.199.1). The echo
request left the web server destined to its default gateway (the DMZ router).
The DMZ router forwarded the packet toward the inside router, per its static
route:

S 10.0.0.0/8 [1/0] via 10.50.1.254

Because the 10.199.199.0/24 network does not actually exist anywhere,
the inside router simply follows its default route and sends the packet to the
FWSM's inside interface:

S* 0.0.0.0/0 [1/0] via 10.20.1.50

Likewise, the FWSM also does not have a route for the destination
network. Therefore, it selects the outside interface as the egress interface
and builds an identity xlate from inside > outside:

S 0.0.0.0 0.0.0.0 [1/0] via 192.168.100.254, outside

The second xlate (inside > inside) was built when
the web server tried to access the DNS server while the inside router's
10.40.1.254 interface was temporarily down due to a link flap. The DNS request
left the web server destined to its default gateway (the DMZ router). The DMZ
router forwarded the packet toward the inside router, per its static route:

S 10.0.0.0/8 [1/0] via 10.50.1.254

However, the inside router's interface connected to the 10.40.1.0/24
network was temporarily down and its directly connected route for this network
was missing. Therefore, the only matching route in the routing table was the
default route back toward the FWSM:

S* 0.0.0.0/0 [1/0] via 10.20.1.50

The packet was routed to the FWSM's inside interface.
The FWSM's routing table indicated that the destination network of 10.40.1.0/24
existed behind the same inside interface:

S 10.40.1.0 255.255.255.0 [1/0] via 10.10.1.254, inside

Because the same-security-traffic permit
intra-interface command is enabled, the FWSM will allow the
u-turned xlate to be built.

There are many different possible solutions to this problem. First and
foremost, deleting the xlate from the table should allow traffic to start
working again until the xlate is rebuilt. This can be done with the
clear xlate command. For example:

FWSM# clear xlate interface inside local 10.30.1.1 global 10.30.1.1

Note: Any connections that are using the deleted xlate(s) will also be torn
down.

Once that is complete, the focus should be on preventing the xlates
from returning. Often times, the most preferred way to do this is to fix the
routing configuration in the environment to prevent traffic from arriving on
the wrong FWSM interface. The FWSM also offers a handful of configuration
options to address these issues.

This solution takes careful planning and a deep understanding of the
network environment. In the first example above, the 10.0.0.0/8 route on the
DMZ router is technically incorrect since the entire /8 network does not exist
beyond its 10.50.1.253 interface. Instead, some options that exist are:

Eliminate the 10.50.1.0/24 network all together and simply route all
traffic through the FWSM. This also provides better segmentation and security
between the Inside and DMZ networks.

Configure a static route on the DMZ for only 10.40.1.0/24 and remove
the 10.0.0.0/8 route.

Use a dynamic routing protocol between the Inside and DMZ routers to
correctly advertise only the networks that actually exist.

There are often many possibilities for adjusting the routing
configuration, but the end goal is to ensure that traffic from a given host is
able to arrive only on a single FWSM interface.

The same-security-traffic permit
intra-interface command allows the FWSM to u-turn or hairpin
traffic on an interface. This means that a packet can enter the firewall on the
same interface it leaves on. This functionality is disabled by default and has
very little use in most FWSM designs. Because the FWSM uses VLAN interfaces,
traffic that stays within the same VLAN should never be processed by the FWSM.

In the second example above, the same-security-traffic
permit intra-interface command allowed a packet to both enter and
leave the inside interface. Disabling
same-security-traffic permit intra-interface would
prevent this behavior and drop the packet before an xlate was ever built:

In both examples above, the xlates were built when a packet from the
web server incorrectly arrived on the inside interface. In
order to prevent the problem all together, the FWSM can be configured to drop
packets that arrive on the wrong interface.

The FWSM requires that all traffic be permitted by an ACL before it can
pass. Therefore, this functionality can be achieved by only permitting traffic
from appropriate source networks on each interface. In the examples above, the
inside interface permits all IP traffic:

access-list inside_in extended permit ip any any

Instead, this should be changed to only permit traffic from the
10.10.1.0/24 and 10.40.1.0/24 subnets:

In some environments, this is not a feasible option due to the size
and/or scale of the different networks passing through the FWSM. However, this
functionality can be achieved more simply using a feature called Unicast
Reverse Path Forwarding (uRPF).

When the uRPF feature is enabled, the FWSM will compare the source IP
address of the first packet of every connection against its routing table. If
the route that is found does not match up with the interface that the packet
arrived on, that packet will be dropped due to a RPF failure.

In the example above, the FWSM has a static route that uses the
dmz interface to reach the 10.30.1.0/24 network. Therefore, if
uRPF is enabled on the inside interface, packets sourced from
the web server (10.30.1.1) that arrive incorrectly on the
inside interface will be dropped.

In order to enable uRPF, apply the ip verify
reverse-path command to each interface in question. For example:

In both of the examples above, the xlates are created with the Ii
flags. These flags indicate that the xlate is an identity translation (I) that
originated on a high security (i) interface. By default, the FWSM will build
these xlates for any traffic that does not match an explicit NAT/PAT rule. In
order to disable this behavior, the xlate-bypass
command can be enabled in FWSM 3.2(1) and later:

FWSM(config)# xlate-bypass

This feature will prevent the FWSM from building identity xlates in the
first place. Thus, the traffic in the examples above would not be redirected to
an incorrect interface due to an xlate table entry. However, the traffic will
still pass through the FWSM untranslated.

In order to determine the egress interface for a packet, the FWSM will
always consult its xlate table before looking at its routing table. If that
packet matches an existing xlate, the egress interface is selected based on the
xlate's associated interface. This happens regardless of any contradictions
that might be found in the routing table. In this way, the xlate table takes
precedence over the routing table.

Because the FWSM will always build an xlate entry for all new
connections by default, this can cause traffic failures in cases where
incorrectly routed packets cause the FWSM to build an xlate. As outlined above,
there are many possible scenarios where this can occur but all relate back to a
packet being received on an incorrect interface. This document covered these
possible issues:

A broad routing config sends packets in an incorrect
direction

The FWSM is configured to permit traffic from incorrect source
networks

The FWSM is configured to hairpin/u-turn
traffic

In order to quickly restore connectivity for connections that fail due
to a wrong xlate, delete the entry with the clear
xlate command. This document also covered multiple solutions for
preventing these xlates from returning in the future, including: