Cisco ASA Security Troubleshooting Best Practices

Troubleshooting configurations and installations is as much a process as it is an art.
The process has to be followed in order for you to obtain the necessary information to solve the problem .However some problems are not merely solved by process alone you need to have that extra bit that usually helps for the timely resolution of the problem. I have found in my experience that the more knowledge a person has about the underlying principles that govern the network the more successful he is in resolving the problem .That being said you can’t build a house from the roof down you need to know some basic network fundamental principles before you attempt to do advanced troubleshooting .

The document bellow is meant to be used with some caution where specified and if at any time you feel you may be lost in too much technical detail you can always contact us to assist you in your endavours.

Network failure is extremely undesirable and usually accompanied with a lot of pressure to from management and all involved parties to resolve the issue .The best approach is to be prepared and equipped with all the tools and techniques you need to cope with network failure. It is always easier and faster to recover from a network failure if you are fully prepared ahead of time.OF course the best way of handling network failure is to avoid it altogether by eliminating the most common cause for the outage and be prepared to deal with the problem by having:

Proper documentation Clear documentation that is easy to follow is a must for any network and in particular for troubleshooting network devices. It is also important during troubleshooting to document any changes being made to the network so that it is easier to back out of an operation if troubleshooting has failed to identify the problem within the maintenance window. Having good backup copies of all the devices before the start of a troubleshooting session can help to restore the network to working condition more quickly.

Solutions to every problem that is solved should be documented, so that you can create a knowledge base that others in your organization can follow if similar problems occur later. Invariably this will reduce the time to troubleshoot your networks and, consequently, minimize the business impact.

Having a network baseline You must baseline your network, and document normal network behavior and performance as follows: at different times of the day, different days in the week, and different months in the year. This is very important, especially for the security aspects of the network. You might also consider deploying external auditing or logging tools such as Syslog, MCP, and so on, for defining the baseline. Another good practice is to constantly collect logs for comparison with the baseline to uncover any abnormal behavior that requires investigation and troubleshooting. For example, you baseline a network that has connection counts across an ASA firewall of 10K in rush hours. Suddenly, on Saturday night, your ASA experiences connections substantially above 10K. You know something is wrong, and this knowledge should trigger further investigation before a worm could possibly spread and cause more harm to the network.

Backup of working configuration and version information of devices on controlled areas in the network You must back up working copies of every device's configuration and version information, and keep them in a secured location that is readily accessible to selected personnel, so that you can revert back if needed.

Clear, concise and updated network topology Undoubtedly, one of the most important requirements in any network environment is to have current and accurate information about that network (both physical and logical topology) available to the network support personnel at all times. Only with complete information can we make intelligent decisions about network change. In addition, only with complete information can we troubleshoot as quickly and as easily as possible. The network topology should include at a minimum, the names, types, and IP address of the devices. Additionally, it is advisable to know the types of protocols, links, and services configured on the devices.

Tools Readily Available It is not acceptable to have to wait to download or install tools that might be required during troubleshooting. At a minimum, be sure to have available external Syslog servers and sniffer software, in addition to an FTP Server and a Trivial File Transfer Protocol (TFTP) server. For example, most ASA firewall issues can be diagnosed with the syslog server. Under some rare circumstances, you might need the sniffer capture if the syslog is not giving you any conclusive result.

Cohesive teamwork between Security Operation (SecOp) and Network Operation (NetOp) personnel In a mid- to large-sized network, network operations and security operations are divided. As the security is a component that goes hand-in-hand with every technology and product, it is extremely important that you have a cooperative professional relationship with those involved in troubleshooting issues that involve technologies beyond security. For example, if you have a GRE over IPsec tunnel between two sites, Security Operations might be responsible for the IPsec VPN, whereas Network Operations handles routing and switching. If you have problems with unstable VPN tunnel and packet drops, the problem might be either with the VPN or the underlying IP network. If the IP network is not stable, the tunnel will not be stable. Or the problem might simply be with the hardware encryption. In nutshell, to come up with the fastest diagnosis and resolution, both teams should form a cohesive work team.

Change control review You must produce a change control for all the following: every component you add to the network; every new service you turn on; and every new command you add to the devices. Change control includes documentation and thorough review with senior engineers and management. If possible, simulate the setup in the lab network before introducing the changes to production. Schedule a maintenance window within which to perform the task. Formally establish a change control review board with the senior members of the team if required.

Clear and concise escalation procedure This is often one of the most important and overlooked proactive measures. You should document and make available to every member of the troubleshooting team a clear and concise escalation procedure. You should also list the points of contact to external networks, including the connections to the Internet. This not only helps you as a senior engineer in the company, but helps others who are new to your network. Escalation procedures could include information on how to engage the next tier of engineering in your organization, and guidelines on when and how to engage the Cisco Support team.

Its estimated that almost 90 percent of the network problems reside at the OSI Layer 1 namely the physical layer of the network.

That includes the most common culprits

Network cable
Switch port
Switch Patch panel
Interface of the host

And in general anything to do with the physical layer of the network .By eliminating these elements out of the equation you have made a giant step towards a successful resolution of any problem. In these scenarios you can utilize any cable test analyzer in order to eliminate layer 1 problems my personal favorite cab be found here www.flukenetworks.com

After successfully eliminating the OSI Layer 1 as being the problem you move on 1 layer up
Data link layer .
In essence the switch needs to be interrogated for most common problems like
Mac address tables containing Arp entries of the correct host mapped to the correct ip address
Next would be to verify if the appropriate devices belong in the correct vlan

Once that’s eliminated of the troubleshooting process move on to the next step layer 3 networks gateways and routing destinations
At that layer is where you usually find the firewall that governs the network access to different destinations .Bellow is summary of things that can done to perform most common troubleshooting commands on the Cisco ASA

Cisco Security Network devices have numerous integrated commands to assist you in monitoring and troubleshooting your network. The following sections describe the basic use of these commands:
Using Show commands
Show ? On the device will show you what commands can be used to dissect the problem

show xlate detail
show connection
show service-policy
show asp drop
show cpu usage
show traffic

show running-config | include route
show access-list | include 10.1.1.50

When troubleshooting an access through the ASA or generally trying to pinpoint the root cause of the problem its very useful to engage the onboard Cisco ASA logging functionality that can be immensely helpful

taking the message output and running it through Google will most likely give you an idea of what you may be dealing with. That’s the quickest way of obtaining a solution to a common problem .Another one would be to look at Cisco’s web site for a definition of the syslog message itself and then try to understand how the problem you are experiencing may be solved

Debug is a more serious diagnostic output troubleshooting command that will display whatever the issue may contain. You can issue the command debug ? on the Cisco ASA to find out what debugging commands are available .Remember debugging can be quite resource intensive and its only recommended to be run for short periods of time and if at any time you require to gain access back to the device just type

Undebug all

ping Helps to determine the connectivity between devices on your network.

traceroute Provides a method of determining the route by which packets reach their destination from one device to another.

telnet Helps to find out if a TCP-based application is running and listening on a specific port.

nslookup Helps to determine if name resolution for a domain name or IP address is working correctly.

Those captures can be quite useful especially when dealing with a more complicated problem that could not be resolved by any of the above methods .This kind of problem usually will contain the network traffic source and destination in its definition in order to simplify the problem and make it more visible to the troubleshooter.

A bellow is an example of the Cisco Network Capture and it assumes there hasn’t been any captures attempted before on the device.
It’s quite simple basically you define an access list on the Cisco ASA and then you assign it to an interface so that it will capture the offending traffic for review

A network analyzer, also known as a protocol analyzer, decodes the various protocol layers in a recorded frame and presents them as readable abbreviations or summaries. These results detail which layer is involved in the problem physical, data link, and so forth. Most network analyzers can perform many of the following functions:

Filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device can be captured.

Time stamp-captured data.

Present protocol layers in an easily readable form.

Generate frames and transmit them onto the network.

Incorporate an expert system in which the analyzer uses a set of rules, combined with information about the network configuration and operation, to diagnose and solve, or offer potential solutions to, network problems.

Tools like Cisco MARS and Cisco IME can provide valuable input to see the larger picture that may sometimes be missed in all the confusion of a network down emergency.
These tools can be obtained from Cisco if you have a valid CCO login .

Using the Core dump usually involves logging a TAC case and shipping this info to Cisco for further analysis.That can indicate where the cause of the issue lies .Besides its always a good practice to have a second pair of eyes looking into the problem .Especially when you have been troubleshooting straight for hours without any break.

show crashinfo
show tech-support

Protecting the ASA Firewall Itself

The ASA firewall protecting your network should be secured by itself. Following are some of the recommendations for securing the firewall that is protecting your network from the hostile environment:

Physical security As the firewall is one of the most critical network devices in your network, proper action should be taken to protect it from a hacker's physical access. Having console access can help the hackers to change the policy, which can cause severe harm to the network. It is recommended that you turn off the password recovery ability on the ASA

Link bandwidth policing Under no circumstances should you allow more traffic to flow to an ASA firewall interface than it can handle. This can happen when your network or ISP is affected with a virus such as code red, and so on. So, proper QoS needs to be configured to police the bandwidth on both upstream and downstream routers. Though ASA Firewall has a good mechanism for protecting against DoS attacks, the mechanism is for protecting internal network resources, not for the ASA from DoS. Therefore, an effective method to implement QoS on upstream and downstream routers is to police bandwidth.

Secure access for management purposes There are multiple ways to allow access for managing the ASA Firewall. Among them, Telnet is the only one that is not secured. Arguably, the console is not secured either if it is connected via reverse Telnet. So, for remote management, it is best to configure SSH instead of Telnet on the ASA firewall.

Access control and password Be sure to configure access control for the Console port before providing access for managing the ASA. In a non-AAA environment, use passwd, and enable password commands. If AAA is configured on the ASA, be sure to apply AAA for connections. SSH with AAA is highly recommended for device management.

Regular monitoring by an IPS device Though ASA has IPS capacity, it is very limited. IPS devices such as the sensor or IDSM blade could be useful for analyzing the exploitation. In the absence of an IPS monitoring device, you might rely on ASA syslog to analyze the traffic, which is not a very effective solution in a large network due to the amount of syslog.

Syslog monitoring if IPS is absent In the absence of IPS devices, syslog may be used to monitor the traffic coming in or going out of your network. However, while configuring the syslog, you might need to pay extra attention to the level of detail you want to configure to collect syslog. If you leave the debug-level logging on, that might cause slow performance, because of the amount of CPU cycles it uses. Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you do not want to see.

Protecting Network Resources

To protect the network resources behind the ASA firewall, you must undertake the following actions:

Anti-spoofing configuration You can accomplish anti-spoofing configuration with the command ip verify reverse-path on all interfaces of the ASA firewall. This means that the firewall rejects any packet that has a source address that is not expected to be on that interface. If the ASA is an Internet firewall, it should reject all packets coming from the Internet that claim to be from a private network. Similarly, it should reject all packets coming from the private network with source addresses that are not part of the private network, as anti-spoofing is not optional in either direction.

Prevention from DOS attack Set embryonic and maximum connection counts on static and nat statements to prevent network resources from DoS attack.

Credits and References
As always credit must be given where credit is due
The bellow references were used in compiling this document