What does it really take for a Network Intrusion Prevention Systems (NIPS) to defend your business from advanced threats and zero-day attacks? Learn about the features you should look for.

Detection and response methods

NIPS products have evolved over time and will continue to evolve, working to better detect and prevent new threats. Today, it is not unusual for a NIPS product to combine several threat detection methods to create a layered defense and offset weaknesses associated with any single method.

The foundation used by most NIPS is signature-based detection: looking for pre-defined patterns associated with reported vulnerabilities and exploits. Like anti-virus signatures, NIPS signatures should be updated often, driven by new threat intelligence and signatures supplied by your NIPS vendor. In addition, many NIPS can be extended with custom signatures that you may develop to reflect threats unique to your business --for example, detecting exploits against proprietary applications.

It is also common for NIPS to provide protocol anomaly and rate-based detection: watching for spikes in traffic load, out-of-sequence packets, or nonsensical headers that are often in Denial of Service (DoS) attacks. Here, look for tuning parameters that can avoid false positives should you experience a legitimate but sudden increase in traffic.

Together, these methods can detect many threats - but perhaps not brand new zero-day threats that have never been seen before. To this end, a NIPS may offer network behavior analysis: using an established baseline of normal traffic to flag suspicious traffic that could represent intruder activity or trojan back-channels. Behavior analysis can be powerful, but it can also be more difficult to tune and requires a solid baseline. Here, look for automated base-lining, self-tuning aids, manual tuning options, and how easily exceptions can be made to work around false positives without disabling the NIPS.

Finally, consider how a NIPS responds when threats are detected. Policies generally control whether a NIPS just generates alerts or launches automated response (e.g., TCP reset, IP address quarantine, ARP redirection). In addition, look at how much useful information the NIPS provides, whether related alerts are correlated to each other and to users, how well severities and thresholds enable focus on top-priority incidents, and whether forensic details are captured to enable investigation long after the incident. A good NIPS should strike a balance between too much information and not enough, using automation, post-processing, and GUI features to promote operational efficiency.

Evaluation criteria

Start your own product search by getting a handle on where your NIPS will be deployed, the assets it will be expected to protect, the primary risks it must be able to address, and the speed at which it must do so. Given this as a foundation, it's time to start looking at individual NIPS products and their capabilities.

One useful framework for NIPS comparison was developed by NSS Labs, a research firm that conducts annual independent tests on various security products, including NIPS vulnerability, exploit and evasion tests. During its recently-completed 4Q10 NIPS Group Test, NSS Labs compared tested products based on the following criteria:

Security effectiveness: As noted above, many NIPS products are simply deployed with vendor-recommended default settings that provide a basic level of intrusion detection. Consider the level of security offered by those settings, as well as the security that can be achieved by tuning those settings. Consider not just vulnerability/exploit coverage, but also successful attack results (e.g., arbitrary code execution, buffer overflow, code injection, cross-site scripting, directory traversal, privilege escalation).

Effectiveness by attack vector: In the early days, NIPS focused primarily on incoming traffic, sent by external attackers. However, the threat landscape has shifted towards compromised internal systems, requiring analysis of outgoing traffic as well. Consider how well any NIPS handles threats in both directions.

Effectiveness by disclosure date: Every NIPS must be continuously updated to defend against new threats, including rapid response to newly-emerged zero day attacks. But what about sustained defense against old threats? In fact, many of the biggest security incidents begin with an exploit against a relatively old security vulnerability that victims had not yet patched. Consider whether default and tuned NIPS policies cover both old and new threats.

Resistance to evasion: Given commercialization of the threat landscape and targeted attacks, hackers have incentive to go "low and slow" trying to not just avoid raising human suspicion but also evading automated threat detection. Ask how a NIPS deals with well-known evasion techniques, such as IP fragmentation, TCP segmentation, RPC fragmentation, URL obfuscation, and FTP evasion, as well as more advanced techniques like PDF or JavaScript evasion.

Impact of evasion: How a NIPS handles basic evasion techniques like IP fragmentation or TCP segmentation heavily impacts its effectiveness at handling most threats, since it will not be able to look for higher-layer exploits. Consider whether basic evasions would neutralize a NIPS deployed with default settings.

To learn more about the above criteria, consult the NSS Labs NIPS Test Methodology [PDF] or read about a summary of its NIPS Group Test Results.

Example products

According to Gartner, 2010's top-selling stand-alone IPS vendors today include Cisco Systems, HP Tipping Point, IBM ISS, Juniper Networks, McAfee, Radware, Top Layer, and Sourcefire. To illustrate available NIPS products, EnterpriseNetworkingPlanet will profile a few of these product lines over the next few weeks, including Sourcefire Snort and 3D Sensors, HP S Intrusion Prevention Systems, and Cisco IPS 4200 Series Sensors.

Lisa Phifer owns Core Competence, a consulting firm focused on business use of emerging network and security technologies. With over 25 years in the network industry, Lisa has reviewed, deployed, and tested network security products for nearly a decade.