Traditional stateful inspection firewalls, with their port- and protocol-based controls, have limited visibility into the contemporary Web-based network landscape. Thanks to the explosive popularity of Web 2.0, thousands of Web-based business and consumer apps and attacks are launched primarily through the application layer. Stateful inspection firewalls cannot distinguish what applications are passing via http and https over ports 80 and 443. Attackers have become adept at using low-and-slow techniques in targeted attacks that evade intrusion-prevention systems (IPS).

What Next-Gen Firewalls Do

True next-gen firewalls perform deep packet inspection to identify application traffic at Layer 7, performing a single inspection pass that integrates firewall, intrusion-prevention and additional security capabilities in a single high-performance appliance. Application intelligence, combined with user identity information, provides context for highly granular firewall access rules that allow for detection of contemporary Web-based attacks. Enterprises can enforce security and acceptable-use policies in ways that make sense for the business, in contrast to black-and-white policies like "No one can use Facebook" or "We have to let everyone use Facebook."

This is a fast-growing market, created when Palo Alto Networks appeared on the scene in 2007 with the capabilities and feature sets that characterize what are now known as next-gen firewalls. Most other firewall and unified threat management vendors have introduced, or are at least developing, network security products that provide fine-grained application and user controls in integrated, high-performance appliances.

"IPS should have been combined with firewall much sooner," says Greg Young, a Gartner research VP. "IPS ballooned up beyond $1 billion and took on a life of its own; no one was integrating. Palo Alto [Networks' next-generation firewalls] changed the game, and incumbent firewall vendors have been forced to react to meet that threat."

Next-gen firewall adoption was between 5 percent and 10 percent of total firewall appliances in 2010, according to a joint report by Infiniti Research and TechNavio Insights, and is expected to gain significant market share over the next few years. Gartner has predicted that next-gen firewalls will comprise 35 percent of the installed firewall base by the end of 2014 and will account for 60 percent of all firewall purchases.

In some cases, enterprises are deploying next-gen in front of their existing network firewalls and IPS to get the benefits of app-layer and user-ID filtering without a wholesale rip-and-replace. In other cases, they put it behind their firewalls and IPS to see what is getting through.

"They look at it as an adjunct," says Lisa Phifer, president of consultancy Core Competence. "They either want to apply extra granularity or use next-gen to act as a sanity check if something goes through that wasn't expected."

But that's now the exception, says Young. Today, 95 percent of next-gen purchases are firewall replacements, as the newer technology has proven its value and the vendor selection has widened.

Driving the NextGen Firewall Market: Consolidation and Cost Come First

Application-based controls and security provide the flash and the coolness factor, but the business case most often relies on the savings and reduced management overhead that come with consolidating several security products into an integrated platform that meets the needs of highly demanding enterprise networks.

"It became apparent that we could consolidate a lot of the technologies we were looking at," says David Rahbany, director of enterprise IT infrastructure at Hain Celestial Group. Hain purchased and deployed Fortinet next-gen appliances when it consolidated connectivity among its distributed sites and corporate data centers from Internet-based VPN to a multiprotocol label switching (MPLS) network.

"The driver was really the costs associated with the MPLS deployment. "We could focus our gateway security perimeter on a handful of sites, for which next-generation products better suited our needs." Rahbany also cited better management control for a relatively small IT staff.

The end of a normal refresh cycle for perimeter devices is a logical time to look at replacement, but a case can be made for off-cycle next-gen deployment if the savings and benefits are compelling. For example, 24-Hour Fitness, a Palo Alto Networks customer, had a year left in the depreciation write-off for its existing firewalls, but found that the savings in purchasing sooner rather than later more than offset the lost depreciation.

But although consolidation and cost savings are paramount, application awareness and control (what Gartner's Young calls the "sizzle") are a key driver as well. next-gen appliances enable enterprises to create policies and rules that reflect the modern Web-based IT business environment, including the growing use of Web 2.0 for both business and personal use. Just as significantly, the technology can be used to monitor and enforce compliance with these policies. It also provides the ability to identify thousands of individual applications and establish rules governing not only which are allowed, but under what circumstances and by whom.

So, for example, peer-to-peer applications might be prohibited, but Skype might be authorized for users who have a legitimate business need for it (see Skype: Is it safe for business?). All users might be allowed to use Facebook, but might be blocked from accessing the site's applications.

From a security perspective, next-gen appliances provide much stronger filtering and threat detection than the combination of traditional firewalls, standalone IPS and other security products, such as URL filtering. If the appliance is performing deep packet inspection on the firewall, it can more effectively reduce the traffic to authorized applications and users, and simplify detection of potential attacks by focusing on what still gets through. The single-pass inspection up front allows the product to correlate and analyze various security engines.

"In many ways, this is a call for a better IPS that's aware of protocols and applications," says Rick Moy, president and CEO of NSS Labs. "Now it's imperative for the firewall to know more about the applications because it has to work in conjunction with IPS to provide context for IPS to do its job."

For example, Moy says, the firewall can tell the IPS module that the application being used is Skype, and the IPS can focus on detecting known Skype attacks rather than applying all of its thousands of signatures to every packet.

"The flip side to enablement is whether I can limit the number of applications that can penetrate the network, thereby controlling avenues of attack," says Chris King, Palo Alto Networks director of product marketing.

This integrated approach makes it easier to track the source of a potential security event than with separate appliances, and effectively reduces the false positives and false negatives associated with IPS.

"We've mitigated risk in providing access to those applications and gained better insight into who's using what and how," says Rahbany. "We have management oversight that we lacked. We're in a better position to anticipate threats and manage bandwidth and applications."

Evaluating Next-Gen Tools: What to Look For

Next-gen firewalls are complex products, and vendors claim an impressive array of capabilities. Determining how well an appliance meets your needs requires understanding your enterprise's requirements, and a lot of research and testing.

Look under the hood. All vendors will claim to have a special sauce for doing that voodoo that they say they do so well, but next-gen requires sophisticated software and hardware engineering that didn't exist until a few years ago. Hold the vendor's feet to the fire to get them to explain their software and hardware architecture and how it accomplishes the required processing, inspection, correlation and analysis. Consult third-party reviews and analysis as well.

Questions to ask include:

Is there actually only one inspection pass being leveraged by the various engines in the box?

Is inspection taking place on the firewall, where it can effectively pre-filter traffic and provide context for IPS and other integrated tools?

Are the firewall and IPS truly integrated, or simply packed in the same box?

Does the product run on standard hardware or as a dedicated appliance? The general trend in IT has been toward use of standard hardware, but next-gen requires purpose-built appliances that can meet its demands in an enterprise environment.

Have they built truly new products or just adapted existing firewall and IPS technology? Most vendors, with the exception of Palo Alto, have existing firewall and IPS engines, and are now trying to integrate application control and other features with the tools they already have, says Young. "They're not completely integrated, so they have this hair-pinning of traffic between modules," he says. "This is highly inefficient."

Check its performance. All this capability comes at a price. Unlike traditional network firewalls, a next-gen appliances (like standalone IPS) is a "bump in the wire" that can clog the flow of production traffic. Connections per second—throughput with all the security features turned on—must be carefully evaluated and tested in as close to a real-world production environment as possible.

One issue in particular to address with your vendor and in testing is how the next-gen firewall handles encrypted traffic. Can the firewall intercept, decrypt and re-encrypt SSL/TLS, SSH and VPN traffic, and, if it does, at what cost to performance?
Determine realistic requirements for your production environments and test accordingly. Where and how you use the next-gen firewall is a strong factor to consider in assessing performance. Financial transactions, stock trading, and so on, are extremely performance-sensitive. Weigh the criticality of the assets and systems you are protecting when creating appropriate rule sets and deciding which security services to enable. For example, says NSS Labs' Moy, unified threat management (UTM) performance typically drops by 60 percent from 10Gbps to 3 or 4Gbps when IPS is enabled, and there is an even more drastic reduction, to 300 to 400Mbps, when antivirus capabilities are turned on.

"I'd be skeptical about turning on [antivirus] on the firewall," he says. "In front of the data center, probably not, but maybe at the perimeter."

More and more complex rules will also affect performance, so factor that into your testing.

"The deeper the policies, the more you feel an impact," says Core Competence's Phifer. "As you layer on additional checks, it is going to get slower and slower."

There are a number of high-end products on the market that perform load and security testing. These are expensive, but worth investing in if you are going to be doing a lot of network equipment and network security product testing in-house. If not, there are third-party testing providers, many of whom make use of these tools.

"Pilot the heck out of it," says Kwong. "I've dealt with many firewalls, and out-of-box we needed to tune a lot of parameters before we got to the right performance level. From my experience with previous firewalls, I've always found performance didn't quite match the claims."

Be realistic about application control. Before you are blown away by a vendor's assertion that they have so many thousand applications in their library, consider your application policies and practices. Learn which applications your company's employees are using for legitimate business purposes, which are likely to be used in the future, who is using them and how are they being used. Armed with this information, you can create security and appropriate-use policies and evaluate next-gen firewall products on their ability to monitor and enforce policy around these apps.