What the company found would likely startle any existing or potential customers: three of the six firewalls failed to stay operational when subjected to stability tests, five out of six didn't handle what is known as the "Sneak ACK attack," that would enable attackers to side-step the firewall itself. Finally, according to NSS Labs, the performance claims presented in the vendor datasheets "are generally grossly overstated."

"Two major issues were discovered affecting a significant number of firewalls. The first is a stability problem, meaning that an attacker can disrupt communications by sending certain sequences of content to a firewall's external interface, causing it to crash. This cannot only cause productivity loss, but can be a precursor to a larger, more effective penetration of the corporate network. Attackers can develop working exploits from these types of code flaws," NSS Labs wrote in an FAQ about its testing.

Certainly, no one would be shocked to discover that vendors pad the speeds and feeds in their data sheets, anymore than they'd be shocked to find gambling in Las Vegas. However, being able to perform attacks like the Sneak ACK attack that enable attackers to gain a "trusted" foothold is startling for many.

"It's not surprising that network equipment, when targeted, might fail," says Pete Lindstrom, research director with Spire Security. "However, it's hard to tell what would happen in real-world environments rather than lab testing," he says.

What it certainly highlights, Lindstrom adds, is the need for multiple layers of defenses. "You have to design your architectures with failure and high availability in mind," says Lindstrom. "You need to build systems with failover capabilities and go in assuming that devices are individually vulnerable to various forms of attacks. You have to segment your networks and put intrusion detection sensors on the wire."

NSS Labs has some prescriptive advice itself, and that's if a company has one of the firewalls with TCP split handshake (Sneak ACK attack) issues, contact the vendor for remediation guidelines. Also, if an organization's firewall is crashing, locking up, or displaying other unstable behavior, it may be the subject of an attack. "If your organization is extremely risk averse or highly sensitive to down-time, consider migration to one of the more stable firewall platforms in our tests," the report stated.

George V. Hulme writes about security and technology from his home in Minneapolis. He is so paranoid he has four firewalls on his home network, and found it difficult getting through them all to file this story. Fortunately, he doesn't use firewalls on Twitter, where he can be found at @georgevhulme.