When Libyan rebels finally wrested control of the country last year away from its mercurial dictator, they discovered the Qaddafi regime had received an unusual gift from its allies: foreign firms had supplied technology that allowed security forces to track nearly all of the online activities of the country’s 100,000 Internet users. That technology, supplied by a subsidiary of the French IT firm Bull, used a technique called deep packet inspection (DPI) to capture e-mails, chat messages, and Web visits of Libyan citizens.

The fact that the Qaddafi regime was using deep packet inspection technology wasn’t surprising. Many governments have invested heavily in packet inspection and related technologies, which allow them to build a picture of what passes through their networks and what comes in from beyond their borders. The tools secure networks from attack—and help keep tabs on citizens.

Narus, a subsidiary of Boeing, supplies “cyber analytics” to a customer base largely made up of government agencies and network carriers. Neil Harrington, the company’s director of product management for cyber analytics, said that his company’s “enterprise” customers—agencies of the US government and large telecommunications companies—are ”more interested in what's going on inside their networks” for security reasons. But some of Narus’ other customers, like Middle Eastern governments that own their nations’ connections to the global Internet or control the companies that provide them, “are more interested in what people are doing on Facebook and Twitter.”

Surveillance perfected? Not quite, because DPI imposes its own costs. While deep packet inspection systems can be set to watch for specific patterns or triggers within network traffic, each specific condition they watch for requires more computing power—and generates far more data. So much data can be collected that the DPI systems may not be able to process it all in real time, and pulling off mass surveillance has often required nation-state budgets.

Not anymore. Thanks in part to tech developed to power giant Web search engines like Google’s—analytics and storage systems that generally get stuck with the label "big data"—"big surveillance" is now within reach even of organizations like the Olympics.

Network security camera

The tech is already helping organizations fight the ever-rising threat of hacker attacks and malware. The organizers of the London Olympic games, in an effort to prevent hackers and terrorists from using the games’ information technology for their own ends, undertook one of the most sweeping cyber-surveillance efforts ever conducted privately. In addition to the thousands of surveillance cameras that cover London, there was a massive computer security effort in the Games’ Security Operation Centers, with systems monitoring everything from network infrastructure down to point-of-sale systems and electronic door locks.

"Almost everything interesting happening in networking has some DPI embedded in it. What gets people riled up a bit is the ‘inspection’ part, because somehow inspection has negative connotations."

The logs from those systems generated petabytes of data before the torch was extinguished. They were processed in real-time by a security information and event management (SIEM) system using “big data” analytics to look for patterns that might indicate a threat—and triggering alarms swiftly when such a threat was found.

The combination of the sophisticated analytics and massive data storage in big data systems with DPI network security technology has created what Dr. Elan Amir, CEO of Bivio Networks, calls “a security camera for your network.”

"There's no question that within the next three to five years, not having a copy of your network data will be as strange as not having a firewall," Amir told me.

The capability used at London’s Games doesn’t have a billion-dollar price tag. Nearly any organization on a budget can assemble something similar, in some cases with hardware already on hand and a free initial software download. And the potential applications go far beyond benign network security. With the ability to store data over long periods, companies and governments with smaller budgets could not only track what's going on in social media, but reconstruct the communications between people over a period of months or even years, all with a single query.

“The danger here,” Electronic Frontier Foundation Technology Projects Director Peter Eckersley told Ars, “is that these technologies, which were initially developed for the purpose of finding malware, will end up being repurposed as commercial surveillance technology. You start out checking for malware, but you end up tracking people.”

Unchecked, Eckersley said, companies or rogue employees of those companies will do just that. And they could retain data indefinitely, creating a whole new level of privacy risk.

How deep packet inspection works

As we send e-mails, search the Web, and post messages and comments to blogs, we leave a digital trail. At each point where Internet communications are received and routed toward their ultimate destination, and at each server they touch, security and systems operations tools give every transactional conversation anything from a passing frisk to the equivalent of a full strip search. It all depends on the tools used and how they’re set up.

One of the key technologies that drives these tools is deep packet inspection. A capability rather than a tool itself, DPI is built into firewalls and other network devices. Deep packet inspection and packet capture technologies revolutionized network surveillance over the last decade by making it possible to grab information from network traffic in real time. DPI makes it possible for companies to put tight limits on what their employees (and, in some cases, customers) can do from within their networks. The technology can also log network traffic that matches rules set up on network security hardware— rules based on the network addresses that the traffic is going to, the type of traffic itself, or even keywords and patterns within its contents.

“Almost everything interesting happening in networking, especially with a slant toward cyber security, has some DPI embedded in it, even if people aren’t calling it that,” said Bivio’s Amir. “It’s a technology and a discipline that captures all of the processing and network activity that’s getting done on network traffic outside of the standard networking elements of packets—the addressing and routing fields. What gets people riled up a bit is the ‘inspection’ part, because somehow inspection has negative connotations.”

To understand how DPI works, you first have to understand how data travels across networks and the Internet. Regardless of whether they’re wired or wireless, Internet-connected networks generally use Internet Protocol (IP) to handle routing data between the computers and devices attached to them. IP sends data in chunks called packets—blocks of data proceeded by handling and addressing information that lets routers and other devices on the network know where the data came from and where it’s going. That addressing information is often referred to in the networking world as Layer 3 data, a reference to its definition within the Open Systems Interconnection network model.

The OSI Layers of an Internet data packet

OSI Layer

Name

Description

Layer 1

Physical

The format for the transmission of data across the networking medium, defining how data gets passed across it. WiFi (802.11) is a physical layer standard.

Layer 2

Data link

Within a network segment, handles the physical addressing—the media access control (MAC) addressing of devices on the network and their communication. Ethernet and Point-to-Point Protocol are data link protocols.

Layer 3

Network

Handles the logical addressing and routing of data, based on soft-defined addresses. Internet Protocol headers are the Layer 3 data in a packet.

Layer 4

Transport

Protocol information, such as in the Transmission Control Protocol (TCP) and the User Datagram Protocol, provides for error-checking and recovery and flow control of data.

The data sent for specific applications in formats such as HTTP for the request and delivery of Web content, File Transfer Protocol (FTP), IMAP and SMTP mail connections, and other application-specific formats.

Internet routers generally just look at Layer 3 data to determine which network path a packet gets relayed down to. Network firewalls look a little deeper into the data when making a decision about whether to let packets pass onto the networks they protect. Packet-filtering firewalls typically look at Layer 3 and Layer 4, checking what transport protocol (such as TCP or UDP) and which Internet Protocol port number they use (this is commonly associated with a specific application; port 80, for example, is usually associated with Web services).

The structure of an IP packet, and how its services match up to the OSI layers.

Application-layer firewalls, which emerged in the 1990s, look still deeper into network traffic. These set rules for network traffic based on the specific type of application the data within the packet was for. Application firewalls were the first real “deep packet inspection” devices, checking the application protocols within the packets themselves, as well as searching for patterns or keywords in the data they contain.

61 Reader Comments

I wonder how many of these companies we deal with daily (or my customers, to be specific...). Then again I deal with this mostly from a network integrity standpoint that doesn't care as much about the actual content of the data...

agencies of the US government and large telecommunications companies—are ”more interested in what's going on inside their networks” for security reasons. But some of Narus’ other customers, like Middle Eastern governments that own their nations’ connections to the global Internet or control the companies that provide them, “are more interested in what people are doing on Facebook and Twitter.”

Seriously? This technology was developed in the US because our government was the first to want to use it on it's own citizens.

This technology seems really interesting, it can literally cause the next revolution in IDS technology.

this exactly what is needed in this current trend of BYOD. the it people used to be able to enforce policy through active directory, something no longer possible with BYOD, but network inspection on the packet level could ease the problem a bit.

The big takeaway here is that security is a now a big data problem because of increasing scope of investigation. Who would have through we'd be looking for fraud by examining call data records or point-of-sale system data for possible fraud using statistical analysis. Splunk's command set includes 'rare', 'mean,' 'average,' 'standard deviation' among others. Base-lining and watching for outliers and strange patterns has a role to play in security for monitoring for unknown threats not seen by the rest of your security architecture. Access patterns that may indicate attempts at account takeover. URL lengths that are 2.5 times an average length possibly contain command and control. I'm glad I didn't skip my statistics classes.

Splunk really isn't a database its a distributed fully indexed flat-file that uses a late binding schema, search language and utilizes map-reduce for scalability.

I've used Splunk & built custom dashboards for it, but more for application monitoring (e.g. is X still up? how many users are on atm? etc.) It's a great product imho and I'm not surprised it's being used for more "security" and network control in these examples. I just find it to be a fantastic log aggregator, especially for distributed systems or enterprise applications that spit out tons of logs.

It's a pity that activity of any kind on the internet has no reasonable expectation of privacy, unlike a hand written letter or which physical magazine I read. I can understand companies and institutions locking down their own networks but at what cost?

There's a fine line and it's undoubtedly been crossed and probably spat on.

Um, you can't. Remind me again how you can actually place an individual behind the IP! In a household full of people, prove John, not Jane, was behind the PC's keyboard when the PC was engaged in downloading a film!

"Enhanced SSL Inspection As social media networks have embraced encryption of user sessions, the Barracuda Web Filter 6.0 can decrypt HTTPS traffic that is SSL encrypted when deployed in forward proxy mode. Transparent deployment of this enhanced SSL Inspection feature requires deployment of a trusted root certificate on client Web browsers. This capability builds on the existing HTTPS filtering capabilities of the Barracuda Web Filter for inline deployments."

So if I'm reading this right, they can only inspect SSL packets of browsers that the company has "patched" with a special trusted root cert?ie. if I'm using my own system on their network I'll still be able to do my banking without fear?

Why can't ssl, in general, protect against this?I understand decrypting specific things, like fb, can be built in, but surely a network appliance can't decrypt all ssl certs... right?

Yes you can. Even more if you control the users computers. All you need is to install a trusted root cert and do some mitm attacks. There won't even a single message warning the user the SSL communications are being sniffed. That's why technologies like certificate pinning and SSL network perspective tools like "Convergence" have appeared.

SSLsniff, Ettercap, Cain and other tools create certificates for the connections. Even with SSLStrip you can get even, well, strip off SSL altogether from the connections.

So the university (or whoever) sniffs packets, but I can use my phone or computer via tethered phone and get around the sniffing. I must be missing something here.

Really? The whole piont of the endeavour is to avoid liability: if I'm connecting with a cellphone I'm doing so with an ISP account under my name, not the university's, so if I do something stupid/illegal online it's not their problem.

The big danger with surveillance, which people don't seem to get, is that prosecutors in most adversarial justice systems are professionally paid to misconstrue evidence, and the more evidence they have, the more they can lie about your motives and your character.

A good article overall, but I'm not sure you showed how things can be done on the cheap. You left out the significant problem of making sure that your sensors/probes/whatevers can actually get the data. That means every switch in your company has to be spanning to somewhere, either a dedicated circuit or device, which then needs to talk to the center of the system. For something like the London olympics, that certainly wasn't cheap. The government doesn't just magically see all POS transfers, for instance, they need to have something in the POS network monitoring traffic, and a way to get the monitored traffic back to HQ. That's not really something any old company can do, that requires the government to step in. It is difficult to put a pricetag on it, so maybe it's not in the billions, but it's up there.

As someone who works from home on a VPN, it's unfortunate how many providers do deep packet inspection. I have Charter and I'm pretty sure they do it, which I am told affects my performance because it's trying to tear apart each encrypted packet and failing. Is it public knowledge which providers do deep packet inspection and which don't? And is it public knowledge what the performance losses are for VPN connections?

Companies such as Sandvine and Procera Networks built network traffic management systems that used DPI to improve overall network performance by giving priority to specific types of network traffic, performing “traffic shaping” or “packet shaping” to throttle bandwidth for some applications while giving priority to others.

Meanwhile, back in reality, traffic shaping is still being used by ISPs when there isn't any network congestion in an effort to punish users for daring to consume the data they've already paid for. Perhaps if we slap some "network performance improvement" lipstick on the pig, no one will realize that its main purpose is to carry legacy infrastructure further into the future by pissing all over network neutrality.

"Enhanced SSL Inspection As social media networks have embraced encryption of user sessions, the Barracuda Web Filter 6.0 can decrypt HTTPS traffic that is SSL encrypted when deployed in forward proxy mode. Transparent deployment of this enhanced SSL Inspection feature requires deployment of a trusted root certificate on client Web browsers. This capability builds on the existing HTTPS filtering capabilities of the Barracuda Web Filter for inline deployments."

So if I'm reading this right, they can only inspect SSL packets of browsers that the company has "patched" with a special trusted root cert?ie. if I'm using my own system on their network I'll still be able to do my banking without fear?

The big danger with surveillance, which people don't seem to get, is that prosecutors in most adversarial justice systems are professionally paid to misconstrue evidence, and the more evidence they have, the more they can lie about your motives and your character.

Give me six lines written by the most honorable of men, and I will find an excuse in them to hang him. --Cardinal Richelieu

So the university (or whoever) sniffs packets, but I can use my phone or computer via tethered phone and get around the sniffing. I must be missing something here.

Really? The whole piont of the endeavour is to avoid liability: if I'm connecting with a cellphone I'm doing so with an ISP account under my name, not the university's, so if I do something stupid/illegal online it's not their problem.

Really. I thoight the goal was to spy on the "citizens." Look at India and the Blackberry fiasco.

Or to put it in a different perspective--a company using this on their own internal network (and outgoing traffic) to monitor what employees do on company machines and company time is one thing. But when you put this sort of monitoring on, say, an ISP, with rather more nebulous goals than making sure that company equipment isn't being misused...that's when things get disturbing.

4. encourage people to stop using IE or Safari on Windows XP which does not support SNI (virtual host support for HTTPS). Which is one of the reasons why websites do not deploy HTTPS everywhere.

5. when websites use a 'self signed certificate' ask them to use a StartSSL-certificate instead as you have no way to check if you are connecting to the right site. These are are free too. They even get a notification email to help them remember when they should update their certiticate (like ones a year).

6. when you own a domainname, encourage your supplier to support secure DNS which is called: DNSSEC. Because DANE (which depends on DNSSEC) will probably also help to make deploying HTTPS easier.

7. When a website does use HTTPS, encourage them to use HTTP Strict Transport Security.