The New Intrusion Detection: Part 1

Most of you who know me reasonably well know that I have been interested in intrusion detection for the better part of 20 years. I realized how potentially valuable it was after witnessing a myriad of attacks over the years when I was the manager of the Lawrence Livermore National Laboratory (LLNL) based incident response team, CIAC. Great strides in the intrusion detection arena were being made at the time, and when the US Air Force asked me to work on the Distributed Intrusion Detection System (DIDS) Project after I left LLNL, I could not have been happier.

Intrusion detection has come a long way since its inception. Widely available open and closed source intrusion detection systems (IDSs) have become considerably more efficient (as shown by higher correct detection and lower false alarm rates) and usable compared to the previous generation of IDSs. At the same time, however, new types of attacks have also exposed some glaring limitations of IDS technology. Today’s attacks, for example, very frequently involve sending encrypted network traffic, thereby rendering network IDSs mostly ineffective in analyzing the content of each packet. The fact that so many of today’s attacks are at the application, not the network or transport layer, very much exacerbates this problem. Meanwhile, attackers from countries such as the Peoples Republic of China keep sending tiny Microsoft Office, Adobe Acrobat, and other attachments that contain only a few lines of executable code—just enough to cause the victim machine that has an unpatched Office, Acrobat or other vulnerability to visit a malicious web site that injects a much bigger piece of malicious code that allows a perpetrator to completely control the victim system. Most of today’s IDSs do not both observe and correctly analyze what is happening when the tiny attachment makes its way through an internal mail server. Host-based (both not network-based) IDSs can at least detect when an unsuspecting user opens such an attachment, but many such IDSs cannot determine whether or not the attachment is malicious.

Let’s look at well-accepted and frequently used detection methods that are part of today’s generation of IDSs.

Signature Analysis. Signatures (recognizable “fingerprints” that show that a certain kind of attack is occurring) are probably still the most intuitive way to detect attacks. Although Marty Rosch and others who lead the Snort IDS effort claim that Snort has rules, not signatures, Snort as well as most other IDSs still rely heavily on signatures for detecting attacks. Signatures, however, have significant limitations, including the fact that hey are post hoc in nature and thus cannot be used to discover new or zero-day attacks. Some attacks, in fact, do not have distinguishing signatures. As mentioned previously in this posting, they are next to useless in network-based IDSs when network traffic is encrypted. Perhaps worst of all is the fact that powerful tools that easily defeat signature-based IDSs are freely available on the Internet.

Analysis of Connections. Discovering anomalous connections is potentially an excellent way to identify attacks. TCP connections that contain very small amounts of data are, for example, potentially suspicious. The same is true of incoming TCP packets with only the ACK flag set when there is no evidence of previous TCP packets with the SYN and SYN/ACK flags having been sent between the source and destination addresses. Outbound connections that follow an inbound connection from an external source are also suspicious. Stateful analysis of connections does some good. Analyzing the content of packets within connections would be better, but as I just mentioned, this kind of analysis is very often precluded by encrypted network traffic. Additionally, because many anomalous connections are not the result of malicious activity; the false alarm rate from connection analysis thus tends to be unacceptably high.

“Blacklists Based on Reported Bad Source IP Addresses.” There are so many problems with blacklist-based intrusion detection that to belabor them here would cause this posting to become intolerably long. The almost inevitable high false alarm rate is the biggest limitation. For one thing, a machine that was compromised yesterday might be promptly rebuilt, but that machine’s IP address is likely to remain on blacklists for quite a while. Additionally, many individuals who report bad IP addresses too often do not have sufficient training and/or technology.

Deep Packet Inspection. Many so-called intrusion detection experts have pronounced deep packet inspection the ultimate intrusion detection method. The main problem with this method is that such a large proportion of current attacks does not occur at layers 3 and 4. As such, inspection of layer 3 and 4 protocol header fields usually does little good.

Target Detection. Tripwire-like methods, methods designed to detect differences in hashes and cyclic redundancy checks in files and directories from one point in time to another, have proven to be much better than average intrusion detection methods. At the same time, however, halfway decent rootkits routinely alter hashes and other output on machines in which they reside to make target detection tools report that nothing has changed.

The gap between the current state-of-the-art in intrusion detection and what is needed to accurately and reliably discover current attacks has grown to the point that something must change. I’ll propose solutions and their potential advantages and drawbacks in my next blog entry.