Richard Bejtlich's blog on digital security, strategic thought, and military history.

Friday, October 27, 2006

Response to Daily Dave Thread

I don't subscribe to the Daily Dave (Aitel) mailing list, but I do keep a link to the archives on my interests page. Some of the offensive security world's superstars hang out on that list, so it makes for good reading.

The offensive side really made an appearance with yesterday's thread, where Dave's "lots of monkeys staring at a screen....security?" thread says:

My feeling is that IDS is 1980's technology and doesn't work anymore. This makes Sourcefire and Counterpane valuable because they let people fill the checkbox at the lowest possible cost, but if it's free for all IBM customers to throw an IDS in the mix then the price of that checkbox is going to get driven down as well.

First, it's kind of neat to see anyone speaking about "IDS" instead of "IPS" here. I think this reflects Dave's background working for everyone's favorite three letter agency. The spooks and .mil types (like me) tend to be the last people to even think about detection these days.

Second, it seems to be popular to think of "IDS" as strictly a signature-based technology, as Gadi Evron believes:

IDS devices are signature based and try to detect bad behaviour using, erm, a sniffer or equivalent.

That's hasn't been true for a while, even if you're talking about Snort. Sure, there are tons of signatures, but they're certainly not just for content matching. If you're thinking about Bro, signatures aren't really even the main issue -- protocol anomaly detection is.

Making IDS part of a defense in depth strategy is giving it some credit for actually providing defense, which it doesn't do. The people who win the IDS game are the people who spend the least money on it. This is why security outsourcing makes money - it's just as worthless as maintaining the IDS yourself, but it costs less. Likewise, Snort is a great IDS solution because it does nothing but it does it cheaper.

The technology curve is towards complex, encrypted, asynchronous protocols. The further into time you look, the worse the chances are that sniffing traffic is an answer to anything.

The market is slowly realizing this technology's time has past, but in the meantime lots of people are making giant bus-loads of cash. Good for them. But IDS technology isn't relevant to a security discussion in this day and age and it's not going to be anytime soon.

I will agree that many commercial managed security monitoring services are worthless, to the extent that they are ticket- and malware-oriented. However, the idea that Snort "does nothing" is just wrong. Hopefully Dave is just being inflammatory to spur discussion. Sure, Snort is not going to detect an arbitrary outbound encrypted covert channel using port 443. That doesn't mean Snort isn't useful for the hundreds of other attack patterns still seen in the wild.

Since the majority of the posters to this thread are offensive, I doubt they have read any of my books. For example, reverse engineering guru Halvar Flake follows up with this insight:

I still agree with the concept of replacing an IDS with just a large quantity of tapes on which to archive all traffic. IDSs will never alert you to an attack-in-progress, and by just dumping everything onto a disk somewhere you can at least do a halfways-decent forensics job thereafter. Since everybody and his dog is doing cryptoshellcode these days you won't be all-knowing, but at least you should be able to properly identify which machine got owned first.

Welcome to network security monitoring, albeit at least a decade late. The fact that the criminal underground is using covert and encrypted channels now doesn't mean they weren't used 10 plus years ago, when smart people in the spook and .mil worlds needed a way to gain some sort of awareness of network activities by more dangerous adversaries.

I am waiting for someone to tell me the story about how an IDS saved their bacon. I'm not interested in the story about how it found the guy with the spyware infection or the bot installation; secops teams find those things all the time in their firewall logs and they don'tfreak out about it when they do.

The last times I manned a console full-time as a "SOC monkey," for the Air Force in 1998-2001 and at Ball Aerospace in 2001-2002, we found intrusions all the time. I expect several people in the #snort-gui channel where I idle on irc.freenode.net also have stories to share. I'll have more to say on this later.

Tom continues:

This "signature" vs. "real intrusion detection" thing is a big red herring. Intrusion detection has been an active field of research for over 15 years now and apart from Tripwire I can't point to anything operationally valuable it has produced.

This sounds like the "Snort is worthless" argument Dave proposed. Finally:

Halvar, when you figure out how to parallelize enough striped tape I/O to keep up with a gigE connection, then, Halvar, then I will respect you.

This is another common argument. Most every detection critic argues their pipes are too big to do any useful full content collection. Let's just say that is not a problem for everyone. Many, many organizations connect to the Internet using OC-3s (155 MBps), fractional OC-3s, T-3s (45 Mbps) and below. Full content collection, especially at the frac OC-3 (say 60 Mbps) and lower, is no problem -- even for commodity hardware, if you use Intel NICs, a solid OS, and fast, large hard drives. Even if you drop some small percentage of the traffic, so what? What are the odds that you drop everything that is relevant to your investigation, all the time?

What if your pipes really are too big for full content collection, say in the core of the network? I would argue that's not the place to do full content collection, but let's say you are told to "do something" about detection in a high-bandwidth environment. That's where the other NSM data types come into play -- namely session data and statistical data. Can't save every packet, or you don't want to? Save sessions describing who talked to who, when, using what protocols and services, and how much data was transferred. That is absolute gold for traffic analysis, and it doesn't matter if it's encrypted. At the very least you can profile the traffic statistically.

The root of this problem with this discussion is the narrow idea that a magic box can sit on an arbitrary network and tell you when something "bad" happens. That absolutely won't be possible, at least not for every imaginable "bad" case. The "IDS" has been pigeonholed in the same way the "firewall" has -- as a product and not a real system.

A standard "IDS" isn't an "intrusion detection system" at all; it's an attack indication system. Snort gives you a hint that something bad might be happening. You need the rest of your NSM data to determine what is going on. You can also start with non-alert NSM data (as described in this war story) and investigate intrusions.

Similarly, a firewall isn't necessarily stopping attacks; it should be enforcing an access control policy.

A real detection system identifies deviations from policy, and perhaps should be called a network policy violation detector. A real network policy enforcement system prevents policy violations. The point is that neither has to be boxed into an appliance and sold as a "NPVD" or "NPES". (As you can see, acronyms which tend to accurately describe a system's functionality are completely marketing-unfriendly.)

I'll conclude by saying that I agree with Dave about "monkeys" staring at screens. Many of those sorts of analysts are not doing NSM-centric work that would truly discover intrusions. Yes, the network is a tough place to detect. However, I've argued before that in an age of ubiquitous kernel-mode rootkits, NSM is needed more than ever. If you can't trust a rootkit-controlled host to tell you what's happening, why would you ignore the network? Sure, the traffic could be covert, encrypted, and so forth, but if the pattern of activity isn't normal you can verify that at least something suspicious is happening.

13 comments:

Anonymous
said...

Richard, you are on the right track here. I've been having conversations like this at work along the same lines. I *agree* that relying on an IDS and it's alerts is pointless. I disagree that an IDS is useless. Those that feel it is useless have not explored its full potential. Reading your books and your blogs, I have learned the importance of getting the session data and content data when possible. Together, these add up to a tremendous amount of intel on your network. By the way, I'm glad to hear you remind people that you don't *have* to capture full data if it's not cost effective. I have some IDS boxes that can capture EVERYTHING, and some IDS boxes that can only log session data due to high bandwidth. It's a hell of lot more useful than staring at a BASE webpage...

Also, regarding SSL encrypted sessions, I had an a idea. It's possible for a Bluecoat Web Proxy to do SSL interception. The proxy decrypts and hands off the clear text data to something like BlueCoat's ProxyAV (separate appliance) via ICAP to inspect. Why not setup an Snort/NSM box to inspect the payload? Can Snort speak ICAP? Don't think so. Perhaps a question for Marty and friends...

1) Does your opinion shift at all when talking IDS versus IPS? Or do you refer to both as IDS?

2) I tend to agree with you that security should be moving towards the switch. But I also feel that communications, especially ones that want to remain hidden, will continue to trend towards being encrypted and performed over otherwise expected ports like port 80. What can a switch offer to this other than src, dest, and protocol?

Richard- i think the real question here is realistic expectations of what the technology can do and how we use it. Inflated expectations will always result in disappointment. I have written on this here

I work at a public institution in the USA. We have a snort box on our edge, monitoring close to an OC-3 on an OpenBSD system with high-powered hardware. It can be labor intensive, but in an environment like this one where decentralization is rampant and people are constantly trying to cut corners, are spread too thin, and don't have enough staff or training to do the job properly - the IDS helps show us certain types of incidents that would be more difficult to determine otherwise. Sure, a bot infection spewing forth 10,000 packets is going to be obvious in firewall logs or netflow (and IDS in some cases). I hvae found the the sourcefire rules + bleeding edge threat rules, plus careful tuning has turned the IDS into an item of value on campus. My expectations of what IDS can do for us are realistic. We don't mention it to the admins, because we know that the "Oh, we've got a {firewall/IDS/IPS} now, we can wait to {patch/harden/securely configure} our systems" mentality would become further inflamed. In a well managed network, with proper controls over all hosts, adequate staff resources and adequate skillsets applied to every system, an IDS is going to perhaps more work than it's worth. However, I feel blind without the IDS. It's proven it's value to us, on numerous occasions. A bleeding edge threat rule to detect the passing of SSN's in clear text has been of great value to us in helping to find that type of policy violation. A firewall log won't tell you that and netflow won't tell you that. Ptacek and other high-end offensive computing types that have multi-processor assembler code wallpaper in every room of their house (or the malicious blackhat equivalent) and dream in hex aren't going to be stopped or even detected by an IDS (newsflash!!! IDS can be evaded LOL). But guess what? Joe Scriptkiddie DOES often get detected, at least until they are smart enough to start encrypting everything or evading. Steve the SDBOT will also often get detected due to the noise of the scans and the cleartext traffic to the command & control. There can be a certain arrogance in the offensive computing world. Some of those points are clearly valid - are there marketing machines at work, offering false promises? Sure, you bet. But show me other areas of computing that don't have the same problem. The recent signature for MS06-040 that sourcefire published has saved us the pain of further infection by helping us to find the problem quick - an array of ancient NT 4.0 boxes (considered a policy violation - no one is supposed to run that hunk of exploitable crap on our network!) that got 0wned by some sdbot variant. Getting them shut down quickly was the key to containing the mayhem that was caused. Without the IDS, it would have taken us longer to determine this. So to all the offensive computing types (selected individuals only) that think you are so cool because you can bash on I{D/P}S, don't forget about people out there in the trenches that maybe aren't as smart as you but have to face the implications of your "research" on a regular basis. What's abstracted for the "security researcher" is often a needle in the foot of the in-the-trenches network defender. Hell, the IDS at the .edu still picks up people trying old old bugs that are no longer cool for the trendy researchers, but guess what, those in the trenches that don't have control over their networks are still getting burned by both whitehat research and blackhat attacks on a regular basis. "Just get control over your network" - sounds great, but sometimes that's not possible. Higher education is a great example of where an IDS or IPS makes sense. Let's see one of these high-falutin' researchers get tasked with defending a /16 (or larger) network where central control is impossible. You'll no longer have time for developing a 16-way polymorphic multi-stage shellcode because you'll be drowning in real-world issues that jump right in your face and won't go away until you do something about them.

All I can add is that most of us on the defence side (if one can say that) are not protecting against the extremely knowledgable and dedicated attacker, most of us are primarily concerned with the more pedestrian threats.

Ptacek &co are deserving of respect for both their skill and professional behavior, but I cant say the same of alot of the other clever folks.

Richard, didn't you mention some network monitoring and network forensic tool that supposedly keep statistical and meta-content, hashes etc. sometime back? (i couldn't find the post and the name escapes me.) a link to this tool will be appreciated.

I can confirm the 1:01AM Anonymous experience with the usefulness of IDS/IPS here at an .edu.

We were an early adopter of IPS technology several years ago and it has both prevented and contained compromises countless numbers of times.

Its not perfect, nothing is. It can be bypassed as can most security measures. But it stops or contains a compromise here at least once a week and probably once a day.

We have had instances where dozens of computers would have been compromised in a short period of time had it not been for the IPS. We have had instances where dozens of computers were compromised and turned into BOTS but prevented from effectively communicating with their C&C nodes by the IPS and notifying us in the process.

Sure we look at screens. There is a lot of operational information there. While we're not looking at the screens, the IPS is preventing and/or limiting the scope of compromises. We are also e-mailed and paged when incidents happen. Find a BOT on the network, get an e-mail and quarantine it while the IPS keeps it from effectively communicating. Find a BOT C&C node on the network, get a page and take it down while the IPS keeps it from issuing commands to clients. See a site trying multiple SSH or FTP sessions across the network. Get an e-mail while the IPS wards it off.

Do some get by? Sure. But that is no reason to discount the value of the technology in our environment.

Can false positives be a problem? Sure. But minimally so with some educated work and disbelief in magic box marketing claims.

Its another defense in depth layer.

Would it be more secure if everyone that operated or administered a computer fully understood it and the internet, if everyone that wrote programs and web sites fully understood the implications of what they were doing, if all vendors produced defect free products, if everyone that made policy understood technology, if everyone obeyed policy, if every decision went through a formal risk analysis?

Sure, let me know when it happens. In the meantime, I've got a network to run and if an imperfect device will help me protect an imperfect network run by imperfect people using imperfect products over which people do imperfect things for an imperfect amount of time then so be it.

BTW, at least two IPS devices allow the importing of server private keys which allow them to inspect and manage SSL sessions.