Monday, June 27, 2011

Snort's output methods

Ever since the beginning of Snort, one of the main concerns was "how do I get data out of Snort". Some of the options that are available have their advantages and disadvantages:

There's some that aren't used.

There's some that cause Snort to be slow.

There's some that we don't maintain and don't frequently test.

There's some that we want to get rid of.

One of those output methods is the "spo_database" module. The module within Snort that directly inputs data from Snort into a mysql, postgres, or an Oracle database. This logging method was written back in the late 90's by a college student (along with the db schema and the interface ACID) as a project for his thesis.

It hasn't been very well maintained since then. In fact, we don't test against it, and we don't recommend it for use. It makes Snort, which is a high-speed data processor, have to stop doing what it's doing (being an IPS), and insert data into the database. While Snort is inserting into the database, this stops inspection waiting for the database connection.

In order to provide the type of functionality we'd like to provide with Snort in the next few releases (more data for you!), we needed someone to take over the maintenance of the db schema that is shipped with Snort as well. As a result of the discussion on the Snort-devel list, the team members over at the barnyard2 project have agreed to take over the maintenance of these schemas.

It is our intention to distribute the unified2 format as our official output method, provide our documentation for it, and the u2spewfoo tool within Snort so that anyone is able to read it. We are going to keep some other output methods as well, but...

At this point I'd like to hear from the community as well. So please leave comments.

What output plugins do you use?
Will you be affected by this change (we hope a lot of you aren't using the spo_database method)?
What other output plugins do you think we can "show the door"?

16 comments:

As a snort package maintainer for a Linux distro, anything that reduces the number of --enable-/--disable- or --with-/--without- options to ./configure will reduce the delta between when Sourcefire releases a new version of snort and when I can release an updated package. The simpler dependency tree that would result from removing these options would also reduces the amount of time it takes for the new package to move from being marked as testing to being made stable.

But, you can't get rid of everything. There are valid use cases for a number of the output options.

Keep----unified2 - This is obvious. Listed here for the sake of completeness.

log_tcpdump - There are many people who use Snort just to log network traffic. There are other tools that do this, but snort is one of the most actively developed tools that do this and can daemonize.

alert_fast/alert_full - These are extremely useful to people who write rules, especially the sending of alerts to stdout. These let rule writers quickly test, tweak, and retest new rules. It could be argued that making these _only_ write to stdout and removing the option to write to a file might be a good idea. It could also be argued that if these are to be kept solely for rule testing purposes then alert_fast could be removed leaving just alert_full.

With that said, I think it is important that before an output module is decomm'ed from snort one of the following conditions needs to be met:

1. Full, stable support in barnyard2, or2. A general consensus that the output module has out lived it's usefulness and is not being widely used.

Also, time needs to be allotted to warn end users who might not follow the mailing list. It is probably not a good idea to remove anything from snort-2.9.1, but if we know before 2.9.1 is released that in the next release after 2.9.1 the database module will be removed, package maintainers can warn their user base (via distro forums, mailing lists, 2.9.1 post-install messages, etc) to start planing for that removal.

unified(1) should be kept on, barnyard2 has issues with logging tagged packets to db and also the csv output is not properly implemented on barnyard2 and also for backwards competability with other custom applications built to read these files(siem parsers and the like)

A bit late to the discussion, but whatever.MySQL is a very valid way of handling data and output, especially when you have multiple servers and need a WEB based front end to handle things.

There's no reason to penalize people who prefer mysql (or any database for that matter), just because you don't want to maintain the output for it. Binary output sucks, it's NOT human readable, and it requires further configuration of MORE applications to just understand it.

We've made the output plugins dynamic. So even though we are removing the plugin from Snort, the developer is welcome to maintain and distribute the output plugin itself after the release of Snort 2.9.3.0.

has the output database after version 2.9.1 removed? I'm currently installing snort version 2.9.4, but there's no output database. If I want to do some logging into database, do I need to use barnyard2 so that I could get the database schema for snort? or is there any solution for my problem? I want to use multiple sensor on snort, but Barnyard2 cant config sensor more than 1.thanks.