Monday, July 30, 2012

How to use Suricata 1.3 IDP with Unified2 alarms and AlienVault 4.0 (OSSIM)

What is Suricata ?

In short: Suricata it's a next-generation high performance open source IDP engine with many plus respect to other open source solution and I think also to commercial one.

Suricata is the Open Information Security Foundatation's (OISF) Intrusion Detection and Prevention Engine.

Suricata ( Suri for the friends ) have many improvements and new functionality in respect to other similar software.

First of all it's was developed as a multi-thread engine from the beginning, only for that functionality Suricata can be called a next-generation IDP, at least in the open source softwares as far I know .

Suricata can for example decode TCP, UDP and ICMP as usually do all IDP software and also HTTP (with a special purpose HTP library), TLS, FTP and Samba version 1 & 2 !

As input method Suricata can catch packets with a standard raw socket ( interface in promiscue mode ) or with NFQueue, Pcap and IPFring. The last one methodologies is created by PF_RING a type of network socket specially crafted to improves the packet capture speed.

As output method Suricata have many formats: fast, unified2, http specific log, pcap-info, pcap-log, alert-debug, alert-prelude, stats, drop, syslog, and finaly if is not enough, can also extract files from captured streams.

Unified2 format is coming from Barnyard2, a decoupling software written to allow Snort to minimize log output overhead and so work at full speed. Evolved then in a log output converter to allow many software to inter-operate with this type of log format

http-log where all HTTP request details are recorded

pcap-info log, a simple line based log created to facilitate the correlation of pcap data with suricata's alerts.

An interesting conjunction have been scripted in LUA with a Wireshark's plugin, SuriWire

Related to this type of log format there is also a proposed patch that I write to handle the archiving of pcap file. That patch allow to diversify the path where pcap file currently under dump are located and where its are moved after they are closed

file where as mentioned upon, configuration of file extraction functionality option are tweaked.

Ending the section related to the input and output of Suricata there is that related to the engine which deserve some note.

What about the engines ?

A flow stream, obliviously contains a lot of packets.

If you want, for example, track a particular flow stream matching a rule and report an alert if and only if another rule in the same flow stream is fired, FlowBits is what you need. FlowBits allow to associate boolean named variable to the rules and to check that variable allowing to combine their value and act in agreements with that to select if fire or not a rule.

What to do if you want to check, set and compare that named variable as you do with FlowBits but in unrelated flow stream ? Simple, use FlowInt ! It operates like FlowBits but with the addition of mathematical
capabilities. With it an integer can be stored and manipulated,
not just a boolean can be flagged.

What about the rules ?

Obliviously in the open source markets there is de-facto standard rule format and off course, Suricata support it. That mean that it can work natively with Snort's rule and EmergingThreats .

Suricata have many other functionality and it's evolve really fast. At the time of write for example there is under development many useful and good functionality, so, keep an eye on it ;)

What is AlienVault (OSSIM) ?

In short: OSSIM by AlienVault is an open source Security Information and Event Management (SIEM), comprising a collection of tools designed to aid network administrators in computer security, intrusion detection and prevention.

OSSIM is the open source code base of the commercial one version called AlienVault. If you want to test OSSIM without install and configure it, you can try the cloud ( Warning: buzz word detect! ) based demo or download an x86_32/64 ISO with a full installer for AlienVault 4.0 already configured and running.

What I like about OSSIM is off course the wonderful dashboard, full of color that appears so nice when embedded in power point presentation, so all managers, from the last one to the highest one in the company pyramid can enjoy it :)

Emm :-) What I really like about that it's the possibility to full integrate many tools useful to manage in a holistic ( Warning: buzz word detect! ) way all the security aspect of the assets in an organization. Despite the normal SIEM capabilities in fact you can integrate and manage many security tools that in day to day investigation in a security operation center can bee useful if not necessary.

Just to mention some core functionality, in addition to the normal SIEM capability OSSIM have an integrated ticketing system, with that you can easily manage who need to handle the investigation behind an alert and an associate knowledge DB to accumulate know how from time to time.

The commercial version have a Raw Logs Parser to handle I think logs from many sources, I haven't tried it at the moment.

OpenVas integration ? Yes, OSSIM have it. For who don't know what OpenVas is, reporting directly from the project's home page: "OpenVAS is a framework of several services and tools offering a
comprehensive and powerful vulnerability scanning and vulnerability
management solution."

Assets, Assets Search and Asset Discovery. OSSIM response to this necessity with two approaches, a passive one and an active one. For the passive side, at lest in AlienVault 3.1, with the help of PADS (Passive Asset Detection System), it's collect asset evidence from the network. From the active side, its manage the reconnaissance with Nmap, that is a standard to manage network exploration and security / port scanning.

Off course, integrated with OSSIM after you identify what asset do you have, you can monitoring the availability of all of them with the Nagios integration. A first class enterprise monitoring tool.

I think OSSIM can offer much more, but to be honest I've not completely explored the documentation and all it's possibility, what I want just say is that:

OSSIM, like suricata it's a project that need to be followed ;)

But now, lets stop to do propaganda!

How to install Suricata in Ubuntu 10.10

Ubuntu already have it's own version of suricata, but from my point of view, it's better to have the last version. So I've installed suricata from scratch from source.

First of all, you need to install as root some support libraries.

Then we need the sources, unpack it, and lets autotools to do its works:

I've download suricata without checking PGP signatures, then I configure it to be installed in /opt directory and I made a full installation that automatically download the last emergin-threats signatures.

How to configure Suricata to produce unified2 alerts

Suricata have many outputs format for its alerts, what we need it's to setup it to create unified2-alert. When unified2-alert is enable, for each alert a new file is created with all information encoded in it.

To enable unified2-alert you need:

In the "outputs" section of suricata.yaml find "unified2-alert" section.

Then what you need is to fill the "enabled" variable to yes.

The variable "filename" specify which prefix all created alert need to have.Default value is "unified2.alert" so every alert will be created according and they filenames will be "unified2.alert.XXXXXXXXXXX" where XXXXXXX it's the epoch time stamp in seconds.

Another variable that need your interest is "default-log-dir" which behavior is self explanatory. So basically now Suricata for every alerts create a file in "default-log-dir" path.

How to configure AlienVault 4.0 to consume unified2 alerts

Now we need to enable OSSIM unified2 plugin which allow ossim-agent to take care of unified2 alerts. To do it what we need is the command "alienvault-setup". Using this perl script you can configure many options of the whole system with a simple dialog interface.

So, let me see how to do it: Obliviously execute with root privileges "alienvault-setup"

Select "Change Sensor Settings"

Select "Enable/Disable detector plugins"

Press "s" and scroll down until you find "snortunified"

Press space key to activate it

Accept that change pressing "OK"

Select "Save and Exit"

Now the script will reconfigure all the necessary component so the whole collection of software become aware of than change.

Can be useful now to check the config of snortunified plugin in "/etc/ossim/agent/plugins" in the "snortunified.cfg" file.

As you can imagine, "plugin_id" entry is the identifier of unified plug-in, because we already have a snortunified2 plugin active with id 1001 we need to set it to 1002, so the normal snortunified2 for the eth0 can work regularly without generating any conflict. In the "[config]" section we found more options:

"directory" options is the full path where unified2 alerts get consumed, in other words is where OSSIM looks to search for unified alerts. I set it with "/var/log/suricata/ and I create it.

"enable" is straightforward and need to be set with yes

"interface" are the associated interface where the alerts come from and don't need any change

"linklayer" is straightforward too and like interface don't need any change

"prefix" remind me to the "filename" option in Suricata, indeed that option is where we fix the prefix name of alerts, so I need to change it according to previous Suricata option, so I set it with "unified2.alert"

"process" is the process name associated with the plugin, if you fill it with snort, every time the system don't see the snort process ossim will try to start it with the command specified in "startup" supposing "start" options is fixed with the yes value. Because in my test suricata don't run in the same system where ossim run I don't set it and I leave that option empty

"shutdown" like startup is the command called when a stop from the web frontend is requested

"stop" specify if the process need to stop when the system will go down

"type" specify the type of plugin

"unified_version" specify which version of unified data model need to be used, so I put it's value to "2"

"source" I hope it's the name of the input source, but I'm not sure of it. I set it with "suricatalog"

Now we need to manually change "/etc/ossim/agent/config.cfg" and in the [plugins] section add "snortunified=/etc/ossim/agent/plugin/snortunified.cfg" to specify that there is another plugin that need to run.

Not it's time to restart ossim-agent to allow ossim to become aware of new unified2 plugin to consume Suricata alerts. To check if all is done grep "/var/log/ossim/agent.log" and check if you can find somethings that remind to unified or plugin with id 1002.
Because we have not fixed the process value in snortunifed.conf what I find is:

plugin (snortunified) has unknow state

plugin (snortunified) is enabled

plugin-enabled plugin_id="1002"

Now if all is ok, I suppose that if an unified2 alert are moved to /var/log/suricata ossim will parse it and will create a entry in the web frontend.

To sync the alerts created from one machine and copy/move it to the OSSIM machine in "/var/log/suricata" there is a multitude of options available. I get this occasion to report incron :: inotify cron system, as stated in it's web page

"This program is an "inotify cron" system. It consists of a daemon and
a table manipulator. You can use it a similar way as the regular cron.
The difference is that the inotify cron handles filesystem events
rather than time periods." That tool can be used to activate a script and handle a copy of every new alert generated from Suricata.

OSSIM rules VS Suricata rules

A question can arise now: How OSSIM consuming an alert with a specific rule id coming from Suricata can display the right information in his own front-end ? The answer is simple, both Suricata and OSSIM use emerginthreats as rules. Supposing an alert from the rule 666 as been created in Suricata, OSSIM parse that alarm and reference the same evil rule in his subsytem. Obliviously the rules need to be the same version.

To keep in sync the rules a perl script is provided: "/usr/share/ossim/scripts/create_sidmap.pl". It's usage is really simple, what you need it's to call it with the path of the rules directory and then wait for the script parsing all the rules and updating the internal DB of OSSIM. If you have custom rules and you want OSSIM to become aware of that, you need to use it. Please: if you encounter some error or omission keep in touch with me so I can fix it.