If you feel you need big data for security, you are doing something wrong

I have been fighting the same battle for 12 years. It all started with IDS, a passive system for comparing network traffic to a set of signatures and generating alerts every time a match occurred. Because IDS was never put in-line, there was no cost to performance or risk from false positives, so signatures blossomed. Open-source communities scrambled to get the most arcane signatures published, which would be shared with all users of the most popular tool, Snort.

When it became apparent that an admin was spending all of his time tuning the IDS to eliminate alerts and the IT security staff were spending what time they had available investigating alerts that had no real bearing on their security, I declared IDS dead as a functioning security solution.

A typical response to a red alert from an IDS system can take hours if not days to investigate and implement. Any organization using IDS gets millions of alerts a day, and large MSSPs get billions. Obviously, they are not doing anything with them.

Two market segments arose to address the IDS problem. Managed security service providers such as Counterpane, Riptech, and Guardent, offering IDS monitoring for around $1,200 per device per month. These MSSPs built tools to make sense of the millions of alerts they received for each customer. At the same time, the security information and event management (SIEM) industry arose with vendors such as e-Security, Intellitactics and ArcSight selling tools that would collect logs from IDS, firewalls and other devices and consolidate all the data.

Obviously, these solutions addressed the data overload issue but did little to address security. They failed to curtail the rise of targeted attacks that are now wreaking havoc upon businesses and critical infrastructure operators.

What’s the answer to the overload of SIEM data? BIG DATA! Create a distributed processing environment that can sift through the data and identify correlations and trends, as if this was a technical stock market exercise or a scientific endeavor to measure global warming.

It is time to stop the insanity. If you need Big Data to solve your security problems you are doing something wrong. That something is that you are looking at data, not intelligence.

Thankfully, there is a rapid development of security intelligence products that are contributing to the solution. These products are still disparate, but they include:

Threat feeds from research organizations that are identifying sources of attacks and command and control servers for botnets (Unveillance and Seculert).

Vendors that apply those feeds to real network traffic (NetWitness, FireEye and Damballa).

Service providers that will customize research on threat actors and their targets for their customers (iDefense, iSIGHT PARTNERS and others that are working behind the scenes and do not want to be identified).

A new category I call advanced security intelligence (ASI) that ties all the components together to automatically craft responses and defend against the attacks identified by security intelligence awareness.

So, before you make massive investments in Big Data take another look at your security posture. You do not need better ways to handle data, you need more intelligence.

Threat feeds from research organizations that are identifying sources of attacks and command and control servers for botnets (Unveillance and Seculert).

Vendors that apply those feeds to real network traffic (NetWitness, FireEye and Damballa).

Service providers that will customize research on threat actors and their targets for their customers (iDefense, iSIGHT PARTNERS and others that are working behind the scenes and do not want to be identified).