I got a question today on which I would really like some feedback from the community. I know quite a bit about SIEM, but comparatively little about Big Data solutions. Below is my take on the question. If you have another, please comment and let's have a discussion.

Question:

How is a SIEM solution different and similar to a Big Data solution?

Answer:

Although there is some overlap in the area of analysis, there are many SIEM functions that would be difficult to reproduce in a pure analytics platform.

SIEM is a purpose-built solution designed to ingest, parse, store and correlate data from security and other log sources. The primary difference is that SIEM is a near real-time analysis tool, designed to match complex conditions against high volumes of incoming log data and to provide visibility and a response capability. That response could be simple, like an alert that notifies a user by email, or it could be a more sophisticated set of actions through integration with other security products in your environment to provide automated response.

SIEM also provides a set of analysis tools with pre-built content specifically designed to aid in analysis of security information. The user interface is intuitive (YMMV) and allows users to build complex queries, dashboards and reports without needing any particular database expertise. Pre-built reports give the enterprise the ability to quickly see value from the deployment, rather than having to develop all reporting from scratch in-house. Finally, in the case of McAfee's SIEM solution, pre-built integration with other McAfee products allow organizations to maximize their security investment by getting additional value out of already purchased solutions.

Big data solutions are typically batch oriented, general purpose databases not optimized specifically for the purpose of security analytics. There is no pre-built security content that may be leveraged and all views, reports, or other output would have to be designed by users.

Considering the large number of security products in the current enterprise environment, even the most basic task, that of writing custom parsers for every single data type, would be time and cost prohibitive for most organizations. Big data solutions also have no real time monitoring component, no mechanism to alert on matched conditions, no pre-built integrations with other data sources or products.

Big Data is a fantastic tool for analysis of unstructured information. It can be useful as both source and destination for collected event or even SIEM metadata. However, wishful marketing to the contrary, it is not a suitable replacement for SIEM.