Full packet capture provides a method for maintaining a forensic copy of all network conversations. However the reality up until now is that full packet capture and analysis has been bounded by the size of the data, the time to process it and the ability of applications and tools to encode key attack, deviations, mis-use and anomaly data into visualizations.

When you can store all of your network data the issue then becomes how do you analyze it. How do you find the single conversation you are looking for in trillions of conversations?

Big Data has supplied both a method for parallel computation and at the same time the cost of storing all network data (full packet capture) is within reach of all organizations. At the same time threats are becoming more blended, complex and difficult to find. Big Data tools such as Apache Hadoop, PIG and NoSQL databases provide the ability to perform complex network traffic analysis at petabyte scale. These tools can be leveraged using the Amazon Cloud (Elastic Map Reduce) to process, query and persist packet capture data.

With these tools there is no time-cost trade off to analyzing every single conversation on a network, enriching the data, intersecting data sets and sharing anonymised data sets.

Allowing you to answer questions that not many tools can:

How can I find Zero Day attacks in past traffic?How can I better detect attacks at greater confidence?What is normal?What is new (never seen before)?What attackers are similar to other attacks?What is the operating system and patch level of my attackers?What protocols are strongly correlated in relation to sessions, bandwidth and payloads?What sessions are tunnels?After each attack how did the victim's sessions and protocols change?What is a normal HTTP payload for each of my web servers? - - How does an attack differ?What are attackers doing within HTTPS sessions to my websites.How can I intersect white and blacklists with my network packet captures?

Disclaimer: We are a infosec video aggregator and this video is linked from an external website. The original author may be different from the user re-posting/linking it here. Please do not assume the authors to be same without verifying.