How intelligence agencies can reap the rewards of big data

Agencies are increasingly being asked to find ways of doing more with less. The data they hold – from intelligence records to number plate recognition systems – is potentially their greatest asset in fulfilling on this need. But the volume, velocity and variety of that data often exceeds their ability to store, process and analyze it for accurate and timely decision-making.

In the past, most agencies were working in a tightly controlled environment largely with the data that they held internally and intelligence generated and captured in intelligence reports or with information gathered from their investigations. The data growth was manageable.

Over the past twenty years, however, everything has changed – data volumes have increased exponentially. Agencies now have access to data that is increasingly outside of their own organization – from social media, telcos, finance organizations, as well as more traditional intelligence. And much of the data is unstructured – word documents, transcripts, witness statements. All contribute to the big data problem, but these sources play an important role.

Up-to-date technology conquers big data woes

Many agencies are finding that their legacy databases cannot cope with the scale and complexity of the data. They struggle to access the entire big data store quickly and efficiently to identify relevant data. Skilled analyst time is taken up with data management rather than gaining insight from the data. This results in missed opportunities.

And they need speed. They need the latest data storage and processing technology, like Hadoop, that uses lower-cost commodity hardware to reliably store large volumes of unstructured data. Building on this capability, intelligence agencies can start to bring in high performance analytics to uncover hidden patterns and identify anomalies and other useful information that can be used to make better, faster decisions.

High performance analytics can enable agencies to integrate and analyze huge volumes and varieties of data, structured and unstructured, to tackle and anticipate criminal activity. However, the real 'value add' of applying HPA to big data is that agencies don't need to know what they are looking for before they start. They don't have to wade through the haystack looking for the needle. Instead, the analytical techniques will model the data and push information of interest back to users, drawing attention to relevant content.

Without the right tools, pinpointing relevant facts in big data would be resource intensive and unaffordable. With the right solutions, irrelevant information can be sifted out and areas of interest highlighted. The ability to scour big data through big data analytics and data management will be crucial in enabling intelligence professionals to reveal hidden insights, produce better decisions and bring investigations to a successful conclusion.