MI5 collecting “significantly more” data than it can use, new Snowden docs reveal

New documents from the Snowden trove reveal MI5 admitting that it was collecting "significantly more than it is able to exploit fully." The 2010 report, published by The Intercept, describes MI5 as the "principal collector and exploiter of target's digital footprint in the domestic space," and noted that its efforts had "grown significantly over the last few years."

A second secret report released today by The Intercept confirms the fact that the UK's intelligence agencies were drowning in data: "There is an imbalance between collection and exploitation capabilities, resulting in a failure to make effective use of some of the intelligence collected today," the report noted. "With the exception of the highest priority investigations, a lack of staff and tools means that investigators are presented with raw and unfiltered DIGINT [digital intelligence] data. Frequently, this material is not fully assessed because of the significant time required to review it."

Although those reports referred to the situation in 2010, the problems seem to have continued afterwards. As Ars wrote last year, too much data and stretched resources are common problems for European security services. A UK government investigation into the murder of Lee Rigby revealed that his attackers were known to MI5, but that it had not been possible to follow the leads because "MI5 has limited resources, and must continuously prioritise its investigations in order to allocate those resources."

Further Reading

Another project, PRESTON, enabled the National Technical Assistance Centre (a unit under the auspices of GCHQ) to intercept about five million communications during a six-month period in 2008-2009. Later, a top-secret study into PRESTON found that only three percent of those communications had actually been assessed by intelligence officers. "This is of concern as the collection is all warranted," the leaked study says.

Other new Snowden documents reveal the "MILKWHITE" programme, which GCHQ used to gather information about people’s use of smartphone apps like WhatsApp and Viber, instant messenger services such as Jabber, and social networking websites, including Facebook, MySpace, and LinkedIn.

According to the Intercept: "GCHQ made some of its huge troves of metadata about people’s online activities accessible to MI5, London’s Metropolitan Police, the tax agency Her Majesty’s Revenue and Customs, the Serious Organised Crime Agency (now merged into the National Crime Agency), the Police Service of Northern Ireland, and an obscure Scotland-based surveillance unit called the Scottish Recording Centre." A similarly-wide use of highly-revealing metadata is also an important aspect of the Snooper's Charter.

This new information that MI5 has been struggling to master the flow of data is particularly relevant in the light of the current passage of the Investigatory Powers Bill through UK parliament. One of the key features of the proposed legislation is that ISPs can be required to store information about everyone's Internet activities for a year (so-called Internet connection records, ICRs). This will inevitably lead to even more data that needs to be processed, making it more likely that important leads will be missed amidst the data cacophony.