This is, by all accounts, a human-led endeavour. But it may be among the last times that a disaster such as this is analysed without the help of a computer’s eyes. Machine vision techniques are on the verge of being able to slash the manpower needed for such investigations, and surveillance systems are starting to get a handle on real-time analyses which could even prevent some attacks from happening in the first place.

First filter

Automatic image and video processing is already good enough that it could be used as a first pass to pare down huge volumes of footage and help investigators focus their search. Computer scientist Yann LeCun at New York University, says individual people’s faces and forms can be picked out of the library of images using existing machine vision technology, then grouped together on a person-by-person basis.

This could be accomplished with similar algorithms to those that currently run on Google and Facebook’s photo platforms for face recognition and clustering, and would give investigators an overview of who was where before, during and after an attack.

Watch and learn

Such tools can help authorities spot perpetrators faster, but the ultimate goal is to stop attackers in the act. Shuiwang Ji of Old Dominion University in Virginia has trained a computer vision system to recognise basic human activities like answering a cellphone or putting down an object. His approach trains learning algorithms using video of public spaces and build models of common activity in those areas.

Once trained, a surveillance system can then flag up behaviour which falls outside of established norms. Such systems have already been tested with footage from Heathrow and Gatwick airports in London. As the recognition accuracy becomes better for a wider range of actions, these tools could be adapted to automatically flag up when a person leaves a bag unaccompanied in public, for example.

None of the existing software is sophisticated enough to handle the wide variation in video quality captured during the Boston Marathon, nor the sheer variety in human behaviour patterns. So for now, law enforcement officials are still tasked with sifting through footage shot by TV cameras, cellphones and CCTV. But LeCun says it won’t be long before automated systems will be able to step in and help. “There is very, very fast progress being made in computer vision,” he says.