FBI wants to scrape Twitter worldwide

Emma Woollacott, 27th January 2012

The FBI is considering continuously monitoring all social media sites on a global basis, and is looking for help.

It's asking contractors for proposals for a system that could scan the networks, globally and in real time, to identify potential threats. The deadline for responses is 10 February.

According to its Request for Information, the applications must be able to "provide an automated search and scrape capability of both social media sites and open source news sites for breaking events, crisis, and threats that meet the search parameters/keywords defined by FBI SIOC".

"Examples include but are not limited to Fox News, CNN, MSNBC, Twitter, Facebook etc," it says. The system should have the "ability to instantly search and monitor key words and strings in all publicly available tweets across the Twitter site and any other publicly available social networking sites/forums (ie Facebook, MySpace etc".

The document also - in a rare moment of light relief - calls for the creation of "reference documents such as a dictionary of 'tweet' lingo".

Along with other government agencies, the FBI has been wanting to scrape social networks for some time, claiming the information's publicly accessible and voluntarily generated.

Since 2005, the CIA's Open Source Center has been monitoring blogs, chatrooms and social networking sites, as well as radio and television programs, according to documents released to the Electronic Frontier Foundation in 2010 following a Freedom of Information request. It's accessible to 15,000 local, state and federal government employees,

The FBI, says the EFF, has also been particularly interested in the University of Arizona’s Dark Web Project, an attempt to 'systematically collect and analyze all terrorist-generated content on the Web' - which sounds very much like what we've got here.

Somehow, we don't think civil liberties organizations will take it lying down.