[BrieFing]
The “deep web,” a concept more in
keeping with fiction than science,
gained widespread attention after the
FBI shut down the Silk Road, the In-
ternet’s premier international one-stop
shop for all things contraband.
A so-called “anonymous
marketplace,” the site ran on
Tor, free software that makes it
difficult to trace Internet activ-
ity by sending traffic through a
worldwide volunteer network of
thousands of relays.
The deep web traversed by
denizens of the Silk Road makes
up a majority of the Internet
space, according to experts,
who assert that the commer-
cial Internet – the .coms, .nets,
.govs, .orgs and .mils typically
accessed through mainstream
search engines – only consists
of about 5 percent of Internet
traffic, according to a report from
CBS News.
The other 95 percent has
proven to be a cyber safe haven
for all types of illicit activity, from nar-
cotics trade to illegal weapons.
Law enforcement officials have gone
to great lengths to prevent such illegal
activity, but if they don’t know where
to look in the deep web, such market-
places can be next to impossible to
find.
Until now, that is. The Defense
Advanced Research Projects Agency
developed a search engine last year
capable of searching the deep web.
DARPA’s goal for Memex, as the
search engine is called, is to develop
the next generation of search tech-
nologies and revolutionize the discov-
ery, organization and presentation of
search results and along the way, shine
a light into the deep web.
“The goal is for users to be able to
extend the reach of current search
capabilities and quickly and thoroughly
organize subsets of information based
on individual interests, according to a
DARPA report on Memex.
“Memex also aims to produce search
results that are more immediately use-
ful to specific domains and tasks and
to improve the ability of military, gov-
ernment and commercial enterprises
to find and organize mission-critical
publically available information on the
Internet.”
While DARPA intends for Memex to
be used in the public market, initially
it will be used by law enforcement to
combat human trafficking and other
illicit activity by monitoring chat rooms,
online forums, advertisements, job
postings and hidden services.
One of the complexities of the deep
web is that much illicit activity is not
available long enough for search en-
gines to “crawl” them.
As part of Memex, DARPA is working
with 17 different teams of research-
ers from industry and universities
to develop tools to give government
agencies ways to access these dark
reaches of the web.
In a recent success story in Sci-
entific American, Memex was used
by law enforcement officials to help
locate a victim of sex trafficking. The
Memex system incorporates eight
different open-source and browser-
based search and analysis programs to
perform data analytics.
DARPA is still holding much of
the Memex technology close to
its vest, but tidbits of informa-
tion have trickled out since its
inception.
According to the report in
Scientific American, DARPA
researchers have also made
progress in creating tools that
help analysts identify relation-
ships among different pieces of
forensic data.
The software also helps
investigators build data maps
showing visualizations of the
links in hundreds of these data
associations. It can identify rela-
tionships between a single piece
of data – an email address, for
example – and hundreds of
websites.
For instance Memex can create heat
maps that illustrate where other pieces
of forensic data – classified ads, for ex-
ample – are most heavily concentrated.
The visualizations help highlight
associations that might otherwise be
overlooked, according to the Scientific
American report.
The New York County District At-
torney’s Office said it now uses Memex
in every human trafficking case it is
pursuing. “Memex helps us build ev-
idence-based prosecutions,” said Man-
hattan District Attorney Cyrus Vance Jr.
“In these complex cases, prosecutors
cannot rely on traumatized victims
alone to testify. We need evidence to
corroborate.” •
DARPA makes strides in
searching the ‘deep web’
BY MARK POMERLEAU
12
GCN APRIL 2015 • GCN.COM
“Memex
helps us build
evidence-based
prosecutions....
In these complex cases,
prosecutors cannot rely
on traumatized victims
alone to testify. We need
evidence to corroborate.”
– MAN HAT TA N DISTRICT ATTORNEY
CYRUS VANCE JR.
MANHATTANDA.ORG
0415gcn_006-013.indd 12
3/30/15 3:10 PM