This post was written by Abe on November 5, 2013
Posted Under: The Deep Web

Time Magazine’s current issue (November 11, 2013) cover story, “The Secret Web: Where Drugs, Porn and Murder Live Online,” reveals the dark side of the Deep Web, where criminals can hide from surveillance efforts to commit nefarious deeds anonymously. The buzz about the evils of the Dark Web (as Time’s Secret Web is commonly referred to) started early last month when Ross Ulbright was arrested in San Francisco “on charges of alleged murder for hire and narcotics trafficking violation” and identified as the founder and chief operator of Silk Road. Ulbright, known as “Dread Pirate Roberts”, is accused of running what is described in Wikipedia as an underground website sometimes called the “Amazon.com of illegal drugs” or the “eBay for drugs.” And, of course, the government shut down Silk Road.

As founder and president of Deep Web Technologies, I take exception to the article’s referral of the dark regions of the Web broadly as the Deep Web. The term Deep Web, first coined in 2000, refers to huge areas of the Internet that serve legitimate organizations and the public. Not all of the Deep Web is dark. In fact, most of it isn’t. In fairness to the Time Magazine article authors, Grossman and Newton-Small, they do make this point early on:

Technically the Deep Web refers to the collection of all the websites and databases that search engines like Google don’t or can’t index, which in terms of the sheer volume of information is many times larger than the Web as we know it.

I’d like to shine a light on the safe and good parts of the Deep Web, where high quality business, scientific and technical research lives that is hidden from Google, Bing and other search engines. Deep Web Technologies has zero involvement with the “Dark Web” yet we’re a huge player in providing services to mine intelligence from the “Deep Web.”

So, what’s in the Deep Web that law abiding citizens might be interested in? Plenty. Here’s a snippet from a series of articles that I coauthored some years ago, Understanding Deep Web Technologies that gives a hint as to what treasures lie in the Deep Web.

The deep web is everywhere, and it has much more content than the surface web. Online TV guides, price comparison web-sites, services to find out of print books, those driving direction sites, services that track the value of your stocks and report news about companies within your holdings – these are just a few examples of valuable services built around searching deep web content.

But, not only is the Deep Web of interest to consumers, it’s of particular value to academicians, scientists, researchers, and a whole slew of business people who rely on timely access to cutting edge Deep Web content to maintain a competitive edge.

Federated search facilitates research by helping users find high-quality documents in more specialized or remote corners of the Internet. Federated search applications excel at finding scientific, technical, and legal documents whether they live in free public sites or in subscription sites. This makes federated search a vital technology for students and professional researchers. For this reason, many libraries and corporate research departments provide federated search applications to their students and staff.

Hopefully you’re convinced that there’s valuable information in the Deep Web. Now, no one knows exactly how big the Deep Web is compared to the Surface Web that Google, Bing, and the others crawl but it’s likely that the Deep Web is hundreds of times larger. This is great when you have access to tools like Deep Web Technologies’ Explorit search engine but it might also make you nervous wondering how you can find that needle in the haystack in a web that is hundreds of times larger than the one you’re familiar with that is overwhelming you with too much information and too much junk mixed in with the good stuff.

Update: I have also written an email to Time Magazine which I’ve copied below. I don’t know if they will publish it or not, but I certainly hope that they will recognize that the Deep Web is much more than a haven for criminals.

Dear Editor:

As someone who makes his living providing access to the legitimate parts of the Deep Web I am very concerned that your article paints a dark picture of the Deep Web as a whole. The company I founded, Deep Web Technologies, Inc., searches Deep Web sources on behalf of scientists, researchers, students and business people. My concern is that the public, and my potential customers, will equate all things related to the Deep Web with dark criminal activity. Please help me to correct this potential misperception to the reality that the Deep Web is about those areas of the Web that contain high quality content and that the Dark Web is just a fringe neighborhood within the Deep Web that most of us will never venture into.

Sincerely,

Abe Lederman

Deep Web Technologies, Founder and President

Share this:

Related

Reader Comments

A few months ago, when looking for some competitive intelligence, I stumbled upon BizNar.com, a Deep Web Technologies’ site. I was exposed to content in a matter of seconds that I would not have likely found in a day’s work, if at all. Since that day, BizNar has been my first stop, and very often, it is my last.

The DeepWeb vs. Darkweb, seem to be somewhat just terms or labels you have an opinion on.

For example, a criminal group could set up a database with dynamic content generated from forms? Is that considered the ‘Deepweb’ in your opinion or is that the ‘Darkweb’ in your opinion? In your opinion that the ‘Darkweb’ only contains static content? What about static content in the ‘Darkweb’ that is not criminal?

It seems that in your opinion, the definitions should something such as:

‘Darkweb’: A subset of the Deepweb that contains dynamically generated content, not indexable by standard search engines that has information not considered criminal.

Of course, with a definition that considers something criminal, you quickly enter issues of geography where there are different laws. Something in one location may be illegal, but in another it is not. So is that information in the ‘Deepweb’ or not depending on where it is located or where the end viewer is located?

This is interesting, but needs much more clarity and definition in my opinion.