DuckDuckGo: The Search Engine You Need to Meet

Recently, DuckDuckGo has been turning up in my referrers list. Curious about the name, and thinking it was a spam site, DuckDuckGo needed investigation.

Seems I’ve been missing out on what could be the major competition to Google as a search engine.

Here is a quick summary of what I learned about DuckDuckGo.

It is an open source search engine project launched in 2008.

It respects privacy and does not collect data about you as you search and click.

It features a minimalistic interface similar to Google, but the search results pageview is a bit more responsive and informative.

DuckDuckGo uses its own web crawler to index the web, and compiles the results with “structured content like Wikipedia” and Wolfram|Alfa, improving the relevancy of the results.

Search results include concentrated removal of spam and clutter, potentially returning cleaner and more relevant results.

It was a one person project, originally run by founder Gabriel Weinberg for the first three years of the search engine’s life. Hiring the first employee in October 2011, it now has five full-time employees and 10 part-timers.

As of August 2012, DuckDuckGo averages 1.4 million million searches a day. They offer a live traffic report chart with the stats.

DuckDuckGo only accounts for an estimated 0.1 percent of all US search traffic, according to results from tracking firm comScore. Half of its traffic comes from overseas, specifically Europe.

It doesn’t use a “filter bubble” to personalize search results, creating a more stable search result for every user not just you.

In addition to the search engine website, they have an iPhone and Android app, and a variety of plugins and add-ons for most of the major web browsers.

DuckDuckGo offers a DuckDuckGo Goodies page, called “Zero-click” search results, for fun and to speed up access to popular searches.

DuckDuckGo is built with open source tools including FreeBSD, Nginix, Memcached, PostgreSQL, Ubuntu, and Perl. Some of the code is distributed free on GitHub.

DuckDuckGo survives and makes money through venture capital funding, contextual advertising, ecommerce site affiliate links (commission if you make a purchase as a result of your search), and relevant sponsored links to companies at the top of search results pages.

It feels a lot like early Google, with a stripped-down home page. Just as In-N-Out doesn’t have lattes or Asian salads or sundaes or scrambled eggs, DDG doesn’t try to do news or blogs or books or images. There’s no auto-completion or instant results. It just offers core Web search—mostly the “ten blue links” approach that’s still really useful, no matter what its critics say…As for the quality, I’m not saying that Weinberg has figured out a way to return more relevant results than Google’s mighty search team. But Duck Duck Go…is really good at bringing back useful sites. It all feels meaty and straightforward and filler-free…

Our favorite alternative is DuckDuckGo, a search engine that focuses on your privacy, while also packing itself with great features for power users, like searching specific sites with a bang (e.g. !lifehacker jailbreak iphone), keyboard shortcuts that help you navigate your results, and even integration with other services like Wolfram Alpha for quick answers to certain things (like unit conversions, calculations, and other facts).

As a long time fan of Wolfram|Alpha, having it tied into DuckDuckGo scores big points for me.

DuckDuckGo offers the proverbial, “breath of fresh air.” While Google’s results are chock full of biased articles that our friends have recommended on Google Plus, or direct us to websites infected with malware, Weinberg prides his creation on its four pillars of excellence: “Our four primary focuses in this regard have been aggressively ridding spam and irrelevant results, reducing clutter as much as possible, adding instant answers from great sources above links whenever more relevant, and offering real privacy,” said Weinberg in an interview with Digital Trends. In other words, DuckDuckGo’s number one priority is protecting you, the user, from the dangers of the online world.

The search results from DuckDuckGo are clean and easy to read. More importantly, the results are generally just as accurate as a similar search with Google, if not more. DuckDuckGo also makes a point of not collecting any user data. When you do a search in Google, a lot of personally identifiable information is recorded and stored for an indefinite period of time. Google has come under attack by privacy advocates for its stance on the subject, but DuckDuckGo easily sidesteps the issue by making a point of collecting data anonymously from the start.

By far my favorite feature of DuckDuckGo is how hacker friendly it is. You can navigate search results using vi key bindings. I’m a long time vi user, so much so that the h, j, k, and l keys are memorized in the muscle of my fingers. I don’t even have to think about navigating through the DuckDuckGo search results, and I don’t have to lift my hands from the keyboard to use the mouse. Awesome.

Some of the largest companies on the Web have attempted to compete with Google and failed. DuckDuckGo, on the other hand, is about as small as a company can be — it has one full-time employee — and has come up with one of the most appealing Google alternatives to date. It doesn’t involve e-mail, maps, real-time results or social networking. It’s just a simple, straightforward search engine that’s reminiscent of early Google, with a no-nonsense privacy policy (it will not store any information that could tie you to your searches). Best of all, the results are dependably relevant and devoid of spam.

PCMag also included DuckDuckGo in their top 100 websites for 2011, crediting the privacy and extra relevance in search results. They chastised Google by saying, “It may just be following the ‘Don’t be evil’ credo that Google once exposed.”

But it seems to coincide with something bigger, too: the public’s growing awareness of the many trade-offs of online privacy, particularly when it comes to Google. So the chart above suggests on the one hand an upstart search engine growing its base of users. But it also suggests, on the other, a base of users who are panicking about their privacy — slowly, steadily, and then, whoa, exponentially.

It’s fairly common for me to meet with clients and students very cautious and a bit paranoid about Google becoming the next big brother, and many are very wary of privacy issues on social media networks. There is a definite trend of awareness and resistance to requests for personal information beyond the basics. A search engine focused on protecting that privacy could go a long way with savvy web users.

According to DuckDuckGo’s own stats page, the source of the chart used by The Atlantic, they experienced impressive growth in search engine queries and traffic of 227% in the first quarter of 2012. They admit that their own marketing worked, but also claim the growth was fueled by current events related to privacy lawsuits and violations from major social media and Internet companies. It’s impressive no matter what the reason.

So how does it really work?

The specifics are outlined in DuckDuckGo’s Privacy statement. They state that they do not collect or share personal information.

DuckDuckGo does not store your data and it does not send your search terms to any of the sites you visited for that search term. That means it does not filter the results based on the information you have previously given it. You get the same results as everyone else does when searching for a page.

It also stops any searches from being traced back to you. With Google searches, information about your IP address, your User Agent and cookie information is sent to the web site. If you are logged in to your Google account, it sends your name and email address too.

Web sites potentially can gather a lot of information about you. As can the search engines. And they might have to give the information over to the authorities if asked for.

Privacy related search engines such as ixquick are used more and more often. There is a downside though. The tight control on privacy means that you do not get any search history if you are making several related searches. Nor do you get auto correct when you type your search query.

DuckDuckGo’s Privacy statement goes on to explain why you should care about what they call “Search Leakage,” when your search terms are sent to the site you are search, thus exposing your search terms. Along with the search terms, your user agent, IP address, and other information is collected for analytics and statistics.

Web Analytics folks, bloggers, and SEO experts rely on this data to analyze what keywords are bringing visitors to a site. DuckDuckGo prevents search leakage by not passing on the information of your search request. A search engine that doesn’t pass on this information could shake up the analytics market reliant on tracking visitors.

It also includes lengthy discussions on privacy issues for search history, what information they do not collect, and what they do, along with the information they share, or do not share, with others. Sharing and selling collected search data is big money to Google, Yahoo, and Microsoft.

DuckDuckGo does not use cookies by default. They do save searches, but not in a “personally identifiable way” with your IP address, etc. They may add affiliate code to major ecommerce sites to generate commission income when you make purchases at those sites, a simple way of generating income for the search engine.

The more he talked to people on sites like Reddit, the more he began to question Google’s way of doing things. “I thought it was kind of creepy that search engines would have all of that data when someone had not opted in to share it,” says Weinberg. ”There was also this history of search engines being subpoenaed for records of people’s personal search histories. I didn’t really like the idea of being subpoenaed. I figured it would only be a matter of time, so I really wanted to get that out of my hands. That’s why I decided to stop storing IP addresses.”

The idea is that the Feds or police won’t come knocking on the company’s door because it doesn’t have anything to give them. It also means that DuckDuckGo doesn’t have certain resources and features that other search engines do, such as targeted advertising and saved search histories.

In the United States, from January to June 2011 alone, the government served Google 5,950 user data requests, with an astounding compliance rate of 93 percent. What’s even scarier is that Google reported an increase in data removal requests by 29%, compared to the previous data reporting period of July to December 2010. While Google itself may not knock on your door, law enforcement officials one day may.

DuckDuckGo isn’t the only or first search engine to realize the impact of privacy issues for searchers and themselves. For a list of search engines offering anonymous Internet searches, which includes DuckDuckGo, see this list by Hacker 10 – Security Hacker.

The Story Behind DuckDuckGo

I didn’t set out to do a search engine, but experimented a bit with some projects that I thought would be useful in search, things like more aggressively identifying spam and removing that from search results, and then using structured content like Wikipedia to improve results.

About a year into that I figured “hey, I could turn this into a search engine.” I decided to package it up into something I could launch and was just surprised at the response of people, so I decided to really focus on it.

SearchEngineLand says that DuckDuckGo could be the biggest long-term threat to Google, saying it gives more instant answers, less spam and clutter, “real” privacy with promises not to track you, and “lots of goodies.”

The article is an excellent read as it explains the challenges facing search engine competitors. When it comes to consumer search competencies, there are two in which a search engine must excel to succeed: retrieval and presentation. Retrieval is the gathering and processing of the search results. Presentation is the way the information is presented to the user, scored by the speed at which the user can find the information they need.

The conclusion from our Google – DuckDuckGo SERP comparison for several query types is that DDG is clearly innovating in search ‘presentation’ — an area that has arguably decayed over the years as the large search engines rush to cram ever more (images, video, local, advertising, social…) into the search results.

Specifically, DDG has successfully reduced the ‘searcher work quotient’, improving on the ‘get in and get out’ standard. With negative publicity around Google’s recent changes that have not always been described as a positive step forward for the SERPs, one could easily argue that there is pent-up consumer demand for just such a change.

The article also reminded us of when some little whipper snapper search engine called Google showed up 15 years ago, Yahoo, Lycos, AltaVista, and many others dismissed the young upstart.

What are the DuckDuckGo Goodies?

Many of the articles refer to the great “goodies” that DuckDuckGo offers.

In addition to Firefox, Chrome, and other browser extensions and tools and Android and iPhone apps, it offers a DuckDuckGo Goodies page with easy to use calculations for man, square roots and other computations, conversions for sizes, speeds, dates, temperatures, and money, date estimators and searches, entertainment questions, random facts, food and recipes, geography – the list goes on and on.

Location aware tools are also fascinating as you can find out the time, weather, sunrise and sunset times, tides, and more with one click.

It also features what it calls “Zero-click Info” sources directing the user to the most frequent search sources for Wikipedia, lyrics, dictionaries, maps, etc., found through DuckDuckGo search results. There are some fascinating searches including find studs (for the home builder and DIY not the other), Kid rock schedule, i’m a walking contradiction, and a little ego search for DuckDuckGo founder Gabriel Weinberg on Crunchbase.

Plugins and tools created by general hackers and coders receive public credit under each tool on the page. That’s a very nice, inclusive touch.

Why Do You Need to Know About DuckDuckGo

Why should you know or care about DuckDuckGo? Should we be paying attention to this oddly named search engine that makes it difficult to say “Go DuckDuckGo it?”

There are many reasons. The most important is the impact of privacy protection, and violations, for your online life and work.

Bloggers are the first voices in the crowd to cry out against infringements of all kinds, and violation of privacy is a huge infringement.

I’m not paranoid. That’s a fact. Search engines and social media networks dating back to the early days of the web have been tracking your every move.

Facebook has been in trouble for the past few years for switching their privacy statement around almost monthly. In 2006, AOL was sued for “search data leaks” when they were caught releasing private user information to the open web. They weren’t the first, and they aren’t the last.

The concept of the “filter bubble” continues to grow in interest, including advice on how to break out of profiled search results. The Filter Bubble is a site and book by Eli Pariser of Upworthy and MoveOn fame dedicated to filter bubble information, including what you can do to bring back the “neutral, un-filtered, un-personalized web.”

We need to educate ourselves and our readers about protecting our privacy, explaining what is tracked and what is not, and how that data is used as we move through the web.

We also need to explore the impact of a trackless search engine in our SEO driven life and work. If more and more of our visitors are visiting anonymously, restricting the information picked up by web analytics programs like Google Analytics, how will we track referrals and traffic? How will we monitor how we are doing in the search engine results pages (SERP) (formerly PageRank) obsessed industry.

How will this change how you blog and how you fuel search engines?

Lastly, we all love kick starters, whipper snappers, and underdogs nipping at the heels of the monolithic monsters of the world. DuckDuckGo is a modern David and Goliath, a story that never loses its power. That’s a story worth sharing.

I didn’t include that in my research. They index from a variety of sources and if you are in those sources you are already part of the their indexing. What would be more interesting to investigate is if they use XML sitemaps, which is the current method of “submitting” sitemaps since that is rarely necessary any more.

Another good question would be to check Ping-o-Matic!, the default “pinger” to search engines for WordPress sites, to see if DuckDuckGo is included. Checking…they haven’t updated their blog since 2009, and that is an update to let the world know they know that they haven’t updated their site or blog in a while. LOL! I don’t see DuckDuckGo on their list, but I know that they ping more than just the ones they’ve listed. Would be interesting to know that answer.

A prime reason I would switch to DuckDuckGo is because, well, Google is now doing evil. Witness the thousands of people who followed Google’s rules for back-links with scrupulous honesty (not using back-link mills, or other dishonest means), only to have their incomes come crashing down, putting their families at economic risk, simply because the Google techies whimsically decided to change how backlinks affect search rankings.

DuckDuckGo has been one of my favourite search engines for a while now. I gave up using Google as a Search Engine a long time ago as it filters the results far too much for my liking. Getting a clean, unbiased search is almost impossible in Google now days – they insist on searching my local region instead of the entire web, even when logged out of Google. DuckDuckGo doesn’t do this. Other good search engines are https://www.ixquick.com/ and https://startpage.com/.

Reblogged this on nebraskaenergyobserver and commented:
This is very interesting. I haven’t tried it yet but. soon will. If you’re at all worried about Google’s [lack of] privacy, and you should be, you should too.

Great article. A few years back I worked for a technology that focused on search enhancement technology. I had a few emails back and forth with DDG fonder Gabe Weinberg, and he was very interested in finding ways to improve the search engine’s search functionality. The difference between DDG and the big players like Google and Bing is that DDG actually cares about the quality of their results and producing SERPs that give consumers exactly what they are looking for. It’s a great engine, hope it continues to gain traction.