Research from academics, Microsoft, and the EFF finds that 2 percent of US …

Nearly 2 percent of all US Internet users suffer from "malicious" domain name system (DNS) servers that don't properly turn website names like google.com into the IP addresses computers need to communicate on the 'Net. And, to make matters worse, the problem isn't caused by hackers or malware, but by the local ISPs people pay for access to the Internet.

Though the 2 percent number might sound low, it's astonishingly high for a core Internet function, as is clear from the fact that no other country—apart from Haiti—sees more than 0.17 percent malicious DNS servers. What's gone wrong in America?

According to researchers from Microsoft and from the Polytechnic Institute of NYU, the malicious DNS servers exist to make a little extra cash for Internet providers. A detailed experiment (PDF) carried out between September 1 and October 31 last year found that most of these DNS servers stealthily intercepted and redirected search queries and URL mistakes, but only when these were entered from a Web browser's address bar. Go to Bing.com and everything works as it should; search Bing through a browser address bar and you might be surprised at the results.

What commonly happens is that specific search queries (usually for brand names) made from an address bar no longer return the expected Web search results page from Bing or Google or Yahoo. If your ISP has such DNS servers configured, and your computer points to them (most ISP subscribers will by default), typing "Apple" into a browser search bar will take you directly to Apple's webpage, bypassing the expected search results page.

Why would anyone do this? Well—there's money in it. The researchers found that multiple site redirections took place behind the scenes in these scenarios, with the DNS server in question not passing the query directly to the search engine but through a host of other URLs that "are all related to several online advertising companies," said the research paper. "The companies only get paid when their advertisement links are clicked by users. The extra rounds of inserted redirection are used to generate clicks, as if they are from a large number of real users."

Further investigation by the EFF and UC-Berkeley's International Computer Science Institute (ICSI) claimed that the behavior was courtesy of a company called Paxfire, which says it can help ISPs make money from mistyped URLs. Such schemes have been around for years, but Paxfire allegedly goes further.

"Paxfire's product also includes an optional, unadvertised, and more alarming feature that drastically expands Paxfire's window into users' traffic," wrote the EFF and ICSI researchers. "Instead of activating only upon error, this product redirects the customers' entire Web search traffic destined for Yahoo!, Bing, and sometimes Google, to a small number of separate web traffic proxies." (Though Paxfire's own description of itself does say that it "is the proven industry leader in monetizing Address Bar Search and DNS Error traffic for Network Operators.")

170 brand names trigger the automated redirection that ends with users being taken to those brand pages—and with affiliates pocketing some cash for sending them there. The money is presumably split between Paxfire and the ISPs in question. Paxfire did not respond to our request for comment.

Is this legal?

The Microsoft/Polytechnic research named names, compiling a list of nine ISPs who last year seemed to purposely run the malicious DNS servers: Hughes, Frontier, Cavalier, FiberNet, Spacenet, Onvoy, WOW [Wide Open West], Cincy B., and SDN. The paper noted that end users can switch from their ISP-provided DNS server to a public server (Google runs such servers, for instance, at 8.8.8.8 and 8.8.4.4) to avoid the problems.

But there are other avenues for action; the paper also noted that "complaints can be made to regulatory agencies or legal actions can be taken."

New Scientistyesterday noted a new class action lawsuit against Paxfire over the practice on the grounds that it violates the US Wiretap Act. Similar attempts to monitor user searches at the ISP level have been made in the UK by companies like Phorm, though it was widely believed such services made little traction here in the US.

And regulatory interest will probably be coming, too. This morning, in fact, FCC Chairman Julius Genachowski held a ceremony to announce the winners of the Open Internet Challenge, in which 24 researchers submitted tools to "help consumers foster, measure, and protect Internet openness." One goal of the contest was consumer-level tools that "could, for example, detect whether a broadband provider is interfering with DNS responses."

The action most affected the major search engines, who don't take such behavior kindly. Google, for instance, has systems in place to detect "hijacking" and it throws up a CAPTCHA if it suspects that such tampering has taken place. The system caught real users from Frontier, Huges, WOW, and other ISPs back in March, who then took to Google support pages to complain that they didn't like filling out CAPTCHAs to access Google tools.

But Google blamed the ISPs. "I want to assure you that at Google we are following this very closely, and trying to get Frontier to fix the issue," said a Google rep in response to one complaint. "The root of the problem is that Frontier is intercepting some traffic, so when you try to use Google your search actually goes through a Frontier server first. Google's systems detect this and identify the unusual traffic patterns as abuse, which triggers the captchas. The captchas will go away as soon as Frontier stops intercepting traffic intended for Google.... Unfortunately the interception is a business decision, not a systems error, so they are unlikely to be able to change things until Monday."

One Frontier user complained to the company back in April and "heard back immediately from Maggie Wilderotter, the CEO. She said that this had been done by one of their vendors, in violation of Frontier's business rules, and it's been shut down." Other providers also tweaked their systems so as not to trigger Google's "are you human?" checks.

But according to the EFF, "As of August 2011, all major ISPs involved have stopped proxying Google, but they still proxy Yahoo and Bing."

As for Paxfire, the company's website only appears to make reference to a much more limited tool that responds to URL typos with search result pages. But the implication is clear: most users won't mind when you monkey around with arcane backend systems.

"What feedback you do receive typically will come from a small group of highly technical users," says Paxfire. "Even that feedback tends to fall away after just a few weeks—as they get used to the new behavior."

As for the money, "Some of our customers literally generate millions of dollars a year using the Paxfire Look-up Service... It all depends. That said, no matter how you slice and dice it, the Paxfire Look-up Service will generate good money for you."

99 Reader Comments

"What feedback you do receive typically will come from a small group of highly technical users," says Paxfire. "Even that feedback tends to fall away after just a few weeks—as they get used to the new behavior."

And most people don't know what a rootkit is, so why worry about it, am I right?

"the DNS server in question not passing the query directly to the search engine but through a host of other URLs that "are all related to several online advertising companies," said the research paper. "The companies only get paid when their advertisement links are clicked by users. The extra rounds of inserted redirection are used to generate clicks, as if they are from a large number of real users."

This sounds a lot like click fraud to me. I'd think that if brought to their attention the advertisers themselves could bring it down much faster than the feds.

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Approximately a few months ago, TWC started turning failed DNS lookups into websites generated by TWC. So if I type in appple.com and that site doesn't exist, it doesn't return a browser unknown host message, it returns a TWC web page showing "did you mean 'apple.com'?" and a bunch of ads.

I realize this isn't the same level of badness, but it's the same level of "we can make money innocently off customer mistakes" and I think it has the same potential for abuse.

If the Tea Party gets their way, the FCC will have no authority and the telcos will be even more abusive.If the Republicans get their way, the FCC will be sold to the telcos.If the Democrats get their way, any FCC action will languish in Congress forever and be gutted or abandoned.Not much hope, is there?

This describes our firm's experience with Frontier pretty much exactly. We started getting the CAPTCH for Google, and that's been fixed, but I was still getting the DNS redirects with Bing until last week. There's a link at the top of the redirected page to opt out, but it didn't work for me in repeated attempts. I contacted their support, which also did not help, until they escalated it due to my increasing anger. Now they have it turned off for our IP addresses at their end, and finally, everything is working fine. Generally speaking, though, it doesn't appear true that Frontier has stopped this in general, unless they did so within the past week or two.

I'd switch DNS providers, but Frontier's DNS servers have been pretty much bulletproof for years. And I don't feel any better about Google tracking me than I do whoever Frontier uses. I guess the answer is to just use root hints.

jfgilbert: At least if the tea party gets it's way they telcos will have to compete, which will keep their ambitions in check.

But let's remember that Comcast does this by default too. So I'd say that number is rediculously low. It redirects to their search page whenever you don't hit anything real. Really anoying so I use google's dns.

@sachmet - go to the Search Engine website (google.com or whatever) then do your search. Most modern Browsers have a secondary Search Field anyway (Safari / Firefox) or all-in-one like Chrome. So if you're in Chrome - maybe try out another Browser that sepereates these functions. The point is use something that specializes in Search for searching.

@slowcoffee - I have an old version of Chrome v5.0.xx - when I typed "arstecchnica.com" into the URL Bar 2 auto-complete options show up 1- the website for Ars and 2- an item to do a Google Search for the above search term.

I just downloaded a current version of Chrome and it still has the same "feature". I would say to simply be aware of the auto-complete list when it shows up - you may be inadvertently following the gogole search item instead of the direct website item.

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Anyone else experience this?

It's a 'feature' of Chrome. The URL bar is also the search bar. I think FF is trying to do the same thing with FF4, but as I don't use it I couldn't tell you for sure.

Is it just my impression or Microsoft really is working to help make the internet better?

I mean, their crusade against spammers, this piece of news, the security challenge (the one for 200k) they posted and some other news I've seen lately are looking like they have a goal in reaching a better environment for everyone.

Frontier is the worst ISP I've ever had and I'm not at all surprised that they're exploiting their users this way. they charge more than Time Warner for a service that's even worse, and their tech support is thoroughly useless. so's their billing department - we had a recurring issue where they would send a bill and then a shutoff notice two days later so it was impossible to pay before they shut off service, they refused to do anything about this, and I don't even know why they were sending shutoff notices in the first place because we paid on time every month and they still pulled this all the goddamn time.

I'd switch DNS providers, but Frontier's DNS servers have been pretty much bulletproof for years. And I don't feel any better about Google tracking me than I do whoever Frontier uses. I guess the answer is to just use root hints.

I generally use Level 3's DNS severs (4.2.2.2) when I don't trust the provided one. For my home connection I put a caching DNS server between it to decrease the amount of information going out and improve latency. For other places (public wifi, hotels, etc) I just use Level 3's directly. Unlike Google they aren't in the stalking targeted advertizing business, and with the amount of traffic they get they probably don't even keep standard logs for very long.

I run my own DNS server at home as well, but I know that my ISP, Brighthouse, does the search redirection thing as well. Basically, instead of getting a 404 you get dumped to their search page. While this is a good idea for users that can't understand how a browser or search engine works, it's really annoying to me. Also, their opt out method is cookie based, so if you clear your browser's cache, you're opted back in again. Hopefully this gets some legal scrutiny.

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Anyone else experience this?

It's a 'feature' of Chrome. The URL bar is also the search bar. I think FF is trying to do the same thing with FF4, but as I don't use it I couldn't tell you for sure.

Yeah, I get that it's a combined URL and search bar. But once "arstechnica.com" shows up in the bar, not just "arstechnica" I expect to go to the address, not to Google.Just a little too suspicious I guess.

I haven't used my ISP's DNS servers for years, but this is because they are notoriously unreliable. I used OpenDNS for a while, but got annoyed at getting dumped on the failure page since it meant you couldn't be sure whether the site was permanently dead, just temporarily down, or you spelled it wrong. I now use servers from Level3 (4.2.2.2 & 4.2.2.3) and haven't noticed any hijacking issues with them (so far.)

Sites that I access regularly get added to my "hosts" file so that they bypass DNS completely. This avoids hijacking as well as making them much faster. Unfortunately, it sometimes causes issues with HTTPS connections. It also bypasses server load leveling on big sites like Yahoo -- not a good thing.

Am I safe in assuming that using HTTPS to connect to Google shields me from intercepts?

I read the paper's abstract, and laughed. It's so full of terminology and buzz words, that I consider it a miracle that Ars was able to extract anything useful from it.

Quote:

When a user requests content from a cloud service provider, sometimes the content sent by the provider is modified inflight by third-party entities. To our knowl- edge, there is no comprehensive study that examines the extent and primary root causes of the content modifica- tion problem. We design a lightweight experiment and instrument a vast number of clients in the wild to make two additional DNS queries every day.

Cloud service provider? Inflight? In the wild? Lightweight?

With my graduate level CS came papers of my own, so I understand the language. That doesn't mean it's okay to use. Packets do not "fly," experiments do not have "weight," and where exactly is "in the wild" on the Internet? -- if it's "on the Internet" say that! Honestly, I still don't know what a "cloud service provider" specifically refers to since the term "cloud" is such total crap.

How about using regular old English terms, like "in transit" rather than "inflight" (or even "modified as it is transferred from malicious servers").

Orwell discusses this issue at length, and the (ab)use of language play strongly in both his and and Huxley's dystopian novels. Orwell discusses its abuse in academia specifically.

If the Tea Party gets their way, the FCC will have no authority and the telcos will be even more abusive.If the Republicans get their way, the FCC will be sold to the telcos.If the Democrats get their way, any FCC action will languish in Congress forever and be gutted or abandoned.Not much hope, is there?

I don't know which I like less, some ISP hijacking my address bar or a troll hijacking a comment page with some political agenda.

I'd switch DNS providers, but Frontier's DNS servers have been pretty much bulletproof for years. And I don't feel any better about Google tracking me than I do whoever Frontier uses. I guess the answer is to just use root hints.

I generally use Level 3's DNS severs (4.2.2.2) when I don't trust the provided one. For my home connection I put a caching DNS server between it to decrease the amount of information going out and improve latency. For other places (public wifi, hotels, etc) I just use Level 3's directly. Unlike Google they aren't in the stalking targeted advertizing business, and with the amount of traffic they get they probably don't even keep standard logs for very long.

Maybe I'm naïve, but why should people worry so much about their DNS server security? I understand that offering up ads due to DNS redirecting is bad, but why should people worry so much about the logs?

Yeah, I get that it's a combined URL and search bar. But once "arstechnica.com" shows up in the bar, not just "arstechnica" I expect to go to the address, not to Google.Just a little too suspicious I guess.

Once I add the .com, chrome takes me to the correct page.

And OpenDNS was fully forthcoming in their redirect to ad pages if the original host was not available. I haven't looked at the TOS, but years ago they made it clear that was how they made money to offer the free service. IIRC, the paid service now allows you to opt out of that.

Since the report isn't entirely clear, are they saying that ISPs are intercepting dns for the hostnames the browser is using to search? so www.google.com, search.yahoo.com, www.bing.com, etc? And then proxying and rewriting the results?

If so, this sounds way more evil than the travesty that was Network Solutions' Site Finder.

There is a checkbox to disable that if you wish. I use their stuff and disabled that functionality, although to be fair they do a pretty good job of using it to try and help the searcher, or did at least when I did have it enabled.

Maybe I'm naïve, but why should people worry so much about their DNS server security? I understand that offering up ads due to DNS redirecting is bad, but why should people worry so much about the logs?

Mostly tin foil hat on my part. Google knows so much about us between search, tracking cookies, etc. and now DNS gives them every site we visit. It makes me paranoid because of a general feeling of unease that they're interested in all this, moreso than any specific objection. It's sort of like if you thought your neighbor was showing undue interest in your family's comings and goings - probably harmless but creepy nevertheless.

I enabled tracking protection in IE9 on my home PC, and was astonished to see that just about ever site I visit has some kind of tracking enabled. That includes a lot of big news sites like CNN and NY Times where I didn't expect it. Admittedly the only affect I've noticed is that I don't get obviously targeted ads with tracking protection - for example, if I search for info about a router, I no longer get Newegg ads for routers on every page I visit.

@Pavon, thanks for the tip about Level 3 - I might give that a try, although I'm good with Frontier now that I complained enough for them to turn off the redirect.

Approximately a few months ago, TWC started turning failed DNS lookups into websites generated by TWC. So if I type in appple.com and that site doesn't exist, it doesn't return a browser unknown host message, it returns a TWC web page showing "did you mean 'apple.com'?" and a bunch of ads.

I realize this isn't the same level of badness, but it's the same level of "we can make money innocently off customer mistakes" and I think it has the same potential for abuse.

Same issue here, and I switched to Google's 8.8.8.8 DNS servers. I believe I set them in my router at home so everyone gets that as the DNS via DHCP.

The other sin you don't mention is that my address bar gets filled in with a different URL because it was redirected. That means I cannot simply go up and fix the typo (usually 1 letter).

Plus I already pay these clowns, and they are inconveniencing me to make even more money for themselves; reminds me of paying to watch commercials in movie theaters now.

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Anyone else experience this?

I tried this out right now on Google Chrome (13.0.782.107 beta-m) and it goes directly to arstechnica.com. For anything that looks like a URL (ends in a TLD like .com, .net, .org, etc), Chrome first attempts a DNS lookup and only if that fails will it pass the search term to Google Search.

With my graduate level CS came papers of my own, so I understand the language. That doesn't mean it's okay to use. Packets do not "fly," experiments do not have "weight," and where exactly is "in the wild" on the Internet? -- if it's "on the Internet" say that! Honestly, I still don't know what a "cloud service provider" specifically refers to since the term "cloud" is such total crap.-Pie

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Anyone else experience this?

It's a 'feature' of Chrome. The URL bar is also the search bar. I think FF is trying to do the same thing with FF4, but as I don't use it I couldn't tell you for sure.

Firefox does this too. Typing in "apple" takes me to a google for apple. Lame!

Safari and Camino (World's Best Browser, but needs a 64 bit version dangit!!!) both do the right thing and autofill. "Apple" will become "www.apple.com" and it's a boom with less typing, and with no INSANELY TIME CONSUMING redirect to stupid google!

Rogers in Canada (far from a little ISP, at least in relation to Canada) does this.

Misspell a domain name and you're brought to their own little "search" page.

I can confirm this. Rogers' DNS servers have proven unreliable so I have switched to Google DNS and a few other backup servers. I changed this at the router level so that all the computers in my network get consistent DNS replies.

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Anyone else experience this?

It's a 'feature' of Chrome. The URL bar is also the search bar. I think FF is trying to do the same thing with FF4, but as I don't use it I couldn't tell you for sure.

Firefox does this too. Typing in "apple" takes me to a google for apple. Lame!

Safari and Camino (World's Best Browser, but needs a 64 bit version dangit!!!) both do the right thing and autofill. "Apple" will become "www.apple.com" and it's a boom with less typing, and with no INSANELY TIME CONSUMING redirect to stupid google!

-Pie

FF used to do a location bar smart search (via Google search), but FF4 nixed it because some people felt it was too confusing and inconsistent. If you want it back, as I did, go to about:config, search for keyword.URL and copy/paste

Using Chrome, I would enter a web address, say arstechnica.com in the address bar, and instead of getting to.. arstechnica.com, I would be re-directed to Google. It's as if Chrome thinks I'm performing a search for the term "arstechnica.com", when I'm not. <foil hat> This could be Google's way of hijacking my URL and turning it into a "search". Thereby allowing more searches - and hence more ads to be displayed. </foil hat>

Anyone else experience this?

It's a 'feature' of Chrome. The URL bar is also the search bar. I think FF is trying to do the same thing with FF4, but as I don't use it I couldn't tell you for sure.

Firefox does this too. Typing in "apple" takes me to a google for apple. Lame!

Safari and Camino (World's Best Browser, but needs a 64 bit version dangit!!!) both do the right thing and autofill. "Apple" will become "www.apple.com" and it's a boom with less typing, and with no INSANELY TIME CONSUMING redirect to stupid google!

-Pie

FF used to do a location bar smart search (via Google search), but FF4 nixed it because some people felt it was too confusing and inconsistent. If you want it back, as I did, go to about:config, search for keyword.URL and copy/paste