When Monetizing ISP Traffic Goes Horribly Wrong

In seeking to further monetize Web site traffic on their networks, a number of major Internet service providers may be inadvertently exposing their customers to a greater risk of online attack from identity thieves, according to research released today.

Many ISPs have already adopted the controversial practice of serving advertisements when a customer tries to browse to a Web site that does not exist. But a growing number of providers also are serving ad-filled pages when customers request a subdomain of a Web site that does not exist, such as something.example.com. This practice, which experts say potentially introduces new copyright violation claims, also potentially introduces security threats when ISPs outsource the ad-serving process to third parties.

The findings come from Dan Kaminksy and Jason Larsen, security researchers from IOActive, a security company based in Seattle, the site of the Toorcon hacker conference where the two are expected to unveil their research today. Update, 3:52 p.m. ET: The slides from their talk can be found here.

According to the duo, ISPs like Earthlink, Qwest and Verizon have outsourced at least portions of their ad-serving technology to BareFruit, a London-based company that specializes in helping ISPs monetize wayward Web searches. The trouble is that until late this week, BareFruit's ad servers were vulnerable to what Kaminsky called a "trivial to find and exploit" vulnerability that would make it simple for fraudsters to trick users of those ISPs into visiting malicious Web sites that appear to be located at trusted sites.

So, for example, the customer clicks on a link like http://something.example.com, and while that link would indeed load the "example.com" site in the user's browser, the vulnerability would allow fraudsters to load hostile content from another site into the user's browser, such as a fake login page.

Kaminsky and Larsen also found they could use the vulnerability to steal cookies on the user's machine. Cookies are simple text files that many sites store on visitors' machines to record information that identifies the user when they return. By swiping someone else's cookies, it is often possible to log in as that victim at the Web site that issued the cookie.

The researchers said the vulnerability that allowed this kind of access to ISP users resulted from a simple cross-site scripting (XSS) flaw in the Barefruit service. Cross-site scripting vulnerabilities occur when Web sites accept input from the user -- usually from something like a search box or an e-mail form -- but do not properly filter that input to strip out or disallow potentially malicious code. The danger is that phishers and online scammers can exploit these types of flaws to make their scams appear more legitimate, because XSS vulnerabilities allow the attacker to force the target site to load content from somewhere else.

Kaminsky, widely considered one of the foremost experts on the security of the domain name system (DNS, as it's more commonly called, is the method by which Web site names are mapped to numeric Internet addresses), said he discovered ISPs were using Barefruit by mapping DNS requests from BareFruit's servers back to residential customers at various providers. He said the discovery disturbed him because it means many ISPs are placing the security and privacy of their customers squarely in the hands of a third-party ad company.

"This kind of practice means the security of the Web is being limited to the security of this ad server," Kaminsky told Security Fix on Friday. "My work is to secure the Web and other computer infrastructure, but this becomes near impossible when other people are injecting content into domains that I am professionally trying to secure. I can audit every single line of code in the browser and in the Web site, and I still have no idea what the Web site is going to send the browser because who knows what's going to make it through all those devices?"

BareFruit spokesman Dave Roberts said the company fixed the vulnerability this week after receiving word from the IOActive researchers. But the ISPs alleged to be engaging in this process were a bit more cagey about acknowledging their use of BareFruit to monetize traffic to nonexistent domains and subdomains.

Earthlink declined to make someone available to discuss the company's practices on this front, but it did acknowledge that it uses BareFruit's DNS error functionality "to enhance our users' experience," the company said in a statement e-mailed to Security Fix. "We believe that the service provides a positive experience for our Internet users. We continue to watch our system closely, quickly resolve any issues that occur, and listen to what our customers tell us about their online experience."

Cox Communications spokesman David Grubert said the company uses BareFruit through its partnership with search engine giant Yahoo. Grubert said the company currently does not use BareFruit to inject ads when customers request nonexistent subdomains, but that the company was considering implementing that feature in the future.

Kaminsky presented Security Fix pages of records showing numerous Verizon DSL customers being redirected through BareFruit's Web servers. Verizon spokesman Eric Rabe said that while the company does use Yahoo to monetize traffic for nonexistent and subdomain errors, he was emphatic that Verizon does not use BareFruit's service.

Registrar and hosting provider Network Solutions also has acknowledged that it also serves ads on nonexistent subdomains that its customers own. Qwest officials did not return calls seeking comment.

John R. Levine, author of The Internet for Dummies, said Internet users -- at least here in the United States -- can expect to be exposed to more vulnerabilities such as those highlighted by Kaminsky and Larsen, as long as ISPs continue to be given so much leeway with the privacy and security of their customers.

"Large ISPs tend to have terms of service that say whatever we give you is what you bought," Levine said. "The ISPs will say they're doing wonderful favors for users who might have to otherwise go back and type in the real name of the site they're seeking. But the reality is that anytime ISPs add yet another level of complexity to their networks, they necessarily introduce more security bugs."

Kaminsky said the practice of subdomain DNS error hijacking is partly illustrative of what he calls the "Times Square effect:" The ads shown in movie and TV depictions of all the blinking digital billboards in Times Square often are paid for and arranged in advance by advertisers, and don't necessarily reflect the same ads that an average visitor to the physical Times Square might see at any given day or time.

"There's no contractual obligation between, say Earthlink and washingtonpost.com to deliver content in a certain way, and theoretically trademark and copyright law is the only force that prevents [ISPs] from putting in whatever material they want, from adding or removing content to rejecting or replacing ads that were already on the site," Kaminsky said. "What we're seeing here is this first instance, trivially, of the Times Square effect coming into play, where there's no obligation to display content of various trademarked sites in a particular way. And as a side effect, it makes it more difficult to secure the Web when this kind of behavior takes place."

Bret A. Fausett, an intellectual property attorney and blogger at Cathcart, Collins & Kneafsey LLP in Los Angeles, said it's tough to say ISPs are breaking the law when they place their own ads on sites that are for all intents and purposes otherwise owned by companies with a trademark claim to those domains. But that doesn't mean ISPs are legally invulnerable to potential trademark infringement claims for this practice.

"If someone wants to go to Amazon.com and [the ISP serves] something for which there is no [DNS] record configured and the ISP captures that and throws up an ad for a competitor while the browser says I am at Amazon.com, I could make a trademark argument on that, sure," Fausett said.

Most ISPs that use BareFruit's service - either for domain or subdomain DNS errors - redirect the customer to a site that clearly explains that the requested domain was not found. But as long as these types of vulnerabilities are around, ISPs can not effectively control what their customers see in the address field of their browser, Kaminsky said.

"Indeed, it is our sense that the HTTP web becomes insecurable if man-in-the-middle attacks are monetized by providers -- if we don't know what bits are going to reach the client, how can we control for flaws in those bits?" Kaminsky said.

Most Internet providers that hijack errant DNS queries from customers say the service is "opt-out," in that customers can disable the service if they like. Web site owners can create what's known as a "wildcard" DNS 'A' record for their domains, which can be assigned so that any unrecognized subdomains requested by the visitor result in the user being routed to the main Web site. The DNS redirection services being employed by ISPs and hosting providers only work on sites that have not included these "A records."

Just a quick comment: I've spent a lot of time working with large organizations, and it's *really hard* to know what everyone is doing, everywhere. So, I could totally see most of Verizon's network being on a different ad injector than Barefruit, and some people not realizing there's a branch using Barefruit. It happens. It's also possible that Verizon has subdivided a chunk of their network out to a different ISP, who is themselves using Verizon but hasn't updated Reverse DNS. I think that's unlikely but not completely impossible. What's without question is that Verizon is creating counterfeit DNS records in other company's domains, and that this limits the security of those domains to the security of some external ad server. That's problematic, and almost guaranteed to be unintentional. After all, the error is NXDOMAIN -- No Such Domain.

I worked for a small company last year and noticed that Network Solutions did this with our nonexistent subdomains. I called Network Solutions up and asked them to quit doing this with our domain. The first-level tech claimed it probably couldn't be turned off but he passed my request upward and they turned it off.

you know, i just noticed that "error" page recently with cox and was wondering what they were doing. kinda pisses me off, i don't need my ISP deciding how an incorrect URL typed in is redirected. that should be done by the site i'm attempting to get to.

That's a great idea. There's a freeway down the street. All I have to do is break a hole in the sound wall and put in my own offramp. Then I can catch people who think they're going somewhere else and sell them counterfeit goods at my deceptively branded stores. I don't want to run them myself, so I'll farm that out to the lowest bidder. Don't worry though, I'll keep an eye on things. No need to send a patrol by, sheriff. Your bribe will be at the usual rate.

It's disgusting how earthlink frames their use of Barefruit as "enhancing their user's experience". Couldn't they just cut the PR bullsquat and call it what it really is... another source of revenue for their company.
I use them and have always found it as annoying as could be and would switch ISP's if it were an option.

Maybe HTTP should have an UNGET request, or better yet, the FTC should have a "Safe Harbor" page which collects referring page URLs ... like forwarding SPAM to the FTC, this would designate the (previous) referring page as a badly served request. ISP's want revenue, but no supplier of goods and services wants their name on a FTC top ten list of pushed ads.

I inadvertently solved this issue for myself out of disgust with how slowly sometimes my ISP's DNS servers were: I run my own copy of bind9. Any names that can't be resolved on my own server are promoted up to the Internet root servers, NOT to my ISP's DNS servers. Result? Speedy response that I can rely on. The ONLY service I want from my ISP is a clean connection to the highway. None of their content or other services is of any interest to me.

The funny thing about the opt-out is that it includes some ID number in the URL of the opt-out link which uniquely identifies your account (someone mentioned this on their blog). This doesn't sound very secure.

(It also by default turns on the "adult" content filter for search engines.)