As more brand dollars move to digital, and the technology supporting digital advertising evolves, publishers have a tremendous opportunity to boost their revenues and profits. However, capitalizing on this opportunity requires publishers to evolve their sales strategies, fast.

As programmatic buying continues to grow, one of the first steps publishers must take is to adopt a holistic cross-channel ad sales strategy.

A new op-ed by Paul Zwillenberg, Global Leader of The Boston Consulting Group’s Media Sector, explores this idea and shares the strategies and tactics employed by today’s most successful digital publishers.

Today the Trustworthy Accountability Group (TAG) announced a new pilot blacklist to protect advertisers across the industry. This blacklist comprises data-center IP addresses associated with non-human ad requests. We're happy to support this effort along with other industry leaders—Dstillery, Facebook, MediaMath, Quantcast, Rubicon Project, The Trade Desk, TubeMogul and Yahoo—and contribute our own data-center blacklist. As mentioned to Ad Age and in our recent call to action, we believe that if we work together we can raise the fraud-fighting bar for the whole industry.

Data-center traffic is one of many types of non-human or illegitimate ad traffic. The newly shared blacklist identifies web robots or “bots” that are being run in data centers but that avoid detection by the IAB/ABC International Spiders & Bots List. Well-behaved bots announce that they're bots as they surf the web by including a bot identifier in their declared User-Agent strings. The bots filtered by this new blacklist are different. They masquerade as human visitors by using User-Agent strings that are indistinguishable from those of typical web browsers.

In this post, we take a closer look at a few examples of data-center traffic to show why it’s so important to filter this traffic across the industry.

Impact of the data-center blacklist

When observing the traffic generated by the IP addresses in the newly shared blacklist, we found significantly distorted click metrics. In May of 2015 on DoubleClick Campaign Manager alone, we found the blacklist filtered 8.9% of all clicks. Without filtering these clicks from campaign metrics, advertiser click-through rates would have been incorrect and for some advertisers this error would have been very large.

Below is a plot that shows how much click-through rates in May would have been inflated across the most impacted of DoubleClick Campaign Manager’s larger advertisers.

Two examples of bad data-center traffic

There are two distinct types of invalid data-center traffic: where the intent is malicious and where the impact on advertisers is accidental. In this section we consider two interesting examples where we’ve observed traffic that was likely generated with malicious intent.

Publishers use many different strategies to increase the traffic to their sites. Unfortunately, some are willing to use any means necessary to do so. In our investigations we’ve seen instances where publishers have been running software tools in data centers to intentionally mislead advertisers with fake impressions and fake clicks.

First example

UrlSpirit is just one example of software that some unscrupulous publishers have been using to collaboratively drive automated traffic to their websites. Participating publishers install the UrlSpirit application on Windows machines and they each submit up to three URLs through the application’s interface. Submitted URLs are then distributed to other installed instances of the application, where Internet Explorer is used to automatically visit the list of target URLs. Publishers who have not installed the application can also leverage the network of installations by paying a fee.

At the end of May more than 82% of the UrlSpirit installations were being run on machines in data centers. There were more than 6,500 data-center installations of UrlSpirit, with each data-center installation running in a separate virtual machine. In aggregate, the data-center installations of UrlSpirit were generating a monthly rate of at least half a billion ad requests— an average of 2,500 fraudulent ad requests per installation per day.

Second Example

HitLeap is another example of software that some publishers are using to collaboratively drive automated traffic to their websites. The software also runs on Windows machines, and each instance uses the Chromium Embedded Framework to automatically browse the websites of participating publishers—rather than using Internet Explorer.

Before publishers can use the network of installations to drive traffic to their websites, they need browsing minutes. Participating publishers earn browsing minutes by running the application on their computers. Alternatively, they can simply buy browsing minutes—with bundles starting at $9 for 10,000 minutes or up to 1,000,000 minutes for $625.

Publishers can specify as many target URLs as they like. The number of visits they receive from the network of installations is a function of how long they want the network of bots to spend on their sites. For example, ten browsing minutes will get a publisher five visits if the publisher requests two-minute visit durations.

In mid-June, at least 4,800 HitLeap installations were being run in virtual machines in data centers, with a unique IP associated with each HitLeap installation. The data-center installations of HitLeap made up 16% of the total HitLeap network, which was substantially larger than the UrlSpirit network.

In aggregate the data-center installations of HitLeap were generating a monthly rate of at least a billion fraudulent ad requests—or an average of 1,600 ad requests per installation per day.

Not only were these publishers collectively responsible for billions of automated ad requests, but their websites were also often extremely deceptive. For example, of the top ten webpages visited by HitLeap bots in June, nine of these included hidden ad slots -- meaning that not only was the traffic fake, but the ads couldn’t have been seen even if they had been legitimate human visitors.

http://vedgre.com/7/gg.html is illustrative of these nine webpages with hidden ad slots. The webpage has no visible content other than a single 300×250px ad. This visible ad is actually in a 300×250px iframe that includes two ads, the second of which is hidden. Additionally, there are also twenty-seven 0×0px hidden iframes on this page with each hidden iframe including two ad slots. In total there are fifty-five hidden ads on this page and one visible ad. Finally, the ads served on http://vedgre.com/7/gg.html appear to advertisers as though they have been served on legitimate websites like indiatimes.com, scotsman.com, autotrader.co.uk, allrecipes.com, dictionary.com and nypost.com, because the tags used on http://vedgre.com/7/gg.html to request the ad creatives have been deliberately spoofed.

An example of collateral damage

Unlike the traffic described above, there is also automated data-center traffic that impacts advertising campaigns but that hasn’t been generated for malicious purposes. An interesting example of this is an advertising competitive intelligence company that is generating a large volume of undeclared non-human traffic.

This company uses bots to scrape the web to find out which ad creatives are being served on which websites and at what scale. The company’s scrapers also click ad creatives to analyze the landing page destinations. To provide its clients with the most accurate possible intelligence, this company’s scrapers operate at extraordinary scale and they also do so without including bot identifiers in their User-Agent strings.

While the aim of this company is not to cause advertisers to pay for fake traffic, the company’s scrapers do waste advertiser spend. They not only generate non-human impressions; they also distort the metrics that advertisers use to evaluate campaign performance—in particular, click metrics. Looking at the data across DoubleClick Campaign Manager this company’s scrapers were responsible for 65% of the automated data-center clicks recorded in the month of May.

Going forward

Google has always invested to prevent this and other types of invalid traffic from entering our ad platforms. By contributing our data-center blacklist to TAG, we hope to help others in the industry protect themselves.

We’re excited by the collaborative spirit we’ve seen working with other industry leaders on this initiative. This is an important, early step toward tackling fraudulent and illegitimate inventory across the industry and we look forward to sharing more in the future. By pooling our collective efforts and working with industry bodies, we can create strong defenses against those looking to take advantage of our ecosystem. We look forward to working with the TAG Anti-fraud working group to turn this pilot program into an industry-wide tool.

Publishers’ growth in programmatic revenue is outpacing traditional direct sales for desktop and mobile across display and video advertising. New technologies like “programmatic guaranteed” are further blurring the lines between direct and programmatic channels.

A new study by The Boston Consulting Group, commissioned by Google, found that despite this trend, many publishers are failing to appropriately capitalize on the programmatic opportunity. For example, the study found that less than 25 percent of programmatic team time is spent on value-creating activities, causing publishers to miss out on significant revenues.

The study also closely analyzed the operations of those publishers that consistently outperform their peers in terms of value creation and efficiency, and arrived at best practices and approaches other publishers can follow to achieve similar success. Using simpler ad tech stack configurations, best in class publishers were on average 30% more efficient, had up to 24% higher CPMs, and delivered 10% more impressions otherwise lost to discrepancies.

Last week at the DoubleClick Leadership Summit, we announced the availability of Native Ads for Apps on our DoubleClick platforms. In this post, we’ll dive into the details of this new ad format and what it means for our clients.

The mobile revolution has changed the way we engage with content. We check our phones literally hundreds of times a day: to catch up with friends and family, read an article, or watch a video while waiting in line. In these moments, we believe ads have the best chance to be effective when they are placed with respect to a user’s context.

Imagine an ad that pops up in the middle of your game; it would be incredibly disruptive. But, one that appears between levels would feel more natural. Or, an ad that blocks your feed as you scroll through; that would be annoying. But, one that’s stitched within the feed would be almost expected. What’s necessary in today’s environment is native advertising—advertising that’s clearly marked and complements the user experience.

Introducing Native Ads on DoubleClick

Native ads fit in with the look and feel of publisher content, enabling better, more effective ad experiences for users. Context is incredibly important on mobile, and that’s why over the next few weeks we’re rolling out our native ad solution for apps to DoubleClick for Publishers clients globally.

Native ads for apps in DFP provides publishers with the full flexibility needed to create seamless ad experiences for their users. Instead of serving a static banner ad, DFP delivers ad components (headline, image, links, etc) to a publisher’s app where they’re rendered into a native ad defined by the developer’s code. By providing the building blocks of an ad, our native solution allows a publisher to create ads that are seamless with content, can take advantage of mobile features like swipe gestures and 3D animation, and can be adjusted to create beautiful ads for any device or screen size.

Setting up native ads for apps in DoubleClick for Publishers is easy. Publishers can choose from two of the most popular mobile formats, app install ads or content ads, or create fully custom native ads by including any additional fields for DFP to send to their app. In addition, publishers using our standard native ad formats can maximize revenue by accessing demand from our native ads beta in DoubleClick Ad Exchange.

Of course, it’s essential that publishers clearly mark native ads as advertising. Ads that trick users into clicking or are indistinguishable from content are bad for the whole ecosystem, including users, advertisers, and publishers.

Native experiences are essential on mobile

When users pick up their phones, it’s critical that they’re presented with a seamless ad experience. With native ads in DFP, publishers can maintain a beautiful user experience in their apps while providing brands an opportunity to reach their audience on mobile. Advertisers should reach out to their publisher partners to find out how they can use native ads to connect with their customers and reach them when they’re most receptive.

To learn more about native ads in DoubleClick, check out our help center or, reach out to your account manager today. Also, visit the mobile solutions section of our website to see how DoubleClick can help you engage your audience on every screen.

As Neal Mohan announced last week at the DoubleClick Leadership Summit, Marketplace in DoubleClick Ad Exchange is now available to all customers globally, and we’re working to bring Programmatic Guaranteed to more publishers as soon as possible.

Today, the biggest brands and agencies are increasingly running premium digital campaigns across apps, videos, and native formats using programmatic technology, and we’re seeing this shift reflected across our platforms. Over the last year, the overall volume of programmatic transactions on our systems has grown 59%, while programmatic direct deals have jumped 2X. Today, eight of our top 25 publishers are selling at least 10% of their ad inventory through direct programmatic deals.

The growth of programmatic is just part of the story. Programmatic Direct deals, the channel of choice for many premium publishers, are also driving higher inventory prices. Preferred Deals and Private Auctions are generating CPMs that are double or triple what publishers see in the open auction.

Introducing Marketplace in DoubleClick Ad Exchange

We see tremendous value in Programmatic Direct but, finding and connecting with all potential advertisers looking for premium offers is a challenge. That’s why we developed Marketplace, an easy to use interface for Ad Exchange buyers to discover, negotiate, and manage deals with the world’s best apps, sites, and properties. For publishers, Marketplace is where their brand is showcased through a customizable publisher profile, and where their programmatic direct offers are discoverable by programmatic buyers globally.

Marketplace in DoubleClick Ad Exchange is just the beginning of how we see Programmatic sales evolving in the future. The programmatic buying trend shows no sign of slowing and, we believe even more premium deals will happen through programmatic channels. That’s why, over the rest of this year we’ll be rolling out a brand new way to transact through DoubleClick: Programmatic Guaranteed.

Blending Direct with Programmatic Sales

Programmatic Guaranteed allows publishers to offer their reserved, premium inventory via a new programmatic channel in DoubleClick for Publishers, and provide brands an opportunity to buy reservations in a more efficient way. Publishers can lock in revenue, while giving advertisers guaranteed access to premium inventory with programmatic targeting and frequency management. It simplifies the workflow of a guaranteed deal, cutting down the steps it takes to implement, from 40 steps to 4. And the best part, it does this all through our Real-Time Bidding infrastructure.

In our pilot testing with DoubleClick Bid Manager, Programmatic Guaranteed deals have been creating tremendous value. We’ve seen CPMs at 15-times open auction prices - on par with upfront or reserved campaigns. But in the future, we see incredible new opportunities for Programmatic Guaranteed. Since our solution utilizes Real-Time Bidding, instead of just automating line item booking in the ad server, we can open up innovative new deal types to give publishers enhanced flexibility that truly blends direct and programmatic capabilities.

We’re excited about the future of programmatic buying and selling and the possibilities it will bring for all of our partners. If you’re interested in learning more about Marketplace in DoubleClick Ad Exchange reach out to your account manager today, and stay tuned as we look to expand our pilot of Programmatic Guaranteed to more buyers.

This is the first announcement in our post-DLS series. Join us over the next week as we release more details of all our recent product announcements. Next up, Native Ads in DoubleClick

This morning, we’re hosting the DoubleClick Leadership Summit in Key Biscayne, Florida. This marks the 15th year that we’ve been bringing together our large advertiser, agency and publisher partners to discuss the future of digital marketing. And we have a lot to discuss. As we convene today, we’re in the midst of one of the largest shifts in consumer behavior ever, thanks to the rise of mobile devices.

We no longer “go online” but instead, engage in many small moments throughout the day, reflexively turning to the nearest device to solve an immediate need. In these moments, we expect the right answer and we expect it right away. In these I-want-to-know, I-want-to-go, I-want-to-do, I-want-to-buy moments decisions are made and preferences shaped. Brands that are there and relevant for consumers in these moments that matter win.

Today, we’re announcing improvements to our DoubleClick products for advertisers and publishers designed to help them thrive in this moments-driven world. Here is a preview of what we’ll be announcing; you can also join us for the live stream, starting at 9 a.m. ET.

Today, the decision to make a purchase rarely starts and ends on the same device. For example you may first see an ad for a pair of running shoes while scrolling through the news at work, then actually decide to buy them a few days later when you realize your old ones need replacing.

For almost two years, we’ve been investing in solutions that help marketers measure these cross-device conversions in AdWords. And the results have been amazing: marketers are finding that they see up to 16% more conversions from search when they look across devices and take the cross-screen journey into account. Check out our AdWords blog for some in-depth research on that front.

Now, cross-device measurement is coming to all of our DoubleClick advertiser products, so marketers can measure conversions that start as a click on one device and end with a conversion on another for all their campaigns across the web, not just with the ads they buy from Google.

Cross-device measurement is already generating important insights for our clients. Here is what we heard from John Gray, SVP of Platforms & Partnerships at Team Detroit, a WPP agency:

“Cross-device measurement is a big step towards having metrics better represent what's actually happening with consumers in today's mobile-first world. This is essential insight for marketers. It not only allows us to achieve our clients' goals more efficiently, but helps us better understand the consumer journey, and reach them with the right messages as they take that path. In our early cross-device measurement tests, we're already seeing a material increase in conversions attributable to mobile, and making changes as a result.”

Support for native ads across DoubleClick

As consumers spend more and more time on their mobile phones, providing ad experiences that respect their context are more important than ever--for example an ad that shows up between levels of a mobile game, rather than disrupting the game itself. Publishers are embracing this change and introducing different types of “native” ad units that are designed to work within their environment, but it’s been a challenge for marketers and publishers to create and deliver these ads at scale. So we’re announcing support for native ads in DoubleClick.

With this new solution, publishers can create their own clearly labeled custom native formats, and decide what ads on their apps should look like. DoubleClick then automatically generates the ads, serves them at scale, and provides metrics to fine tune those native ad campaigns. We will also provide standardized native templates that work across publishers at scale to make native ad buying easy for advertisers.

Introducing Programmatic Guaranteed

In the early days of programmatic, it was all about direct response advertisers, buying and selling banner ads in an open auction. Today, however, programmatic is relied upon by major brands and premium publishers, across apps, mobile web, video and now native formats.

Along with this expansion, we’re seeing that direct deals, those favored by brands and large publishers for premium inventory, are fast on the rise. Just across our own platforms, the volume of programmatic direct transactions has jumped 2x in the past year and eight of our top 25 publishers are now selling at least 10 percent of their impressions via programmatic direct.

To reflect this new reality, we’re introducing a new way of buying: Programmatic Guaranteed--guaranteed inventory, sold programmatically at pre-negotiated rates.

The early numbers show that this is truly premium inventory, on par with reservations--with CPMs 15x the open market rate.

After all, connection is what we’re all about: connecting advertisers and publishers, brands and consumers, messages and moments. We’re excited about the updates announced today and can’t wait to see how marketers and publishers use these new tools to own THEIR moment.

Posted by Neal Mohan, Vice President, Video & Display Advertising

Check back next week for our post-DLS blog series where we’ll dive deeper into each of these announcements.