Posted by Tom SatherNovember 17, 2009

As discussed in my previous post, email providers have long used engagement metrics to determine email reputation and deliverability. These metrics include “This is Spam” (aka TiS) data, “This is Not Spam” (aka TiNS) data, panel data, Trusted Reporter data, and Inactive Account data. In this post, we’re going to focus on the largely unknown, yet highly effective use of panel data in Microsoft’s anti-spam filtering technologies.

Panelists, or Windows Live Sender Reputation data as Microsoft calls them, are a small group of Windows Live Hotmail users that come from every corner of the world, and classify up to 300,000 messages per day as “junk” or “not junk.” These users are selected at random (sorry, invite only!) and can come from any type of email service that Microsoft offers, like Windows Live or MSN. These panelists are actual subscribers on your lists and are asked to evaluate messages that either have landed in the spam folder or in the inbox and determine if it’s something they expected to receive or not.

Microsoft’s use of panelist data helps with sender reputation scoring in its own anti-spam filtering product, SmartScreen, as well as Return Path’s Certified product. Microsoft factors panelists’ votes to determine a sender’s reputation. Their decisions can negatively or positively affect your email deliverability. Let’s say, like in this example for Coach, that most of your email gets delivered to the spam folder. Randomly selected panelists, who are also subscribers in your program, are asked if your message is something they expected to receive in their inbox, or something that they didn’t expect to receive. If a majority of panelists vote “Not junk e-mail,” your chances of reaching the inbox are greatly increased to prevent a false positive. As a result, panelist data had a direct positive impact for your email delivery.

The same holds true for the inverse. If it lands in the inbox and most people respond by saying it’s a message that they didn’t expect to receive, like this message which is clearly spam, future messages will more likely be delivered to the spam folder. False negative averted!

Microsoft uses panelists for training SmartScreen as it’s more reliable than Bayesian content filters or even “This is Spam” and “This is Not Spam” votes. What’s more, it’s a system that can’t be gamed or abused. The message from Microsoft is clear. You can jump IPs, hop domains, try to manipulate complaint data, but you still can’t hide from panelists. Since these users are not known, are selected randomly from Microsoft’s most active users from around the world, and are given messages at random to vote on (without consideration of the sending domain or IP), the chances of changing the outcome of this filtering process are highly improbable. Additionally, panelists can only vote on a message when it was received within 24 hours, so the timing of the send and complaint can’t negatively impact the sender’s rating, something that frequently happens with other feedback loops.

Marketers can access this data in two ways. The first is through Return Path Certified as it is part of the compliance checks to remain in the program. The second way is through Microsoft’s Smart Network Data Services (SNDS) through its filter results. While they don’t display the actual panelist votes, they do have a color coded result (green, yellow, and red) which is based off of their SmartScreen filtering product. SmartScreen is widely used in most of Microsoft’s email products ranging from their free webmail services, Exchange, Hosted Exchange, as well as Outlook. SmartScreen looks at things like a properly configured mailing infrastructure, authentication, content, spam traps, unknown users, complaints, as well as panelist data. If the data you have access to is showing that you are fully within compliance at all of these metrics, and your filter result is red, you most likely have a problem with panelist votes.

What can you do to make sure that panelists recognize your email as wanted and vote you into the inbox? In Part 2 of this post my colleague Bonnie Malone will discuss strategies to help.

About Tom Sather

Email data and deliverability expert Tom Sather has worked with top-tier brands to diagnose and solve inbox placement and sender reputation issues as a strategic consultant with Return Path. As the company’s senior director of research, Tom is a frequent speaker and writer on email marketing trends and technology. His most recent analysis of new inbox applications’ effects on consumer behavior was widely cited across leading business media outlets including the Financial Times, Ad Age, and Media Post.