CircleID: Emailhttp://www.circleid.com/topics/
Latest Email related postings on CircleIDenCopyright 2015, unless where otherwise noted.2015-03-03T10:12:01-08:00CircleID13045http://www.circleid.com/images/logo_rss.gifhttp://www.circleid.com/
Who Is Sending Email As Your Company?http://www.circleid.com/posts/20150115_who_is_sending_email_as_your_company/http://www.circleid.com/posts/20150115_who_is_sending_email_as_your_company/
You might expect that the IT department or security team knows who's sending email using your company's domains. But for a variety of reasons these groups are often unaware of many legitimate senders — not to mention all the bad actors. Fortunately you can get a more complete view by using DMARC's reporting features.

How does it happen? Product teams managing a new product launch or customer survey hire marketing consultants and Email Service Providers (ESP). Affiliate programs or strategic partnerships lead to new domains or sub-domains being created. Employee benefit programs are outsourced, and the vendor wants to use a sending address in your domain. All too often these things are done quietly as part of a small project, without consulting anybody in another department or division.

And then there are all the bad actors using your domains without asking permission…

Many times the IT or security groups can see some of what's happening because of the "backscatter." These could be customer complaints sent to the company's abuse mailbox, bounce notifications from forged messages, or incoming messages getting flagged by your company's email filtering products. But frequently these indications are like an iceberg, and only show 10% of what's actually going on. Fortunately you can use DMARC to uncover the other 90%.

This is what we did at one global financial institution. As we tracked down all the domains that we owned and published DMARC records for them, we were able to see all the sources of email using those domains: corporate gateways, subsidiaries and partner firms, ESPs. In fact in one case we were able to see that messages sent from one host at one ESP were failing to authenticate properly 5-10% of the time. That may not seem like a lot, but this host was sending around 100,000 messages per day, so those failures represented a lot of customers who weren't getting important notifications about their accounts and transactions.

Then there were all the mailbox providers and university alumni programs that were receiving messages we sent to a customer at one address, and forwarding them on to a different destination. And the random news, image hosting and social media sites that were sending email using our domains in response to employee activity. Beyond these fairly benign uses, there were literally hundreds of thousands of random PCs sending spam and phishing messages that used our domains.

And now we could see them. Not only could we monitor our authorized vendors, we could quantify the unauthorized messages using our domains, and see what impact it would have if we published a DMARC policy that would block them. It does take a little work to make use of DMARC this way, and the reports only reflect messages received at 80+% of consumer mailboxes in the US, or 60+% worldwide. But that's a night-and-day difference compared to what we could see before DMARC.

To start receiving DMARC reports you'll have to publish a few DNS records for each domain. An email address and mailbox to receive the reports is necessary, as well as a program to process the XML files. But the report format is documented (seeAppendix C) and there's free software available to help you get started (seeCode & Libraries). There are also a number of free and commercial services that will process reports for you, and most of those companies can be hired to provide assistance in all aspects of deploying DMARC if you choose (seeProducts & Services).

For links to tutorials, and a series of videos explaining DMARC provided by M³AAWG, see the listings at DMARC.org.

]]>2015-01-15T17:38:00-08:00internetemailspamWhen DNSBLs Go Badhttp://www.circleid.com/posts/20150113_when_dnsbls_go_bad/http://www.circleid.com/posts/20150113_when_dnsbls_go_bad/
I have often remarked that any fool can run a DNS-Based Blacklist (DNSBL) and many fools do so. Since approximately nobody uses the incompetently run black lists, they don't matter. Unfortunately, using a DNSBL requires equally little expertise, which becomes a problem when an operator wants to shut down a list.

When someone sets up a mail server (which we'll call an MTA for Mail Transfer Agent), one of the tasks is to configure the anti-spam features, which invariably involves using DNSBLs. Some software (notably Spamassassin) comes with DNSBLs pre-configured, some require you to choose the ones you're going to use.

Since most MTA admins are not mail experts, they ask around, find a set of DNSBLs their friends use, put them in the configuration, and forget about it unless something goes so badly wrong that they have to fix it. Once they're set up, the MTA sends out DNSBL queries for the sending IP address of every incoming message. Depending on the way the MTA is set up, if a DNSBL lists an IP address, the MTA may outright reject the message, or it may use the DNSBL result as part of a scoring system.

The problem arises when one of the DNSBLs wants to shut down. Running one competently is a lot of work, and it's not surprising that people run out of time, retire, or otherwise can't do it any more. The goal is to get all of the people who are querying the DNSBL to stop. How do you do that?

The usual first approach is to notify everyone you've told about the DNSBL, send mail to lists about mail management, and so forth. Then you list nothing, return NXDOMAIN to every request, for a while. This will cause a small drop in query volume, since none of those systems on autopilot will notice or care.

Eventually desperation sets in, as the queries just keep rolling in. I run a small DNSBL (korea.services.net) and although I have no plans to shut it down, I can see from the query logs that some system admins can't type, and are querying kroea.services.net, or in one case http://korea.services.net, with a colon and slashes in the domain name. I would prefer that if they want to query my DNSBL, they query the actual DNSBL, so I've tried various techniques to get their attention. I've introduced very long delays, or used DNS delegation to point to name servers that don't exist on networks that don't exist. These should cause very slow responses on the mail servers, which you'd think that someone might notice and investigate. Nope.

The final step is to put in a wildcard to list the world, answer every query saying that the address is listed. This is somewhat controversial. RFC 6471 on Overview of Best Email DNS-Based List (DNSBL) Operational Practices says not to do it, on the theory that it will cause needless disruption. I used to agree, but now I'm thinking that we need to force those dusty mail systems to fix their software so they'll automatically stop using dead DNSBLs. I put in wildcards for those misspelled korea names several years ago, and also returned deliberately insulting TXT messages that might appear in their logs, e.g. "your mail has been rejected due to sheer incompetence." The abandonware is hammering as hard as ever.

The AHBL list, a widely used list for over a decade, was shut down last year. Predictably, the queries continued to hammer on their web server even though nobody was getting any useful responses, so in exasperation they listed the world last week to try and get the queries to stop. As far as I know, it didn't help much either.

The solution is to automate the client side of shutting down a DNSBL. It is not hard to tell whether a DNSBL is operating. By long tradition, every list contains a test entry for address 127.0.0.2, and does not have an entry for 127.0.0.1. At least since I wrote RFC 5782, on DNS Blacklists and Whitelists, a similar convention for DNSBLs that handle domain names rather than addresses is that the name TEST is listed and INVALID is not.

A client MTA can check the two addresses, and if it gets the desired response and not the undesired one, the list is operating. If it gets neither or it gets both, the list is dead. This catches both DNSBLs that are shut down and don't respond at all, DNSBLs that deliberately list the world, and the common case that a list is abandoned, its domain name registration expires, and is then picked up by a speculator that sets up wildcard DNS pointing to a "for sale" page.

The wildcard looks enough like listing the world to confuse a lot of people. For example, I recently saw complaints about this page which claims that a provider uses defunct DNSBLs called webmail.rhs.mailpolice.com and dynamic.rhs.mailpolice.com. The mailpolice.com domain is owned by a speculator with wildcard DNS and a for-sale page. A few moments' thought should confirm that nobody can be using it to block incoming mail, since if they were, they'd get no incoming mail at all. Nonetheless, someone was sure that they were listed and it needed to be fixed, now.

What needs to be fixed is all those abandonware MTAs. (I realize this is not going to be easy.) If spam filters checked their DNSBLs once a week, and didn't use the ones that failed the check, that would both relieve the load on the nameservers of dead DNSBLs, and it would make the MTAs work better and faster, since they would not be wasting time asking DNSBL questions to which they will never get useful answers. I suppose I can start by sending in patches for Spamassassin.

]]>2015-01-13T23:45:00-08:00internetemailspamDave Crocker and John Levine Discuss Current Dealings With Spam (Video)http://www.circleid.com/posts/20141230_dave_crocker_and_john_levine_discuss_current_dealings_with_spam/http://www.circleid.com/posts/20141230_dave_crocker_and_john_levine_discuss_current_dealings_with_spam/
During the M3AAWG meeting in Brussels earlier this year, Dave Crocker and John Levine were asked to step into an impromptu video studio and talk about how email has changed over the past several decades and whether we are any closer to resolving the spam problem. Discussion begins at :48.

]]>2014-12-30T20:04:00-08:00internetemailtelecomEmail Vendors: Time to Build in DMARChttp://www.circleid.com/posts/20141216_email_vendors_time_to_build_in_dmarc/http://www.circleid.com/posts/20141216_email_vendors_time_to_build_in_dmarc/
DMARC is extremely useful, yet I've heard some vendors are putting their implementations on hold because of the IETF DMARC working group. You really shouldn't wait though — it's been in wide use for nearly three years, enterprises are looking at DMARC for B2B traffic, and the working group charter is limited in it's scope for changes.

Let's compare this to a similar situation in the past. When I was on a panel at the INBOX Conference in 2006, I told the audience — which included email appliance makers, software vendors, and email service providers (ESPs) — that they should already be offering SPF and be preparing to support DKIM, which would be finalized Real Soon Now (in fact it took another year). But I did not tell them they should be implementing DomainKeys…

DomainKeys had been announced two years earlier and was an effective technology, but it was not widely deployed — a number of senders were using it, but few mailbox providers or companies were using it to filter inbound messages. Meanwhile the IETF DKIM working group was creating something different from and incompatible with DomainKeys. So the decision to wait for the DKIM working group to finish was reasonable.

Today, the circumstances around DMARC are very different. DMARC has been filtering the email sent to over 80% of consumer mailboxes in the US alone for almost three years now, and over 80,000 active domains have published DMARC records. It's already popular with saavy email senders and domain owners for the light it sheds on where email using their domains is coming from. And just as enterprises became enthiastic users of TLS for B2B email, DMARC is being evaluated as an additional protection for sensitive B2B email channels.

The IETF DMARC working group is chartered to fix some important interoperability issues with forwarded email, mailing lists, and other "indirect" mailflows. However the working group is not chartered to make major changes to the protocol, the way the DKIM working group was — maintaining interoperability with existing implementations is a key objective.

Several vendors and services have already integrated DMARC filtering into their products (list at dmarc.org), and you can bet more have it in the pipeline. So if you're planning the next release of your email appliance, MTA, or related software, you should make sure you've got DMARC support in the works too.

]]>2014-12-16T17:59:00-08:00internetemailThe EFF and Hanlon's Razorhttp://www.circleid.com/posts/20141112_the_eff_and_hanlons_razor/http://www.circleid.com/posts/20141112_the_eff_and_hanlons_razor/
The EFF has just posted a shallower than usual deeplink alleging an "email encryption downgrade attack" by ISPs intent on eavesdropping on their customers.

Outbound port 25 blocking is a best practice, which is enforced by several large providers around the world and is recommended, for example by M3AAWG. Port 587, the SMTP submission port, has been recommended for outbound SMTP since it was first defined in 1998, in RFC 2476 (now obsoleted by RFC 6409). An older but still relevant Best Practice document from 2007 is RFC 5068. These RFCs are explicit that port 587 is to be used for mail submission, and that it MUST NOT (capitals as used in the RFCs) be subject to port blocking.

However, airport and hotel wifi networks, and other networks with a large number of transient users, tend to filter outbound port 25 rather than follow the commonly accepted best practices of blocking port 25 outbound traffic, a large part of which is malicious, originating from virus infected hosts on a network. This might be a well intentioned measure (possibly to decrease tech support costs) but it is certainly not a best practice, this is well on the "ignorance" rather than "malice" side when you slice it with Hanlon's razor.

It is certainly not appropriate to conflate this, as the EFF has done with their FCC filing, with other practices allegedly adopted by ISPs to track their users or slow down sites they see as competitors. And it is certainly not new, in fact it is about a decade old, for the EFF to equate spam filtering of any sort with censorship or worse.

That said, it does appear to be high time to update existing best practices on port 25 management to explicitly recommend that proxy filtering port 25 by turning off TLS to allow content inspection is not a privacy friendly alternative to blocking port 25 outright. That blocking rather than suppressing port 25 will avoid frivolous FCC filings targeting an ISP is perhaps an additional icing on the cake.

]]>2014-11-12T21:35:00-08:00internetcensorshipemailnet_neutralityspamA Look at the Origins of Network Emailhttp://www.circleid.com/posts/20140903_a_look_at_the_origins_of_network_email/http://www.circleid.com/posts/20140903_a_look_at_the_origins_of_network_email/
The history of long distance communication is a fascinating, and huge, subject. I'm going to focus just on the history of network email — otherwise I'm going to get distracted by AUTODIN and semaphore and facsimile and all sorts of other telegraphy.

Electronic messaging between users on the same timesharing computer was developed fairly soon after time-sharing computer systems were available, beginning around 1965 — including both instant messaging and mail. I'm interested in network mail, though, so we need to skip forward a few years.

You need a network. And a community.

Around 1968 the initial plans for "ARPANET", a network to link the various ARPA-funded computers together were underway. Local mail between users on the same system was already a significant part of the nascent community.

Q.What did you discover?

A. The thing that really struck me about this evolution was how these three systems caused communities to get built. People who didn't know one another previously would now find themselves using the same system. Because the systems allowed you to share files, you could find that so-and-so was interested in such-and-such and he had some data about it. You could contact him by e-mail and, lo and behold, you would have a whole new relationship.

Q.Were these first communities built around mailing lists?

A. Well, you had a directory, so you knew how to send them e-mail, but you might only know 2 or 3 people, and you would meet 20 or 30 through the computer.

It wasn't a static medium. It was a dynamic medium. And that gave it a lot of power.

Q.It was all about community?

A. There was one other trigger that turned me to the ARPAnet. For each of these three terminals, I had three different sets of user commands. So if I was talking online with someone at S.D.C. and I wanted to talk to someone I knew at Berkeley or M.I.T. about this, I had to get up from the S.D.C. terminal, go over and log into the other terminal and get in touch with them.

I said, oh, man, it's obvious what to do: If you have these three terminals, there ought to be one terminal that goes anywhere you want to go where you have interactive computing. That idea is the ARPAnet.

Linking those local messaging systems over ARPANET to create inter-computer mail was one of the earliest uses considered, even before the network was up and running.

J.C.R. Licklider mentioned inter-computer mail to me in a conversation about 1968, when he asked me if I was interested in a project he had in mind, "to connect all the ARPA-funded machines together, and see what they said to each other." He was recruiting folks for what became the Project MAC networking group. By that time, many time-sharing systems had some kind of internal mail. Discussion of linking the mail systems on various computers was one of the earliest applications proposed for the ARPANet.

In 1971 Ray Tomlinson was enhancing SNDMSG, the tool used to send local mail on TENEX systems. There had been discussions about how to implement a standardized form of network mail, but they were perhaps overly complex and too paper-based. Something less weighty was needed.

BBN had two TENEX systems, side-by-side, both attached to ARPANET. Ray realized he could extend SNDMSG to use an existing TENEX tool. CPYNET, that copied files from one machine to another, allowing it to send mail over the network. Sometimes you need a simple hack to demo a concept.

The "@" sign was a character not used in names, and wasn't really used for anything on TENEX, so Ray chose to use it as a separator between the user and host. It's a happy accident that BBN used TENEX systems; if they'd had the other type of system used on ARPANET, Multics, the "@" key would have already had a reserved use — deleting the current line — and we might not have seen it used for email.

It'd be nice to have a standard protocol about now

To make network mail work between different machines, you need a way to transfer the message that's a bit more portable than CPYNET. File Transfer Protocol (FTP) was being developed as a standard way of copying files from one machine to another. At one of the FTP development meetings, someone proposed sending mail via FTP. Someone from BBN mentioned that Ray had done something similar, using user@host addresses. A standard for email transfer began to be born.

It's a two-way conversation

SNDMSG already used "business memo" style messages — To:, From:, Subject:, Cc:, Date: etc. — so the messages it sent looked like modern email in many respects; that format was adopted, and began to be standardized.
At first the user interface was crude, allowing recent messages to be dumped to the users terminal or new messages to be composed. Fairly quickly tools were developed that allowed individual messages to be read or deleted.

There was still a critical element missing. That was the ability to reply to an email.

John Vittal combined and extended existing tools to create MSG, adding several new features including "Answer" — a command to automatically create a message formatted as a reply to the current one. MSG was the first integrated modern email client.

It included a command for replying to a message, similar to what is used today. My impression at the time was that, as soon as this command became widely available, e-mail use exploded. As one of MSG's early users, I found that it permitted quick, convenient, and sustained conversations similar to the conversations we have come to expect over e-mail today.

Written by Steve Atkins, Founding partner of anti-spam consultancy & software firm Word to the Wise

]]>2014-09-03T08:10:00-08:00internetemailCall for Nominations: M3AAWG J. D. Falk Award Seeks Stewards of a Better Online Worldhttp://www.circleid.com/posts/20140806_call_for_nominations_m3aawg_j_d_falk_award/http://www.circleid.com/posts/20140806_call_for_nominations_m3aawg_j_d_falk_award/
Anyone seeking to honor a groundbreaking contribution toward a better online world should submit a nomination for the 2014 M3AAWG J. D. Falk Award. Presented to people whose work on specific projects made the Internet a safer, more collaborative, more inclusive place, the J. D. Falk Award has recognized leaders and pioneers who saw elements of the online experience that needed improvement and took action to fix them. The nomination process is simple, open to anyone, and free of charge. To be considered for the October 2014 J. D. Falk Award, nominations must be completed by 5 September, 2014.

This year's program marks the third presentation of the M3AAWG J. D. Falk Award. Past winners include Gary Warner, director of research in computer forensics at the University of Alabama at Birmingham's Center for Information Assurance and Joint Forensics Research (2013), and FBI Supervisory Special Agent Thomas Grasso (2012). Warner is credited with developing a university lab project into one of the world's preeminent training programs for cyber intelligence analysts. Grasso led the DNS Changer Working Group, assembling a partnership of industry, government and academic experts to organize a response to a global malware attack that jeopardized Internet access for millions of people.

The 2014 M3AAWG J. D. Falk Award winner will be announced in October at our 32nd General Meeting in Boston. The award includes a small honorarium that is funded by Falk's previous employer, Return Path Inc. Nominees' contributions can take virtually any form, in the public or private sectors, individually or with groups, from research and software innovation to organizing or mentoring communities dedicated to improving the online environment for all. The only broad requirement is that the nominated project should reduce online abuse and make the Internet experience better. The nomination form can be found at HERE.

The award commemorates J. D. Falk's commitment to addressing specific issues, working professionally, as a volunteer and as an organizer to find solutions and strengthen the Internet as a public resource. In addition to the J. D. Falk Award for distinct projects or accomplishments, M3AAWG annually presents the Mary Litynsky Award for lifetime achievement, recognizing people whose collective body of work has made online communication safer and more trustworthy. The Mary Litynsky Award nomination form can be found HERE.

]]>2014-08-06T14:06:00-08:00internetcyberattackcybercrimednssecemailinternet_governancemalwaremobileprivacyregional_registriessecurityspamNon-English "IDN Email" Addresses Are Finally Working!http://www.circleid.com/posts/20140705_non_english_idn_email_addresses_are_finally_working/http://www.circleid.com/posts/20140705_non_english_idn_email_addresses_are_finally_working/
Late this afternoon (Tuesday, August 5th, 2014), Google announced a long-awaited and highly anticipated change that took place today with Gmail, which will surely be remembered as one of the most important milestones in the history of internet email: The "client-side" implementation of non-Latin, character-based Internationalized Domain Name (IDN) email.

To say this announcement is "huge" is putting it lightly.

While the world's relevant standards-setting body, the Internet Engineering Task Force (IETF) has had a ratified standard for non-English email since 2012, not a single provider of email services — not Google, not Yahoo, not Hotmail, not Microsoft's Outlook, not any of the Chinese providers — actually supported the IETF standard.

Before today, this meant that while you could get an "IDN email" address for your non English domain name — say, 夏明@域通联达。在线 (translated: simon@tldregistry.online in Chinese), nobody could actually send you emails to that address!

Today, thanks to Google, this English-specific email address barrier has been broken, and will now allow non-English speaking email users to email in their native language, making the process of emailing faster, easier, and more efficient. And that's the way the internet should be for everyone, regardless of what language you speak.

In the announcement, Software Engineer Pedro Chaparro Monferrer states, "Less than half of the world's population has a mother tongue that uses the Latin alphabet. And even fewer people use only the letters A-Z."

Testing IDN Email in Gmail

Within minutes of learning about Gmail's new multi-lingual capabilities, we tested the system. In the screenshot below, we have addressed a new standard Gmail message to an IDN Email address. Note that we used the "Chinese dot" (。) rather than the ASCII/English dot (.) which is what a typical Chinese user would do when typing a Chinese email address.

Then we hit send. The message was accepted by Google's email server, and the message was on its way!

In the screenshot below, you can see that Gmail was particularly nice and converted the Chinese dot to an ASCII dot, so as to not break the IETF standard for IDN email. Nice touch, Gmail!

Today, the internet changed for the better for around half of Earth's internet users

Google understands that the number of people around the world that experience difficulty using email simply because of what language they speak, is exponentially high.

Thanks to Google, the confusion, hassle, and inconvenience of using English-only email addresses is now considerably diminished, and will eventually be completely dissolved.

We applaud Google and are very excited that our many owners of Chinese domain names can finally use email. As the first email client technology provider in the world to support the IETF standard, the company has taken a big step towards breaking the unfair hegemony of English in email, and the internet in general.

"Language should never be a barrier when it comes to connecting with others and with this step forward, truly global email is now even closer to becoming a reality," Chaparro Monferrer wrote.

We agree.

]]>2014-08-05T16:14:01-08:00internetdomain_namesemailmultilinguismtop_level_domainsGmail Now Supports Internationalized Domain Nameshttp://www.circleid.com/posts/20140805_gmail_now_supports_internationalized_domain_names/http://www.circleid.com/posts/20140805_gmail_now_supports_internationalized_domain_names/
If your first language isn't English and you don't use the Latin character set you can and will run into barriers. While Internationalized Domain Names (IDNs) i.e. domain names where either the left of the dot, the right of the dot or the entire string is in characters other than Latin ones, do exist and have existed for a number of years not all services work well with them.

The video on the right produced by ICANN a couple of years ago helps illustrate what IDNs are and why they're important.

Potentially the biggest issue for IDN domains is email. Several times over the last year or so various Google staff have hinted at Google working on supporting IDNs in their products and services. It's a logical step, as the company has applied for several IDN top level domain names and not being able to use them with their own products would be rather "silly".

Google announced earlier today that they've added IDN email support to Gmail:

"Starting now, Gmail (and shortly, Calendar) will recognize addresses that contain accented or non-Latin characters. This means Gmail users can send emails to, and receive emails from, people who have these characters in their email addresses."

They've also announced that they're adding it to other services, though, for now at least, you won't be able to use an IDN email to signup for services.

]]>2014-08-05T15:54:00-08:00internetdomain_namesemailmultilinguismDealing With DMARChttp://www.circleid.com/posts/20140604_dealing_with_dmarc/http://www.circleid.com/posts/20140604_dealing_with_dmarc/
DMARC is an anti-phishing scheme that was repurposed in April to try to deal with the fallout from security breaches at AOL and Yahoo. A side effect of AOL and Yahoo's actions is that a variety of bad things happen to mail that has 'From:' addresses at aol.com or yahoo.com, but wasn't sent from AOL or Yahoo's own mail systems. If the mail is phish or spam, that's good, but when it's mailing lists or a newspaper's mail-an-article, it's no so good.

The mailing list community has been gnashing its teeth for the past month trying to figure out the least bad ways to deal with the problem.

To keep track of all the ways of avoiding or limiting the damage, I've made a page on the ASRG wiki. (The ASRG is gone, but the wiki lives on.)

]]>2014-06-04T12:03:00-08:00internetemailUniversal Acceptance of All TLDs Now!http://www.circleid.com/posts/20140519_universal_acceptance_of_all_tlds_now/http://www.circleid.com/posts/20140519_universal_acceptance_of_all_tlds_now/
Universal acceptance of top level domains hasn't really meant much to most Internet users up until now. As long as .COM was basically the default TLD, there wasn't much of an issue.

No longer. With 263 delegated strings (according to ICANN's May 12, 2014 statistics) adding to the existing 22 gTLDs that were already live on the net after the 2004 round of Internet namespace expansion, the problem of universal acceptance gets very real.

What is the issue? Being able to use any TLD with any browser (Safari, Chrome, Firefox...) or email client (outlook, Apple mail...), no matter what the environment (Mac, Windows...). Non-ASCII Internationalized Domain Names (IDNs) have made the issue even more crucial because universal acceptance is needed to fulfil their promise: giving millions of people who use different keyboards and scripts access to an "own language" Internet use they've never been able to experience up until now.

Non-Latin script TLDs are considered one of the most important Internet naming innovations of the last decade because they are key to reducing the digital divide between those who are comfortable typing ASCII web addresses and those who are not. Those whose native alphabets are Chinese, Arabic, Cyrillic or others have jumped at the chance to type their own scripts into the address bars. The high registration volumes reached by IDN ccTLDs like .рф (Romanised as .RF for Russian Federation) or more recently the Chinese character new gTLDs launched by TLD Registry .在线 (.ONLINE) and .中文网 (.WEBSITE) are clear indications of the pent-up demand for non-Latin script web addresses.

Ensuring new web addresses can be used everywhere, by everyone

Because they are that important, ICANN's new gTLD program has given deliberate priority to IDNs. The 104 applications for non-Latin script TLDs were allowed a faster track through the evaluation and delegation process if they wanted it.

Despite that, most new gTLDs that have gone live so far are of the traditional ASCII kind. But as TLD Registry's suffixes show, the Internet is not going to stay all-Latin for long. Others also have ambitious plans to bring Internet navigation to new audiences using different alphabets. Stable Tone is one such registry operator. It will soon be launching 世界, (.WORLD in Chinese) with an innovative plan to bring international brands to Chinese audiences.

Without universal acceptance, those plans will suffer. The products will be out there, but their intended audiences won't actually be able to use them. Why? Because in many ways, the Internet still works as if there had never been an expansion of its namespace. There are 3 main issues.

Internet browsers tend to carry out summary checks when a destination is typed into their address bars. Some browsers will actually filter out anything that doesn't resemble the "classic" TLDs. Type www.example.guru, one of the new TLDs that is already proving very popular, and browsers expecting .COM or something similar will either tack that suffix on the end and search for www.example.guru.com, or take the user to a search engine to try and "help".

The rate of expansion is also an issue. In 2004, only a very limited number of TLDs were added to the Internet root. So as suffixes like .BIZ and .INFO came online, browser developers had time to adjust. But the new gTLD program means an increase of several orders of magnitude for the number of TLDs becoming valid Internet namespaces. How can developers of browser, email and other Internet services keep up?

IDNs have brought a third universal acceptance issue. Users must be able to type, and see, their URLs in the intended native script. If it's a Chinese address, you should be able to type that in as easily as if it was ASCII. Having to resort to some kind of encoder to get the Chinese script complicates the user experience to such an extent that there's a risk users simply won't bother. And that could ultimately mean the failure of IDN TLDs that, ironically, Internet users actually want.

There's a fourth, slightly separate but equally important issue: mobile device use. Browsing and emailing environments tend to be even more technically limited on smartphones and tablets than they are on desktop computers. Yet those are the devices which the next billion Internet users, those that stand to benefit from new TLDs and especially IDNs, are most likely to use.

All that leads to a situation where currently, there is a lack of universal acceptance for these new TLDs. Take those mentioned earlier. A web address like 房地产.在线 (RealEstate.online) simply isn't reachable using the majority of web browsers running on PCs and mobile devices in China. A part-ASCII address like nic.在线 has the same issues.

Chinese data shows that around 50% of the browsers used in the country are not compatible with Chinese IDN gTLDs and that on mobile devices, tha vast majority of the browsers available aren't compatible.

Single focus

So what to do? Awareness of the problem isn't new. ICANN, for instance, has been working on it since 2004, when it opened a discussion forum on universal acceptance, and even before. Technically there are no major hurdles to universal acceptance. For example, the technology needed to correctly render IDN web addresses was developed even before the new gTLD program, when work was being done on the IDN ccTLD program designed to give a path to non-ASCII country codes to countries such as Russia, where .рф is considered the second national suffix after the ASCII .RU.

The major stumbling block for universal acceptance today is probably coordination. And let's face it, that's an area where organisations like ICANN have traditionally not been very strong. ICANN, the technical community's IETF, and others, are very good at taking on a variety of problems at once. They are not so good at focusing their collective energies on a single problem.

Yet that's exactly what's needed to guarantee universal acceptance. Across-the-board efforts targeting all aspects of this one problem. For example, if technical operators such as browser developers are not made sufficiently aware of new gTLDs by the governance community, then how can they be expected to adapt their products to suit?

It's incumbent on the technical and governance communities to ensure that demand is met. Disappoint these non-ASCII hopefuls now, and new gTLDs as a whole will suffer the consequences.

]]>2014-05-19T08:08:00-08:00internetdnsdomain_namesemailtop_level_domainswebAOL Has a Security Hole, and It's Our Problemhttp://www.circleid.com/posts/20140423_aol_has_a_security_hole_and_it_is_our_problem/http://www.circleid.com/posts/20140423_aol_has_a_security_hole_and_it_is_our_problem/
Two weeks ago I wrote about Yahoo's unfortunate mail security actions. Now it's AOL's turn, and the story, as best as I can piece it together, is not pretty.

Yahoo used an emerging system called DMARC, which was intended to fight phishing of often forged domains like paypal.com. A domain owner can publish a DMARC "reject" policy which, oversimplifying a little, tells the world that if mail with their name on the 'From:' line didn't come from their servers, it's not from them so you should reject it.

While this policy is fine for Paypal, it was bad news when Yahoo did it over the weekend a few weeks ago, because there is a lot of mail with 'yahoo.com' on the 'From:' line that doesn't come from Yahoo. The highest profile is discussion mailing lists, but there are others like newspaper mail-an-article buttons that put the address of the person sending the article on the 'From:' line, and various mail consolidation setups, like the one at Gmail that lets you collect mail from all your accounts in your Gmail inbox, and also send mail with the addresses of those accounts. (They check it's really you before they let you do it, of course.) All those now don't work reliably any more, when sending from a yahoo.com address, and as I explained in the previous article, cause considerable harm to innocent bystanders on other mail systems.

Now AOL just did the same thing, at least sending out a press release so we didn't have to reverse engineer what they did. But it's also now clear why Yahoo and AOL did this: it's a band-aid for big security problems. I've been getting a lot of spam from aol.com addresses of people I know, with other people I know on the 'To:' line. This means that the crooks were using the AOL user's address book. And when I look at the spam, most of it doesn't come from AOL, but instead from other ISPs such as Orange, a French phone company, which means that the horse has left the barn, the crooks have stolen the AOL users' address books and there's nothing AOL can do about it. I am hardly the only one to noticethisproblem.

A friend (the always perceptive Steve Atkins) pointed out that Yahoo had the same problem earlier this year. This makes the DMARC actions more understandable, if not necessarily more excusable. Crooks are using the stolen address books to send spam, and the DMARC policy stops a fair amount of it, which is good. Unfortunately, in the process of slamming the barn door after the horse left, AOL and Yahoo have crushed a lot of small animals on the way.

At this point, I channel Colin Powell's Pottery Barn Rule: You break it, you own it. It's their mess, due to their security breaches, so it's their responsibility to deal with the damage. All of their suggestions so far have the consistent theme that everyone one else spend their own time and money to change the way they work, and remove features their users depend on, to circumvent the DMARC damage. Yahoo estimates this affects 30,000 small mailers. Uh, no. Speaking as a mailing list operator, I'm willing, not eager, but willing, to work with the DMARC cartel to mitigate the damage, but they have to be willing to put some effort into it beyond waving their hands and telling us our mail volume is so tiny that we don't matter. (If that's the argument, the total traffic volume of all mail including the spam is a rounding error compared to video and P2P, so we should just shut all mail down, thereby solving the spam problem at a stroke. So let's not go there.)

In the next and I hope final installment, I'll suggest some short term and longer term ways that ISPs can use DMARC and preserve the mail services their users and everyone else counts on.

]]>2014-04-23T20:43:00-08:00internetemailYahoo Addresses a Security Problem by Breaking Every Mailing List in the Worldhttp://www.circleid.com/posts/20140408_yahoo_addresses_a_security_problem_by_breaking_every_mailing_list/http://www.circleid.com/posts/20140408_yahoo_addresses_a_security_problem_by_breaking_every_mailing_list/
DMARC is what one might call an emerging e-mail security scheme. It's emerging pretty fast, since many of the largest mail systems in the world have already implemented it, including Gmail, Hotmail/MSN/Outlook, Comcast, and Yahoo.

DMARC lets a domain owner make assertions about mail that has their domain in the address on the 'From:' line. It lets the owner assert that mail will have a DKIM signature with the same domain, or an envelope return (bounce) address in the same domain that will pass SPF validation. The domain owner can also offer policy advice about what to do with mail that doesn't have matching DKIM or SPF, ranging from nothing to reject the mail in the SMTP session. The assertions are in the DNS, in a TXT record at _dmarc.domain. You can see mine at _dmarc.taugh.com.

For a lot of mail, notably bulk mail sent by companies, DMARC works great. For other kinds of mail it works less great, because like every mail security system, it has an implicit model of the way mail is delivered that is similar but not identical to the way mail is actually delivered.

Mailing lists are a particular weak spot for DMARC. Lists invariably use their own bounce address in their own domain so they can collect the error reports from list mail, so the SPF doesn't match. Lists generally modify messages via subject tags, body footers, attachment stripping, and other useful features that break the DKIM signature. So on even the most legitimate list mail, most of the mail fails the DMARC assertions, not due to the lists doing anything "wrong".

The reason this matters is that over the weekend Yahoo published a DMARC record with a policy saying to reject all yahoo.com mail that fails DMARC. I noticed this because I got a blizzard of bounces from my church mailing list, when a subscriber sent a message from her yahoo.com account, and the list got a whole bunch of rejections from gmail, Hotmail, Comcast, and Yahoo itself. This is definitely a DMARC problem, the bounces say so.

The problem for mailing lists isn't limited to the Yahoo subscribers. Since Yahoo mail provokes bounces from lots of other mail systems, innocent subscribers at Gmail, Hotmail, etc. not only won't get Yahoo subscribers' messages, but all those bounces are likely to bounce them off the lists. A few years back we had a similar problem due to an overstrict implementation of DKIM ADSP, but in this case, DMARC is doing what Yahoo is telling it to do.

The DMARC mailing list issue has been argued at length among us nerds, and one of the counter arguments has been that mail systems know who is sending mailing list mail, so they will whitelist those lists or otherwise avoid the problem. We now know that is not true. I've been running lists for years, no spam at all (they're all noncommercial stuff like my church, CAUCE's newsletter, and a group of folk dancers), and every possible technical feature including DKIM, SPF, correct forward and reverse DNS, you name it. As noted above it didn't help, and I have heard from many other list managers with the same problem, thanking me for explaining what happened.

I understand, from the always interesting Word to the Wise blog, that Yahoo has severe phishing problems, with crooks sending mail to Yahoo users, pretending to be yahoo.com administrators. Yahoo chased the crooks off their own servers, so now the crooks are (as I understand it) sending mail to Yahoo from the outside, pretending to be Yahoo. While I sympathize with their problems, and this is not exactly swatting a fly with a sledgehammer, it's a nail that needs a regular hammer, and the sledgehammer is demolishing the surrounding plaster every time it whacks the nail. Concretely, Yahoo should be able to figure out ways to reject non-Yahoo mail going into their own servers without abusing DMARC to screw up everyone else.

I hope they get their act together, but in the meantime here are some suggestions for people who run mailing lists or other mail software that might legitimately pass on a yahoo.com message:

• Suspend posting permission of all yahoo.com addresses on any mailing lists you run, to limit the damage.

• Tell Yahoo users to get a new mail account somewhere else, pronto, if they want to continue using mailing lists.

• If you have source code for your list software, as a band-aid, see if you can add a hack to check for yahoo.com From: addresses and change them to something like "Address redacted", which will avoid triggering DMARC. I did that on my lists.

• If you know people at Yahoo, ask if perhaps this wasn't such a good idea.

]]>2014-04-08T07:50:00-08:00internetemailFine Grained Mail Filtering With IPv6http://www.circleid.com/posts/20140123_fine_grained_mail_filtering_with_ipv6/http://www.circleid.com/posts/20140123_fine_grained_mail_filtering_with_ipv6/
One of the hottest topics in the email biz these days (insofar as any topic is hot) is how we will deal with mail on IPv6 networks. On existing IPv4 networks, one of the most effective anti-spam techniques is DNSBLs, blackists (or blocklists) that list IP addresses that send only or mostly spam, or whose owners have stated that they shouldn't be sending mail at all. DNSBLs are among the cheapest of anti-spam techniques since they can be applied to incoming mail connections without having to receive or filter spam. On my system about 85% of incoming IPv4 mail connections are handled by DNSBLS, and I gather that number is pretty typical.

On IPv6, DNSBLs can't work the same way.

The problem is that the IPv6 address space is much, much larger. The IPv4 address space has 232 (4 billion) addresses of which maybe half are in use. Two billion is a lot of data for people, but not a big deal these days for computers, so large mail systems track every IP address that sends mail and has an opinion (the jargon word is reputation) about what kind of mail each IP sends.

IPv6, on the other hand, has 128 bit addresses. Usually each individual network, such as a home LAN or a single customer at a hosting provider, is assigned a 64 bit prefix, which means that each LAN has 264 addresses, way too many to track individually on any plausible computers. On the assumption that all of the 264 addresses are under the same control, a common plan for DNSBL operators is to disregard the low 64 bits of an address and just filter on the high half. Unfortunately, that doesn't really help, for two reasons. One is that even the remaining addresses are way too big to track. Currently there is about 53 bits allocated IPv6 networks (disregarding the low bits), and 253 of anything is still way too big to track. The other is that some hosting providers have ignored the standard configuration advice, and have put multiple customers in the same 64-bit LAN, so the 64 bit rule will treat all those customers as one. (The providers claim their routers made them do it.)

The thinking is that since IPv4 mail will continue to work for a very long time, we can be pickier about IPv6 mail and only accept it if it has DKIM signatures or otherwise makes itself easy to recognize. While I expect I'll be doing that, it occurs to me that the vast IPv6 address space offers senders and receivers a lot more finegrained whitelist and blacklist opportunities than we had before.

If I were providing public mail server like Gmail or Yahoo or Hotmail, there's plenty of bits to give every user a unique IP in a single /64. A recipient can treat the whole /64 as a unit, or if they want, they can track individual addresses, maybe all of them, or maybe just ones that come to their attention via spam complaints or the like. Recipients can use IP addresses to block mail from senders with a bad history, or maybe slow it down, or send it to different servers.

For ESPs (bulk mail service bureaus), we can add an extra level and assign IP bits both to users and mail campaign per user, perhaps like this:

|nnn--64--nnn|xx-8-xx|uuu--40--uuu|ccc-16-ccc|

The high 64 bits is the network number, same as any other IPv6 address. The low half has 8 spare high order bits for future cleverness, 40 bits of user (a trillion users per provider should be enough), and 16 bits to identify the campaign. If you want to handle one campaign specially, block, delay, or reroute or whatever, it has a unique IP. If you want to handle all of the user's mail specially, just ignore the low 16 bits and do something with that user's block of IPs.

Maybe this particular bit setup isn't ideal, but it's definitely worth thinking about what to do with all those address bits, beyond ignoring most of them and recreating techniques from IPv4.

]]>2014-01-23T11:46:01-08:00internetemailipv6spamThe Naive Arrogance of FUSSPshttp://www.circleid.com/posts/20131226_the_naive_arrogance_of_fussps/http://www.circleid.com/posts/20131226_the_naive_arrogance_of_fussps/
Everyone who's been in the e-mail biz long enough knows the term FUSSP, Final Ultimate Solution to the Spam Problem, as described in a checklist from Vern Schryver and a form response that's been floating around the net for a decade.

FUSSPs fall into two general categories, bad ideas that won't go away, and reasonable ideas that are oversold. The classic bad idea is e-postage, charging for e-mail. I wrote a white paper about it a decade ago, and nothing of importance has changed.

Reasonable ideas that are oversold are much more frustrating, and there are a lot of them in the e-mail authentication field. One of the bad ideas is that if we just knew who was sending every mail message, we'd have no spam, which is silly for many reasons.

It's not at all silly in weaker versions. There are plenty of partial authentication schemes that let us identify interesting parts of our mail stream so we can whitelist mail from some senders we know. Attempts to give an identity to each user, PGP and S/MIME, have never been widely adopted because it's too hard to distribute the keys into each user's mail program. Less ambitious schemes where the granularity is the domain name are quite workable for parts of the mail stream, but suffer when they're oversold.

A sure fire way to tell when an authentication scheme is turning into a FUSSP is when naive enthusiasts demand that other parts of the mail system that work perfectly well have to change what they do to deal with limitations of the authentication scheme. Invariably, the mail system keeps doing what it's always done, and people learn to live with the limitations of the authentication scheme, but teaching the enthusiasts to curb their enthusiasm is very frustrating.

About a decade ago, the new authentication scheme was SPF. For some kinds of mail, particularly bulk mail all sent from a known set of servers, it works quite well. For others, such as mail systems with human users who send mail in a variety of hard to predict ways, it works much less well. SPF has ways to deal with this limitation, by allowing various intermediate results between pass and fail, but by golly the enthusiasts wanted to fix the e-mail world so that all legitimate mail would pass. (Presumably they would reject all mail that didn't pass, and solve the spam problem.) To try to force the issue, some of them had their mail systems reject any mail that didn't pass, thereby losing lots of perfectly ordinary mail that didn't happen to be sent in a way that SPF can describe. Add-ons to SPF with names like SES were supposed to let the rest of us "upgrade" our mail systems so that forwarded mail, a particular sore point, could be SPF compliant, but in practice what happened was that people adjusted their SPF to avoid needless damage and the mail system works the same as always.

The next round was with DKIM. While SPF tries to validate the path a message takes, DKIM puts a cryptographic signature in the message. That doesn't depend on the path the message takes, so forwarding isn't usually a problem. For DKIM, the FUSSP issue arrives with mailing lists. Most mailing lists make small changes to the messages for the convenience of the list subscribers, such as adding a list tag to the subject line, or a message footer that says what the list is and how to subscribe or unsubscribe. If the incoming message has a DKIM signature, these changes invalidate it. The mail system running the list software can add its own DKIM signature, recipients use that signature to recognize list mail, and everything works.

Some DKIM enthusiasts insist that mailing lists need to preserve DKIM signatures from incoming mail by turning off the convenience features that break signatures, even though that would solve no problem whatsoever. (I can't wait to see the comments on that one.) List operators, of course, have ignored them, since the subject tags and message footers are useful, and lists work just fine as is.

This year's authentication hack is DMARC. It sits on top of DKIM and SPF and allows senders to describe their sending policies, and offer advice to quarantine or reject mail that doesn't match the policy. For domains like Paypal that are heavily forged and send all their mail from a small set of known hosts, DMARC is very useful to tell people how to detect forged mail. For domains like mine, who are forged only by the occasional random spammer who picks fake return addresses out of the spam lists, and whose mail goes through forwarders, mailing lists, newspaper send-to-a-friend, and so forth, DMARC has some interesting ways to gather statistics, but the policy parts are not useful.

Predictably, there are DMARC enthusiasts who want every message ever sent by anyone to have a "reject" policy, which will solve the spam problem by rejecting all spam. (Perhaps they are too young to remember the past two iterations.) This again brings us back to mailing lists. DMARC policy assertions can't describe list mail for the same reasons that list mail fails DKIM, and it doesn't matter for the same reason, receivers can recognize mail from lists their users subscribe to. But DMARC adds an extra level of excitement: overzealous use of DMARC by list contributors can make receivers reject valid list mail, which list software interprets as bad recipient addresses, and removes entirely valid subscribers from the list.

You will probably not be surprised to hear that the enthusiasts blame the problem on the mailing lists, which should turn off features that break DKIM signatures, or need to be rewritten with complicated heuristics to recognize and ignore DMARC bounces, or something, rather than the actual problem, which is that they're publishing the wrong policy. List software will of course not change what it does; at most it will recognize and reject incoming mail from subscribers that have strict DMARC policies. There are already patches to GNU mailman to do this, and I've found I can adjust my majordomo2 lists do to so with a one line configuration change.

I suppose it's nice to know that we're still getting new enthusiasts in the anti-spam biz, but it sure would be nice if they'd read the history before making the same mistakes over again.