Thursday, August 16 2012

Wherein we think about ways to get your mailing lists clean and keep
them clean.

First let us define the characteristics of a clean mailing list:

All recipients are confirmed opt-ins

All addresses are deliverable (valid domains, no bounces)

No addresses are spamtraps

No addresses are temporary, time-expiring addresses

All opt-ins have been reconfirmed by user action within the past
year

If you've read Part Two then
most of those are pretty obviously vitally necessary.

Those of my clients who've had the greatest problems with blacklisting
are those who've employed downright lousy bulk mailing programs, and
most of them were of the mindset that throwing mail at as many people as
possible was the key to increased sales. It's not, but it seems
plausible enough.

Monday, August 06 2012

I periodically look through my web server logs to pick out things that
are not as they should be. You might recall from previous blog entries
that I operate spam traps and so on -- last night I picked out of my
server logs that some critter calling itself MJ12bot was going where no
legitimate bots belong. But it's apparently trying to be a good bot
because it leaves its calling card:

So off I go to that URI, and find that the folks who run the thing have
said "If you have reason to believe that MJ12bot did NOT obey your
robots.txt commands, then please let us know via email..." And so I
did.

We discussed the matter via email a bit, and it seems probable that
their bot encountered some kind of network error when it tried to grab
my robots.txt file. Not an error response from my server, but a failure
to even contact my server. To my way of thinking, in a case like that a
properly designed bot will try again to get that file, and will not
crawl the site until it gets either the file or a verifiable 404 Not
Found. Not MJ12bot, though. The network failure is treated as if it
were a 404, and is taken to mean that the whole darn site is wide open
to them. Here's what their guy Alex said:

Sadly it's very difficult for us to diagnose this case - as
you can see from your logs our bot grabs robots.txt, so we are not
intentionally breaking your directives, it's just if bot could not get
robots.txt then it could not obey it :(

Huh? Your bot encounters a network error and that gives you
license to crawl my site in violation of my terms of service? It seems
to me that if you know your crawler is broken in that way, which you do
now, and you continue to run it knowing it's broken in that way, then
what you're doing is willful negligence and that makes it
intentional.

No worries here. I've informed the folks behind the thing that their bot
is no longer welcome here and any connections it makes will be
considered trespass. The fun part? When their bot comes around it will
not see my web site. It will instead see a very, very long joke that
will be delivered very, very slowly. How slowly? From start to finish
will take from an hour and a half to more than six hours.

If you've seen a bad bot in your logs and want to punish it in this way,
feel free to hit my contact form to inquire
about it. It's a freebie if all you need is the application itself and
very minimal installation/configuration instructions. After all: Bad
Bots Must Be Punished!

In our previous
installment we established that being SMTP blacklisted sucks and
that if you've been blacklisted you probably deserved it. I promised to
explain to you how those SMTP blacklists work, and being a man of my
word here is that explanation.

SMTP blacklists serve two very important purposes. The first is to keep
spam out of the inboxes of human users who don't wish to be bothered by
it. The second purpose is to conserve the resources of the email service
provider so the spam doesn't consume any more network, processing, and
storage capacities than are absolutely necessary. Those resources are,
after all, costly, finite, and better used for purposes other than the
delivery of mail no one wants to see anyway.

Modern SMTP blacklists collect and collate data from many sources, some
of which you may never have considered. When you send email of any kind,
whether bulk or individual, you are providing information to them.
Always. Always, always, always. You decide what that information will
be but not how they will use it, so it behooves you to take care about
what that information says about you/your mail server/your domain name.

Saturday, August 04 2012

So, you ran a bulk email campaign that got your mail server blacklisted,
eh? I don't envy you, fellow space traveler, don't envy you at all. SMTP
blacklisting is unpleasant, and it can be costly.

I'd like to share some hard won wisdom with you. It's wisdom that you're
going to come to eventually one way or another, at least in part, so you
might just as well take the easy way and take heed of my words. You've
already started on the course of learning the hard way -- you're smarter
than to pursue that course all the way to the bitter end, aren't you?

Before we get into the meat of it, a brief aside: In this article I may
make you uncomfortable by challenging your self perception or your
concept of fairness. Know that I intend no insult or offense, and that I
speak plainly only because I believe I would be doing you a disservice
if I were to do otherwise. It's time to confront this SMTP blacklisting
problem head on, solve it once and for all, and never have to deal with
it again. That's what we both want, and the only way we're going to get
it is to observe reality as it is rather than as we wish for it to be.
No matter how you choose to come to the wisdom I'm sharing with you, you
are going to be uncomfortable at times. You might as well get it out of
the way right now and then get on with solving the problem.

We can, between us, solve your problem.

The top four things that I always hear from those whose servers are or
have recently been blacklisted:

I only send email to those with whom I have a provable and
legitimate business relationship, customers and those who have
contacted me in the past requesting product information.

People can opt out of my mailing at any time. There are always
opt-out links in the messages, and there's an opt-out form on
my web site, too.

All of my promotional mail is CAN-SPAM compliant.

I am a legitimate businessman. I am not a spammer!

First things first: Let us review the definition of spam as it is
accepted by most technically savvy individuals. Spam is unsolicited bulk
email, typically though not strictly commercial in nature. This is the
definition used by those who make the internet work. Please make note of
the fact that this definition does not address how the sender came into
possession of the email addresses in his list.

The second thing: The only thing being CAN-SPAM compliant gets
you is freedom from prosecution in the United States for the bulk email
you send. It does not obligate anyone anywhere to conduct or receive
your internet communications.

Let us now apply the previous two paragraphs to the Top Four things that
I hear from those whose servers have been blacklisted.

"I only send email to those with whom I have... a business
relationship...", every time I've heard it, has meant that the speaker
has sent bulk email to everyone whose email address he's acquired in the
conduct of his business and most of them did not give their express
consent to receive that bulk mail. Hmmm. Lack of express consent.
Didn't ask or agree. Synonym: Unsolicited. Unsolicited bulk email, the
very definition of spam.

"People can opt out..." That's what blacklisting is. It's those
recipients of your unsolicited bulk email acting en masse to opt out in
a way that does not leave you the option to disregard their will. In my
experience, for every spam complaint sent there will be two to four
opt-out requests, so overall most people do have a sense of fair play.
But those who fire the spam complaint instead? They have evidence that
you have no regard for the rules so they have no reason to believe that
you will keep your word that opting out will stop your bulk mail. They
didn't break the rules. You did.

Rules? But "all of my promotional mail is CAN-SPAM compliant". Again,
all that gets you is freedom from legal prosecution and does not
obligate anyone to conduct or receive your communications. The rules for
gaining access to another's network are whatever they say they are, and
most publish their rules for the world to see. The one rule that's most
prevalent is Do Not Spam Our Users. Unsolicited bulk mail is spam, and
you've sent it to their users. Of course they've blacklisted you.

"I am not a spammer!" Unless all of those people who receive your bulk
email have asked you to send it to them, yes, you are. And yes, that
blacklisting was an appropriate response whether you like it or not.
People don't like receiving unsolicited bulk email. They react badly to
it, and almost everyone I've ever discussed the matter with has told me
the same thing: If some online vendor spams me once, I will never go
back again. It's a simple enough concept, one that most of us employ
without even thinking about it: Do not reward bad behavior.

Tangentially: Some years ago I was in a position that required me to
attend lots and lots of seminars, several of them being on the topic of
how customers behave. Somewhere in the neighborhood of 95% of the
customers who have a problem with your company will never give you the
opportunity to make it right. They just go away. Then they tell nine or
ten of their friends about their bad experience -- and these days, some
number of those friends are likely to be social media sites where tens,
hundreds, or thousands will view the complaint. When sending unsolicited
bulk email, that's the fire you're playing with.

In Part Two, we'll go into how blacklists decide what is and is not spam
and who is and is not a spammer. If you've got a dirty mailing list,
you're a spammer even if unwittingly so. We can fix this problem. Stay
tuned.

Thursday, August 02 2012

After a long hiatus I've brought my blog back to life. The world may
never be the same again. It's changed in the past few minutes, even.
All of the working clocks now show a newer time so clearly we have got
a little further into the future.

I will be updating fairly regularly with my inconsequential thoughts
on life, conducting business on the internet, doing the things I do,
and whatever else might hold my interest long enough to be written
about.