Domain auto-renew mechanisms

Rick Moen [rick at linuxmafia.com]

Mon, 6 Aug 2007 15:32:50 -0700

I figured this will be of general interest. Ben and I (with Ben doing
the real work) have been working, off and on, on a scriptable mechanism
for checking a list of Internet domains for impending expirations, and
then do some sort of appropriate notification. Ben's Perl script,
"domain-check", for this purpose relies on the commodity open-source
client for the TCP port 43 "WHOIS" information service, offered by most
domain registries.

We've run into a fascinating tangle of complications, most of them
(primarily) non-technical. It's been a lovely case study in all of the
manifold ways that the interface between people and automated systems
can go wrong.

Domain registrars (typically) don't want their customers' domains to
expire: That leads to loss of business, after all -- though most are
pleased to put expired domains in a "held" state for a while, in which
one can ransom them back for significant money. Registrars have a few
ways to help customers avoid accidental expiration: One of them is
called "auto-renew". With this feature enabled for one's domain (now a
default setting at many registrars), the registrar will attempt to
charge a renewal fee to the customer's credit card of record, at some
time prior to -- or exactly at -- the time of expiration.

Most registrars' implementation of that idea involves, say, three
successive attempts to charge the customer's credit card, at, say,
30 days from expiration, 20 days, and then 10 days -- or something like
that. At least, I recall seeing such a policy once or twice: I think
Joker.com does that, for example.

But then there's Gandi.net, of France. Look at the first line of the
domain-check report below:

I've corresponded with Chaz Boston-Baden (a friend), owner of the
laconiv.org domain (which was the site of the recent World Science
Fiction Convention in Los Angeles, L.A.con IV): He thanked me for
keeping laconiv.org on my monitoring list, and assured me that he's
arranged with Gandi.net, his registrar, to auto-renew.

So, holy mama! Expiration is set for 10:18 PM UTC today, which is 3:18
PM Pacific Daylight Time -- and it's 3:26 PM PDT right this moment.
(Probably some timing glitch.)

If one assumes that Chaz is correct, and Gandi is set to carry out the
auto-renew action, then Gandi's cutting things awfully close -- and with
vanishingly little opportunity for recovery if, e.g., the credit card
charge is temporarily not possible, and so on.

If there are significant numbers of registrars who wait until the last
few minutes of registration -- a Bond-flick-style countdown -- before
running their autorenew scripts, then that makes it all the more
difficult for third parties, running things like domain-check, to
know whether domains are in danger of expiration, or not.

> I figured this will be of general interest. Ben and I (with Ben doing
> the real work) have been working, off and on, on a scriptable mechanism
> for checking a list of Internet domains for impending expirations, and
> then do some sort of appropriate notification.

Rick, meanwhile, has been doing yeoman duty testing and retesting my
script against the hundreds of domains, and reporting the failures in
enough detail that I can work on improving the parsing. Development
would have gone nowhere without him.

I must say that hunting down all the odd little cases and writing the
regexes to parse all the weird little date formats (the human mind can
be a very twisty little thing) has been a fun experience.

I've also discovered an interesting fact WRT the 'whois' client as
compared to 'jwhois': the latter isn't hard to rewrite in Perl - it just
creates a query (a text string formatted per RFC 954), pings port 43 on
a server, and prints out the returned results. The former, however, is a
bit more interesting: it does all of the above plus precedes that
output with a block of information that contains even more useful data
(often, that's the block I end up parsing.)

I haven't had the time to explore all the possible WHOIS queries in
depth, so I don't know where they're getting the info - but it's a
pretty nice feature of the 'whois' client.

> Rick, meanwhile, has been doing yeoman duty testing and retesting my
> script against the hundreds of domains, and reporting the failures in
> enough detail that I can work on improving the parsing. Development
> would have gone nowhere without him.

Eh, you flatter me.

> I must say that hunting down all the odd little cases and writing the
> regexes to parse all the weird little date formats (the human mind can
> be a very twisty little thing) has been a fun experience.

My friend Chaz, owner of laconiv.org, resolved part of the mystery about
what was happening with that site. He said, quite understandably:

No, I don't have any of them set for auto-renew. I don't like having
an auto-payment of any kind on my credit cards.

So, I really did in this case help a friend avert an accidental domain
expiration (and this wasn't a case of Gandi.net doing a last-minute
autorenew) -- and got thanked, for a change.

You may recall that I had checked, yesterday, on the domain status and
found that it was some number of minutes clearly past the indicated
expiration time. I remember that the whois record still showed
laconiv.org as in status "Active", rather than any variety of expired
status. It would have been interesting to watch more closely and see
when the DNS ceased working, and when the status was changed from
"Active" -- bearing in mind, on the latter item, that there's always
some lag between the time the TLD zone (in this case, .ORG) implements
changes such as domain deactivation, and the time those changes
propagate to the public WHOIS data.

However, I missed my chance to watch that unfolding disaster, and
instead chose to help a friend. Damn, messed up again! ;->

> I've also discovered an interesting fact WRT the 'whois' client as
> compared to 'jwhois': the latter isn't hard to rewrite in Perl - it just
> creates a query (a text string formatted per RFC 954), pings port 43 on
> a server, and prints out the returned results. The former, however, is a
> bit more interesting: it does all of the above plus precedes that
> output with a block of information that contains even more useful data
> (often, that's the block I end up parsing.)
>
> I haven't had the time to explore all the possible WHOIS queries in
> depth, so I don't know where they're getting the info - but it's a
> pretty nice feature of the 'whois' client.

So, again mostly to fill everyone else in, and recap, the two WHOIS
clients (cited above) each have drawbacks in being not able to deal
intelligently with particular top-level domains (TLDs). Among other
things, out of the box, each fails to know the correct TLD-specific
WHOIS server to consult, on a bunch of domains.

IANA (the Internet Assigned Numbers Authority) maintains a supposedly
authoritative list of where the WHOIS service (if any) resides for each
of the 250 country-code TLDs (ccTLDs) and 20 generic TLDs (gTLDs), and
post that data to public Web pages. In an ideal world, the two WHOIS
client utilities' authors would at least bother to ensure that they
benefit from that public information. Alas, they seem not to. Also,
IANA's data are incomplete and sometimes wrong.

To be fair, each featherless biped on this planet probably only cares
about at most a dozen TLDs at a time, and some of the TLDs strain
even my internationalist instincts past the breaking point. (On the
other hand, it's kinda cool to discover that Antarctica's domain, .aq,
actually has at least one Web server, www.com.aq, even if the
corresponding IP is probably somewhere in Texas.) However, in trying to
make Ben's script be as universally useful as possible, we've ended up
trying to "fix" or work around the two client programs' respective TLD
blindnesses.

The "jwhois" client has the advantage of having a very flexible conffile
format, in which among other things you can supply an explicit WHOIS
server FQDN to consult for any TLD. Which suggests a subproject....

I've started a modest little list of test data at:
http://linuxmafia.com/pub/linux/network/domain-check-testdata
When complete, it will then serve as the basis for an optimal
jwhois.conf file (optimal in the sense of returning real results from
all TLDs that offer any useful data on a known TCP-43 WHOIS server --
which in turn Ben and I can then use to ensure that Ben's domain-check
script works on all TLDs that provide domain expiration data in their
WHOIS output.

Amazingly enough, most TLDs provide either no expiration data at all,
or expiration data that obviously has no real-world consequence, since
various domains shown as expired are still functional. It's a wacky
world, out there.