Thursday, 6 October 2011

Time-zone database down

It is perhaps easy to read that line, think it doesn't affect you, and then move on. But thats just not the case.

Update 2011-10-07: A backup copy of the mailing list and subscribers has now swung into operation run by volunteer Robert Elz. Please only join this list to discuss changes to time-zones - its not a "fighting the lawsuit" list. New tz list

Update 2011-10-07: More details on the history of ACS, Astrolabe and how this came about from a competitor. Worth reading.

Update 2011-10-10: A new version of the time-zone database 2011l has been released. This news indicates how other members of the community have stepped up and continued the work, so the database can no longer be considered to be "down". However, we all need to ensure that we continue to support Olson and Eggert in their fight with the foolish people at Astrolabe. My blog about the rebooted database.

The time-zone database (sometimes referred to as the Olson database) is the computing world's principle source of time-zone data. It is embedded in every Unix and Java for starters, and will be used by many websites and probably by your iPhone. You may know it via the IDs, such as "Europe/London" or "America/New_York".

But, perhaps you're thinking that time-zones don't change? Well that may be true for America and the EU right now, but certainly isn't for the rest of the world. Governments change their time-zones all the time, and the decisions are frequently very political. I'd estimate there are between 20 and 100 separate changes made around the globe each year. And these can be at very short notice, triggered by earthquakes for example.

The time-zone database tracks all this information and creates a standard format file that describes it. I would show you an example of the file, but then perhaps I'd be sued....

The database itself was run as an open source project, led by Arthur David Olson, supported by many others. The data was published as a set of files about 15 times a year, and then picked up by users everywhere.

The complaint itself comes from Astrolabe, Inc, whose website looks like a company I would avoid doing business with.

The complaint is that Astrolabe produce a work, the "ACS Atlas", which is referenced by the time-zone database (some sources suggest that Astrolabe may have recently purchased the work). Astrolabe claim copyright over their work and thus believe that the time-zone databse should not have released their information to the public domain. The case is targetted at two private individuals - Arthur David Olson and Paul Eggert, who have hosted the website for many years.

The key passage in the time-zone database files is this:

# From Paul Eggert (2006-03-22):
# A good source for time zone historical data in the US is
# Thomas G. Shanks, The American Atlas (5th edition),
# San Diego: ACS Publications, Inc. (1991).
# Make sure you have the errata sheet; the book is somewhat useless without it.
# It is the source for most of the pre-1991 US entries below.

For obvious reasons, I'll refrain on commenting on the rights and wrongs of the case, although I will note that facts like the phonebook cannot be copyrighted. A detailed response from one site taken down is now available. Instead I'll focus on the impact.

The impact of this is severe for anyone that uses it - whether via Java, Unix or some other means. This really is the key tool used by everyone to tell the right time globally. We all owe a debt of gratitude to the database maintainers who have worked on this for many, many years at zero cost to the industry and for zero financial gain.

So, right now the global situation is that there is no longer a single central location for time-zone information for computing. I'm sure that each major user project (like the Unix distros) will patch their own versions as best they can, but the stricter ones might argue that the current data is tainted and want to remove even that. This could get very messy very quickly.

Both Joda-Time and ThreeTen/JSR-310 use the data to build timezone information. ThreeTen/JSR-310 in particular provides this information in huge detail to applications. The worst case scenario is that multiple groups start up to provide this data in the future, and applications are then responsible for handling multiple competing data sources.

This data is so key to the world at this point that it needs to be formalised and run by a group with more legal and financial backing. Efforts had been ongoing to achieve this, but they may now be in jeopardy - who would want to take on a project being legally attacked?.

I hereby call on the industry leaders to help sort this out - IBM, Oracle, Apple, Google, RedHat I'm looking at you.Update: I didn't include Microsoft here because Windows has its own time-zone data files.

In the meantime, could I please ask that anyone thinking of patching the data on a temporary basis, or trying to recreate it from scratch, re-uses the existing file format. There is no reason to believe that the C code or file format is tainted by the lawsuit, just the data. So, lets all please try to minimise the mess that could happen if everyone starts to go their own way.

67 comments:

Hi Stephen - I'll have a chat to the JCP EC about this - we're not the right body to host the data, but the EC contains the range of big industry leaders that can form something to host this. Drop me a line if I don't get back to you within the week about this. - M

David Braverman mention that he "...will be contributing to Olson's and Eggert's legal defense fund once they get it set up." If you happen to find out the details for contributions to the fund, could you please post them.

carterson2, this wasn't a database that was copied (they even wrote that the compilation alone is useless - you needed the errata sheet). From your link:

The Copyright Act specifically states that the copyright in a compilation extends only to the compilation itself, and not to the underlying materials or data. 17 U.S.C. § 103(b). As a result, compilation copyrights cannot be used to extend copyright protection to ideas or facts that are otherwise unprotectable (it is a basic premise of copyright law that there is no copyright protection for ideas and basic facts

It's not that Microsoft isnt an industry leader per se. They are just a notable exception in that they have compiled their own proprietary time zone database and don't rely on the tz data base that is the subject of this post.

.. who knows: maybe microsoft simply has a program to convert that unix tz database into the internal propriatary microsoft format. Would save them quite a bit of work to simply do it that way round. And still maintain their incompatibility with *nix.

Seriously. Not the end of the world. Someone will simply work around it or create an open database. The judge will probably give them a slap on the wrist for the infringement and the world keeps turning. I think OSX, Linux, Solaris, etc will be just fine. Sometimes I think the open community automatically assumes a company is out to get the little guy just because. Has anybody considered the fact that maybe they DID infringe on something owned by the company? I doubt it is the evil map making giant trying to destroy the world with miscalculated time in *nix Operating Systems. If they own it... then it's theirs. Make your own.

I've updated the post to explain the absence of Microsoft from the list of industry leaders (they have their own TZ database).

A number of people have opined various legal theories. I'm staying clear of that as I'm not a lawyer and I don't live in the US.

While logically we might assume that removing data before 1991 would solve the problem, in all likelihood it wouldn't immediately stop the lawsuit. Whether that is sufficient is open to interpretation.

A key goal of mine is simply to ask all participants to try to work together on this, and not simply to create lots of forks. If a new database has to be created, lets figure out a way to do it together, perhaps at CLDR.

There are two products here. The book (ACS publications) and the computer program of the book (Astrolabe). I've checked the address of the Astrolabe website and complaint, and it is definitely Astrolabe (computer program producer) making the complaint.

Microsoft don't seem to *have* a correct TZ database. At the moment, Outlook is sending out meeting invitations in US Eastern Daylight Time, which say 'GMT-0500' when in fact EDT is GMT-0400.

This problem is well-known; they even hacked Exchange to add a disclaimer to meeting invites that it sends out, stating explicitly that the timezone information may be incorrect. One of the most unprofessional and incompetent 'solutions' to a problem that I've ever seen :)

I understand that the lawsuit is probably ill conceived, however why was the project completely removed from the web? In a case this egregious shouldn't the project managers seek out a bit of help before shutting everything down?

I think taking it down from the web is a good. It makes visible how the recent copyright and patent trolling (especially in the US) is destroying internet freedom. And hey, why innovate if you can do rent-seeking on ages-old stuff?

Astrolabe's future prediction technology must have told them they will win the case or they would not have spent money on the lawyer. If the court finds that the ACS is a work derived from public legislation on timezones and they lose, they will have falsified astrology. I for one welcome their application of the scientific method and eagerly await the result.

http://www.astrology-x-files.com/terranatlas.html has a nice write-up on some of the history relating to his, and would seem to support the idea that while some data may have been checked against the ACS data, what has been 'copied' has also been corrected? Perhaps a 'bugs list' with the tz corrected data would be a useful document for the court case? The more evidence that the tz database IS the more accurate work would mitigate against the problem of a straight copy?

I want that company to just die. TZ Data is the more correct and reliable source, they should be proud to be part of this by being quoted. Isn't it prior art to just collect this data and print a book?

I think that the most interesting point is that the work these people at Astrolabe, Inc. have done has rested on the backs of work shared freely between Astronomers and Astrologers for centuries. I suppose that this is the difference between the people who start the company (with the best of intentions) and the business people who invest and direct the affairs.

It's not like that Astrolabe invented all those timezones, they just talked about it in the past and now nobody else may do so? Now that would be funny.

If the content of something that profane can be copyrighted I want a copyright on the information that a day has 24 hours. That would just be fair, after all.

Keep the work up guys and let us know where we can donate to, I'm pretty sure that you won't be left alone. After all this is very important for all the companies out there, too. Imagine that - as a result of this case - you would not be able anymore to talk about history, after all it was written in a book before and as such there must be a copyright on it.

The real shame to me isn't that the TZ database is down (if nothing else, we programmers will find a replacement) - but that Olson and Eggert are being sued for all their great work. Especially with Olson about to step down - we can't let them treat people who spent so long on an open source project badly.

The writer of a trivia book suit the makers of Trivial Pursuit for lifting questions out of his book. He lost because the courts ruled that facts are not protected by copyright. A compilation of trivia questions has an unarguably higher level of creativity than a compilation of historical time zone data, and it still didn't meet copyright criteria.

"that while some data may have been checked against the ACS data, what has been 'copied' has also been corrected? Perhaps a 'bugs list' with the tz corrected data would be a useful document for the court case? The more evidence that the tz database IS the more accurate work would mitigate against the problem of a straight copy? "

Copyright protects the expression of an idea, not the idea itself. ---That is the only way to express that idea-- is a defense, as well as -- I derived that expression of the idea on my own, independently of the other person's expression thereof-- (See the case of Phoenix Bios and the IBM PC where Phoenix created a chinese wall with one group of people reading the IBM bios, abstracting the ideas and required actions, and another group who had never read the IBM bios coding those requirements into a bios.) --I read and changed his expression of the idea, eg, correcting mistakes-- is not a defense, it is a derived work which still falls under the original copyright (and your copyright for the changes).

You really should get Astrolabe's software and have a look at it. Probably the most beauty has is included in the Imperial Astrologer 1.0.6 (I guess about 25 MByte of brilliance). I got my copy! Get yours.

Well, I think we have to fight this absolute nonsense. If Astrolabe prevailed in this case, which they will not, because their claim is ridiculous, douchebag companies all around the world would try to do the same thing with other opensource projects.

If you want to set up a mirror yourself, to help spread around the data all over the world, have a look here:

Earler versions of the ACS Atlas did cite their sources, but later copies apparently expunged this data. The 1978 edition shows the following acknowledgements:

Major contributions have been made by the following published materials and journals: Doris Chase Doane, "Time Changes in the USA"; Curran and Taylor, "World Daylight Savings Time"; documents published by the Interstate Commerce Commission; National Railway Publication Co.; "The Official Guide of the Railways ... of the United States..."; "The Mercury Hour"; "AFA Bulletin"; and "Astro Psychological Problems".

If you can't leave a message to these nice Astrolabe people on facebook then perhaps the good old fashioned email way will suffice. Let's send them some emails of appreciation to the following addresses - like one every minute :)

I'm not sure why people who make a living from lying to other people about "the stars" should be allowed the protection of the courts, or indeed food and water.

Leaving aside that the threat has come from professional fraudsters, why is it that an accusation like this is so damaging? Why is it so easy to take down someone with a baseless accusation and why are there not more punishments for frivolous or malicious prosecution?

What gets me is not the stupidity of these astrologists - but the stupidity of the US Judge who actually ordered the TZ database shut down simply on the basis of the dubious complaint. Couldn't he have at least waited until the matter was finally decided?