It was a problem with the USNO servers (I.e. tick.usno.navy.mil, tock.usno.navy.mil etc.) being rebooted and starting to hand out the wrong time. Very few downstream startum 2 NTP servers should have accepted such a large skew, although they may have lost accuracy.

Amusingly I happen to work with an ex. USNO NTP admin, so I'll be sure to take the piss for the rest of the week.

Yes and no.This article is interesting: http://support.microsoft.com/kb/884776Summary: Windows can do it, but before Server 2008 it defaulted to not doing any sanity checks.Since 2008 it still is quite generous, allowing 48 hour jumps.If you don't like it you have to adjust the value in the registry.I guess it still shows that the Internet was an afterthought for Microsoft...

Windows implements basically Daytime using NTP. What this means is instead of NTPd trying to adjust the clock to be in sync in small increments, Windows does what you do with Daytime instead - query the Daytime server, and set the clock. Except that since very few servers provide Daytime, Microsoft uses NTP to fetch the time and date.

Even the latest Windows still does it - though if it drifts too far, it stops syncing. Very annoying as it doesn't stay synced and quietly fails.

Had your home router sync'd with such a server and been rebooted, it would have booted with the wrong time, and then wouldn't have accepted the corrected time when the error was fixed...

Why would it have accepted the big skew the first time, but not the second?

A home router would typically not have a battery backed clock, and so on boot would simply accept whatever time it was given by the ntp server. Once it's up and running though it shouldn't accept a 12 year skew. My home router (openwrt) boots to something like 1/1/1970 I think.

As it happens I recently implemented the RTC in an embedded product and used a simple trick to reject obviously wrong times. I compare the date with the firmware build date and reject it if it is earlier. In the case of a router it could easily try other NTP servers in that case.

That simple test rejects 99% of incorrect dates where something has been reset and started from zero again (which in this case would seem to be 01/01/2000).

Honestly not sure... it did hit us on a couple of old boxes, but fixed itself pretty quickly, and we definitely weren't using the affected navy servers. tick and tock are supposed to be used only by gov, and stratum 2 servers. Maybe some servers forwarded the wrong information?

Yes, this is all Microsoft's fault because with UNIX apps, you're expected to deal with this sort of bullshit.

You SHOULD expect this kind of bullshit. Because in real life, shit happens. Servers do reboot. Servers do see their clocks reset. Yes, even time servers.

In general, YOU are responsible for your own system. You may query other servers to check their time, but in the end, it's your program that sets your computer's time, so it's its own responsibility to do the right thing.

Generally xntpd and its ilk will not step the time more than a small amount, but will rather give up and quit instead. One machine such as (say) the RPi with no RTC that use ntpdate to *set* rather than *adjust* the clock, this is harder to avoid, but servers should not be automatically running ntpdate when configured properly.

So there may be *two* bugs here to get the problem: one in an NTP implementation and one in use of ntpdate for anything other than manual updates on servers. But I haven't read TFA yet.

(On the little bit of work I put into the ARCRON driver I inserted extra code to look out for year rolls, etc, on top of whatever xntpd would do. In part because ARCRON only sends 2-year dates IIRC.)

Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard. Celebrate success on occasion, sheesh.

Unfortunately most of the general public think that because nothing really went wrong there was not a problem in the first place, and that it was all hyped up by the media. Some of this is the simple truth that it was over-hyped by the media who over-hype everything so people are growing desensitised, some of it is people not bothering to research their opinions or properly engage their critical thinking abilities.

We have a built-in bias against successful disaster planning: since the planning was successful, the disaster didn't happen, and hence to the average observer, it looks like there wouldn't have been a disaster. The reasoning is flawed, of course, but apparently very hard to resist. This is why governments are only ever harmful—if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort. This cognitive flaw is why people who do diving catches are respected, and people who plan for the future and avoid problems are ignored, and why blithering idiots keep getting control of the reins and breaking things.

This is why governments are only ever harmful&#226;&#8364;"if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort

That's funny, because I always see it in reverse with the US government: if we hadn't had the stimulus, unemployment would have been much worse, or if we didn't have federal loans, no one could afford college, or if we hadn't passed the Patriot Act, we would have had lots of terrorist attacks...

Yeah, it cuts both ways. The fact that nothing happened doesn't mean that the preventative worked. But the fact that nothing happened also doesn't mean that the preventative wasn't needed. It's important to understand why the reasoning is flawed, and not just that it is flawed, or else you wind up falling into different flawed reasoning.

E.g., the reason people believe the stimulus worked is not that things might have been worse without it, but rather that things _are_ demonstrably worse in places wher

To be fair, it makes a bit of sense. I know, it sounds dumb, but the horrible truth is that being prepared can be rather expensive and the cost of preparing for every possibility is utterly infeasible.

To be evolutionarily successful, you need to be prepared enough to survive long enough to reproduce, and perhaps to see that your offspring reproduce. Being prepared to live 100 years when you won't breed much past 35 means that there's up to 65 years of time where your cost/benefit to the gene pool drops to n

Just because you're not punching kids out THIS VERY MINUTE doesn't mean your cost/benefit to the gene pool drops to near zero. Humans in modern societies take 2 decades or more to become self-sustaining, to say nothing of the aid (financial, emotional, or educational) provided by grandparents and then the very act of going to work in the morning, living somewhere and paying property taxes for schools (ideally) educating kids still adds benefit, regardless of whether you have kids of your own or not. And f

Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard.

Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four. But those of us that actually programmed in the days when 4K was a LOT of RAM, know that we never used two ASCII chars to store a year. We used a single byte to store the offset from 1900 in binary. So there will be no overflow until 2156.

Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four.

While I'm sure plenty of organizations did nothing and were fine, I know for a fact that storing years as two bytes was not a myth. At my organization I fixed such a problem in critical systems. We would have had people collecting GPS data in the field with no way to verify it's quality if I hadn't. Not doom for our organization, but not a "myth" as you call it.

I lost several days of simulation work that had been running over Christmas. Again, not a big deal because I restarted it when I got back in January, but still a real Y2K bug that we hadn't found or fixed.

Actually, most TRS-80 operating systems didn't even keep TRACK of time. It's the Disk Operating System (LDOS) that you're talking about. The TRS-80 didn't even have a built in clock / timer board / dedicated bits or bytes for knowing the time. And Level I and Level II Basic on it don't ask about the time or save it on the cassette tape system. And neither did the Apple ][+. You had to buy add-in clock boards to keep track of the time.
Even the IBM PC did not have a
real-time clock [wikipedia.org] until the
IBM-AT version [wikipedia.org] in 1984. [Thank you wikipedia for prior info]. If you boot up an old IBM machine, one of the first things that the system asks after boot-up is to enter the time and date. [actual info from turning the old computers on in the garage].

The PC and PC/XT were very commonly retrofitted with an AST Six Pack [artofhacking.com] or generic equivalent. It was expensive, but still cheaper than buying each function on a separate card (and using six slots, which the PC didn't even have, it had just five). Even then it required running a program at startup to read from the RTC and load it into the motherboard clock. Setting the time on the Six Pack also took an executable, as simply setting the time the normal way didn't write it back to the board.

It was a problem in COBOL and some databases. If you had no COBOL you would usually be fine. Many banks however have very old code, and they needed to have this problem fixed in 2000, and they got it fixed. Checking for Y2K problems was checking for COBOL code and particular patterns of programming and database-design where 2 chars were used for years.

It was a problem in many places. Y2K bugs did crop up. Some cropped up as early as 1996, a lot more showed up in '98 and '99. By then most of the major stuff had been figured out.

COBOL is only significant here because it is the most common language used in some financial operations. However the problem existed in C, assembler, Fortran, Java, databases, etc. Yes, I hear someone say "but my favorite language has a Date type!", which really only matters if the original programmer used the date type correc

I know of at least one organization which had a significant Y2K problem, even after making preparations.

Sadly, the preparations were "Hire someone to take the fall when the shit hits the fan so we can continue with business as usual. Er... hire someone to ensure Y2K preparedness."

The fatal glitch in the plan was that the person who got hired made friends with an exec in the parent company before the ball dropped. So, when things went south the hire got a silver parachute while the rest of the company folded

I beg to differ about not experiencing significant problems on 1/1/2000. We had significant issues that caused all our approximately 2000 store servers to repeatedly shut down until we unloaded the offending software.

I was working for Hollywood Video in the Tech Support department (support for the rental stores and all their computer equipment) leading up to and after Y2K, in Wilsonville, Oregon at the corporate office. (Of course this was before their two bankruptcies.) The software development department performed extensive (and probably expensive) testing on every facet of our current in-store software and hardware setup (custom COBOL software running on DOS 5.0 on NetWare 3.1 if you can believe it.) They were even going to scrap NetWare in favor of a brand spanking new Windows NT Remote Desktop-type setup, but we were highly disappointed when NetWare came up with a patch for NetWare 3.X series to make it Y2K compatible, so they scrapped the NT plans. But I digress...

Came in to work on 1/1/2000 a couple hours after midnight (yep they pretty much forced us to come in, and for very little extra pay - I may have been a bit drunk still.) Everything was already chaos: Almost every single store's NetWare server shut itself down at midnight, thinking there was a power outage. And since our stores' computers ran as dumb network-booted terminals to the main server, that means all the computers were down and rentals couldn't be performed except by writing the rentals on paper.

Problem was, in the test lab someone had commented out the UPS backup auto-shutdown software line in the servers' autoexec (or its NetWare equivalent, might have been autoexec.ncf or something.) And yes, I do know who that someone was (wasn't me.):) So I guess no one thought to test that particular software. So all the servers would boot up, immediately think there was a power outage, and immediately shut themselves off. We did have a manager's station computer in each store that had its own hard drive and could be used in emergencies, and had pcAnywhere and a modem, so we manually dialed into each of our approximately 2000 stores (at 14.4 kbps.) Then we walked a bunch of clueless managers and minimum wage kids through taking the new autoexec we had copied to a floppy on their manager's station (and a bunch of the stores had to run out and buy a box of floppies on New Year's Day) and booting up their servers using the floppy.

I think we got the last few stores up and working by 2 or 3 pm Pacific. And before you say "who rents movies on New Year's Day?" - EVERYONE did. New Year's Day and Christmas Day were two of our biggest movie rental days of the year. People are home with their families, the festivities are over, everyone wants something to do and streaming from the internet didn't really exist yet. What did everyone do? Rent a video or go to a theater. I'm not sure how many tens of thousands of dollars in rentals we lost that day, but I'm sure it was significant.

TL;DR: Just because you didn't hear about any significant losses due to Y2K bugs, doesn't mean they didn't happen. It's not like businesses were eager to admit they screwed up and forgot to test something.

We used a single byte to store the offset from 1900 in binary. So there will be no overflow until 2156.

The vast, vast majority of software that ran infrastructure was COBOL, and it almost always uses PICture statements, not single bytes, to define dates. Those PIC statements most commonly defined dates using 2-digit years. Stupid and short sighted, yes. But also a very real issue that took gajillions of man hours to fix.

Y2K disasters would have been very real (not as real as the Press would have had us believe, but very painful) if all that code hadn't been hacked. Note that the problem wasn't actually f

I worked on a system processing medical records that used a number of distinct methods of storing dates

* British 2-digit years : dd-MM-yy* British 4-digit years : dd-MM-yyyy* American 2-digit years : MM-dd-yy* ISO 8601 style : yyyy-MM-dd (this is the correct form for all cases where you have to store the date as text)* An integer offset in minutes from a custom epoch value

The rule I tried to adopt was to always store the date in an unambiguous format, and to present it to the user in whatever format was ap

The fact is, when we put all our data on 80-column punch cards, we'd use only two columns for the year. It would have been possible to encode it in one EBCDIC column as you say, but the cards were keypunched by women* who wouldn't generally know how to do that, and sometimes spot-verified by office staff who could read a two-digit number much easier than a format putting 256 values into one column. (Not to mention that, if you did too many fancy things, you could conceivably wind up with a lace card that

There are people use a Date type, but then ran into problems when converting back and forth to character types. Silly stuff like "years 0-50 add 2000, years 51-99 add 1900", then using that for birthdates.

Really, this is where experience matters. I see these problems from junior programmers who've never had the experience of dealing with long range compatibility problems, or who are naive enough to treat library routines as appropriate for all situations, or just not the mindset for thinking ahead about h

However time_t, whether 32-bit or 64-bit, was intended for file dates. It was not intended for general all purpose date handling. And yet people do this. They use the signed 32-bit time_t for storing birthdays for example. Or the 64-bit time is used without accounting for historical calendar changes.

And MANY embedded systems out there use a signed 32-bit Unix time (some even with unsigned 32-bit, which is perfectly logical for file creation times). There are even Unixes still running that don't do 64 b

Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard. Celebrate success on occasion, sheesh.

Because if you accept that the problem was real and the only reason life doesn't suck as badly as predicted is because the fix was real means you are going to have a LOT more trouble sticking your fingers in your ears and chanting LALALALA I CAN'T HEAR YOU whenever people discuss Global Climate Change.

For those of you who didn't read the article; US Naval Observatory rebooted one of their NTP servers (tock.usno.navy.mil I believe, though not sure), and the server's time reverted to 2000.. This time change got pushed out, and thus people freaked out claiming that NTP was broken.

A lesson can be learned from this: change your BIOS battery when it's dead (alternatively, never reboot your servers).

Or, better. Reboot your servers one at a time, a couple of times a year. That way, when a problem happens (may it be the RTC battery failing, bad init scripts or watever else) you'll catch it in a much more safe way.

Believe it or not I still see applications being developed with reliance on two-digit years.

People say that Y2K was the result of computers not having enough memory to store 4 digits. That might have been true once upon a time for a few systems, but for the most part it is due to laziness. Ugh, can't you shorten that field? And so on...

Just as a single word can have multiple meanings, a single phrase can have multiple meanings as well. The Y2K bug that a lot of people (including me!) helped fix, yes, that was real. However, the Y2K bug that some nutcases hyped, where everything that had a clock chip in it would go haywire at 00:00:01 Jan 1, 2000, did not exist.

It's like person A saying "dragons don't exist", and person B replying, "Yes, they do! The komodo dragon is a real animal!" The komodo dragon is a dragon, but it's not the kin

did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

Yet lots of other people did absolutely nothing, and the disaster didn't happen for them either. I worked on Y2K. When we were testing, we tried booting early versions of MSDOS, Windows, OS/2, Linux, FreeBSD, MacOS. The older the better. None of these had a problem when the date transitioned. Or at least (in the case of Windows 3.0), nothing that a reboot wouldn't fix. The situation was similar with applications. We found a few applications that needed to be restarted after the date transitioned, but

did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

Yet lots of other people did absolutely nothing, and the disaster didn't happen for them either. I worked on Y2K. When we were testing, we tried booting early versions of MSDOS, Windows, OS/2, Linux, FreeBSD, MacOS. The older the better. None of these had a problem when the date transitioned. Or at least (in the case of Windows 3.0), nothing that a reboot wouldn't fix. The situation was similar with applications. We found a few applications that needed to be restarted after the date transitioned, but there were no show stoppers.

Y2K was hyped up by the press and by software vendors peddling "critical" upgrades.

the realtime clock in your PC bios in that era stored the date in BCD form (4 bits per digit) and often either had a problem at transition, or just didn't work. Easily fixable with a boot-time patch or something, but still a problem that needed to be addressed.

The satellite images at www.bom.vic.gov.au showed the year as 19100 for a while. Again, not a show stopper but still an indication that someone didn't fix a problem that should have been fixed.

Last night AD went to year 2000. Linux clients joined to AD freaked out right away with scary security warnings. A reboot fixed them right away after AD time was set right. Or setting time manually.

Windows had authentication trouble so Shares, Printers, etc stop working for clients. For some reason I had to rebooting them many times fixed them, no idea why one wasn't enough. Had trouble reported all today even though it was only off for 2 hours last night.

Our NTP sync came from a remote site and they were doing Y2K testing. Clocks jumped forward to March 2000 and screwed up the timestamps on our binaries. In the end we had to touch backwards to get our build system working again.

I distinctly remember on multiple occassions feeling irked by what I assumed were tinfoil hat wearing fools who required you to get the time kind of sort of right first before clocks would sync up from an external time source...especially within windows.

I could swear windows and the standard unix ntp systems explicitly check for and do not allow such shenanigans...yet this happened and as I hear from a few sources real systems were effected... Maybe there is some nuance with stratums or something which come

At my last employer, they had the brilliant idea to use 111111 as infinity for time in their database. When 2011-11-11 happened, it was very ugly. Some of the software had been fixed AFTER to use 2911-11-11 as the new infinity date, but not all of it. I was still fixing software when I left in October.

My suggestion to make the field NULL didn't go over well. They couldn't figure out how to represent NULL in their C programs that accessed Ingres.:)