Beating the Daylight Savings Time bug and getting correct file modification times

Windows reports erroneous file modification times, which change according to daylight savings. This article describes why this is so and how to determine correct file modification times and avoid the DST bug.

Not many Windows developers seem aware of it, but Microsoft deliberately designed Windows NT to report incorrect file creation,
modification, and access times. This decision is documented in the Knowledge Base in articles Q128126 and Q158588. For most purposes,
this behavior is innocuous, but as Microsoft writes in Q158588,

After the automatic correction for Daylight Savings Time, monitoring programs comparing current time/date stamps to reference data that
were not written using Win32 API calls which directly obtain/adjust to Universal Coordinated Time (UTC) will erroneously report time/date
changes on files. Programs affected by this issue may include version-control software, database-synchronization software,
software-distribution packages, backup software ....

This behavior is responsible for a flood of questions to the various support lists for CVS, following the first Sunday in April
and the last Sunday in October, with scores of people complaining that CVS now reports erroneously that their files have been modified.
This is commonly known as the "red file bug" because the WinCVS shell uses red icons to indicate modified files.

Over the past two years, several people have made concerted efforts to fix this bug and determine the correct file modification times for
files both on NTFS and FAT volumes. It has proved surprisingly difficult to solve this problem correctly. I believe that I have finally
gotten everything right and would like to share my solution with anyone else who cares about this issue.

An Example of the Problem

The batch file listed belowFor example, run the following batch file on a computer where C: is an NTFS volume and A: is a FAT-formatted floppy disk.
You will need write access to C:\ and A:\. This script will change your system time and date, so be prepared to manually
restore them afterwards.

On 27 October, Windows correctly reports that Bar.txt was modified half an hour after Foo.txt, but
the next day, Windows has changed its mind and decided that actually, Bar.txt was modified half an hour beforeFoo.txt. A näive programmer might think this was a bug, but as Microsoft emphasized, this is how they want
Windows to behave.

Solution to the problem

Having spent a lot of time thinking about this problem, I wanted to share the information with people who want to get to the bottom of it,
but I realize that most readers here will just want to grab my solutions and use them. Thus, I am putting the instructions here for using the
solution.

The library I supply contains one exported function: BOOL GetUTCFileModTime ( LPCTSTR name, time_t * utc_mod_time ). Just pass this function a filename
(it can be a fully qualified path or just a file name in the current directory). The function will return TRUE for success, FALSE for failure, and will store
the UTC file modification time in * utc_mod_time. Link with the JmgStat.lib library and you're off and running.
Oh, yes. I have wrapped the library in the namespace Jonathan_M_Gilligan_95724E90_4A88_11d5_80F3_006008C7B14D.
Someone else might have a function called GetUTCFileModTime() and I don't want to collide with it, so I have concatenated my initials with a GUID to
produce a unique, but recognizable namespace. Rather than typeing such a long string for every invocation of the function, you may want to assign a namespace alias,
namespace jmg = Jonathan_M_Gilligan_95724E90_4A88_11d5_80F3_006008C7B14D; and then you can call jmg::GetUTCFileModTime();
Example:

Why Windows has this problem

The origin of this file-name problem lies in the early
days of MS-DOS and PC-DOS. Unix and other operating systems designed for
continuous use and network communications have long tended to store times in GMT
(later UTC) format so computers in different time zones can accurately determine
the order of different events. However, when Microsoft adapted DOS for the IBM PC, the
personal computer was not envisioned in the context of wide-area networks, where
it would be important to compare the modification times of files on the PC with
those on another computer in another time zone.

In the interest of efficiently using the very limited resources of the
computer, Microsoft wisely decided not to waste bits or processor cycles
worrying about time zones. To put this decision in context, recall that the
first two generations of PCs did not have battery-backed real-time clocks,
so you would generally put DATE and TIME commands
into your AUTOEXEC.BAT file to prompt you to enter the date and
time manually when the computer booted.

Digression on systems of measuring time...

By the time of WinNT, wide-area networks and had become
sufficiently common that Microsoft realized that the OS should
measure time in some universal format that would allow different
computers to compare the order (and separation) of events irrespective
of their particular time zones. Although the details vary (different time
structures measure time relative to different events), the net effect is that
all times used internally in Win32 measure time with respect to UTC (what used
to be called GMT).

Having once worked down the hall from the master atomic
clock array for the United States at the National Institute of Standards and
Technology in Boulder, I feel obligated to add a few words about time and
systems for reporting time. Long ago, we used to refer time to GMT, or Greenwich
Mean Time, which was kept by the Royal Observatory in Greenwich, England and was
ultimately referred to the position of the sun as measured by the observatory.
When atomic clocks became the standard for timekeeping, a new standard, called
UTC emerged. UTC is a bastard acronym. In English, it stands for "Coordinated
Universal Time," while in French it stands for "le temps universel coordonné."
Rather than using either CUT or TUC, the nonsense compromise acronym UTC was
adopted.

To understand UTC, we must first understand the more abstract International
Atomic Time (TAI, le temps atomique international), which measures the
number of seconds that have elapsed since approximately 1 Jan 1958, as measured
by caesium atomic clocks. The second is defined to be the amount of time
required for 9 192 631 770 cycles of the caesium hyperfine
frequency. However, neither the day nor the year are exact multiples of this
number, so we take TAI and correct it so that it corresponds to the actual
motion of the earth by adding corrections such as "leap seconds."
TAI measures raw atomic time. UTC measures time coordinated to the
motion of the earth (i.e., so we don't end up having midnight while the sun
is shining or January in midsummer). Details of what UTC
really means, together with a more detailed history of timekeeping, can be found at http://ecco.bsee.swin.edu.au/chronos/GMT-explained.html.

UTC, time zones, and Windows file times

So what does this all have to do with file modification
times on Windows computers? Windows is stuck with some serious problems
integrating FAT and NTFS files compatibly. FAT records file modification times
with respect to the local time zone, while NTFS records file modification (as
well as creation and access times, which FAT does not record) in UTC. The first
question you may want to ask is, "How should Windows report these file
times?" Clearly it would be stupid for dir
and Windows Explorer to report FAT file times in the local time zone and
NTFS file times in UTC. If inconsistent formats were used, users would have
great difficulty determining which of two files was more recent. We
must thus choose to translate one of the two file time formats
when we report to the user. Most users are likely to want to know the file
modification time in their local time zone. This keeps things consistent with
what people learned to expect under DOS and Win16. It also is more useful to
most users, who may want to know how long ago they modified a file without
looking up the offset of their local time zone from UTC.

It is straightforward to translate UTC to local time.
You look up the offset, in minutes, between the local time zone and UTC,
determine whether daylight savings is in effect and add either the standard or
the daylight offset to the UTC time. However, daylight time throws a subtle
wrench in the works if we try to go backwards...

The problem with daylight time

If you want to translate a
time in your local time zone into UTC, it seems a straightforward
matter of determining whether daylight time is in effect locally and then subtracting
either the standard or the daylight offset from the local time to arrive at UTC.
A subtle problem emerges due to the fact that the mapping from UTC to local time is
not one-to-one. Specifically, when we leave daylight savings time and set our clocks
back, there are two distinct hour-long intervals of UTC time that map onto the same
hour-long interval of local time.
Consider the concrete case of 1:30
AM on the last Sunday in October. Let's suppose the local time zone is
US Central Time (-6 hours offset from UTC when daylight time is not in effect, -5
hours when it is). At 06:00 UTC on Sunday 28 October 2001, the time in the
US Central zone will be 01:00 (1:00 AM) and daylight time will be in effect.
At 06:30 UTC, it will be 01:30 local. At 07:00 UTC, it will
be 01:00:00 local and daylight time will not be in effect. At 07:30 UTC, it
will be 01:30 local. Thus, for all times 01:00 ≤ t < 02:00 local, there will
be two distinct UTC times that correspond to the given local time. This
degenerate mapping means that we can't be sure which UTC time corresponds to
01:30 local time. If a FAT file is marked as having been modified at 01:30
on 28 Oct. 2001, we can't determine the UTC time.

When translating local file times to UTC and vice-versa, Microsoft made a strange decision.
We would like to have the following code procduce out_time equal
to in_time

The problem is that if the local time zone is US Central (UTC - 6 hours for standard time,
UTC - 5 hours for daylight time) then in_time = 06:30:00 Oct 28 2001
and in_time = 07:30:00 Oct 28 2001 both map onto the same local time, 01:30:00
Oct 28 2001 and we don't know which branch to choose when we execute
LocalFileTimeToFileTime().
Microsoft picked an incorrect, but unambiguously invertable algorithm: move all times up an hour when
daylight time is in effect on the local computer, irrespective of the DST state of the time
being converted. Thus, if DST is in effect on my local computer, FileTimeToLocalFileTime
converts 06:30:00 Oct 28 2001 UTC to 01:30:00 CDT and 07:30:00 Oct 28 2001 UTC to 02:30:00 CDT. If
I call the same function with the same arguments, but when DST is not in effect on my local computer,
FileTimeToLocalFileTime will convert 06:30:00 UTC to 00:30:00 CDT and 07:30:00 UTC to
01:30:00 CDT.

It may seem strange that this would affect the C library call stat, which allegedly
returns the UTC modification time of a file. If you examine the source code for Microsoft's C library,
you find that it gets the modification time thus:

// pseudocode listing
WIN32_FIND_DATA find_buf;
HANDLE hFile;
FILETIME local_ft;
time_t mod_time;
// FindFirstFile returns times in UTC.
//// For NTFS files, it just returns the modification time
// stored on the disk.
//// For FAT files, it converts the modification time from
// local (which is stored on the disk) to UTC using
// LocalFileTimeToFileTime()
//hFile = FindFirstFile ( file_name, &find_buf );
// convert UTC mod time to local...
FileTimeToLocalFileTime ( &find_buf.ftLastWriteTime, &local_ft );
// Now use a private, undocumented function to convert local time to UTC
// time according to the DST settings appropriate to the time being
// converted!
mod_time = __secret_microsoft_converter(local_ft);

For a FAT file, the conversions work like this:

raw file modification is converted to UTC by LocalFileTimeToFileTime()

UTC converted back to local by FileTimeToLocalFileTime(). Note that this
exactly reverses the effect of step 1, so we are left with the correct local modification time.

local time converted to "correct" UTC by private function

For an NTFS file, the conversions work like this:

raw file modification is already in UTC, so we don't need to convert it.

UTC converted to local by FileTimeToLocalFileTime(). This applies a DST correction
according to the DST setting of the computer's system time, irrespective of the DST setting at the
file modification time.

local time converted to "correct" UTC by private function. Note that this does not
reverse the effect of step 2 because in step 3, we use the DST setting for the file modification time,
not the system time.

This explains the problem I showed at the top of this article: The time
reported by dir for a file on an NTFS volume changes by an hour
as we move into or out of daylight savings
time, despite the fact that I haven't touched the file. FAT modification times
are stable across DST.

Categorizing the problem

There are 3 possible ways I can think of where this inconsistency in
reporting file times may cause problems:

You may be comparing a file on an NTFS volume with a
time_t value stored in a file (or memory). This is frequently seen in CVS and
leads to the infamous "red-file" problem on the first Sunday of April and the
last Sunday of October.

You may be comparing a file on a FAT volume with a
time_t value.

You may be comparing a file on a FAT volume with a file on an NTFS
volume.

Solutions:

For case (1), it's simple. Get the file times using the Windows API call GetFileTime()
instead of using the C library stat(), and convert the FILETIME
to time_t by subtracting the origin (Jan 1 1600) and dividing by 10,000,000 to
convert 100-nanosecond units to seconds.

For case (2), stat()
will work and return a time_t that you can compare to the stored one. If you must use GetFileTime()do not
useLocalFileTimeToFileTime(). This function will apply the the daylight state of the
current system time, not the daylight status of the file time in the argument. Fortunately, the C library
mktime() function will correctly convert the time if you correctly set the tm_isdst
field of the tm struct.

There is a bit of a chicken-and-egg problem here. Windows does not supply a good API call to let
you determine whether DST was in effect at a given time. Fortunately for residents of the US and other countries
that use the same logic (Daylight time starts at 2:00 AM on the first Sunday of April and ends at 2:00 AM on
the last Sunday in October), in which case you can set tm_isdst to a negative number and
mktime() will automatically
determine whether daylight time applies or not. If the file was modified in
the window 1:00-2:00 am on the last Sunday in October, it's ambiguous how
mktime() computes the modification time.

People in time zones that do not follow the usual US daylight rule must brute-force the daylight time
problem by retrieving the applicable TIMEZONEINFO structure with
GetTimeZoneInformation and manually calculating whether daylight time applies.

For case (3), the best bet is to follow the
instructions for case 2 above and compare the resultant UTC time_t with the time
for the NTFS file determined in case (1).

The library (available for download from the link at the top of this article implements this solution
with checking for the filesystem the file is stored under.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Daylight savings time SUCKS. I had the fortune of going to college in one of the counties in Indiana that forbids DTS, then I moved to Arizona for 8 years.. so I did not have to think about that abominable holdover from pre-industrial times for a blissful 12 years. Now I spend two hours a year resetting clock times.

Look guys, we can moan and bitch about it all we want HERE,
and nothing is ever going to get done about it .. I suspect
to do what you all suggest, you're going to have to become
politicians / at least political lobbyists, because it seems
that it has commercial implications too, which means to say
in the States and here in Australia too, we have
DIDDLY SQUAT chance of getting rid of it ...

What could make more sense than adjusting your clock so that you don't sleep through an hour of sunlight and have more time in the evening during warmer months to take your kids to the park &tc ? We're computer programmers here, how damn hard is it to reset you clock twice a year ?

Sheesh !!

Christian

#include "std_disclaimer.h"

People who love sausage and respect the law should never watch either one being made.

The things that come to those who wait are usually the things left by those who got there first.

Do you really only wake up or go to bed when your clock tells you to do so?
Might it be that you're kinda addictive or at least overly dependent on your clock?

If I were in a position to take my kids to the park (have none yet), I'd base that
decision on the current level of brightness outside, not on what my watch is telling
me (it could be lying). OK, I like my watch; Omega really made good stuff a hundred
years ago, but if I forget to wind it up ... well, it gets less precise somehow.

Besides, the sudden shifting of time for 1 hour causes slight jetlag problems.
(Wow! This one without even going to the airport). What about having six different
DST intervals shifting in 10-minute amounts?

Whatever.

This 'adjusting the clock' would no big deal if it wouldn't involve that many (say: all) clocks. That is, every device that comes with a clock has to be re-set twice a year.
Every one. Twice. Every ticket seller. Every cash dispenser. Every ... whatever.

And consider data collected by computer systems. Well, it could be possible to ignore
DST completely (GMT rules), but most customers want to see their own personal local
time information ... and here even MS errs (as the original article here describes).
Or for instance a system that is steering events to happen at a certain time ...
What should happen in the 1-hour interval that is either missing or occuring twice?
Omit the events? Repeat them?

All these problems can be solved, but they would not be problems in the first place if
this (expletive) DST were inexistant.

> Do you really only wake up or go to bed when your clock tells you to do so?
Might it be that you're kinda addictive or at least overly dependent on your clock?

No, I get up every day at about 6 am, and go to bed when I feel like it ( or later, if we are busy ) How does the question of when I go to bed relate to available sunshine in the evening ?

>If I were in a position to take my kids to the park (have none yet), I'd base that
decision on the current level of brightness outside, not on what my watch is telling
me (it could be lying). OK, I like my watch; Omega really made good stuff a hundred
years ago, but if I forget to wind it up ... well, it gets less precise somehow.

When you have kids you'll find they are creatures of routine when it comes to food and are also required by the state to attend school. I can't pick them up an hour early because it's a nice day, even if my wife and I could walk out of work at 4.00 ( hell, it's 8 pm and I'm still here )

>Besides, the sudden shifting of time for 1 hour causes slight jetlag problems.
(Wow! This one without even going to the airport). What about having six different
DST intervals shifting in 10-minute amounts?

I'm 32, and I don't have any DST jetlag. You young fellas are a bit soft, I reckon...

>Whatever.

>This 'adjusting the clock' would no big deal if it wouldn't involve that many (say: all) clocks. That is, every device that comes with a clock has to be re-set twice a year.
Every one. Twice. Every ticket seller. Every cash dispenser. Every ... whatever.

Wow. Every one ? *sarcasm off* OK, I see the hassle, but unless you are personally responsible for hand setting them all in your local area, I still don't see the hassle.

And consider data collected by computer systems. Well, it could be possible to ignore

>DST completely (GMT rules), but most customers want to see their own personal local
time information ... and here even MS errs (as the original article here describes).
Or for instance a system that is steering events to happen at a certain time ...
What should happen in the 1-hour interval that is either missing or occuring twice?
Omit the events? Repeat them?

>All these problems can be solved, but they would not be problems in the first place if
this (expletive) DST were inexistant.

DST was a good idea before computers, and the issues it raises for us do not negate it's benefits.

Christian

#include "std_disclaimer.h"

People who love sausage and respect the law should never watch either one being made.

The things that come to those who wait are usually the things left by those who got there first.

> Well, I guess I was begging for a forceful response ;0)
Slightly sorry for that. But being called a 'pussie' (whatever that might be on the
piece of planet you're sitting on) ...

> No, I get up every day at about 6 am, and go to bed when I feel like it ( or later, if we are busy > )
> How does the question of when I go to bed relate to available sunshine in the evening ?

Ehm. Well. I sometimes "feel like it" depending on the sunshine. If it's dark and depressive
beds are even more attractive. (Where I live winter really sucks.)

> When you have kids you'll find they are creatures of routine when it comes to food
> and are also required by the state to attend school.
> I can't pick them up an hour early because it's a nice day, even if my wife and
> I could walk out of work at 4.00 ( hell, it's 8 pm and I'm still here )

OK, this 'attending school' thing is a valid point.
Food ingestion with kids occurs unrelated to the time of day but more or less related
to feeding time intervals (at least the kids I know operate this way).

> I'm 32, and I don't have any DST jetlag. You young fellas are a bit soft, I reckon...

Tread carefully I'm 33. Jetlag is something highly subjective; I don't have great jetlag
problems either, but DST affects _everyone_. The first day after a switch, approximately
five to ten percent of all people are either too late for whatever they plan or too early.
I remember sitting for one hour in a small greek village in torrents of rain waiting for a
bus. My radio-controlled clock was just too far away from the sender, so was rendered useless.
And this was supposed to be a holiday vacation; I don't run around on these thinking about
bloody (expletive) DST things, I'm happy enough when I know it's Monday. Bugger the year.

> Wow. Every one ? *sarcasm off* OK, I see the hassle, but unless you are personally
> responsible for hand setting them all in your local area, I still don't see the hassle.

Well, consider not carrying your personal time piece around. Strange thought? I second that,
but for some mechanical reason, my beloved pocket watch went defective some time ago
and I had to find a competent "pocket watch repairman" (don't know the correct english term).
Since I dislike wrist watches, I was dependent on watches for public display, which are scattered
throughout the town I live in. These display approximately four different time informations:
1) the correct one.
2) something completely different. Time in Moscow? On the Moon?
3) the correct one +/- 1 hour.
4) nearly the correct one.
Try to catch a train/bus with that.

Another aspect in the "hassle" is that the date for the time shift is not always the same.
I live in europe, and here it is not a straight rule-of-thumb as "always the last weekend in
October", even though it seems to be that way.
No, the european union's council just declares the dates for several years in advance.
When I first heard of that I nearly fell off my tree. (Ehm. Make that a chair).
Some years ago the date of the end of DST was shifted around about one month;
before that, the UK and the EU had different DST schemes.
If you're developing an embedded device that is supposed to run unattended for the next
ten to fifteen years (my company does such things), you're quite close for a good and hearty puke.

> DST was a good idea before computers, and the issues it raises for us do not negate it's benefits.

You're the first living person I came across (in a virtual way) that offers this opinion.
What virtues do you see in it? (DST, not the opinion)

Claudius - I can see Christian's point of view - I used to live in Tasmania,
Australia, and because of DST and the lower latitude, summer the sun
wouldnt go down till 9pm !! - that means after work, you go home, have dinner,
then go sailing/swimming/fishing ... isnt sleep an acquired habit left
over from our distant cave-man relatives who had nothing to do when the
sun went down anyway ??

My 'internal clock' gets shitted off by other people saying its time to eat/
sleep/drink because thats what their watch is telling them to do !!!

Not hard at all. But we are programmers. That means: getting up late and working late. I always get up when the light is shining, but then the Daylight saving Time kicks in I suddenly get home when it is already dark! I used to do all sorts of stuff after work when there was still an hour light (e.g.: riding a bike). On winter daylight saving time, I only ride half as much and have to use lights. And for what? So that I have an hour more sunlight when I sleep in the morning. Time should be like it is in summer all year long!

Daylight savings was introduced so that people would have light in the morning in winters, not to have more light in the evening in the summer (original time is summer time!). People used to get up a lot earlier (and go to sleep a lot earlier too).