DateTime FAQ Entries

I have recently created some new DateTime FAQ entries to address some questions people have about using DateTime on blogs. Our web site is in transition, so I'm posting some of these into the blog. However, they will eventually be rolled into the DateTime FAQ on the BCL web site.

Why is DateTime not always UTC internally even for cases when it is used a local time?

People often ask why DateTime is not always UTC internally, or why we don’t change it to be so now. This would address a number of problems with the DateTime, but there are various reasons why we can’t do it. The short answer is that we probably could have designed DateTime this way from the beginning, but it would not possible to now transmute the DateTime into working this way without creating unacceptable compatibility problems.

In V1.0 the decision was made to have a purely numeric DateTime, and the context of the time zone was to be external. The driving reasons behind this were compatibility, simplicity and performance. Compatibility is relevant because the key legacy systems that .NET needed to interoperate with, COM, Win32 and databases, also had numeric-only instances. Marshaling from one to the other would have required a guess about the time zone context that would be wrong in many cases. There was no easy guess either because the convention for Win32 times was universal and the convention for COM times was local.

Simplicity and performance were relevant because it is much simpler and faster to be dealing with a purely numeric value, rather than having many methods converting back and forth between a local view and a universal representation.

After shipping V1.0 it quickly became obvious that the combination of this simple DateTime representation, combined with the fact that most public APIs were using local instances created a big problem. Local instances have reliability problem during daylight savings boundaries and whenever time zones change.

Serious investigation was done as to whether DateTime could change to be UTC internally and represent an absolute point in time. However this invariably resulted in behavior changes from V1.0 that would cause reasonably functioning code to break. It also created a “performance compatibility” problem in that it would not be possible to retain the performance characteristics of the class to any reasonable degree with all that extra logic going on.

Instead we chose a solution in Whidbey that solved the most difficult problems in the class and presented minimal compatibility risk. Local instances can now be converted back to universal time without data loss, and you can annotate whether the time is local universal or unspecificed. Unfortunately compatibility has also meant that using this updated information has to be opt-in in most cases, so you need to pass in new flags and values to consume this information in technologies like XML Serialization.

Admitted, this is a very unfortunate situation. You still need to know about a lot of the details and pitfalls of DateTime in order to use it correctly. If we could have seen the full implications of the earliest design decisions, we may well have begun with a UTC based DateTime instead. As it is, it will be possible to get reliable DateTime usage in Whidbey, but you have to know what you are doing. We are planning some FxCop rules to help out with this.

All this being said, it may be possible to create an entirely new class that does represent an absolute point in time and does have time zone context. This may be considered for a post-Whidbey release.

What is the Recommended Way to Store a DateTime in Binary?

The way most people serialize a DateTime in Binary form is to use ticks. This the simplest and fastest way to store a DateTime, although this is not the recommended practice for local times, which are discussed below.

public Int64 StoreDateTime(DateTime value) {

return value.Ticks;

}

public DateTime ReadDateTime(Int64 value) {

return new DateTime(value);

}

In Whidbey, the DateTime has additional information about whether it is Local, Universal or Unspecified. To preserve this information there are new ToBinary and FromBinary APIs:

public Int64 StoreDateTime(DateTime value) {

return value.ToBinary;

}

public DateTime ReadDateTime(Int64 value) {

return DateTime.FromBinary(value);

}

These techniques will not work well for local times, and will be inconsistent with recommended techniques for storing a DateTime in Text (see below). It is not recommended because a Local DateTime instance exists in the context of the current machine’s time zone, so if this changes while the DateTime is persisted it will render the data invalid. The most typical scenario here is if the reader and writer are different machines in different time zones.

If you deal with Local DateTime instances, the recommended way to handle this is to store them as UTC:

public Int64 StoreDateTimeLocal(DateTime value) {

return value.ToUniversalTime().Ticks;

}

public DateTime ReadDateTimeLocal(Int64 value) {

return new DateTime(value).ToLocalTime();

}

This obviously makes things difficult if you don’t know whether the data is local at compile time. A binary serialization API that is more consistent with the new text serialization APIs is still being considered for the Whidbey release.

What is the Recommended Way to Store a DateTime in Text?

Most of the standard formats for the DateTIme do not preserve all the information in the DateTime. The also vary from one culture to another. Thus, the recommended format for storing a typical DateTime is like so;

public string StoreDateTime(DateTime value) {

// e.g 2003-10-26T14:33:41.1234567

return DateTime.ToString(“yyyy-MM-ddTHH:mm:ss.fffffff”,

CultureInfo.InvariantCulture);

}

public DateTime ReadDateTime(String value) {

return DateTime.Parse(value, CultureInfo.InvariantCulture);

}

This format is useful, both because it is a standardized format, used in XML among other standards, and because with the 7 decimal places, it retains all the precision of the DateTime.

There is a critical piece of information that is lost here, which is the context of the time zone of the DateTime, if it is available. If you are communicating with a 3rd party system, or if your application has writers and readers on different time zones, it is recommended to use a slightly different format.

If you handle your Dates in Universal time, use this format:

public string StoreDateTimeUtc(DateTime value) {

// e.g 2003-10-26T14:33:41.1234567Z

return DateTime.ToString(“yyyy-MM-ddTHH:mm:ss.fffffffZ”,

CultureInfo.InvariantCulture);

}

public DateTime ReadDateTimeUtc(String value) {

return DateTime.Parse(value, CultureInfo.InvariantCulture,

DateTimeStyles.AjdustToUniversal);

}

Note that it is important to pass AdjustToUniversal to this routine, because the output format identifies the time as UTC, and converted to local time by default by Parse if they have any sort of time zone marker.

If you handle your Dates in local time, use this format:

public string StoreDateTimeLocal(DateTime value) {

// e.g 2003-10-26T14:33:41.1234567-07:00

return DateTime.ToString(“yyyy-MM-ddTHH:mm:ss.fffffffzzz”,

CultureInfo.InvariantCulture);

}

public DateTime ReadDateTimeLocal(String value) {

return DateTime.Parse(value, CultureInfo.InvariantCulture);

}

Note that in this case, if the reader and writer are in different time zones, because Parse converts to local time by default, it will be effectively adjusted to the local time as it is parsed in.

Beware: do NOT use a UTC format to store a local time or a Local format to store a UTC time. It will actually seem to work at first, but it will not correctly adjust when time zones change, and 3rd party systems reading the information will get the wrong date.

In Whidbey from Beta 1 onwards, there is a simpler way to do all this:

public string StoreDateTime(DateTime value) {

return DateTime.ToString(“o”, CultureInfo.InvariantCulture);

}

public DateTime ReadDateTime(String value) {

return DateTime.Parse(value, CultureInfo.InvariantCulture,

DateTimeStyles.RoundTripKind);

}

The “o” format is a new shortcut. In Whidbey you can store whether the DateTime is Local, Utc or Unspecified in the DateTime instance. The “o” format will pick a different format depending on what the Kind property of the DateTime is. The RoundTripKind style will effectively preserve what the kind was when persisted based on the format read in. The above formats are not invalid, but since they can only handle one type of DateTime, this the preferred means of persistence if you get DateTime instances from unknown sources, such as if you are writing a serialization engine.

I have some questions about the comments here. Pawel, you say that we have guidelines that are incorrect. Please indicate which ones are incorrect. We recommend converting to UTC before doing any arithmetic. Is that something you would consider incorrect? If you mean that it does not take relativety into account or that standard clocks occaisionally get tweaked by a few seconds, I would contend that this is beyond the scope of a DateTime representation for normal business use.

We are aware that ToBinary is not currently suitable for persistence accross time zones. This is something that should be addressed before we ship. If people confuse this with the Ticks, they should promptly get an argument exception.

There are number of problems in your approach, and no, my concept applies to business applications as well as other common problems associated with time. Coincidentally it also allows you to correctly deal with time relativity (curved time field), but this is not the argument I make. Although, I can see nothing wrong if NASA could use DateTime and be rest assured to land on Mars and not Neptune ;-) However, the applications I had in mind are more down to earth.
I explained that your recommendation is not deterministic, therefore not possible (without at least an occasional error).
One has to understand what time is in order to come up with a right model. For example, using ticks for that purpose is like counting wrinkles on your bum to tell the age - easy but not very reliable (I have many wrinkles on my bum and I still feel young). The best way (that I found) is to model the time as a tensor object - just like gravity, stress, displacement, coordinate or even the amount of money Microsoft has in it's account (I told ya it’s generic). Having tensor object representation, it allows for certain operations, some of which may be linear, some may be continuous, and some may be reversible. Such ability allows consistent, correct and well-defined operations (such as conversion to Sydney daylight saving adjusted time, or a number of seconds since Bush became the president). Using your approach the number of seconds that Bush was the president measured in Sydney and in New York would be different! This is a fault (one of many) in your approach and you do not need to land on Neptune instead of on Mars to find it out.

I would be happy to get involved in providing a better solution for Microsoft. You can contact me directly off-line.

There is one serious problem in your recommandation to store date time format as "yyyy-MM-ddTHH:mm:ss.fffffff". This format is not acceptable by lots of COM code written in VB6, VBScript and VBA function (you always find these things in enterprise environment in obscure places like VBA functions in Access queries). The reason these functions fails is because,

1. They don't like fractional seconds part.
2. The "T" as seperator between date and time is not acceptable in some non-English local like Dutch. For example DateValue() call in VBA/VB6/VBScript getting executed in Netherlands will fail if you feed data generated by .Net program in above format.

The format that you should use if you want your data to be readable in all worlds is this:

"yyyy-MM-dd HH:mm:ss"

Yes, you loose fractions of seconds but in a realistic world, this is the only format I've found that works in all different locals with .Net and with VB6 COM and VBScript and VBA.

It would be really good to add a shortcut for this format in Whidbey because most people aren't aware of this. Its too late when you get a call from Japan that your new data generated from glorious .Net app doesn't go hand in hand with their queries fired on linked tables in Access :).

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Whenever people say "we mustn't be sentimental", you can take it they are about to do something cruel. And if they add, "we must be realistic", they mean they are going to make money out of it.