Thursday, January 6, 2011

Hyper V Time Synchronization on a Windows Based Network

A very common query that I get asked to help out with in relation to Hyper-V installations is the small (but very important!) task of getting time synchronization working between the physical environment and the virtual environment.

There are a lot of blog posts out there at the moment covering this topic but I thought I'd throw my 2 Cents in based on the solution that works for me.

Background

Firstly, I need to explain a little about how time synchronization works in a domain environment.

When you install the first domain controller in your domain, this DC is the owner by default of all 5 FSMO roles within Active Directory. The PDC Emulator Role is the one that is responsible for Active Directory Time Synchronization.

When you add member servers or client computers to your new domain, they then default to using the PDC Emulator Role holder as their time synchronization source. If you move the PDC Emulator Role to another Domain Controller, then this will become the time synchronization source for that domain in place of the original DC.

When you install a Member Server into Active Directory and add the Hyper-V role onto that member server things start getting a little bit more interesting!

The physical Hyper-V host that is now a member server in your new domain will use the PDC Emulator Role holder as its time synchronization source as you would expect it to, however, when you add virtual machines to that Hyper-V host and the 'Hyper-V Integration Services' are installed onto each virtual machine, then that VM starts to synchronize its time from the Hyper-V host it is based on and not from the PDC Emulator Role!

This 'VM to Host' time sync generally isn't a problem when all of your domain controllers are physical machines as the Hyper-V host updates itself from a physical machine separate from the virtual environment and this keeps the internal VM's up to date.

What happens though when your PDC Emulator Role holder Domain Controller (gasp for air!) is based on the Hyper-V host as a virtual machine? You guessed it - time sync issues occur and the virtual DC's clock starts to move out away from the time on the Hyper-V host, which in turn then knocks the rest of the domain off-skew as they are- as mentioned above, configured by default to sync from the PDC Emulator Role Holder.

The Wrong Way

A lot of people - including myself for a long time - took the stance that to resolve this problem, the best and easiest solution was to just remove the 'Time Synchronization' tick box from the 'Integration Components Offered' section on the virtual Domain Controller that holds the PDC Emulator Role. Once this step was completed and the time within the virtual DC was configured correctly, this nearly always resolved the time sync issues on site. I use the word 'nearly' in that sentence though as this workaround didn't always work.

The Right Way

So what's the solution? A couple of months ago, I came across a blog post from Ben Armstrong -Virtualization Program Manager at Microsoft -covering this subject and specifically around the Hyper-V 'Time Synchronization Integration Service'. In this post, Ben states categorically that you SHOULD NOT disable the 'Time Synchronization Integration Service' on any virtual machine within Hyper-V and instead you should manipulate the Windows Time Service (w32tm service) from within the virtual DC to get the results that you need for a coherent time sync within your domain!

To summarise his post and outline my steps when faced with time sync issues in a Hyper-V environment, I carry out the following procedure on all my VM's (most importantly, my virtual DC's):

Enable all of the Hyper-V Integration Services (see below)

Check each virtual domain controller's time source using the "w32tm /query /source" command

If the virtual DC is using the 'VM IC Time Synchronization Provider', you need to type the following commands into the command line within the virtual DC to leave the Hyper-V time sync enabled for VM reboots but not for when the VM is up and running:

(This command queries the Windows Time Synchronization Service again and should now detail the internal time server - hopefully the DC with the PDC Emulator Role)

At this point, your Virtual Domain Controller should use itself for time sync when it is online and it will use the Hyper-V host for when it is rebooting or coming out of a 'Saved State' and before the Windows O/S loads

Finally, all that is left to do now is to configure the Virtual Domain Controller and each Hyper-V host that you have in your domain to synchronize with an external time source provider (NTP Provider)

Note: Make sure to follow the instructions in this next paragraph exactly as you read them as clicking on the 'Internal Time Source' option will not give you the desired result!

Use this link from Microsoft to automatically configure an external authoritative time source by selecting the 'Microsoft Fix It' button half way down the page under the 'Configure the Windows Time Service to use an External Time Source':

The 'Microsoft Fix It' button should initiate a wizard that will prompt you to enter your external NTP servers (you can enter two in here for redundancy) and will then configure the registry to reflect these changes automatically.

As I'm based in Ireland, the NTP provider that I prefer to use is - ie.pool.ntp.org

Hopefully this will help you to get a better handle on Time Sync in a Hyper-V environment.

To clarify, everything will synchronise with the PDC Emulator as it normally should do however, if your PDC Emulator is a virtual machine based on a Hyper-V host or cluster, then your Hyper-V host machines will need to be configured to synchronise with an external time source to compensate for the possibility that the virtual PDC Emulator might be powered down or rebooting and the virtual PDC Emulator will have to take it's initial time settings from the Hyper-V host before it boots into Windows to access it's external NTP source.

I have been having several time-related errors, as well as errors documenting failed connection to the PDS, that I believe are related to the time errors. (however, upon checking time sync the computers appear to be syncing perfectly anyway?)

When I first ran the /source command I was indeed syncing to the VM IC TIme Provider, so I followed the instructions and the /source command showed me that I was indeed now syncing from my PDC, however the time errors persisted.

So I ran the fix-it from Microsoft successfully, but now when I check /source it tells me I'm running from the Local CMOS Clock, so I guess not syncing at all? I gathered that it would still try to sync from the PDC first.

Have any idea what's up?

Also something that may be of note, my Hyper-V host machine is running server 2008 core, is not part of my domain, and I prefer to leave it entirely disconnected from the network when not connecting to it for security reasons. So it is obviously not syncing to any external time source.

Thanks very much for your comment, i will try my best to assist you with your problem.

It is a long time since I had to virtualize an SBS 2003 server with Hyper-V, but I do recall having a lot of strange issues with a number of different virtualized SBS 2003 instances based around time sync and also the 'Server' service crashing / hanging too.

In the cases with time sync issues, I recall having to actually 'Untick' the 'Time Synchronization' option in the Hyper-V Integration Components (contrary to what I have blogged about above in fairness) and then ensure that the SBS 2003 server was syncing with an external time source. Once this was configured, the time sync issues in the virtualized SBS 2003 VM went away.

Ok, I have been working with Server 2008R2 and various Hyper-V machines.

I followed everything here as best I could.

It appeared to be working wonderfully. I got the primary DC (Which is a virtual machine) to connect to an external time server, it looked like it was updating and all the other VMs and the physical machine were using the DC as it's time host.

I found a problem today that took 2 weeks to develope. The physical machine was too fast, it had gained 4mins. The DC that was hosted on it took that time and not the external NTP time.

I forced an NTP update on the DC (w32tm /resync /force) and it got the NTP time but within 10 seconds, it has reverted back to the Hyver-V host's time.

If you want to get the VM to stop taking the Hyper-V hosts time, then you need to untick the 'Time Synchronization Integration Service' from within the Virtual Machine properties. However, the point I make in this blog post is that we leave this option enabled and we ensure that the physical Hyper-V host is configured to sync with an external NTP time source.

Hello and thanks for the great post. I am confused though. In my environment, all my DCs are virtual. So I should disable time sync partially as you described?

Then,I should run: w32tm /config /syncfromflags:DOMHIER /update on all DCs (even the PDC)?

And finally, why "..all that is left to do now is to configure the Virtual Domain Controller and each Hyper-V host that you have in your domain to synchronize with an external time source provider (NTP Provider)" Why do you do this on each host and DC. If I just set an external time source on the PDC, shouldn't every other member of the domain sync from the PDC?

HI KevinThanks for taking the time to publish this excellent article.In my case I have A windows2008R2 DC in a Hyper V VM and one other DC (win2008R2) in a separate Physical machine.Once I transfer the PDC role to the separate Physical DC do I need to synchronize with an external time source?Also Do I need to leave the Hyper v Time synchronization service enable in the Hyper V Integration service?Thank You for any help you can provide!!!

If you move the PDC role to a physical computer, then I'd still recommend that you sync it with either an external ntp source or at least a hardware time provider that is onsite. Time sync is such and important part of Active Directory so ensuring your DC's are up to date is crucial.

In relation to Hyper-V, as per my post, you should still leave the Hyper-V Integration Service enabled for all VM's. Just follow the steps above and it'll work perfect.

I have a single physical server running 9 VMs (Two of which are my primary and secondary AD DC). I originally followed your post and got it working wonderfully.

3 months in, I found that the time had crept out of line.

I checked the physical machine, ntp was not set correctly. It was set to "Local CMOS Clock". I thought it strange, used the MS Fix It to set an external NTP server and got it updating from that. So the physical machine works. The other VMs sync to my primary DC. There lies the problem.

My DC is not doing what it should. I tried following all the steps in this post again to get my DC to syncronise with an external NTP server (as the post suggests) and yet nothing I do can get it to change from "Local CMOS Clock". I can set it to an internal machine, but not an external NTP.

Any ideas? I have tried the MS Fix-It, restarting the machine, running through all the steps again and yet nothing changes.

I'm pulling my hair out because this is causing a massive issue, I can't RDP to any desktops because it can't find DNS names, Kerberos tickets aren't being created correctly, and like I said w32tm /resync /force works for about .5 seconds and then it throws it back out of whack.

Only thing I can think is that somehow, AD03 thinks its a PDC or something along those lines?

Review the MS FixIt link again and ensure that you've carried out the correct fix against your servers. There are two fixes, one for synching with an internal time source and one for synching with an external one.

I guess you'd need to use the internal option as you have the Symmetricom device. Try to redo the steps again with this option. If you've already tried that FixIt, then do the opposite and try the other one instead.

The IPv6 resolution might be something to look at though. I'd always recommend leaving that setting enabled on DC's as it's the primary path for Win 2008 DC's to resolve to each other. If you simply uncheck that box in TCP/IP it doesn't actually turn off IPv6 - there's a whole lot more that has to be done (which isn't recommended) to properly disable it.

At worst though, DC's are easy enough to demote and promote again. Why not simply DCPROMO AD03 and leave AD off of it for a day or so and then see if the rest of the servers synch properly. If they do, then you know that you've an issue with AD03 and can then investigate further.

Kevin, thanks for your thoughtful reply! What I did and has worked now, is I disabled time sync on the hyper-v settings for the two ad servers and so far, so good. It has kept time.

The servers have no internet connection, it's in a lab with no access to the outside world so, promoting / demoting is fine since it isn't production, and in production it's not hyper-v it's a real actual blade so, hopefully this isn't something we need to worry about there :)

Why do you recommend not turning off ipv6 via netsh? I don't want any IPv6 happening, nothing, nada. Reason being a certain device that I'll leave unnamed doesn't cooperate well with it so we've disabled it across the network.

Any other ideas as to why though? I didn't want to disable the time sync, but well, it seems that it was necessary.

When I follow this procedure on my virtual DC which is running the The PDC Emulator Role I get stuck when running the following command: w32tm /resync /force

It says 'the computer did nog resync because no time data is available'. I've tried to unregister and register the time service, this worked but still I get the same message when I use the command: w32tm /resync /force.

I noticed the same exact thing in our environment. If it helps, what I did was leave our PDC VM set to use VM IC Time Sync Provider (I just skipped/removed the registry entry step Kevin included). The reason being since we configured the Hyper V host server to sync to an external source, I want the PDC to sync it's time to it essentially using the Hyper V host as it's time server. If you sync that to the NTP Pool server properly and set the time skew parameters in the MS KB, it should not skew.

I've read you post many times, but to me it remains contradictory, sure because I'm not native speaker. You say:

1. Leave Time Synchronisation between DC and Host via Integration Services intact2. Let DC use itself for time sync ("At this point, your Virtual Domain Controller should use itself for time sync when it is online...")3. "...configure the Virtual Domain Controller [...] to synchronize with an external time source provider."

I read that like DC should sync with host via Integration Services, then it should sync with itself and finally it should sync with external source? Hope it's clear what I mean. I'm trying to understand that topic now for days and I would appreciate it if someone could clarify this to me.

Just a comment to thank you for this blog. I was trying to fix an out of sync time on my secondary virtual domain controller running 2012 R2. I followed the instructions on other sites and ended up being stuck on "Local CMOS clock" for the time source. After running the second download option in the linked Microsoft support article, it is now sourcing from the PDC. Thanks again.

Preferred Product

Translate This Page

Speaking at Experts Live

Speaking at Cloud Camp

Speaking at CDC 2018

Total Pageviews

Subscribe To My Blog

LinkedIn

Twitter

My Books

About Me

I'm a Microsoft MVP in the Cloud and Datacenter Management space and work as a consultant for Ergo based in Dublin, Ireland.

I really enjoy the fact that we are in a constantly evolving industry that requires us to up-skill as much as we can and one that provides us with new products and solutions each year to make the job all the more interesting and ever changing!