Friday, June 28, 2013

Safely Demote a Windows 2008/r2 Core Domain Controller

Have you installed shiny new 2012 domain controllers? Do you have old, musty 2008r2 domain controllers laying around? Do you want some french fries?

Me too. Let's take a holistic look at decommissioning a domain controller. This article will not only walk through the actual demotion process but also list important pre and post steps to any successful domain controller removal. Most of the steps outlined here should actually work on 2000 up, but I've focused on 2008r2. Note that doing this in a production environment is something you may want to do over the course of a couple weeks due to reasons that will shortly become apparent. That said, let's go.

Assumptions

New domain controllers are installed and ready to go

Domain Admin Privs

A backup of the server should something go wrong. Note a snapshot will not suffice; as a general practice one should never use snaps on a DC.

Ensure Replication is Working Correctly

Removing a domain controller can be dangerous if your replication isn't appropriately healthy.Check the following event logs on EACH Domain Controller for issues regarding replication/DNS:

Application

Applications and Services Logs->DFS Replication

Applications and Services Logs->Directory Service

Applications and Services Logs->DNS Server

If there are any issues, sort them out prior to continuing.

Re-Point Clients

Next we need to ensure all clients are pointing to DNS servers other than the one you intend on taking down. Do the following:

Update your DHCP scopes with the addresses of the DNS servers you intend on carrying forward

Your DHCP lease duration will determine how long you need to wait for the next step. The DHCP RFC tells us that clients will attempt to re-up their lease at the halfway point of the lease. Therefore, if your lease is 7 days you'll need to wait 3.5 days from when you make the change (Assuming the DHCP is fully available). Once you think all the clients have been moved, then move to the next step.

Enable DNS Logging on the DNS Controller to be Decommissioned

Now that we think the DNS server isn't used, we enable debug logging to ensure that nobody is referencing this server for DNS look-ups. Avoiding outages is more fun than fixing them. :)

Edit the DNS server properties for the server to be decommissioned.

Click the "Debug Logging" tab.

Check the "Log packets for debugging" and set a location for the log. Generally the defaults for this application will be sufficient. Click "OK"

Restart the DNS server service or reboot the server when your environment permits to start logging.

Wait however long you would like before continuing; the longer you wait
the more opportunity you give for devices still pointing to this server
to issue a query. You make the call here depending on your environment.

After that wait period: Open the log you specified above; here we'll be reviewing the log for clients that still need their DNS server settings updated. At the top of the log you'll see a couple pages of initialization information that can be safely ignored. After that there will be a log of the DNS queries that the server has received. The first IP address listed in each entry is the IP of the requesting client. You may see some requests from other domain controllers; so long as the other DCs have their "client" settings configured correctly you can ignore these entries as some DC->DC DNS traffic is normal. Taking that into account, look for machines querying this DNS server and re-mediate them prior to moving on.

Transfer FSMO roles off of the DC

Before demoting the DC you need to ensure there are no FSMO roles served by it. I won't cover this in a step by step at this time to keep the article focused. To do this, refer to one of the two following links:

You may need to add other options as well, especially if this is the last DC in the domain, your Domain Admin privs are associated with a different account, or similar for the DNS service. Also, this assumes you have a DNSDelegation to remove. If you do not you should take that line out. Refer to this technet article for a full guide.

Save the file to the domain controller to be demoted.

Log on to the DC to be demoted.

At an elevated command prompt (default on core) execute:

dcpromo /unattend:"(path to file)"

where path to file=the file you just saved, i.e. dcpromo /unattend:c:\temp\demote.txt

After successful execution the machine will successfully reboot. If you intend on re-using the machine for other operations you can log back in and execute dcpromo /uninstallbinaries to remove unnecessary files. If taking the server down don't bother.

Note: If you cancel a demote operation using this file the
administratorpassword field will be intentionally blanked out by the
installer. You'll need to reset if if you start again.

After the demotion, open up %SystemRoot%\debug\DCPROMO.LOG on the old DC and ensure the operation didn't have any major errors. After confirming the demotion was successful...

Clean Up DNS

After waiting for a replication cycle (should be 15 minutes, but wait an hour to be safe) it's time to ensure AD DNS was cleaned up successfully. Open the DNS manager and point it to one of your new domain controllers. Browse through your zones, including your _msdcs.domain.ext zone. Pay particular attention to the listed NS records in each zone. Those can also be manipulated by right clicking the zone->properties->Name Servers. Ensure only the new DNS servers are listed. Fix any other records that are incorrect; common clean up items are the _msdcs delegation in the primary zone and the sites container under the _msdcs zone. Before making changes to anything other than the the NS records (Name Servers GUI) make sure you fully understand what function the records serve. If you're unsure, this will take some study; here's a good place to start.

It should also be safe to remove the machine from the domain now should you choose to do so. To remove the machine, execute the following on a command prompt:

netdom remove /d:%USERDOMAIN% %COMPUTERNAME%

Also, be mindful that if you were using the old DC in your Windows Time infrastructure you may want to revisit that. Here is an article on how to setup an authoritative time source.

Test

Now that the machine is gone and DNS has been cleaned up, it's time to test your domain. To do so, perform the following tasks:

Run DCDiag on a new DC with the following command line: dcdiag /e /c and review the output for any issues.

Review the event log on each domain controller. Refer to the "Ensure Replication is Working Correctly" section above for the log locations.

That's it! You've successfully removed a DC and made sure nothing will come back to bite you. If you have questions/comments please feel free to leave the below.