Symptoms

On a domain controller (DC) that is running Windows Server 2003, Windows Server 2008, or Windows Server 2008 R2, you may experience the following symptoms:

The server runs slower than expected every several hours.

The server stops responding every several hours.

This issue lasts for several minutes and then disappears.

Note If you use Performance Monitor (Perfmon.exe) to monitor the domain controller, you find that the CPU usage and the disk I/O are very high when this issue occurs.

Note If you enable the garbage collection event logging according to KB314980, you find that the garbage collection process runs when this issue occurs. However, tombstone objects are not deleted.

When this issue occurs, one or more of the following events are added in the Directory Service log:

Event Type: Error

Event Source: NTDS ISAM

Event Category: (14)

Event ID: 623

Date: <date>

Time: <time>

User: N/A

Computer: <computer name>

Description: NTDS (432) NTDSA: The version store for this instance (0) has reached its maximum size of <number>. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.

Possible long-running transaction:

SessionId: <id>

Session-context: <context> Session-context ThreadId: <id> Cleanup: 1

Event Type: Error

Event Source: NTDS General

Event Category: Internal Processing

Event ID: 1519

Date: <date>

Time: <time>

User: NT AUTHORITY\LOCAL SERVICE

Computer: <computer name>

Description: Internal Error: Active Directory could not perform an operation because the database has run out of version storage.

Additional Data Internal ID: 2080490

Event Type: Error

Event Source: NTDS Replication

Event Category: Replication

Event ID: 1479

Description: Active Directory could not update the following object on the local domain controller with changes received from the following source domain controller. Active Directory does not have enough database version store to apply the changes.

Object: <object DN>

Object GUID: <object GUID>

Source domain controller: <DC GUID-based DNS name>

User Action

Restart this domain controller. If this does not solve the problem, increase the size of the database version store. If you are populating the objects with a large number of values, or the size of the values is especially large, decrease the size of future changes.

Additional Data

Error value: 8573 The database is out of version store.

Cause

This issue occurs because there are many inconsistent objects or reference phantoms in the Active Directory. The DELTIME attribute of these objects is set. However, the isDeleted attribute of these objects is NULL. Because of the DELTIME attribute, garbagecollector includes these objects in its database scan. Therefore, these objects cannot be deleted.Many inconsistent objects can cause the garbage collection process to use up version storage space.

This problem occurs during the garbage collection process of the tombstone objects. The tombstone cleanup code in the garbage collection process will update the DELTIME for these inconsistent objects, because it considers them reference phantoms. In this state, the tombstone cleanup code builds one database transaction for these updates and for many consecutive objects. Therefore, the size of the transaction exceeds the size of Version Store.

The Garbage Collector treats this as a fatal error and stops. Any database records that require cleanup will not be looked at, for example tombstones that are due for removal. The procedure is retried on the next run of the garbage collector. Therefore, the next attempt may also fail.

These inconsistent objects are not easy to identify, as DELTIME is an internal database column. They do not appear as such objects in the User Interface or in a database export. If you suspect you are affected by this problem but you want to be positive that applying this hotfix will help you, we recommend that you contact Microsoft Customer Technical Support to investigate the state of the affected databases.

In addition, these events can be caused bytoo many reference phantoms. These database records are placeholders for one-way object links (that have no backlink) in which the target object was deleted and the tombstone removed from the database. However, the reference is not changed. These references have to be cleaned up. You can do this by deleting the attribute value or by changing the reference to a valid object. One example attribute in this class is the "secretary" attribute.

Werecommend that you contact Microsoft Customer Technical Support to help you identify and remove these reference phantoms.

Resolution

The hotfix for Windows Server 2003 is available now. If you encounter this issue on a computer that is running Windows Server 2008 or Windows Server 2008 R2, you can follow the steps provided in the Workaround section. It is recommended that you contact Microsoft Customer Technical Support for further assistance.

Hotfix information

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing the problem described in this article. This hotfix might receive additional testing. Therefore, if you are not severely affected by this problem, we recommend that you wait for the next software update that contains this hotfix.

If the hotfix is available for download, there is a "Hotfix download available" section at the top of this Knowledge Base article. If this section does not appear, contact Microsoft Customer Service and Support to obtain the hotfix.

Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that do not qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, visit the following Microsoft Web site:

Note The "Hotfix download available" form displays the languages for which the hotfix is available. If you do not see your language, it is because a hotfix is not available for that language.

Prerequisites

To apply this hotfix, you must be running Windows Server 2003 Service Pack 2 (SP2). Additionally, Active Directory must be installed.For more information about how to obtain a Windows Server 2003 service pack or a Windows XP Professional x64 Edition service pack, click the following article number to view the article in the Microsoft Knowledge Base:

Installation instruction

At a command prompt, type the following commands, and then press Enter after each command:

ntdsutil

semantic database analysis

go fixup

After the command is executed, you find a file "dsdit.dmp.x" in the folder from which NTDSUTIL was executed that lists all the objects that were corrected. "x" is the execution number of the semantic database analysis, please inspect the newest file.

Restart your computer in normal mode.

File information

The English (United States) version of this hotfix installs files that have the attributes that are listed in the following tables. The dates and the times for these files are listed in Coordinated Universal Time (UTC). The dates and the times for these files on your local computer are displayed in your local time together with your current daylight saving time (DST) bias. Additionally, the dates and the times may change when you perform certain operations on the files.

Windows Server 2003 file information notes

In addition to the files that are listed in these tables, this hotfix also installs an associated security catalog file (KBnumber.cat) that is signed with a Microsoft digital signature.

For all supported x86-based versions of Windows Server 2003 SP2

File name

File version

File size

Date

Time

Platform

Ntdsutil.exe

5.2.3790.4650

312,832

15-Jan-2010

12:42

x86

For all supported x64-based versions of Windows Server 2003 SP2

File name

File version

File size

Date

Time

Platform

SP requirement

Service branch

Ntdsutil.exe

5.2.3790.4650

503,296

15-Jan-2010

15:14

x64

SP2

Not Applicable

Wntdsutil.exe

5.2.3790.4650

312,832

15-Jan-2010

15:14

x86

SP2

WOW

For all supported IA-64-based versions of Windows Server 2003 SP2

File name

File version

File size

Date

Time

Platform

SP requirement

Service branch

Ntdsutil.exe

5.2.3790.4650

807,424

15-Jan-2010

15:14

IA-64

SP2

Not Applicable

Wntdsutil.exe

5.2.3790.4650

312,832

15-Jan-2010

15:14

x86

SP2

WOW

Workaround

To work around the issue, increase the size of version storage by changing the value of the following registry item:

(Note The machine's native pointer size (4 bytes on 32-bit CPUs, 8 bytes on 64-bit CPUs,) is also a factor in calculating version store size. Therefore, using a 64-bit CPU and OS effectively quadruples the default size of the version store.)

Note You cannot set an arbitrary high value, especially on an x86 operating system. You may run out of memory in the LSASS process space for other tasks.

Note Setting the value of "EDB max ver pages (increment over the minimum)" to anything less than 6400 will have no effect. 6400 is the default and the minimum value.

We recommend that you approach the required value in increments of 50MB. The value that enables garbage collection to work should then be the base value tried for additional affected DCs. If the garbage collector does not run successfully when the value is "19200" (302 MB), Werecommend that you contact Microsoft Customer Support Services for help.

Increasing the size of version storage is only a temporary method. It enables the garbage collection code to pass these inconsistent objects. However, this issue may occur again after a tombstone lifetime interval, or when the objects are in scope of the Garbage Collector scan again.

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

More Information

This issue may occur after you conduct an authoritative restore. The problem occurs with databases with 100,000 affected objects or more. You may proactively run the procedure in the resolution section to fix the affected databases, or you may use it reactively when you encounter the Version store errors in the Symptoms section.

For more information about the Active Directory database garbage collection process , click the following article number to view the article in the Microsoft Knowledge Base: