Amazingly, this article has already been viewed over 260,000 times! An incredible response. Of course the Microsoft documentation in this area is weak at present, and UPS is what you could call a “rough edge” of SharePoint 2010. But even so this is an amazing number. Furthermore the amount of time I spend helping others with UPS related problems as a result has inspired this follow up article which will cover the most common issues people experience.

Before I dive in, I must stress that the number one reason people have problems is that they do not follow the procedure! No really, I can’t count the number of times steps have been missed or I get a response like “oh, I didn’t think I needed to do that”. If you follow the procedure you will be successful unless you are hitting an environmental or other known issue.

I won’t detail all of the known issues in this article, but rather cover the most common ones. The general idea is to have a reference for common queries and so on. I will update this article over time as “new” things are “discovered”.

Once again I cannot stress how important it is to follow the procedure in the Rational Guide. If you haven’t, go there first!

This article will use the following definitions for the various components involved:

UPA: The User Profile Service Application (in Manage Service Applications)

Why does UPS get “stuck on starting”?

Why does it get stuck? Well, 15 provisioning runs will be attempted before it gives up. When we hit Start in Services in Server a series of tasks are run, which are akin to running the second stage of the Forefront Identity Manager setup. Those include things like setting up certificates, windows services, installing database schemas so on. If something is incorrect, it will try all 15 times, and more often than not, fail 15 times. That’s why it appears to be “stuck” when in fact its attempting each of the runs. Of course this can take a while, hence the appearance of being “stuck”. After around 20 to 45 minutes the status will return to Stopped, and you can attempt to start it again.

In some cases, the status will remain as “Starting” even after reboots. In this case something very bad has happened and you are best off throwing away the UPA service application and starting over. Hacking out a solution is not a good idea, and will almost certainly cause you further problems in the future.

You can however, manually kick things back to the state before the starting of the UPS service instance. This can be necessary in some cases, but again, I strongly recommend you start clean by deleting and recreating the UPA. Steps to reset the state can be found here.

Incorrect Permissions

Incorrect Permissions are the number one cause of the infamous “Stuck on Starting” behaviour. There are no ways around the account requirements. If you don’t set the permissions correctly things won’t work, its that simple.

There are two accounts we need to configure. The UPS Service Instance Service Identity, i.e. the account that the two FIM services run as (which is the Farm Account). And the Synchronization Connection account, this is the account which actually performs the sync.

The UPS Service Instance Service Identity.

Firstly, you must run the UPS Service Instance as the Farm Account. Just because its possible to change the service identity in Manage Service Accounts, doesn’t mean it will work, and more importantly it is unsupported. So don’t change it.

Next, the Farm Account must be a local administrator of the machine running the UPS Service Instance during provisioning only. When we hit Start in Services in Server a series of tasks are run, which are akin to running the second stage of the Forefront Identity Manager setup. Many of these tasks require local machine administrator rights. You must grant this right before hitting start, and more importantly they must be applied. As the Farm Account is running services on your box (SPTimerV4 and the Central Admin app pool) we must simulate a log off and log on for the change in rights to be applied. You can do this by restarting SPTimerV4, or better yet, rebooting the machine. Once UPS is provisioned you can remove the Farm Account from the local administrators group.

[Update]Please note that any event in your farm that requires the UPS service instance to be provisioned will require the Farm Account be a local admin. Such events include the re provisioning of the service instance following the deployment of a SharePoint Cumulative Update and performing a Farm Backup from Central Administration (which stops and starts the UPS service instance). Don’t forget to ensure that the correct rights are assigned (and actually taking effect) when planning and scheduling your farm operational maintenance tasks.

Also, the Farm Account must have Log on Locally rights on the machine running the UPS Service Instance. Now be careful, you may think the account has these rights, and it will do if you made it a local administrator. However once you remove the admin rights, you may not have Log On Locally. If you don’t various operations will fail.

The Synchronization Connection Account

The Synchronization Connection Account must have Replicating Directory Changes on the Domain you are syncing with. It is the only way sync will work. It’s a hard requirement. Despite the scary name the Replicating Directory Changes permission makes zero changes to AD. It provides a change log capability, which improves the speed of operations such as Sync.

If you are syncing with a Windows Server 2003 functional level domain, the Synchronization Connection Account must be a member of the Pre-Windows 2000 Compatible Accessgroup.

If you wish to do an Export, the Synchronization Connection account must have Create Child Objects and Write permissions on the OUs you are syncing with. Watch out for a common gotcha, that these rights are set on the container level only, you must ensure that This object and all descendants is selected.

So there you have it, a summary of the rights required. What if you are not the domain administrator? Well before you start, ask for proof in the form of a screenshot that they are set correctly. Make sure you verify these permissions before you start working in SharePoint. Over and over again I’ve seen people waste hours (or days) on this because they weren’t.

It’s also very common to mix up these accounts. Remember the Farm Account runs the service, and the Sync Connection account does the sync. Its not very sensible to use the same account for both.

You don’t need to log on as the Farm Account ever to get UPS (or anything else) working. Logging on as the Farm Account is a very, very bad idea and you shouldn’t be doing it.

You of course require Farm Administrator rights to perform the actions related to the farm, service application administrator rights to perform actions related to the service application, and local machine administrator to perform actions related to the machine configuration detailed below.

Health Analyzer Warning Please note, that the Health Analyzer will report an error: “The server farm account should not be used for other services”. This will pop up every week by default. Unfortunately, this Health Analyzer Rule is not smart enough to understand that you have to run UPS as the Farm Account. Therefore this error (with respect to UPS only) should be ignored. You could choose to disable this rule if you are confident that your farm administrators won’t use the farm account elsewhere. One hopes that this will be modified in a future update.

Configure Service AccountsWhilst it is possible to change the UPS account using Configure Service Accounts, this is NOT supported and it will not work properly. You may think it’s working, but you cannot have 100% fidelity UPS under any account other than the Farm Account. It is a bug that UPS shows up in this screen.

If you can’t get the UPS provisioning to even kick in and attempt to work, its likely you have run the Farm Configuration Wizard (FCW) and are attempting to start the UPS Service Instance associated with the User Profile Service Application created by the FCW.

Whilst it is possible to get UPS working in this configuration, you shouldn’t be doing it, and I’m not going to detail the steps.

Don’t use the Farm Configuration Wizard! That’s all I have to say on this issue.

If the NetBIOS domain name and it’s fully qualified name do not match there is additional configuration necessary. This does not effect provisioning, but it will prevent sync from working. You must do the steps below in the correct order, otherwise you will encounter problems with the SyncDB. Do them in the correct order!

Additional Permissions (Do this first)

The Synchronization Connection account must have Replicating Directory Changes on the cn=Configuration naming context. You can also perform this using the Advanced Features view of ADUC if you wish.

Start… Run… ADSIEdit.msc

Connect to the Configuration Partition

Right click the configuration partition and choose properties

From the Security tab, add the Synchronization Connection account and give it Replicating Directory Changes permissions

Configure the User Profile Service Application to support NetBIOS names

You do this after creating the service application, but before provisioning the UPS Service Instance.

Run the following Windows PowerShell:

$upsa = Get-SPServiceApplication –Id <GUID of User Profile Service Application> $upsa.NetBIOSDomainNamesEnabled=1
$upsa.Update()
# To get the GUID of the User Profile Service Application run Get-SPServiceApplication.

Now we can go ahead and provision UPS and configure our Synchronization Connections.

[UPDATE]
Note: the December 2010 Cumulative Update breaks this capability and after setting NetBIOSDomainNamesEnabled, you will not be able to create Synchronization Connections. If you need this capability, do not install the December 2010 CU!

This issue is resolved in the February 2011 CU. Once you have applied the CU and then set the property of the UPA, perform an IIS Reset before attempting to create sync connections.

If you are hosting your SharePoint databases on a SQL Server Named Instance there is additional configuration required.

The SharePoint Configuration Wizard (PSConfig) supports named instances (e.g. SQLServerName\InstanceName), but UPS does not. Therefore if we wish to use named instances and UPS we must configure an Alias, and we must do this before we run PSConfig to create the farm.

Note: this issue is fixed in the June 2010 Cumulative Update for SharePoint Server. Thus if we are running the June CU or later, we do not need to configure an alias. However, I strongly recommend you use an alias, as it is a best practice for addressing SQL named instances regardless of this issue.

To do this we should run the SQL Server Client Network Utility (which is installed on every SharePoint machine).

Start… Run..

Type cliconfg and click OK.

Click TCP/IP and then the Enable >> button.

Click the Alias tab.

Click the Add button.

Select the TCP/IP radio button.

Enter the alias you wish to use (e.g. SHAREPOINT) in the Server alias text box.

Enter the address of your instance (e.g. SQL1\SHAREPOINT) in the Server name text box.

Deselect the Dynamically determine port check box.

Enter the port of your instance (e.g. 1433) in the Port number text box.

Click OK to save the alias.

Click OK to save the configuration and close SQL Server Client Network Utility.

Once we have an alias we can create our farm using it. However there is also another step necessary for reliable start up of the UPS service instance. Basically what happens is that we can provision UPS, but when we restart the machine (for example after patching the box) the UPS services will fail to start. We should configure this before starting the UPS service instance for the first time to avoid the issue completely.

We need to open up network access to the Local DTC on the machine hosting the UPS Service Instance, which is done using the Component Services MMC Snap In:

Now we can provision UPS and it will start reliably following a machine restart.

Note: it is common to see an Event in the Application Event Log complaining about lack of Local DTC Network Access during UPS provisioning. However the only time you need to alter this configuration is if you are using a SQL Server Named Instance.

If you have created the User Profile Service Application (UPA) using PowerShell there is a bug in the creation of the Sync DB which will prevent the UPS provisioning from succeeding. Before we start UPS we must fix this up.

The issue is that the default schema for the Farm Account in the Sync DB is set incorrectly. When UPS provisioning attempts to install the schema it will fail. This is one of the times we MUST touch the SharePoint databases. At present this is the ONLY way to get this working.

We simply need to change the Default Schema of the Farm Account to be DBO after we create the User Profile Service Application and before we start the UPS Service Instance. We do not need to make the Farm Account a DBO.

Please note that altering the Database in this fashion is unsupported. Another workaround here is to create the UPA from Windows PowerShell whilst logged on interactively as the Farm Account using UAC elevation. This effectively does the same thing as creating the UPA with Central Administration (which of course runs under the Farm Account).

So you have a choice about how to fix this issue. Either run the Windows PowerShell while logged on as the Farm Account, or alter the database user default schema. Whilst the database approach is unsupported, logging on as the Farm Account is a very bad idea, and something you shouldn’t be doing. If you do this to avoid this issue cool, but you shouldn’t be doing it generally. Of course the logon as Farm Account approach has serious implications for any automated build scripts you may be pursuing.

An alternate to the log on as the farm account approach is to make use of the Get-Credential and Start-Job Cmdlets when creating the User Profile Service Application. This allows us to avoid the default schema problem without logging on interactively or manually touching the database. It’s not ideal, but it is a fully supported approach, and it works!

If you have already created the User Profile Service Application, you either have to throw it away and start over using the above, or manually fix up the default schema for the farm account. Note that this is NOT supported.

If you are running your SharePoint Central Administration over SSL (which you should be in production) on the default zone, profile synchronization will fail and you will notice events 6801 in your event log. This is a bug where the core FIM Management Agent is configured with incorrect connection information. Actually it’s not incorrect, but it doesn’t account for a SSL connection.

Note: this issue is fixed in the October 2010 Cumulative Update and later.
The following steps are for those who cannot move to the October 2010 CU or later. This approach is NOT supported.

To fix this we must manually edit the Management Agent Connection Information. We must do this after UPS provisioning.

When you attempt to Populate Containers or save a Synchronization Connection you may receive a client side timeout error. This can be due to a large number of containers in the domain, or other domain connectivity issues. We can adjust the timeouts used using Windows PowerShell.

Firstly, the Populate Containers timeout, which by default is 30 seconds. We set this property on the User Profile Service Application Proxy:

$upaProxy = Get-SPServiceApplicationProxy -Id <GUID of User Profile Service Application Proxy>
$upaProxy.ImportConnAsyncTimeout = 45
$upapProxy.Update()
# To get the GUID of the User Profile Service Application Proxy run Get-SPServiceApplicationProxy

Next, the Save Synchronization Connection timeout, which by default is approximately 17 minutes. We can adjust this value (in milliseconds this time) on the Service Application:

$upsa = Get-SPServiceApplication –Id <GUID of User Profile Service Application>
$upsa.FIMWebClientTimeOut = 240000
$upsa.Update()
# To get the GUID of the User Profile Service Application run Get-SPServiceApplication.

Lastly you may receive timeouts when simply connecting to the domain. By default the maximum time is 30 seconds. To alter this value, we must install the June Cumulative Update or later. Once we have done that we can modify the connection timeout on the Proxy:

$upaProxy = Get-SPServiceApplicationProxy -Id <GUID of User Profile Service Application Proxy>
$upaProxy.LdapConnectionTimeout = 45
$upapProxy.Update()
# To get the GUID of the User Profile Service Application Proxy run Get-SPServiceApplicationProxy

I strongly recommend that you slowly increase the timeouts until they work, rather than simply entering a very big number! : )

If you are hosting everything (Active Directory, SQL Server, SharePoint) on a single server Virtual Machine, it is very common for the UPS service instance to be stopped after a restart of the machine. This is commonly because the two FIM services need SQL Server up and running and responding to connections. If SQL Server takes along time to come up as is often the case on a domain controller, the FIM Synchronization service will fail to start.

You can avoid this issue by changing the start-up behaviour of the FIM Synchronization service to “Delayed Start” within Services.msc. You could configure the FIM Synchronization service to have a dependency on SQL Server, but I don’t recommend that. Of course for production environments you shouldn’t be making either change, but then for production you should never host AD and SQL and SharePoint on the same machine!

After the installation of Cumulative Updates, the UPS service instance needs to be re-provisioned.

After the installation of Cumulative Updates for SharePoint 2010, the UPS service instance will require re-provisioning, and it’s state will be stopped. This behaviour is by design. When CUs are applied, we run PSConfig(UI), which amongst other things will upgrade the schema of the SharePoint databases. However PSConfig doesn’t touch the Sync DB. Hence we need to re-provision UPS to update any changes to it’s schema. Re-provisioning is significantly quicker than the initial provisioning but does require the same rights.

Re-provisioning the UPS service instance does not affect the User Profile Service Application (UPA) it simply means repeat the process of Starting the UPS Service Instance.

When attempting to Manage the UPA, you receive an error, Not Found, Correlation ID: [guid], or Could not load file or assembly 'Microsoft.ResourceManagement’

This can occur once you have installed the June 2010 Cumulative Update (CU) for SharePoint Server 2010. The June CU actually requires installation of two different hotfixes (in addition to the SPF June CU). You must install both of these. If you install just KB983497, you will receive the above errors.

I strongly recommend you deploy the August 2010 CUs instead, which offer a much better installation experience.

After the installation of the August CU, Event ID 3 is repeatedly logged to the Application Event Log

This is a known issue with the August 2010 CU, or later, which is due to a version mismatch in the Sync DB. The error can be safely ignored, and UPS will continue to work as expected. However there are cases where this mismatch causes operational problems. This error does not occur if you install the August 2010 CU or later before provisioning the UPA Service Application and UPS service instance.

If you want to get rid of this message you can also delete and re-create the UPA service application. This is a pretty drastic action for an erroneous error! and you will need to consider the data you are throwing away in this procedure carefully. I do not recommend this course of action. Whilst you can do this and retain data, its not clean. If you can, wait for a future fix which will resolve the version mismatch and therefore the erroneous event log entry.

However, this is not a fix that can be easily delivered by PSConfig. As mentioned above in the section about CUs, PSConfig does not touch the Sync DB, and it’s the provisioning of UPS that fixes up the mismatch, as you can see below:

Therefore don’t expect a fix for this. Get your UPA in overall good shape now, don’t wait thinking this will be sorted in the near term.

After clicking Start in Services on Server, nothing happens for 20 minutes and UPS is not provisioned. No ILM Configuration steps are logged to the ULS

This condition occurs when a Fully Qualified Name (FQDN), for example, SQLSERVER1.domain.com, or IP Address has been used as the address for the SQL Server in the SharePoint Configuration Wizard (PSConfig) when creating the Farm. UPS (amongst other things) has major issues with FQDNs and IP addresses. Don’t use them! Use a NetBIOS name (e.g. SQLSERVER1) or a SQL Server Alias when creating the farm.

The best way to view progress of UPS provisioning, and identify the root cause of any failures is to examine the SharePoint ULS logs. All events related to the provisioning of the UPS service instance are logged with a Category of User Profiles. Therefore we can filter the logs for these entries. I strongly recommend ULSViewer which allows you to view events in real time, alongside a host of other features. ULSViewer is the single most important tool in the box for a SharePoint practitioner.

If you are hitting problems and contact me for help, I will ask you for the ULS.

Here you can see the events which take place after hitting Start in Services on Server for the UPS Service Instance in ULSViewer, you can click the image for the full size.

A successful provisioning, as you can see above, takes around three minutes.

Any errors will be highlighted in here which will point you to the appropriate troubleshooting steps above, which is more often than not (over 75% of all queries I have got), incorrect permissions!

In some cases, the UPS status will remain as “Starting” even after reboots. In this case something very bad has happened and you are best off throwing away the UPA service application and starting over. Hacking out a solution is not a good idea, and will almost certainly cause you further problems in the future.

You can however, manually kick things back to the state before the starting of the UPS service instance. This can be necessary in some cases, for example if you have provisioned the UPA with Windows PowerShell, but not set the correct default schema for the Farm account before attempting to start UPS.

I strongly recommend you start clean by deleting and recreating the UPA.

The following Windows PowerShell can be used to reset the Sync Machine Instance:

Add-PSSnapin Microsoft.SharePoint.Powershell
# these will only work if there is one DB and one SA on the box.
# If more than one, then use Get-SPDatabase and Get-SPServiceApplication
# to grab the GUID and pass that in instead of the pipebind
$syncDBType = "Microsoft.Office.Server.Administration.SynchronizationDatabase"
$upaSAType = "User Profile Service Application"
$syncDB = Get-SPDatabase | where-object {$_.Type -eq $syncDBType}
$upa = Get-SPServiceApplication | where-object {$_.TypeName -eq $upaSAType}
$syncDB.Unprovision()
$syncDB.Status = "Offline"
$upa.ResetSynchronizationMachine()
$upa.ResetSynchronizationDatabase()
$syncDB.Provision()
# we MUST restart the timer service for the state to be reflected in
# Services on Server and Manage UPA
restart-service SPTimerV4
# at this stage we MUST add the Farm account to the SyncDB (the above
# steps remove the user) remember the default schema must be 'dbo'.
# If we don't do this, UPS provisioning will fail.

This is due to insufficient privileges for the SharePoint Farm Account on the Sync DB (not the Config DB!). You need to add the farm account to the Sync DB users as DBO with a default schema of DBO and then start UPS again.

After installing the June 2011 CU the User Profile Synchronization service instance will show as Stopped within Services on Server. Additionally while the Forefront Identity Manager service will be started in Services.msc, the Forefront Identity Manager Synchronization service will be stopped.

You must restart the UPS service instance from Services on Server after installing the June 2011 CU. Remember to ensure that the Farm Account is a local administrator on the machine running UPS!

Profile Synchronization fails with Stopped Extension DLL errors reported in the Synchronization Service Manager (miisclient.exe) and System.PlatformNotSupported exceptions are written to the Event Log. This is due to a change made to the FIM config files.

Remove the following lines from %PROGRAMFILES% \Microsoft Office Servers\14.0\Synchronization Service\bin\miiserver.exe.config

<supportedRuntime version="v4.0.XXXXX"></supportedRuntime>

Remove the following lines from %PROGRAMFILES% \Microsoft Office Servers\14.0\Synchronization Service\bin\mmsscrpt.exe.config

<supportedRuntime version="v4.0.XXXXX"></supportedRuntime>

Restart the FIM and FIM Sync services using Services.msc or the command prompt

[UPDATE: 12/07/2011]The June 2011 CU was “re-released” today. If you deploy the new version of this CU, the problem does not occur.

Conclusion

There we have it. Details of some common problems with the User Profile Synchronization capability in SharePoint Server 2010, which I hope will be useful. I will be updating this article from time to time in the future, hopefully by adding “install this patch” : ) but also as new issues come to light.

.

Feedback

Thanks for being the source on the UPS goodness. My early attempt at UPS setup failed miserably even with outside assistance. Your work surrounding this service has given me a worry free UPS, thus far.

Thank you so much. We've been stuck with our planned deployment because things were timing out when trying to create our connections. I opened up a case with MS last week about it, and conveniently, they came back with the suggestion of increasing the LDAP timeout on our User Profile Proxy this morning. I'd lay a significant chuck down betting that the tech just grabbed the answer from you.

Michhes, you should only delay start the FIM Sync service in non production environments. If you are having issues with startup in production, you should check out the DTC task - this can be done at any stage - you don't need to do it before you provision the farm.

Spence, thanks a lot for this great post and also for the rational guide...When you say that the UPS service instance needs to be re-provisioned after the installation of CU, does it mean that the user profile application service must be removed and recreated with previous databases?

This is in fact the best hand on Labs and reference resource about the UPS, this is for sure, thanks for the info it is greatly appreciated, i owe you at least a dinner, so if sometime you come to Barcelona tell me and at least some beers will be cool to celebrate a working UPS ;)

Hi, I have tried all the steps above i still get the following errorI tried it 4 times in test and the production environment. It always gets stuck while starting the service. and then gives me this error.

1) .Net SqlClient Data Provider: System.Data.SqlClient.SqlException: HostId is not registered at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) and also this error

UserProfileApplicationProxy.InitializePropertyCache: Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application. at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.GetProfileProperties() at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.RefreshProperties(Guid applicationID) at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValueNow(K key) at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValue(K key, Boolean asynchronous) ......at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.InitializePropertyCache()

Hi Spence.Thanks for everything you've done for me over the years.I've been busy building farms for the last few months and in all that time I've had no issues with UPSA; until now.I've restored the Profile and Social DB's from one of my other farms over the top of the most recent one to get user data and user properties in place rather than manually adding the properties and risking adding problems through human error.Since I did this (I've done it numerous times before) the UPSS will not start up.I've checked and double checked the database permissions and everything is identical to the other farms where things are working fine.I've rebooted the servers in the farm (1x WFE and 1x App server) and I've got to the point where I'm ready to blow the Service App away and start over but I don't want to admit defeat.Any ideas?

I think it might be a bit too late for you as you most likely blew away your databases but hopefully this will help others. I was having the same issue of the synchronization service not starting after being unprovisioned. I was reading this article to see what might be the problem and I came across this section:

"Firstly, you must run the UPS Service Instance as the Farm Account. Just because its possible to change the service identity in Manage Service Accounts, doesn’t mean it will work, and more importantly it is unsupported. So don’t change it.Next, the Farm Account must be a local administrator of the machine running the UPS Service Instance during provisioning only. When we hit Start in Services in Server a series of tasks are run, which are akin to running the second stage of the Forefront Identity Manager setup. Many of these tasks require local machine administrator rights. You must grant this right before hitting start, and more importantly they must be applied. As the Farm Account is running services on your box (SPTimerV4 and the Central Admin app pool) we must simulate a log off and log on for the change in rights to be applied. You can do this by restarting SPTimerV4, or better yet, rebooting the machine. Once UPS is provisioned you can remove the Farm Account from the local administrators group."

Because we have had problems in the past with the UPSS I tend to watch services.msc as the service is being provisioned to watch it set the service accounts, startup type and then start the services. This time I noticed it was trying to start the FIM sync service as the synchronization account and not the farm admin, this made me go check settings. It turns out that SharePoint changed the manage service account entry to our sync account. I have a feeling there is some bug in the UPS provision process that checks for synchronization connections and if there is one it changes the UPSS managed account to the account used there instead of the farm account. Which according to the above is a no-no. I am about a year behind on my blogging but if I get a chance i'll write this up on mine in more details.