Issues with roaming profiles on 2003 TS /citrix environment.

Hi Guys, I am trying to resolve a issue that is really doing my head it. I have a 2003 r2 x86 server that is being used as a citrix presentation server 4.5. It has hotfix rollup 7 applied. This server has been alive for approx 4 years now and never had a issue.

2 weeks ago multiple times when i log in id see the following in c:\documents and settings

Username
Username.domain
Username.domain.000
Username.domain.001 etc

Now i have gone through the usual steps of a week full of process monitor/explorer and modifying the LogoffCheckSysModule under the HKLM\System\CurrentControlSet\Control\Citrix\wfshell\TWI.

I am at the stage where now i can see users log in and out with absolutely no remaining processes yet the the profile remains. In the task manager tab it shows the user log in and leave the server. Process explorer shows all processes that started as the user logs in, terminate when the user logs out. There is no reason i can see why the profile remains.

I have then installed the User Profile Hive Cleaner and am presented with the following errors
The following handles in user profile hive Username (user guid) have been remapped because they were preventing the profile from unloading successfully:

Now i have done some research and i have found one of the old IT admins has modified some of the policies in terms of internet settings. Problem is this guy "left against his will" about a week after making these changes. The problem is i know the changes were not malicious and more likely error in creating the policy. My problem is i am trying to identify what the code in the ( ) above refer to so i can try and resolve this issue.

Does anyone know what the (0x6c8) , (0x6d0), (0x5c4), (0x648) refer to in regards to the above registry error codes.

I found this support article from microsoft http://support.microsoft.com/kb/975619 however a call to microsoft and i have been advised this hotfix is no longer valid and microsoft not willing to provide me with the original file/fix.

Yep, ran a RSOP and came back clean no issues with policies or errors relating to application of policies to users.

GPO is set for delete cached copies of roaming profiles and as mentioned above this has been working fine for a couple years right now.

We have a group of about 15 people all in the same groups and permission set and of this group this issue is only happening to about 5 of them. The weird thing is i got one of there account details and spent 6 hours logging in via rdp and citrix applications. I sent emails as the user printed and used pdf however could not replicate the issue. I logged in and out of the machine approximatly 50 times and no double profiles. The next morning i logged in again about 4 or 5 times and and no double profiles and its cleaned as it should be.

The way we know this issue occurs is the users use a particular citrix application that requires email. The email component is scripted in the usrlogon.cmd to write a certain reg key calling on a required dll key as needed by application. When this works fine the user can email and login and out without issues with the profile on the TS box. However as soon as the user gets a duplicate profile on the server the email functionality stops and the user alerts us to the problem.

I have looked for commonalities and symptoms however this seems completely random. They may be able to log in approximatly 3-5 times over the hours and everything is fine. Then out of the blue this issue will occur where it cant write the profile back. I have checked following event logs for about 30 minutes before and after this occurs to find some common ground but there is absolutely no logs showing any errors or warnings. I check the group policy logs for the same and again no signs of errors or not.

to me this seems strange that the available space is a negative number. I confirmed that the network share that the profile hive is sitting on has atleast 400gb available left, and that there is no quota settings or restrictions on any way. Thats the first thing that caught my attention.

2) shortly after the above lines it then starts saying that it is leaving behind various parts of the profile

I havent gone into to much review as yet but attached is the last logoug from levans who logged out at 5:02pm (17:02)userenv-debug.txt

I have promised my family to go out just right now - so I had a quick look on the file.

I usually would look at the interface where you will find an abrupt change in the time stamp recorded. However, couldn't find such thing.

There seems to be a lot of delnode in the log file. There might be many possible reasons to this (especially that a negative value of available size is recorded):
-> I would think that you might be having network congestions during some time in the day
-> Network file share (where you store your profiles) is hanging during some time in the day
-> I would also definitely look at the size of the profiles of those users having the issues and at their network connections from their PCs.
-> Perhaps try recreating a fresh profile for one of them and test.

Ok i have narrowerd it down a bit. I have checked the networking side and the file share and there is no congestion at all. If anything its is currently underutilised. One of the factors that point away from this is that there is approximatly 30 people on this server at any one time. All 30 have there profiles on the same file share and all 30 inherit 99% of the same policies. Yet it only seems to be the same people day in and day out with the issue (about 8 days and running). That sort of takes the randomness out of it.

Now i have done some more investigation into found some of the original changed made by the employee that is no longer here.

Please have a look at the attached image. The lines highlighted in the red box's were already there. The rest of them have been newly added. Now the User-Opt-AppConfig_MSIE policy is applied to everyone however since not everyone is having the issue i believe that it may just be caught up in the works.

The User-Mand-Flt-AppConfig-MSIE-ANZTransactive policy is the new one. The image shows the relevant policy applications that sort of relate the error code in my first post. I suspect it may have something to do with the url extensions that have been put in.

The four users that are having the main issue are all part of a security group that inherits the second policy. This narrows down my searching a bit more but then raises a few more questions. It seems the second policy mentioned is causing this issue (not sure why yet) but it doesnt answer the question why only 4 or 5 of these users out of a group of about 15 are being affected. The only other commonality between the affected users is that they are using a windows xp desktop instead of a windows 7. Being a citrix / ts session with the issue im not really sure how the local desktop could be cauising it. Both the windows xp and windows 7 are running same version RDP client and exact same version of Citrix Online Plugin.

Confusing issues.... The more i seem to make progress on this one the more questions it is raising and confusing me.

OK. I need to clarify on certain points (I think we are getting somewhere).

1. The policies you are referring to are GPO or local security policies (I guess you are referring to GPOs)

2. When you are applying the policies (if they are GPOs), are you using group filtering (cause you are saying the security group is inheriting the security policies)? Otherwise, those policies are affecting solely the containers they are linked to (OU).

1) Yes they are group policies applied and managed at the domain level.

2) Yes security filtering is applied so only members of this group inherit the changes. The problem only exists with users from within this group and of those users currently only 4 out of 15. As far as i can see though the policy in question is not applied on the users OU it is applied on the OU where the citrix servers are kept. Loop back processing is enabled to ensure the correct users get this policy.

3) The windows XP and Windows 7 computers are kept in the same OU
In terms of testing I am 2 steps ahead of you on that. Testing i have implemented last night to identify the issue is have one of the affected users from the xp machine swap with a non affected user on a windows 7 machine. Waiting to see if the problem re-occurs.

Second testing i am completing is removing one of the users from initial security group to see if

ok i have finally found the issue I dont know exactly why this is being caused but I have found the problem and the soluton. I was very close with the gpo's in the first place however i was looking at the wrong one.

I found a URL that was set to be added to the IE trusted zone by GPO. the url was set by a over eager engineer as

that this will allow any of the government sites in Australia to be added to the trusted sites list. Unfortunately Microsoft doesn't like adding a top level domain to a trusted list. Once this was changed to https://*.ato.gov.au etc the gpo errors disappeared both on the domain level as well as the TS level.

Now I am no longer getting any userenv events in the eventvwr and the profiles are no longer getting caught up. Its funny as i found alot of articles about how the urls should be specified in the gpo but could not find many on restrictions relating to it.

Mutawadi thank you for your assistance. I dont think i could have solved this without you as if it wasnt for bouncing idea's off you i dont think i would have persevered for as long as i did. Hence i am giving you the points. ;)

This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller.
Log onto the new domain controller with a user account t…

How to install and configure Citrix XenApp 6.5 - Part 1. In this video tutorial we have explained step by step installation of Citrix XenApp 6.5 Server on Windows Server 2008 R2 is explained in this video.
We have explained the difference between…