Hello, this is Sabin and Shravan from the Microsoft Directory Services Support team once again. We will continue our discussion regarding slowness/delay experienced by clients in accessing a DFS Namespace. In Part 1 of this blog, we reviewed the referral process for Domain Based Namespaces and took a closer look at a working scenario where clients were able to access the DFS Shares without delays.

Here in Part 2 of this blog for troubleshooting slow access of DFS Shares, we will review a scenario where a user is seeing a delay while trying to access a DFS share (\\contoso.com\rootdfsn\data) where both DFS Servers –MS1 and MS2 are in the same site as the client.

DFS SETUP:

STEPS TAKEN:

We reproduced the issue with slow DFS access and ran the following tools:

1. In this case, as you can see the root referral (0x81) shows a TARGETSET of two servers in the referral list ordered as –

a. MS1 b. MS2

2. As we can see, MS2 is set to ACTIVE. Ideally, the client should have chosen the first root target (MS1) in the domain-based root referral, connect to the root server and navigate the subfolders under the root folder. On encountering a link folder, the root server should send a status message to the client, indicating that this is a link folder that requires redirection.

3. If the link referral is not in the cache, the Vista client should have connected to the IPC$ named pipe of the root server in the user’s context (or in the context of the LocalSystem account for pre-Vista clients) and request a link referral from the root server which in turn returns a list of link targets.

4. However, as we can see, MS2 (second server in the list) is set as the ACTIVE server since the client was unable to connect and/or request a link referral from the MS1- first server in the targetset.

Note: It’s important to call out here that the client spent some time trying to traverse the root referral list and finally get to the link target. This can contribute to the delay in accessing a DFS share.

5. Finally, as we can see in the referral cache, the client gets a link referral from MS2 and eventually accesses the target on MS2.

Question:

Why was the client not able to access MS1 DFS root?

Step 2: The following is an excerpt from dfsdiag /testdfsconfig /dfsroot:\\contoso.com\rootdfsn

DFSDIAG_ERROR - SYS - The RPC server is unavailable. DFSDIAG_ERROR - SYS - The specified domain either does not exist or could not be contacted. DFSDIAG_WARNING - APPL - Unable to access dfs metadata of \\MS1\rootdfsn.

Finished TestDfsIntegrity.

Analysis:

More evidence on accessing/binding issues with MS1 root server

Note: Alternatively, we could run dfsdiag /testreferral /dfspath:\\contoso.com\rootdfsn\data /full which collectively runs all the above tests.

1. Packets 18-19 – Shows the DFS referral request from Vista client to domain controller DC2. Then DC2 comes back with a referral response and provides a list in the following order:

TargetPath: Index:1 \Ms1\rootdfsn TargetPath: Index:2 \Ms2\rootdfsn

NOTE: The referral order is important. As we can see, MS1 is the first server in the list followed by MS2.

2. Packets 20-30 - Shows the Vista client trying to access MS1 (first server in the list) and eventually failing over to MS2.

NOTE: In frame 23 you can see MS1 sends a Reset to client before it fails over to MS2. You may not always see as this particular problem is specific to the lab environment and might differ from the network captures taken in your environment.

3. Packets 31-108 – Shows the client trying to connect to the ipc$ of root (MS2) and navigating shares before getting a STATUS_PATH_NOT_COVERED response (expected) from server indicating the share is a “link folder” as against normal share and requires a link referral.

4. Packets 109-100 – Shows a link referral request from client and MS2 responds with the following referral order:

TargetPath: Index:1 \Ms2\DataReplica TargetPath: Index:2 \Ms1\Data

NOTE: As you can see, MS2 is the first in the list. This is not always the case. Sometimes, the client will be pointed to MS1 again and that could lead to more delay if client is not able to access the shares and fail over to MS2.

5. Packets 111-148 – Shows the client accessing the \\MS2\datareplica (target replica for data on MS1) and the user navigates/accesses the files (file.txt) as needed.

Netmon 3.3 Tip: You can parse the network captures using a combination of the following filters:

SMB DFSIPV4.address == a.b.c.d && IPV4.address == w.x.y.z

{Where a.b.c.d and w.x.y.z are DFS client/Server IP addresses}

RESOLUTION:

Based on the above steps, we can clearly see that the client is unable to access MS1 via DFS and NetBIOS/FQDN. The traces also indicate the failure and eventual failover to MS2 server causing the delay. When we checked the services on MS1, we found that the DFS and Server services had stopped/disabled. Once we re-started the services, normal operation resumed and client was able to access the MS1 for DFS services.

Other ways to reproduce the problem:

- Turning off MS1 (Note: There is a 50% chance to reproduce the problem using this method since the DC’s will randomly order the DFS referral within a targetset. If MS2 was listed first, the client may not experience the delay as it never attempts reaching MS1)

or

- Configuring Target priority of the servers to list MS1 before MS2 and turning off MS1.

That’s it for this time. See you next time where we will go into troubleshooting access to DFS shares where clients access DFS data in remote sites resulting in slow responses.

Hello, this is Sabin and Shravan from the Microsoft Directory Services Support team. We are here today to discuss one of the common customer issues where they are seeing a delay in accessing a DFS Namespace. Some of these issues may manifest as affecting user’s experience on the machine where redirected folders may be slow to access or home drives don’t work. You may need to perform other troubleshooting which may bring you to the point where you are now left to troubleshoot slow DFS access. In essence, users can get to the shares but it’s slower than usual – in the order of minutes – resulting in lower productivity. Accessing “\\contoso.com\DFSroot\share” is slower as compared to directly accessing the closest DFS server using the UNC path, i.e. “\\DFSserver_netbios Name\share” or “\\DFSserver_fully qualified domain name\share”.

This is Blog 1 of the 3 part series:

- Part 1: Understanding the Referral Process for Domain-based Namespaces - Part 2: Client experiencing slowness though DFS server accessed is in the same AD site - Part 3: Client experiencing slowness due to DFS server accessed in a different AD site (out of site)

Part 1: Understanding the Referral Process for Domain-based Namespaces

Before we start trouble shooting slow access for DFS, it’s important to understand the following:

DFSN terminology

Referral process

Referral cache (viewed by dfsutil /pktinfo)

UseCount and Type

All these are described in details on the technet site under How DFS works? Understanding the DFS referral process is critical.

Simplified Referral Process for Domain-based Namespaces

A user attempts to access a link in a domain-based namespace, such as by typing \\Contoso.com\Public\Software in the Run dialog box.

The client computer sends a query to the active domain controller to discover a list of root targets for the domain-based namespace.

The domain controller returns a list of root targets defined for the requested namespace.

The client chooses the first root target in the domain-based root referral. The client connects to the root server and navigates the subfolders under the root folder. When a client encounters a link folder, the root server sends a STATUS_PATH_NOT_COVERED status message to the client, indicating that this is a link folder that requires redirection.

The client checks its referral cache for an existing link referral. If this link referral is in the cache, the client computer connects to the first link target in the list. If the link referral is not in the cache, the client connects to the IPC$ (named pipe) of the root server in the context of the Current User in Vista clients (LocalSystem account in case of Pre-vista clients) and requests a link referral from the root server. The root server constructs a list of link targets in the referral depending on the target selection method and sends the referral list to the client.

The client establishes a connection to the first link target in the list.

NOTE: The Referrals in steps 3 and 5 above are grouped as a TARGETSET which indicates set of targets of equivalent cost. By default the servers are grouped in two TARGETSETS – in site and out of site.

Now that you have a better understanding of the working scenario, this will greatly help in understanding the problem cases where the expected response from DFS shares is slower. Be sure to check out the next part in the series on slow DFS access where we actually get into the troubleshooting steps in different scenarios. Ciao!

Hey everyone, Mark from Directory Services again. Just the other day I ran across something that may be useful to the public. Here in Directory Services we support the Authorization Manager snap-in, aka AzMan.msc. This tool can configure role-based access control on applications using an AzMan store. The Azman store can be in an XML file, Active Directory, ADAM/AD LDS or in SQL. In this article we will focus on using XML for a store. Unless you have an application that can use the AzMan store it is hard to really know how this works. Well you are now in luck!

Before we go any further I want to make sure everyone understands that this article is not how to write an application to use the AzMan store. This article will not involve any coding at all, script writing, etc. Please note that all the sample applications provided are just that, samples, and are not supported by Microsoft.

Setting up a sample AzMan Store and Application

Ok, with that being said lets learn a little about Azman.

1) First thing we need is a Windows Server 2008 member server, an Active Directory domain and a client machine. I will be using a Windows Server 2003 domain where the member server is running Windows 2008 and the client is running Windows XP SP3. Both of these computers are joined to the domain.

2) Download the Windows 2008 SDK from here. It does not matter which server we download it to and install on. By default it will install to the Program Files directory so open windows explorer and drill down to:

Microsoft SDK\Windows\v6.1\Samples\Security\Authorization\AzMan

Inside this folder you will see a folder called WebExpense. Copy this entire folder to your Windows 2008 member server so that the folder resides underneath the inetpub\wwwroot directory. This sample application will only work on Windows 2008 using the steps described below.

3) Next run the Server Manager snap-in on the 2008 server and install the Web Server (IIS) role. During the installation of the role when you get to the Select Role Services page, select the ASP.Net under Application Development. This will cause a window to popup stating it needs to install some other services. Go ahead and select the Add Required Role Services. Before clicking on Next, make sure you check the box under Security for Windows Authentication. Now click Next, confirm your selections and then click on Install. Once the install is done, go ahead and close the installation page.

4) Now go to the Administrative Tools and open the Internet Information Services (IIS) Manger. In the left hand pane expand the connections and drill down so that you see the WebExpense folder underneath the Default Web Site. Right click the WebExpense folder then select Convert to Application. In the Add Application popup window for the Application Pool click on the Select button, then in the drop down box select Classic .Net AppPool then Ok twice.

5) With the WebExpense highlighted in the center pane (/WebExpense Home) find the Authentication icon and double click on it. Verify that Anonymous Authentication is Disabled and Windows Authentication is Enabled.

6) Now we are moving right along, go to Start – Run – azman.msc – Ok on the 2008 server. Right click in the left hand pane on Authorization Manger then Open Authorization Store. Verify we have XML file selected and then click on Browse. Browse to the folder:

inetpub\wwwroot\WebExpense folder

Select the AzStore XML file. Let’s leave this for now and come back to it shortly.

7) Go to your XP client and open IE then go to the 2008 server’s URL as:

http://<2008servername>/webexpense/index.asp

The page should open as Trey Research and WebExpense –online expense reporting and auditing. Then you should see Welcome <username>: Sorry <username> you do not have permission to use this application. Please contact your manager. This is telling us that we do not have the proper permissions to run the web application.

Using the Sample Application

Before starting to learn how to use the MMC let’s learn a little bit about Groups, Tasks and Operations:

Groups: We can create different type of groups. We can create a Basic Application Group which is much like a local group. We can add users/groups from Active Directory or another group from Authorization Manager. We can even exclude users/group as well. Say you have a group of 200 users in AD where you want to add all the users but one user. We could add the group then go in and exclude the one user.

An LDAP Query Application Group could be created that determine who is a member by running a LDAP query. If the user matches the query then the user is deemed to be a member of the group. You could use a simple LDAP query or you could make it more complex to fit your needs.

Business Rule Application Group is the third type of group that can be created. In this type of group you will need to have a script that defines the members of this type of group. The script will need to be constructed either in VB or Java.

Role Definitions: A role definition is defined as a set of permissions that a user must have to perform a job or task. A role can have lower level tasks, roles and operations associated with it.

Task Definitions: A task definition is smaller than a role definition and can be use to define other roles and tasks.

Operation Definitions: This can only be seen in Developer mode. Operation definitions are small computer level actions that are used to define tasks and usually are not relevant to an administrator.

1) Now let’s go back to the 2008 server and the Authorization Manger MMC. In the MMC in the left hand pane under the AzStore.xml go to Groups. This is where you can create groups for different uses in AzMan. The default ones here should be Managers and Admins. Let’s experiment here a little and right click Groups and select New Application Group. Call the new group Submitters, in the Group Type select LDAP Query Application Group then ok.

2) Now right click the new group you created and select Properties and go to the Query tab. Click inside the box and type in: (objectSid=*) then Ok.

3) Now expand Expense Web in the left hand pane, expand Role Assignments then go to Submitter. Right click Submitter and select Assign Users and Groups – From Authorization Manager. Place a check mark in the box next to the group we just created then Ok.

4) Now let’s test this from the XP client. If the IE window is still open close it. Open IE again and go to the WebExpense URL like we did before. You should see the icon and Submit an Expense. Now you have permissions to submit an expense!

Note: The reason you can see this now is that we authenticate to the web site via integrated authentication meaning we are using our logged on credentials at the XP client to authenticate with to the application. We gave the Submitters group we created the permission on the Submitter Role. When we created the Submitters group we went back into the properties of the group to the query tab. There we defined the query to Active Directory (AD) as objectSid=*. This is basically saying if the user has an objectSid in AD then they meet the requirement of being a member of this group. By setting it up this way we are basically saying everyone has the permission to submit an expense. You can get more granular if you want by changing the query for the Submitters group.

5) Now let’s try something a little different. Under your Role Assignments – Submitter, right click the group we added and delete it. Go to one of your domain controller and open Active Directory Users and Computers. Create a new Global Group, I called mine AzManSubmitters. Add a user to the group, I added the account I was logged onto the XP client with named Mark.

6) Now add this new group to the Submitter role in AzMan. Go to the XP client and log off then back on. We need to do this to update our security group membership as we just added ourselves to a group.

7) Now go back to the URL for the WebExpense application. You should still see the Submit an Expense. The reason why we can still see it is due to we are a member of the AzManSubmitters group and we had added that group to the Submitters role.

8) Let’s explore even more now. In Azman MMC, under the Expense Web expand Definitions and go to the Role Definition. Here you see several roles. Take the Submitter for an example. Go to the properties of Submitter then to the Definition tab. Here you can see what Tasks the Submitter role can perform. In this case the Submitters can Submit Expense.

9) Now go to the Task Definitions folder under Definitions. Go to the properties of Submit Expense and its Definition tab. Here you can see that within the Submit Expense task it breaks it down further to Read Expense and Submit operations. All this defines what the particular Role can perform within the application.

10) Take a look on the Definitions\Task Definitions\Definition Tab\Submit\Authorization Rule. Here you can define some script code and tweak the rule to fit your business needs. In this example you cannot submit an expense if the amount is 500 or greater. You can try this by connecting to the application and if you have the Submit permission click on the link then fill in all the boxes. Enter an amount that is greater than 500 then click on the Submit button. This should return a runtime error due to the application is configured to use a backend SQL server to store the data and we do not have one configured. Again, we are not concerned with that in this article.

AzMan can be run in one of the following two modes, Developer or Administrator. To select or find out which you are running right click Authorization Manager in the left hand pane and select Options. The default mode is Administrator. When in Developer mode you can create, deploy and maintain applications. You also have unrestricted access to all of AzMan features. When in Administrator mode you can maintain and deploy applications. You still have access to all AzMan features except you cannot create new applications or define operations.

Summary and Notes

Let’s recap some of the information and make a few notes.

Now that we know a little bit about the type of groups that can be used, here is a little bit of low level troubleshooting information. When using an LDAP Query Application Group the web server will need to make an LDAP call to a DC (or ADAM as the case may be) to verify that you meet the requirements set in the LDAP query for the group. If you take a network trace of the user connecting to the web application from the web server then you will see the LDAP traffic. You will not be able to see the exact LDAP query being made in the network trace as the information will be encrypted. Other than that you should authenticate using Kerberos when using either Basic Application Groups or if you use Active Directory groups directly on the Role. Another piece of advice is to pay attention to the Role Definitions and Role Assignments. Let’s say you have a user that needs to approve expenses and you add the user one way or another to the Approver role. They connect to the web application and they state that they cannot see the Approve Expenses icon. Look at the Role Definitions. You will find that the Approver role only has the Approve Expense task. You will need to add the user to the Expense Manager role as well as they have View Pending Expenses. You need to be able to view the pending expenses in order to be able to approve them.

To dig in and learn more about AzMan see the help file that is associated with the MMC. We have come a long ways with the help files over the years and they can be a good resource for you. Here are some other good references:

Bob Drake again and welcome back for the second part of the slow logon series. In this next part I want to dive into some simple troubleshooting techniques to assist you in isolating the cause of your slow logon.

Is it slow from when you hit the power button to the point where you get to the login screen?

Is it quick to get to the login screen but then hangs for a while to get to the desktop?

Is it all users, and not administrators?

All the above

Troubleshooting will be dictated by the answers to those questions. We will start off with a slow boot that occurs when the power button is hit and it takes forever to get to the logon screen (even though a slow boot is NOT a slow logon). If the slowness occurs when the machine first boots up before you get to the login splash screen, then typically there is either an issue with the core OS, the applications installed, or a combination of both. A great place to start troubleshooting is to enable verbose startup, shutdown, logon and logoff messages (providing the operating system is XP or higher) according to KB 325376. With this enabled you will receive additional information during the boot/login process:

Policy location (XP and 2003)

View of additional messages (XP and 2003)

Policy location (Windows 7, Windows 2008)

View of additional messages (Windows 7, Windows 2008)

The first thing that should be determined is if the delay happens when the machine is “clean booted (Windows XP/2003)(Vista/2008/Win7)”. With MSCONFIG you can selectively disable all third party services and applications from loading. Now before the bashing begins here about the necessity of the applications that are on the machine, this step is quite essential to know if the OS is the issue or the applications that are installed on the OS. Here is how it’s done:

1. Click Start then Run and type “MSCONFIG”

2. Select the “Services” tab as displayed and check the “Hide all Microsoft Services” and click “Disable All”.

NOTE:When you “Hide all Microsoft Services” you will see the applications that are installed on the system. Often times there are applications that are crucial to the boot/logon process (like drive encryption software) which will cause the machine even more problems. You will need to review the applications and disable what you can (more the better).

Select the “Startup” tab and click “Disable All” once again. If you find disabling all the third party applications causes a bigger issue you can press F8 at startup and select “Last Known Good” or “Safe mode” to back out the changes to “msconfig”.

3. Once the third party services are disabled you will need to reboot (a window will appear stating you need to reboot. Once the machine comes back up another window will appear when you logon, just click “OK”).

4. Test and see if the boot time is the same or not.

When you disabled the third party services did the computer boot and logon faster or normally? If the answer is yes then you have a conflicting piece of software on the machine that is causing the delay. To find out which one can be a little more laborious but the quickest way will be to enable half the services (making sure to list which ones you are enabling) and see if the delay comes back. If not, repeat process by enabling half of the other services you haven’t tried yet until you get issue to return. Once you identify which one it is, try updating that application or components. You may also take a quicker approach if you believe that one particular application is the issue (due to a recent install) and simply disable that one only for confirmation. You will have to reboot several times during this process to be confident that you have discovered the cause. When you believe you have discovered the application causing the delay, re-enable it and see if the delay comes back just to be sure.

Once you identify the application with the issue call the vendor and explain why it has been pinpointed as the issue and seek guidance from them. Often times there are updates or hotfixes that will resolve the conflict.

NOTE:Some antivirus applications will still load filter level drivers even though their services are disabled from starting with MSCONFIG. The only way to truly rule out antivirus as a possible contributor to a slow logon is to uninstall it during the test.

So it’s still slow!?!

If the boot up is still slow check your client DNS configuration. DNS servers along with other hardware (like switches, routers) could also be the source of the problem. If you find that one section of your network is having the issue but other portions don’t, there is a good chance that you may have some network issues. A good way to determine network and DNS type issues is to take a network trace using a packet capture application like Netmon . The hard thing to do is capture a trace when the machine is starting up. This can be accomplished by monitoring the computer with another computer that is plugged in to the same hub or switch (port mirroring). You enable the network capture utility from one and filter for the other’s IP address.

Delayed or unanswered responses (both from the DNS servers to domain controllers)

Kerberos failures

SMB failures

Numerous TCP resets or retransmits

A specific Domain Controller consistently used when the issue happens (possible issue with the Domain Controller itself)

If there are any network related issues you should see them stand out without having to be a master at reading traces.

If you have disabled all the third party software using the above method and still find that you have a lengthy boot process, then the next step is to look at what policies are being applied to the machine. A great place to start troubleshooting group policies is the technical reference.

In most environments there will be numerous policies applied, so how do you rule them out as being an issue? The quickest method is to create a new “TEST” organization unit (OU) in the Active Directory Users and Computers snap-in and block policy inheritance to the OU. Once this is done, you can move the problem computer to that OU. Verify that there are no policies being “enforced” or set to “no override”. If there are polices with those settings, they will still be applied to an OU where policy inheritance is blocked.

Before you move the computer you should run the following command to find out exactly what group policy objects are linked to it:

When you open the “gp.txt” you can view the policies as shown:

You can see that the only policies that are applied to the machine are the “Default Domain Policy” and “Local Group Policy” (The above snip was shortened here to show user and computer).

Once you have the policies identified you can move forward with creating the test OU. Here is a step by step on creating the OU and blocking the inheritance:

Note: You will know that you are blocking inheritance when the OU icon has a blue exclamation as seen:

3. Once you have inheritance blocked and the computer moved to the OU, reboot the computer at least two times to clear the previously set policies.

There are times where you cannot remove all policies if they are enforced. At least you will have a short subset of policies at this time. Time to test again…

If the computer boots fast now then you have a group policy (or combination of policies) that ARE causing the delay. To find which policy is causing the issue you will need to link them one at a time rebooting in between and monitor when the delay occurs again. This is done by selecting the “Link an Existing GPO” as seen in above picture. Once you have the policy identified a thorough audit should be done to determine which setting in the policy is causing the delay.

So it’s still slow…!?!

If you have gone through the above steps and were not able to find why the boot up is slow and you were not able to disable all software or policies then the next step to do is enable debug logging. There are a few ways to enable logging depending on which operating system you are using.

For 2000, XP, 2003 you can enable logging by following the article: “How to enable user environment debug logging in retail builds of Windows “http://support.microsoft.com/default.aspx/kb/221833. Lucky for me one of my co-workers has already written a blog on how to interpret the output (find more in his two part section: Section 1Section 2). For Vista, 2008 and Windows 7 Microsoft has changed the debug logging format to what is called “Event Tracing”. Basically the data output is the same as the output from the above KB article once it is converted from the binary output. You will need Microsoft’s assistance with converting these files since they contain source code (to view the policy portion without the profile use gpsvcdebug logging).

Another great tool to use is “Autoruns”. This utility will show you programs that are configured to run during system start or during login. One of the best features of this tool is the “Hide Microsoft Entries”:

It also allows you to select the autoruns per user as seen:

Autoruns may also find items not normally seen with other applications.

To wrap it up, most of the time a slow boot or slow logon happens there is a conflict with an application, restrictive group policy or configuration issue. With the above troubleshooting and a little homework you will be able to identify where and why you have a slow logon and be able to resolve it in minimal time.

Hi, Bob Drake here again after a short blogging hiatus. I have put this two-part blog post together with hope that it will save you countless hours and a few aspirin when troubleshooting a slow logon. I have had the luxury of working many different slow logon cases and I have to say that these can be the most grueling to handle, depending on how they are approached and what information you have. There are multiple reasons why slow logons can occur and sometimes they are a result of multiple issues masked as one.

For this first part in the series I want to cover some well-known causes of slow logons, optimizing logon for your environment, and assist you with documenting your baseline to identify when you really have a slow logon issue. But before I do we need to set some expectations.

The “logon process” (I use this term to encompass both the boot up of the workstation and the user login that is completed with a functioning desktop) has a lot of moving parts. The most important question to address is “What is an acceptable logon time to you?”If you have expectations that your logon should only take 3-5 minutes from the time you turn your computer on to the point you get your desktop, you will have a brief window to perform all tasks. Your business requirements will dictate what you will be able to accomplish during the logon, so a thorough understanding of your goals is needed before moving forward.

Once you have your logon task list, then you can start testing the logon time frame. If all is configured and you are over your accepted limit, then an adjustment will need to be made by either limiting your tasks or accepting the lengthier time. There is a saturation point that will be reached when you try to accomplish too much in too little time.

So you want to know what the top items are that will definitely slow your logon process? Here’s a list of configurations that will have an impact on your logon time:

Outdated operating system (OS) patch level: Your operating system should have the latest service pack installed from windows update

Roaming user profiles: Roaming profiles change the way group policy processing is performed. When roaming profiles are configured the processing is changed from “asynchronous” (background processing or multiple at a time) to “synchronous” (foreground processing or one at a time). This disables “Fast logon Optimization” which will delay the user getting the desktop by waiting for the network to initialize first.

Note:This is really important to understand that when roaming profiles are implemented, group policy software installations and folder redirection requires that the user is NOT logged on before the network is initialized and processes policy synchronously- ONE AT A TIME. This is the default behavior and changing it could cause inconsistencies with your logon.

Home folders: This could impact your logon times because instead of looking at a local location for system DLL’s, the client machine will look for them in the home folder instead. If that mapped network share is across a wide area network (WAN) link that is slow you can bet that your logon time is going to suffer even more.

Note:If home folders are needed with roaming profiles there is a registry key tweak (SafeDllSearchMode) that can be added that will change the behavior. If you’re not sure that this is an issue in your environment, take a network trace at logon and see if DLL’s are being queried across the network to the home folder. There is also another tweak on the same page (StartRunNoHOMEPATH) that will assist with applications doing this behavior.

Start up applications: Applications that are configured to automatically run during startup will slow the logon down.

Profile scanning: There are many antivirus software applications that will scan profiles at login and at their home location if they are roaming. This is not limited to just antivirus software but other applications will as well. (In the troubleshooting section we will review how to discover if this is happening)

Excessive group policies: Having a ton of group policies that perform extensive tasks or configurations (like software restrictions) will increase your logon time. A few policies that accomplish everything are better than many policies that do a handful of things each. If possible consolidate your group policies.

Excessive startup/logon scripts: Scripts that run at logon or start up can delay the process significantly if they perform a lot of tasks or use inefficient code

No local domain controllers: Not having local domain controllers (users authenticating across a wide area network-WAN) will cause a logon delay

Before we get into troubleshooting a slow login we need to first identify what is a slow login and whereisit slow. To be able to say a logon or boot up is slow you must know what a normal logon or boot time looks like in YOUR environment. With the above expectations the next step is to document the time a logon takes under normal conditions and under load (morning and afternoon rush hours). This should be done with all the different operating system builds in your environment (desktops, laptops, servers, XP, Vista, Win 7, 2003, 2008, 2008 R2) to have a standard baseline to work with.

Here is a short starter list of things to include in your baseline documentation:

Network topology

Active Directory Topology

User and computer group membership

Operating system and service pack level

Installed applications

Network bandwidth and latency )

NIC driver information

“UserEnv” log (from several users who are members of different security groups) from XP or 2003, and ETL logs from Vista, 2008 and Win7

Network traces

Group Policy information (both computer and user)

Once you have a solid baseline of average times, then you will know right away when logon times increase and where to narrow your search for the culprit. With the above documentation in hand the issue will be resolved much quicker. Without the documentation you’re setting yourself up for hours of agony and a costly resolution.

Be sure to check out the next part in the series on slow logon where we actually get into the troubleshooting steps.

As you can see, they implemented a one-way forest trust between their internal network and the DMZ where the DMZ AD trusts the internal AD. Their ADFS-enabled application was not claims-aware. It was an NT token-based application, so it needed to be protected with the NT token-based Web Agent. The application also did Kerberos delegation to a backend SQL server (not shown). They had user accounts in both the internal AD and the DMZ AD which needed access to this application.

The forest trust is utilized by ADFS only when both the Account Partner (their internal AD) and Resource Partner (their DMZ AD) ADFS consoles are configured as follows:

-On the Account Partner:

-On the Resource Partner:

When ADFS is utilizing the forest trust and a user from the internal AD accesses the application, the user SID from the Account Partner AD is now available on the Resource Partner side where the NT token is built. The NT token-based Web Agent builds an NT token local on the web server. Once the NT token is built and used to access the application, ADFS authentication is complete. Next, their application needs to access SQL, so we must transition to Kerberos to do delegation to the backend SQL service.

Whenever we need to do Kerberos delegation with Federated WebSSO with Forest Trust it must be configured for Kerberos Protocol Transition and Constrained Delegation. If you have read documentation on Kerberos Constrained Delegation, you will remember that cross-forest authentication requires a two-way forest trust. Having ADFS authentication in the picture does not negate the two-way forest trust requirement.

During this process of delegation, a S4U2SELF ticket is requested and we also need to be able to get a TGT for the Account Partner domain. Without a two-way trust in place, this TGT request will fail with S_PRINCIPAL_UNKNOWN.

Rather than use the Federated WebSSO with Forest Trust , you have two other options:

Option 1: Create resource accounts in the Resource Partner forest

Create an Alternative UPN suffix in the Resource Partner forest that matches the UPN suffix of the Account Partner forest

Create user accounts in the Resource Partner forest with the same usernames and UPN suffixes as the Account Partner forest

In the right pane, select the UPN Identity Claim, right-click, Properties

Select the Groups tab and configure an Organizational Claim to map to a user account

Disadvantage of using this option: The identity of individual users is lost. All users from the Account Partner are authenticated to the application with the UPN of the single resource account.

For more information about resource account mapping methods, check out this TechNet Article: http://technet.microsoft.com/en-us/library/cc779214(WS.10).aspx

To conclude, the Federated WebSSO with Forest Trust scenario will work just fine with a one-way forest trust when there is no Kerberos delegation involved. As soon as you need to delegate Kerberos with your ADFS-enabled application and you are using Federated WebSSO with Forest Trust, you must be using the NT Token-based Web Agent and you must have a two-way forest trust in place. If a two-way trust will not work for your environment, consider the alternative options described above.

It’s Randy again, here to discuss LDAP security. Lightweight Directory Access Protocol is an interface used to read from and write to the Active Directory database. Therefore, your Active Directory Administration tools (i.e. AD Users and Computers, AD Sites and Services, etc.) as well as third party tools are often going to use LDAP to bind to the database in order to manage your domain. As you can imagine, we rely on Windows security to authorize what users can do when accessing this important database. Access to this database is controlled by the Directory System Agent (DSA), which is represented by Ntdsa.dll on a domain controller. Ntdsa.dll runs as a part of the Local Security Authority (LSA), which runs as Lsass.exe.

The process of granting access is a two step process; Authentication and Authorization. This blog post will focus on the Authentication portion (verifying the user’s identity.) If you would like to learn more about the Authorization process, please read my post on security tokens. Objects in the Active Directory database conform to the same rules as other Windows objects. They have permissions and privileges that govern what the authenticated user can do.

I use the LDP.EXE utility in Windows 2008 to reproduce all of the scenarios that follow. This tool is a client GUI to connect, bind and administrate Active Directory. You also want to download Network Monitor if you are troubleshooting an LDAP problem or want to follow along in your own network traces.

If you open up LDP and choose “Connect…” under the Connection menu and do not put in any parameters:

And choose bind as currently logged on user:

The first thing that you will notice is that it just works. We locate a valid domain controller when we do not specify a target DC by leveraging the DC Locator process and we bind as the currently logged on user by leveraging the security packages included in Windows. So what do we see in a network trace? We will be filtering on LDAP when opening in Network Monitor 3.3.

Why do we get results in LDP.exe before we even Bind?

The first set of traffic that we see is a couple of LDAP Search Requests and Search Responses. You can see this in the Description field of the Frame Summary:

So why do we see an LDAP search request before we do the LDAP bind? This is because we first access the “RootDSE” partition of the Active Directory Database as an anonymous connection to query information on the rules to bind and authenticate to this DSA. We allow anonymous access to this root, but in Windows Server 2003 and later we deny anonymous access to all other partitions in Active Directory. You can change this setting by changing the DSHeuristics value; this will be a forest-wide change for the behavior of all domain controllers.

You can see the LDAP request parameters as “BaseDN: NULL” if you look at the Frame Details pane of the LDAP search request. Expand the “LDAP: Search Request “ , then expand the “Parser: Search Request” , then expand the “Search Request”:

“BaseDN” is the container where the search begins in the LDAP query. NULL is going to be the RootDSE of the domain controller. This is why results return in the right hand pane of the LDP utility before we even bind.

The “LDAP: Search Result” shows this same data if you expand the “LDAP: Search Result”, then expand the “Parser: Search Result”, then expand the “SearchResultEntry”:

Notice that the results in the trace are identical to what displays on the LDP.exe result pane. One important piece of information that we get is “supportedSASLMechanisms”. These are the supported security mechanisms that we can use in our LDAP bind. We will discuss this later in the post.

Why do we log on as ‘Null’ when we select to pass credentials?

So next we bind and select the default of currently logged on user. In the result pane of LDP it shows that we are logging on as ‘NULL’.

I consider this more as an idiosyncrasy of the tool and not completely accurate. When we perform the Bind, we do present ‘NULL’ credentials, but specify SASL as the authentication type. At this point, we are passing the authentication process off to the SASL mechanism to complete. We can see evidence of this in the network trace.

Looking at the Bind Request in the frame details pane, you will see some interesting information as you expand out the LDAP packet:

Here we see that we are passing NULL credentials, but negotiate SASL authentication. Simple Authentication and Security Layer (SASL) is a method for adding authentication support to connection-based protocols. So basically, LDAP binds with NULL credentials because we are handing off the logon process to SASL and letting it do all the work. We see details of the negotiation process in the bind request and where we present the Kerberos session ticket as a result of selecting the GSS-SPEGNO SASL mechanism:

So how does SASL provide authentication?

For detailed information, you can check out the IETF explanation of SASL. Similar to how the Security Support Provider Interface (SSPI) uses security packages, SASL will use registered SASL mechanisms to complete the authentication process. During the initial LDAP query for supportedSASLMechanisms described earlier. We retrieve a list of all the mechanisms from which the client and server will choose. Below is a list of the default mechanisms in Windows Server 2008.

Authentication Protocols

Comments

GSS-SPNEGO

GSS-SPNEGO, in turn, uses Kerberos or NTLM as the underlying authentication protocol.

Used to allow a client to authenticate itself using information provided outside of the LDAP communication.Can be used to present certificates to establish TLS or IPSec.

DIGEST-MD5

Encrypts authentication using Digest-MD5

The LDP tool allows you to choose various mechanisms and is a great tool to test connections when other tools fail. You can select the appropriate bind specifications in order to closely simulate what your application is trying to perform.

The application will decide how it will bind to the database by what functions are used to establish the connection (i.e. LDAP_Simple_bind, LDAP_Sasl_bind, etc )

What about LDAP signing?

If you have ever looked through security settings in Group Policy, you may have stumbled on a couple related to LDAP.

Signing LDAP traffic is a way to prevent man-in-the-middle attacks. By signing the LDAP traffic, this guarantees that the LDAP response did originate from the DC of whom the request was made. With these settings enabled, computers would not be able to intercept the traffic and modify the data on the wire. You can see the digital signing value in the network trace below. This value will be on every LDAP response from the DC where signing is done.

What is a Simple Bind?

An LDAP Simple Bind will send username and password in clear text to provide credentials to the LDAP server. This is not secure, so if your application is using simple binds it needs to be reconfigured or updated. You can create a Simple Bind using this option in LDP.exe or calling the LDAP_Simple_bind function in your code. In a network trace we can see the username and password in the LDAP BIND Frame.

With Signing required in my domain, I was unable to perform a simple bind with LDP.exe. I received the error “LDAP_Simple_Bind failed: Strong Authentication Required”.

What is an Anonymous Bind?

As mentioned earlier, every LDAP connection is going to perform an anonymous bind to query RootDSE of the LDAP server. Also mentioned earlier, by changing the DSHueristics value on your AD Forest, you can permit Anonymous Bind LDAP searches for all the AD Partitions. The anonymous bind is still restricted by the permissions on the Directory Objects themselves. I did an anonymous bind and performed a search for all the user objects in the domain and resulted in only one user because I had manually granted permissions on that User object.

I also had to grant permissions on the Organization Unit that contained the user; otherwise the anonymous logon was unable to traverse containers where it had no permissions. I did not have to grant access to the Domain Naming Context at the root. I was able to set permissions in “AD Users and Computers” by selecting the “Advanced Features” option in the View Menu.

What about LDAPS?

LDAPS uses SSL/TLS technology to establish an encrypted tunnel between the client and the LDAP server. The tunnel is encrypted with the LDAP server’s PKI Certificate, this way no one else can read the traffic except for the client and LDAP server so the Client is free to perform a simple bind and safely pass the credentials in clear text. LDAPS is an entirely different subject that deserves its own post. Luckily James Carr has written it, check it out here.

Developers have a plethora of ways to bind to an Active Directory database. This provides supportability for numerous flavors of LDAP clients, but it does require some configuration on the part of the AD administrator. Farewell for now…