The errors listed above is very typical when deploy linked server with delegation. They actually are thrown by the linked server and pass by middle server to the client application. In this post, I will discuss how to properly configure SQL instances and Windows environment in most common scenario and try to make configuration steps as explicit as possible.

By using delegation in distributed query, such as linked server query, the SQL instance obtains impersonated token of the user logon credential to gain access to resources of another SQL instance, the linked server. In delegation setting, the client connection and linked server object are configured to use integrated authentication in SQL Server’s term as opposed to SQL login. Some time integrated authentication also referred as trusted connection or Windows authentication. Linked server login can also use SQL login, but it is not discussed here.

To simplify the discussion, let’s assume two SQL Server instances are installed on machine A and B respectively. Also, let’s assume A is the middle server that has a linked server object configured which points to a SQL instance on machine B. If the client is on machine C different from A, we call it double-hop setting; if the client is collocated with middle server machine A, we call it single-hop setting. In single-hop setting, it is relatively straightforward to configure linked server to work. Believe or not, double-hop setting requires more careful configurations as you will see. This is because in single-hop setting, windows NTLM authentication, which is available in most common setting if all machines are windows, is sufficient for delegation; while in double-hop setting, Kerberos authentication is mandate for flowing user’s credential through machine boundaries from the client to the linked server. It requires windows domain, correct DNS name resolution, proper account setting in both Active Directory and SQL Server. To make sure Kerberos delegation [1] is correct becomes vital to operate distributed query with delegation. The authentication scheme required by delegation in different setting is illustrated by the following table.

Authentication scheme

C to A

A to B

Single hop

NTLM or Kerberos

(C is on the same box as A)

NTLM or Kerberos

Double hops

Kerberos

Kerberos

[Create Linked Server Object on Middle Server]

Before getting into details on how to configure other components, since I am talking about delegation in the context of SQL Server distributed query, let’s first give an example on how to configure a linked server object on A and set up its login to use delegation. To do so, you need the following two steps.

(1) Use sp_addlinkedserver to create a linked server object and name it “LinkedServer” which points to the SQL instance on machine B, SQLB.

“EXECsp_addlinkedserver @server=’LinkedServer’,

@srvproduct=''”,

@provider='SQLNCLI',

@datasrc=’SQLB’,--the data source

@provstr="Integrated Security=SSPI; "

“

To verify if the command is executed correctly, run query

“select*fromsys.serverswherename='LinkedServer'”;

(2) Use sp_addlinkedsrvlogin to configure login to use self-mapping as following

“execsp_addlinkedsrvlogin ‘LinkedServer’, 'true'”

Step (2) makes middle server A try to use impersonated token of user to authenticate to server B. To verify that the linked server is setup for “self-mapping”, run query

“select uses_self_credential as delegation

fromsys.linked_loginsas L,sys.serversas S

where S.server_id=L.server_id

and S.name=N'LinkedServer'

The resulting table should show the delegation column is “1”.

[Test Linked Server Query in Single-hop Settting]

Before test-drive a link server query in single-hop setting, you need also make sure that the client user can make direct query to the SQL instances on both A and B. This means that the user account, either windows domain account or a machine account, must have permission to access both SQL instances.

(3) To verify the user domain account has permission to access both SQL instances, use your favorite client tool, for example,“osql –E –S SQLA” and “osql –E –S SQLB”. If you are failing for whatever reason, please refer to [5][6].

(4) To test linked server query, run query at SQLA,

“select * from LinkedServer.master.dbo.sysdatabases”.

[Configure and Test Double-hop Setting]

To deploy delegation based linked server in double-hop setting, the followings need to be configured correctly.

(3)Kerberos in Windows mandates Windows domain. Therefore the user account needs to be a domain account and middle server and linked server need to join a domain. All machines involved in the delegation, including client machine, middle server and linked server, must have good TCP/IP connectivity between each other and to the domain controller and Active Directory. To not complicate thing further, we assume that A, B and C are in same Windows domain D and the user account is a domain account in D.

(4)The user’s domain account must NOT select “Account is sensitive and cannot be delegated” in its Active Directory properties of domain D. Please refer to [1] on how to configure this on Active Directory machine.

(5)The service account under which the SQL instance is running must be “trusted for delegation”, configured in Active Directory. If the service is running as “NT AUTHORITYSYSTEM” or “NT AUTHORITYNETWORK SERVICE”, the computer must be “trusted for delegation”. Please refer to [1] on how to configure this on Active Directory Machine. You need to have domain admin privilege to do/verify so.

(6)The user domain account must have permission to access both SQL instances from C. To verify, use your favorite client tool, for example,“osql –E –S SQLA” and “osql –E –
S SQLB”. If you are failing for whatever reason, please refer to [5][6].

(7)If the SQL connections are to use TCP/IP connectivity, configure and verify that SQL connections from C to A and A to B are using Kerberos authentication. Please refer to [2] on how to configure Kerberos for SQL. In a nutshell, both services on machine A and B need to have a SPN created in Active Directory for SQL service. If the service running account is not “NT AUTHORITYSYSTEM”, you need to configure the SPN on the Active Directory Machine with domain admin privileges. To verify that every hop is using Kerberos and TCP connectivity, run query

when (a) connect to A from C (b) connect to B from A. To make Kerberos work for both hops is crucial and some time it might not be very straightforward on what goes wrong. If fail to get Kerberos to work, please (a) verify If the SPNs are configured and well-formed according to [4]; use “setspn –L acccoutname” to verify the SPN (b) verify if DNS reverse lookups of both machine A and B return well-formed FQDNs. Use “ping –a machinename” on machine A, B and C to verify DNS works as expected, i.e. returning FQDN; (c) make sure that there is no cached Kerberos ticket on machine A and C. Use “klist purge” to purge all tickets. There might be delay before Windows local security authority (LSA) requests a new ticket from Active Directory. Sometime, you need to log out and log back in again before a new Kerberos ticket can take effect. For more Kerberos troubleshooting techniques, please refer to [3].

(8)If the SQL connections are to use Named Pipe connectivity, SQL level Kerberos is not required as opposed to TCP connectivity. This is because Windows named pipe protocol can use Kerberos to authenticate logon user under the cover. You need to verify that both machine SPNs “HOST/machinename” and “HOST/ machineFQDN” is well-formed use “setspn –L machinename”.

(9)Since double-hop can have combination of Named pipe connectivity on one hop and TCP on the other, the following table is valid configuration for delegation. Run query

Thank you for the article. Can you talk a little bit about when you have trusted domains. Our cluster resides in a "Domainlet" and we are using trusted domains. We are having some problems getting our linked server to use domain authentication.

TCP Provider: An existing connection was forcibly closed by the remote host.

means that the remote host RESET the socket from client to server. This can happen for example if domain2 does not trust domain1 and IPSEC is enabled in your domain. IPSEC can automatically RESET untrusted socket attempts.

The other two errors are consistent with double hop failures.

With these types of errors the basic rule of thumb is be systematic and eliminate one leg at a time. Since you can get "TCP – KERBEROS" I am assuming this is from the first leg, so now you need to troubleshoot the second leg.

Start on the middle SQL Server and use OSQL or SQLCMD to attempt to connect to the next SQL, does it work? Log onto the middle SQL Server and connect to SQL locally, then try linked server query, does this work? Etc…

I am still experimenting, but it seems on clustered instances it’s necessary to have each physical node’s computer accounts in AD have the "trusted for delegation" as well as the service’s user account.

OK I get it, I have to enter Remote login and password and not map anything

I get the list of remote tables in Enterprise manager but select from "main" serevr to linked servers tables returns "The operation could not be performed because the OLD DB provider ‘SQLOLEDB’ was unable to begin a distributed transaction

Try running dbcc freesystemcache(‘all’) to free up pooled connections on the server side, then reconnect again. I think what is happening is you are picking up a pooled connection that you made locally.

The case is interesting. Can you provide more detail about the linked server deployment? Basically, how do you configure linked server from C,D to A,B? You client query running on Q, so it was double-hop, correct? and whether you double checked that Q->C, C->A or Q->D , D->A were using Kerberos over TCP? What is the configuration difference of A and B(SQL 2k5), whether they are same SKU? same default instance? Whether A and B are on the same box? With same OS?

1. Same script created linked servers on both ‘middle servers’ A & B. Both have 4 linkedserver accounts authorized for ‘impersonation’…one of those is the ‘serviceaccount’. All linkedservers are set this way.

2. Client was queryanalyzer or SSMS on my workstation, so yes double-hop was required.

3. Yes, from my workstation using q/a or SSMS, i can log directly to any of the servers…

4. I don’t know how to check that i’m using kerboros from my workstation out.

This sounds like linked server connection pooling is making the hop succeed. To prove if this is the case, try running:

DBCC FREESYSTEMCACHE (‘ALL’)

After step 2. If this causes step 3 to fail, then you are just reusing the authenticated connection you created in step 2. Pooling auto shuts down the connection in around 8 mins so this is why it fails later.

Matt, Thanks for your comment. That did help me understand the connection pooling issue.

Ming, (or Matt)

Could there be a difference in SQL server versions linked server functionality? My 3 SQL 2005 Enterprise versions are all able to double hop to named instances. The Standard Edition server is only able to double hop to default instances.

I didn’t see your comment until after i posted my last note. To be more specific: my 2005 Enterprise versions can all double hop to 2000 named instances. My one 2005 Std edition can only double hop to 2000 default instances. The fixes you refer to relate to 2005 to 2005 double hopping.

I am experiencing the same problem. Only the edition I have is Developers Edition. I applied service pack 2 and am still getting the error when I try to run a distributed query from my pc to a SQL server 2005 (developer’s ed) that has a linked server to a SQL server 2000 (standard edition).

Does anyone know if it’s because of sql editions that this problem occurs.

When I make my connection from the client to server "A" and then by linked server to server "B", it doesn´t work, when I run the query "“select net_transport, auth_scheme from sys.dm_exec_connections where session_id=@@spid” I have: NAME PIPES/NTLM.

In the other hand when I make the connection from the client to server "B" and then by linked server to server "A", it works, and my net transport is TCP/KERBEROS.

How can I change the net transport of my server "A" to TCP/KERBEROS?????

Double hopes are working fine in my environment using Domain Authentication. They connect as KERBEROS. The issue is that after 8 hours it seems that the Token expires and the connection falls back to NTML. At that point double hop does not longer work. I need to close the connection and reopen.

The issue I am facing is that it is not possible for us to reset a database connection from a .NET application, and seems that some of those connections are lasting over 8 hours.

For some reason I am now getting the "Login failed for user…" error message, even after I had it working by disabling Named Pipes on Server B. Does this have something to do with tokens expiring? If so, how do I keep tokens from expiring or how can I renew tokens automatically? Any assistance is much appreciated. I thought I finally had this issue worked out but apparently not.

I guess i have double hop im my envoiroment. But the intresting thing is that every thing is working fine (distributed queries) form backend, but when i login through application i get the error mentioned above.

Let me explain my Enviormnet.

Code server: windows server 2003 (ASP application)

DBServer A =windows server 2003 SP1 and sql 2005 Ent E with Sp2.

DBSever B=windows server 2003 SP1 and sql 2000 Ent E with Sp4.

Now first application calls the one or 2 storeprocedure from sql 2005 (server A) and then it calls store procedure that invloves data from sql 2000 (Server B). Here the problem lies and i get the error mentioend above.

But again if i run quires from (backend)sql 2005 every things work fine and if i login from apllication it will work, but after few mins like 15 to 20 mins it doesnot work.

Apllication doest not login because it verfies the logins from sql 2000 (server B) database.

But again if you run queries from sql 2005 (server A),those store procedure that involves sql 2000 (server B) data, it will work from backend and from front end as well. (mean you can login application)

but again after certian period of time application will not work.

My requirment is to create linked server with service account (windows account).

Account has sysadmin rights on both server

i have followd you 9 steps…..as per my network administrator every thing is fine at his end like active directory, domain controller as per your steps.

Infcat i followed ur steps and created linked server as you mentioned but when i ran the query on serve A (Sql 2005)

We’re having problems combining single hop authentication (with kerberos available) with changing a user context on server A using "execute as login". Also the used domain account to which we’re switching the user context has permissions on both servers, we get a connection timeout, when trying to switch the context and query server B via linked server. Both servers are in the same domain and have logins for the used domain user. Is this scenario an impossible design and failed to doom or should we keep trying?

THANK YOU SO MUCH! I’ve been bothered for ages by a non-functioning double hop setup, most searches on the web return info on people trying to resolve trivial single hop setup issues. Your article explains clearly and comprehensively the components required for double-hop setups and I now understand fully why I’ve been having trouble for so long.

I have the same issue with one of my clients, the scenario is complicated. Clients access the Middle tier servers through a hardware load balancer (HLB). The HLB used to distribute the load to the 3 middle tier servers (load balanced) serving the clients. The 2 backend servers are clustered and utilizes a default instance of the SQL 2005 Failover CLuster DB. I am just damn confuse on what to set for the SPNs. hope you can help me. Thanks. But I’m trying to go through your article.

I discovered the working computer was connecting through Shared Memory using NTLM. The computer that failed was connecting through TCP/IP using Kerberos. So I ran SQL Server Management Studio, went to connect to my local database, selected the Advanced tab and changed the Network Protocol to Shared Memory then connected to the database. I then ran the SELECT query and problem resolved. Hope this helps.

I tried to create a linked server on server B to connect to server A which is not in the domain and I can access the linked server succesfully from server B. However, I cannot run query to the linked server from my ASP code and I cannot run query from my Query Analyzer on my machine either.

Note: I have Enterprise Manager installed on my machine and I can I register both server A and B.

When I am trying to test the linked server that connects to Server A from Server B on my Query Analyzer, it fails.

When using Windows Auth for a linked server query which is initially coming from a remote client, as in the situation you describe, it requires Kerberos authentication with these special delegation settings enabled, and Kerberos is not available in a workgroup. So, I think for the scenario you have, the solution that will give you the fewest headaches and which will definitely work will be to use SQL Authentication for your linked server queries, rather than Windows Authentication. As it says in that blog post I pointed to above, we recommend SQL Authentication for connection in workgroups anyways, and your extra requirements mean that it really doesn’t make sense to try to get Windows Authentication.

I have a SQL one node cluster and another SQL one node cluster .. and i want both of them to be connected trough the linked server . I am using Windows auth and i fail with the Login Failed for user Nt AuthorityAnonymous logon…

Interestingly When i am connecting from any other SQL Server instance to one of the cluster instance the linked server runs just fine…

This seems t be onehop as we have 2 cluster node talking to each other…right?

I even created SPNS for the virtual server name on both the clusters and while taking netmon ..i see that NTLM is being used(i read NTLM string in the big Junk). I am also seeing that the error message coming from the target SQL server cluster.

Now i have 3 questions

1. Why would it will still fails in one hop?

2 > Is there something cluster specific i need to do on Active directory.

3. while i am using window authentication ..the SQL server service account should pass ..why would it change to NT Authority Anonymous logon…

I’ve followed the instructions here and made the changes necessary but cannot get the double-hop to work.

Here’s the situation…

From my desktop using SQL2008 SSMS I connect to server A which is SQL2008 on Windows2008 which has a linked server configured for server B which is SQL2000 on windows2003.

Linked server connection is fine from server A to B. this is the sticky part.

From my desktop (C) I connect to server A successfully but fail when making the linked server call to server B. If I RDP onto server A and make the linked server call to B then go back to my SSMS on my desktop I can now double hop. It’s like my credentials are being cached.

AD Settings – we enabled delegation on service account for A for all traffic cos when trying to enable delegation for server B it doesn’t show the mssqlservice in the list of services to trust for delegation. Does this have something to do with this?

Also, the service account on Server A is not a local admin so the SPN was set manually for the service account. The service account on server B is a local admin so it would dynamically set.

in this A is the server for workgroup1 and B is the server for workgroup2, and C is a node in Workgroup2

Here i have Sql server 2000 in A and B and My application is running in C

i have 2 Databases one is in A and another is in B I need to access the tables in A by using linked queries from B. when i try to run my query in my application from Server B it running correctly but when i try to run my application from node C i got the error message is below

I managed to get double hops to work… the key thing is that the SQL Server service account must be set to "trusted for delegation" and this is an AD setting so you need a domain admin to do this for you.

Could you tell me if SQL Server 2008 has double-hop issues? I mean, if we update the sql server from 2000/2005 to 2008, can double-hop issues disappear? Or are there any alternative method(rather than modifying the NTLM to Kerberos) to resolve double-hop issues in Sql server 2008?

On step 5, my AD admin only trusted delegation for SQL services running on Server 2. He changed it to trust for all services on my service account and it worked. Then we narrowed it down to trusting delegation for all SQL services on Server 1, Server 2 and Server 3 and it is working like a charm. Great article Nan. Thanks!

The last two posts were related to our dev environment. Our test environment is segmented from dev from prod at the network layer. We were unable to get it working by delegating just the MSSQL services per server. So, we went back to delegating at the domain account runing SQL server for all services. It is now working in our test environment and expect to have no issues in prod environmant.