Security

Since SQL Server 2005 we’ve been able to grant one login the ability to impersonate another in order to have a different set of effective permissions. More than a few times I’ve found this paradigm useful in order to solve problems while adhering to the principal of least privilege and likewise more than a few times I’ve tried to find information about who has impersonation privileges in SSMS and have been severely disappointed to find that it is buried. This extremely simple script solves that problem and gives all of the information about who has impersonation rights on an instance.

If you want to read more about all the fantastic things you can do with EXECUTE AS and impersonation, MSDN is a good place to start

We kept getting messages that the cluster node was offline because the Quorum was unavailable. This made little sense as both nodes in this cluster were online and the Quorum disk was available. We could ping across the heartbeat, everything looked fine except for these errors.

After a little research we determined that a new version of Symantec Endpoint Security had been pushed to these servers. Even with the new version of endpoint security, we could establish communication across all networks between the 2 nodes so we were a little stumped. Eventually we ran across a policy that was being enforced from the Symantec central management server/policy/whatever its called!

As it turns out, Symantec endpoint security by default blocks all IPV6 traffic. If you’re like me, I didn’t even realize that a windows 2008 cluster would use IPV6 for the heartbeat communication. After disabling the rules that were preventing IPV6 traffic everything returned to normal.

So, the moral of all this is nothing new… NEVER trust anything new getting pushed to your servers..

The infamous SSPI Failed error strikes again!

One of our SQL servers was generating these errors for “some” Windows logins but not all.

Error: 17806, Severity: 20, State: 2.

SSPI handshake failed with error code 0x8009030c while establishing a connection with integrated security; the connection has been closed. [CLIENT: 192.168.1.1]

Error: 18452, Severity: 14, State: 1.

Login failed for user ”. The user is not associated with a trusted SQL Server connection. [CLIENT: 192.168.1.1]

After exhausting all of the normal troubleshooting for this error (accounts locked, disabled, Sql Service accts, bad connection strings, SPN’s, etc.) I spent the next few hours learning more about the way SQL handles authentication requests than I had ever wanted to know.

The Scenario –

A couple of separate individual Windows ID’s started generating these errors while attempting connections, all other windows logins were working properly. The connections were initially happening through applications, but also occurred through sqlcmd. When logged in to the server locally with the offending ID’s the connections to SQL would succeed.

The Troubleshooting process –

Check all the regular SSPI issues, I wont bore you with the details as they are easily searchable

A relatively easy way of checking the “easy” authentication issues If possible/appropriate is to log into the SQL Server locally with the offending ID and fire up sqlcmd and connect to the server via sqlcmd –Sservername,port –E (by specifying the port you force TCP/IP instead of LPC, thereby forcing the network into the equation)

Verify whether the login is trying to use NTLM or Kerberos (many ways to do this but simplest is to see if there are any other KERBEROS connections on the machine)

SELECT DISTINCT auth_scheme FROMsys.dm_exec_connections

If Kerberos is in use, there are a few additional things to verify related to SPN’s, since only NTLM was in use on this server I skipped that

Determine if the accounts were excluded from connecting to the machine through the network through a group policy or some other AD setting

After all of these checked out OK, I began to try and figure out what the error code 0x8009030c meant, turns out, its fairly obvious what the description is : sec_e_logon_denied. This description was so helpful I thought about making this server into a boat anchor but, luckily for my employer the server room is located many miles away and has armed guards.

Since I knew we could logon locally to the SQL Server with the ID that SQL was rejecting with logon denied something else was trying to make my life miserable.

We didn’t have logon failure security auditing turned on so, I had no way of getting a better error description, As luck would have it though this would prove instrumental in finding the root cause. To get a better error message, I found this handy KB article detailing steps needed to put net logon into debug mode.

Say hello to my new best friend! — nltest.exe

After downloading nltest & using it to enable netlogon debugging on the SQL Server, I got this slightly better message in the netlogon.log file

Even though this command returned “act found” it was returning from DC2. (I dont exactly understand why the same account would authenticate against 2 different DC’s based on a local desktop login or a SQL login but it apparently can)

After asking the AD guys about DC2 the light bulbs apparently went off for them as that server actually exists behind a different set of firewalls, in a totally different location. While DC2 would return a ping, the console wouldn’t allow logons for some reason. After a quick reboot of DC2, and some magic AD pixie dust (I am not an AD admin, if it wasn’t totally obvious from my newfound friend nltest) the windows Id’s that were having trouble started authenticating against DC3 and our SSPI errors went away.

Interesting tidbit — During troubleshooting, I found that this particular SQL Server was authenticating accounts against at least 5 different DC’s. Some of this might be expected since there are different domains at play but, I haven’t heard a final answer from the AD guys about whether it should work that way.

The solution

Reboot the misbehaving DC, of course there may be other ways to fix this by redirecting requests to a different DC without a reboot but, since it was misbehaving anyway, and the AD experts wanted to reboot so we went with that. A reboot of SQL would have likely solved this problem too but, I hate reboot fixes of issues, they always seem to come back!

When creating a new application, after going through the entire business analysis & requirements gathering process, normally you wind up with a datamodel that includes many tables and relationships. By this time, depending on the size of the datamodel/system there has been considerable amounts of time invested on all sides. We need a way of preserving this investment of time while still allowing developers to do their thing!

Deploy

Most shops have policies in place for what level of access developers can have in each environment. In many places I’ve seen, developers are allowed DBO access in development, and some lesser access in the higher environments (read only usually).

After you’ve deployed the datamodel to the physical database in a development environment, before you grant the developer group dbo access consider all of the time/effort that has been spent making the datamodel what it is. In order to allow the developers to do their jobs but not allow them to modify the actual table/schema layout you can grant a combinations of privileges.

Grant Alter Schema on the schemas where the developers will need to modify database objects (for instance stored procedures and functions)

Grant db_datareader –to allow read access

Grant db_datawriter –to allow write access

Grant Create Procedure, Function, Default, Etc — Allow developers to do whatever you are comfortable with

Important** Alter Schema permissions will allow Alter of ANY object type in the schema that you havent explicitly used a Deny on

Principle of least privilege

This method has proven effective to allow developers to write Stored procs, Functions & Views while still keeping the actual datamodel (tables and relationships usually) in pristine shape. You could also mix and match your own grants/denys on certain object types to allow for unlimited configuration without granting the almighty DBO. Yes, you might say that I’m a paranoid DBA who restricts permissions even in DEV! Of course my great developers would never change a modeled database thereby forcing my hand into figuring out this lockdown of privileges

What is the DAC?

The Dedicated Admin Connection, Commonly called the DAC is used to manage SQL Server when a regular connection wont succeed. Here’s what SQL Books Online (BOL) has to say about the DAC “This diagnostic connection allows an administrator to access SQL Server to execute diagnostic queries and troubleshoot problems even when SQL Server is not responding to standard connection requests.”

DAC Errors

Occasionally, while troubleshooting SQL servers in a large environment, especially one thats managed from many different geographic locations you could come up with this error, if more than 1 person is using the DAC. It should also be noted this only happens if you have remote DAC enabled in your environment

Error 17810

Could not connect because the maximum number of ‘1’ dedicated administrator connections already exists. Before a new connection can be made, the existing dedicated administrator connection must be dropped, either by logging off or ending the process. [CLIENT: 127.0.0.1]

Since I could still connect with a regular connection currently, I set out looking for a query to determine who was using the DAC connection. I whipped this up, and since I couldnt find anything in search, I thought id blog it