Obviously, if I turn off pooling, I don't get these exceptions - but performance suffers noticeably.

These exceptions don't always happen. The web app will run fine for hours before an exception is thrown. It could also be as short as the next request. . It randomly occurs and in random places in the code. Once thrown, every request afterwards throws the "Access Denied" exception.

Notice this has a waitHandle, this is the actual semaphore handle, examine this:

0:011> !handle 123c fHandle 0000123cType Event…

Note this handle should be a semaphore handle in normal operating circumstances but what I found is the handle is corrupted, it's now an Event handle! (or some other random sort of handle for example a file handle).

This can happen if something inside the same process inadvertently closes the wrong handle. Windows will recycle the handle and the next caller to open a handle will get the same handle value. When we use this handle with semaphore functions it reports E_HANDLE or potentially SemaphoreFullException when ReleaseSemaphore is called on the handle (this is what .NET Semaphore class raises when ReleaseSemaphore fails for ANY reason).

Hence the root cause of this problem I believe is something inside the process inadvertently calling CloseHandle on our internally held handles. This invalidates _poolSemaphore and all calls into pool fail (until pool is cleared).

To debug further you can use Application Verifier and track handles to capture the caller who is calling CloseHandle.

To do this you download Application Verifier and enable handle tracking for w3wp.exe. Then attach Windbg to running w3wp.exe process and run !htrack -enable to enable handle tracking. Then capture another dump when SemaphoreFullException occurs and run !htrack to see who closed the handle (the call stacks are stored in the dump and tracked when CloseHandle is called).

I was thinking we could harden our pool handles by calling DuplicateHandle to increase handle ref on our side, but in general this would not solve the general problem that customers need to resolve anyway. The general problem is code inside the process inadvertently closing handles it does not own, this is a very serious problem that will cause other things in the process to abnormally fail so you need to get to the root cause of this problem in any case, even if I harden my pool handles.

All replies

This exception is being thrown from the pool's syncronization primitives, and should not occur in normal circumstances. A serious configuration or installation error might cause them. It could also be a bug in the pool itself -- the randomness of time before hitting the problem would be consistent with a race condition, although something this far into the core of the pooler should have shown up before now. I'll look into the this possibility.

If this isn't a production machine, I suggest trying to eliminate the config/installation problem by re-installing the .Net Framework (there are probably less drastic measures -- anyone who knows one is urged to chime in :-).

Is there anything else you can tell about your code or environment that might be significant? Does the problem happen only under a heavy load? Is your app multi-threaded and sharing state between threads, either explicitly or via sharing connections between host threads such as ASP.Net?

While it is on a production machine, it is in its own application pool and only a handful of people are using it (usability testing). It is also on my local develeopment machine (XP) and I get the same exceptions, albeit much less frequently. Therefore, I do not believe it to be an installation/configuration issue - althought I won't dismiss any suggestions of something to try. I will re-install the .net framework during the next downtime we have for that box.

I'm not sure what would be significant, but....

I wrote a connection string provider library the keep connection string information in either the registry or in a heavily protected database. The password is encrytped with some custom encryption.

I wrote a facade class around SqlConnection to make working with the connection string provider easier for the other developers. They only need to know the database alias. I also used this facade class to do custom serialization (serializes the database alias) so that the ManagedConnection:SqlConnection can be serialized with the objects that use them. My money is on this class causing the problem, although I don't have a clue how.

The ManagedConnection class constructors create a single SqlConnection object that is used for the life of the object. Each of the methods that use the SqlConnection uses a try/catch/finally block to Open/Close the connection properly.

The problem happens with only 1 user (me) running it. There isn't much of a load yet. The application is a trouble ticket system that integrates our billing system. The application connects up to 8 different databases on two different servers. Tickets and the billing system information are represented as objects that are collections of objects that are collections of objects. In addition, I wrote a custom security library to enable applying permissions on the various objects/functions. Each permission check is a hit against the security database. It's nothing for a single http request to generate 50 hits to various databases for permission checks, building the current ticket, building the current billing customer/plan.

I'm not sure what you mean by "via sharing connections between host threads such as ASP.net". Is a host thread the worker process thread or the thread for a particular request? Is there only one connection pool for the entire application or does each request get its own pool? Even if they all share the same pool, I seem to remember reading somewhere that each connection in the pool is not only associated with a connection string, but the user/identity/whatever as well.

I'm relatively new to multi-threading and discussions about it are still above my head. Should I do some kind of synchonizing on the ManagedConnection objects? I don't think the same ManagedConnection object would be used by more than one thread at a time, but I don't understand enough to know for certain.

I've been running with pooling turned off for the ManagedConnection connections. However, I have the SessionState mode set to SqlServer and pooling is not disabled. I just got the access denied exception:

Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

ASP.NET is not authorized to access the requested resource. Consider granting access rights to the resource to the ASP.NET request identity. ASP.NET has a base process identity (typically {MACHINE}\ASPNET on IIS 5 or Network Service on IIS 6) that is used if the application is not impersonating. If the application is impersonating via <identity impersonate="true"/>, the identity will be the anonymous user (typically IUSR_MACHINENAME) or the authenticated request user.

To grant ASP.NET access to a file, right-click the file in Explorer, choose "Properties" and select the Security tab. Click "Add" to add the appropriate user or group. Highlight the ASP.NET account, and check the boxes for the desired access.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Just a note that re-installing the .net 2.0 framework did not fix the problem.

So.. If it's not an installation/configuration issue, that brings it back to my code.

But since I get the same exceptions thrown when using the session state (mode = SqlServer), how can it be my code?

A bug in the framework? Then why does it appear that I'm the only one experiencing this evil?

The ETW trace doesn't seem to show anything that helps.

Are you still reading this Alazel? What else can I try? What other information do you need? I don't understand why this is happening, but I really need to get this resolved soon as this app needs to go live.

There are two ways that this problem occurs:1. a call to DbConnectionPool.CleanupCallback(Object)2. a call to DbConnectionPool.PutNewObject(Object)

It would seem that the PutNewObject is a more likely candidate for a problem since it seems that it just does a semaphore Release with no apparent matching WaitOne.

Here are two stack traces that show the problem:=============I. First StackTrace=============System.Threading.SemaphoreFullException was unhandled Message="Adding the given count to the semaphore would cause it to exceed its maximum count." Source="System" StackTrace: at System.Threading.Semaphore.Release(Int32 releaseCount) at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state) at System.Threading._TimerCallback.TimerCallback_Context(Object state) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading._TimerCallback.PerformTimerCallback(Object state)

.... It could also be a bug in the pool itself -- the randomness of time before hitting the problem would be consistent with a race condition, although something this far into the core of the pooler should have shown up before now. I'll look into the this possibility.

I've opened up a case with Microsoft. No progress on creating a test case. The application can be modified to reproduce the problem in around 10 minutes (usually between 12 and 18 minutes) but it's pretty hard to send our entire application to Microsoft as it's an enterprise app with a rather difficult setup.

If you can make a test case that causes the exception, it sounds like Microsoft would fix it.

Can you post the connection string you are using? There are a couple of places that could match the call stacks given, and this might help track it down. I don't need actual values for sensitive stuff like password, username & server, but I would like to see what options you are specifying and values for the non-sensitive options.

One possibility: could the app be openning many connections in parallel? The connection pool throttles connection attempts via a semaphore, and you can get exception call stacks something like this if there is a timeout while waiting for the semaphore.

My app opens a lot of connections sequentially. It would be nothing for a single request to generate 10-30 connections to using 3 databases (2 of which are on the same server).

Each database has its own DAL-like class to access it. Each instance creates a single SqlConnection object that it uses for the lifetime of the object (web app, so until the end of the request?). At first, I would leave that SqlConnection object open for the lifetime of the DAL object thinking to avoid any overhead of opening/closing connections continuously. After the exceptions were thrown I started doing research. Common thought appears to be to let the connection pool handle everything. So, I changed to the code to call Open/Close on the (still) single SqlConnection object as needed. The exceptions did not go away. I then started posting on various forums and newsgroups to no avail. I've gotten nothing helpful.

After posting here, I decided to not use pooling and disabled it. The exceptions are no longer thrown from my code. However, I occasionally (rarely) get the exceptions from the session state (I have mode = SqlServer and the connection string is like above).

Can you open a case with CSS? They'll have the tools and personel to
dig into this fairly quickly and help you capture the information we'll
need solve the problem. This appears almost certainly to be a serious bug in System.Data code at this point, but I'm unable to pin it down with the information I've got so far.

Basically, an .aspx form submits and processes data with about 9 connections opened and closed between 3 databases.

We get all 3 of the errors you mentioned. Even with Pooling=False in the Connection String we were still able to generate anerror. Also, we get errors with a very light load of users 2-10 people max.

I have searched all over to no answer. Did you ever solve this issue? We have a ticket open with Microsoft to fix thisbut so far they seem to be stumped. It's been 3 days now. This is a very important app that needs to be live again.

Please let me know if you have the answer or workaround...I'm going to try the last suggestion mentionedof setting Min Pool Size = 1.

We too are facing the same problem. We get both the COM Exception and Semaphore Exception atleast once a day. Ours is a windows application. We are using SQL Express 2005 as database. We are using Data Access Application Blocks and we are using DatabaseFactory.CreateDatabase to get reference to a Database object and are calling either ExecuteNonQuery or ExecuteDataSet methods on it to call stored procedures. The problem occurs in a method of the code that saves each user action to local database. Its called "Journal Entry". So every button click is saved to the local database by calling a stored procedure available in local database(SQL Express). I believe DAAB automatically handles connection pooling. But as suggested it could be a bug in connection pooling or DAAB code. Could any one help us. We have deployed the application on site and the application crashes multiple times a day on site.

One thing to watch out for when using some versions of DAAB in a multi-threaded applications -- you can end up using the same connection from multiple threads, especially if you set up some of the objects as static fields. Multi-thread use of ADO.Net connections will definitely causes random problems, including failures to properly release semaphores.

So are you saying that there are some versions of DAAB that do not have this problem in a multi-threaded application? I'm experiencing the SemaphoreFullException problem while using the SqlHelper class from DAAB version 2.0 (most of the functions in that class are defined as Static).

Is there a more recent version of this class that doesn't have this problem, or is there a replacement that does not invovle restructuring my entire application?

Just a follow up folks. I've been investigating this one for a while, creating stress tests and modifying internal timers to increase the likelyhood that the code paths in question would trigger something but thus far have not been able to replicate. Testing on a quad amd64 machine and giving it lots of stress.

From code examination alone I don't see any issues yet the code looks very solid.

As a response to :It would seem that the PutNewObject is a more likely candidate for a problem since it seems that it just does a semaphore Release with no apparent matching WaitOne.

Actually the PutObject is releasing a semaphore ref gains in a call to WaitForMultipleObjectsEx in DbConnection.GetConnection, this call passes in array of handles one of which is the PoolSemaphore in question, so this code is ok, I verified this.

I have some theories on how to replicate this one however, will post back once I have a repro. If anyone has a nice repro let me know.

I am experiencing this same behavior. The most common error I see is [COMException (0x80070006): The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))]. However, I have also ran accross this error [SemaphoreFullException: Adding the given count to the semaphore would cause it to exceed its maximum count.] , but less frequently.

I have the Min Pool Size = 1 in my connection string and I still get his error. Hs anyone found a way to solve this issue? The only way I've found to restore the client once they get the 0x80070006 exception is to recycle the application pool in IIS, which seems to be a rather drastic and broad reaching approach. Any help would be much appreciated as this is running in a live environment and I cannot replicate the behavior in my test environment.

Just as a reply to above post, I examined this code very closely and verified we don't have a problem here. Note the semaphore is add-ref'd earlier by a call to WaitForMultipleObjects, so there is not a WaitOne/Release mismatch here.

Notice this has a waitHandle, this is the actual semaphore handle, examine this:

0:011> !handle 123c fHandle 0000123cType Event…

Note this handle should be a semaphore handle in normal operating circumstances but what I found is the handle is corrupted, it's now an Event handle! (or some other random sort of handle for example a file handle).

This can happen if something inside the same process inadvertently closes the wrong handle. Windows will recycle the handle and the next caller to open a handle will get the same handle value. When we use this handle with semaphore functions it reports E_HANDLE or potentially SemaphoreFullException when ReleaseSemaphore is called on the handle (this is what .NET Semaphore class raises when ReleaseSemaphore fails for ANY reason).

Hence the root cause of this problem I believe is something inside the process inadvertently calling CloseHandle on our internally held handles. This invalidates _poolSemaphore and all calls into pool fail (until pool is cleared).

To debug further you can use Application Verifier and track handles to capture the caller who is calling CloseHandle.

To do this you download Application Verifier and enable handle tracking for w3wp.exe. Then attach Windbg to running w3wp.exe process and run !htrack -enable to enable handle tracking. Then capture another dump when SemaphoreFullException occurs and run !htrack to see who closed the handle (the call stacks are stored in the dump and tracked when CloseHandle is called).

I was thinking we could harden our pool handles by calling DuplicateHandle to increase handle ref on our side, but in general this would not solve the general problem that customers need to resolve anyway. The general problem is code inside the process inadvertently closing handles it does not own, this is a very serious problem that will cause other things in the process to abnormally fail so you need to get to the root cause of this problem in any case, even if I harden my pool handles.

1. Handles are reused. Therefore, if you have an IntPtr to a handle and it gets closed, your IntPtr is still (or will be soon) pointing to a valid handle.

2. If you call LogonUser (from the advapi32.dll) twice for the same user, your handle returned from the first call is automatically closed (if it wasn't already). If you call CloseHandle on the IntPtr returned from the first call again, you will screw things up as your IntPtr may now be pointing to a registry handle or a file handle or an SQL connection handle or it may even be the same value as the one returned from the second LogonUser call, etc.

3. If you close an SQL connection handle, you will get one of the above post's three errors. The actual error depends upon whatever state the pool happens to be in. To debug one of the above three errors, search for all uses of CloseHandle in your code -- one of those is likely your culprit -- one of those is likely closing a handle that is no longer pointing to the handle/token you thought it was.

4. The WindowsIdentity token/handle constructor duplicates the handle internally. Therefore, you can and should CloseHandle your token from LogonUser immediately after you construct the WindowsIdentity object. There are some rumors that you need to duplicate your handle before passing it into that constructor, but that issue was fixed in .Net 2.0.

5. Do not dispose of the WindowsIdentity object yourself. The Timer class grabs the current one and holds onto it for later use. A number of .Net classes use the Timer class internally, including the SqlClient connection pool manager. Rather, you should cache them so that you only have one for each user, not one per webservice call, etc. You don't want to burden the authentication server anyway -- you should be caching WindowsIdentity objects to avoid network traffic. For more info on this problem see http://winterdom.com/2008/12/crashingyourprocesswithtimers

Adding "Min Pool Size=1" should exacerbate the issue as there is now guaranteed to be at least one active SQL connection handle that can be closed incorrectly through the CloseHandle call. A single threaded environment with connection pooling disabled should not see any of the above errors as it's unlikely the code does anything like this:

using(new SqlConnection) { CloseHandle(a handle/token I thought was open but really wasn't); }