Wednesday, November 02, 2005

ThreadStatic, CallContext and HttpContext in ASP.Net

Summary:Even if you think you know what you're doing, it is not safe to store anything in a ThreadStatic member, CallContext or Thread Local Storage within an ASP.Net application, if there is the possibilty that the value might be setup prior to Page_Load (eg in IHttpModule, or page constructor) but accessed during or after.

[Update: Aug 2008 In view of the fairly large number of people continuing to link to this post I feel the need to clarify that this thread-swapping behaviour happens at a very specific point in the page lifecycle and not whenever-it-feels-like-it. My wording after the Jef Newson quote was unfortunate. That aside, I've been immensely gratified (and flattered) by the number of times I've seen this post cited within design discussions around dealing appropriately with HttpContext. I'm glad people found it useful.]

There's a lot of confusion about using how to implement user-specific singletons in ASP.Net - that is to say global data that's only global to one user or request. This is not an uncommon requirement: publishing Transactions, security context or other 'global' data in one place, rather than pushing it through every method call as tramp data can make for a cleaner (and more readable) implementation. However its a great place to shoot yourself in the foot (or head) if you're not careful. I thought I knew what was going on, but I didn't.

The preferred option, storing your singletons in HttpContext.Current.Items, is simple and safe, but ties the singleton in question to being used within an ASP.Net application. If the singleton's down in your business objects, this isn't ideal. Even if you wrap the property-access in an if statement

Storage in CallContext alleviates some of these problems, since the context dies off at the end of the request and GC will occur eventually (though you can still leak resources until the GC happens if you're storing Disposables). Additionally CallContext is how HttpContext gets stored, so it must be ok, right?. Irrespective, you'd think (as I did) that provided you cleaned up after yourself at the end of each request, everthing would be fine:

"If you initialize a ThreadStatic variable at the beginning of a request, and you properly dispose of the referenced object at the end of the request, I am going to go out on a limb and claim that nothing bad will happen. You're even cool between contexts in the same AppDomain

"Now, I could be wrong on this. The clr could potentially stop a managed thread mid-stream, serialize out its stack somewhere, give it a new stack, and let it start executing. I seriously doubt it. I suppose that it is conceivable that hyperthreading makes things difficult as well, but I also doubt that."Jef Newsom

Update: This was the misleading bit. I do explain further later on that this thread-swap can only happen between the BeginRequest and the Page_Load, but Jef's quote creates a very powerful image I failed to immediately correct. My bad.

Trouble is that's exactly what happens. Trouble is that's almost what happens. Under load ASP.Net can migrate inbound requests from its IO thread pool to a queue taken up by it's worker process thread pool:

So at some point ASP.NET decides that there are too many I/O threads processing other requests. [...] It just takes the request and it queues it up in this internal queue object within the ASP.NET runtime. Then, after that's queued up, the I/O thread will ask for a worker thread, and then the I/O thread will be returned to its pool. [...] So ASP.NET will have that worker thread process the request. It will take it into the ASP.NET runtime, just as the I/O thread would have under low load.Microsoft ASP.NET Threading Webcast

Now I always knew about this, but I assumed it happened early enough in the process that I didn't care. It appears however that I was wrong. We've been having a problem in our ASP.Net app where the user clicks one link just after clicking another, and our app blows up with a null reference exception for one of our singletons (I'm using CallContext not ThreadStatic for the singleton, but it turns out it doesn't matter).

I did a bit of research about how exactly ASP.Net's threading works, and got conflicting opinions-masquerading-as-fact (requests are thread-agile within a request vs requests are pinned to a thread for their lifetime) so I replicated my problem in a test application with a slow page (sleeps for a second) and a fast page. I click the link for the slow page and before the page comes back I click the link for the fast page. The results (a log4net dump of what's going on) surprised me.

What the output shows is that - for the second request - the BeginRequest events in the HttpModule pipeline and the page constructor fire on one thread, but the Page_Load fires on another. The second thread has had the HttpContext migrated from the first, but not the CallContext or the ThreadStatic's (NB: since HttpContext is itself stored in CallContext, this means ASP.Net is explicitly migrating the HttpContext across). Let's just spell this out again:

This is a major PITA, because as far as I can see it mean the only persistence option for 'ThreadStatic'esque behavior in ASP.Net is to use HttpContext. So for your business objects, either you're stuck with the if(HttpContext.Current!=null) and the System.Web reference (yuck) or you've got to come up with some kind of provider model for your static persistence, which will need setting up prior to the point that any of these singletons are accessed. Double yuck.

The key bit is what happens when FastPage's Page_Load fires. The ThreadID is 4748, but the threadID I stored in HttpContext in the ctor is 2720. The hash code for the logical thread is 1703, but the one I stored in the ctor is 1835. All data I stored in the CallContext is gone (even that marked ILogicalThreadAffinative), but HttpContext is still there. As you'd expect, my ThreadStatic is gone too.

27 comments:

Could you clarify if this was reproduced in .NET 1.1 or 2.0? Sorry if I missed it in the details.

I'm anxious to hear if anyone else has come up with any other details/alternatives. For a custom, portable "CurrentRequest.User" singleton I was also looking at Thread.CurrentPrincipal, but I shyed away due to SQL Server Windows auth and CAS issues. For simplicity's sake I'd rather developers know their code is running in a single configurable service account, with a business-domain-specific CurrentRequest.User context fully separated... I wonder if Thread.CurrentPrincipal has any similar issues w/ agile threads...

Everything's 1.1 unless mentioned otherwise at the moment. Sorry I should have made that clearer.

You're right of course that this would apply to the Thread.CurrentPrincipal as well. From memory this gets setup at the end of OnAuthenticate, in which case it would have to be re-setup if the thread migrates in order not to get lost. I'll have another play...

The workaround we've been using is to store all per-user context in a UserContext object. This class just delegates storage either to Thread local storage, or to HttpContext.Items, depending on whether HttpContext.Current!=null. As a result this class insulates our business objects from having to know the mechanics of how per-user context is stored, and doesn't tie them to a web UI (so they can still participate in unit tests, even if there's no WinForms UI on the horizon).

The Thread.CurrentPrincipal gets setup from the HttpContext.User in two different places: in HttpApplication.SetPrincipalOnThread, which fires after the authentication modules (if any) have done their work, and in HttpApplication.OnThreadEnter, which is what's called when a HttpApplication instance starts its execution, or resumes it's execution on a different thread.

So - as you'd expect - there's no issues with Thread.CurrentPrincipal being lost when you migrate to another thread, though if you've overwritten the Thread.CurrentPrincipal directly, rather than setting HttpContext.User then you're not going to get what you want.

I guess you could use the IPrincipal as your user-singleton if you wanted to, but it would tie your business objects to knowing about a particular IPrincipal (or at least a particular interface), so I'm not convinced its a great solution.

Assuming System.Web.HttpApplication.OnThreadEnter is called anytime an HttpApplication instance resumes its execution,

Then: OnThreadEnter of System.Web.HttpApplication calls SwitchContext(System.Web.HttpContext) of System.Web.HttpContextWrapper, which calls SwitchContext(System.Web.HttpContext) of System.Web.Hosting.ContextBase, which switches the HostContext of System.Runtime.Remoting.Messaging.CallContext.

By looking at HostContext getter and setter in CallContext, and since the HttpContext does not implement ILogicalThreadAffinative, we derive that the HostContext will be set via set_HostContext(...) of whatever is returned by Thread.CurrentThread.GetIllogicalCallContext().

GetIllogicalCallContext() returns a System.Runtime.Remoting.Messaging.IllogicalCallContext. Which stores its HostContext in a private m_HostContext field and NOT in its inner HashTable, where data accessed via SetData/GetData go. So what OnThreadEnter does is initialize the HostContext of a new IllogicalCallContext -- hence making sure that the HttpContext is available. But it does NOT copy the whole IllogicalCallContext, and therefore whatever has been set with SetData is lost.

Again: CallContext does not migrates, only CallContext.HostContext.

Now, I'm not quite sure what we can do with this. In an ASP.NET app, HostContext will be of (sealed) type HttpContext anyway so to access its Items collection we'd have to reference System.Web anyway. Unless we use reflection?

What's scary -- unless we're missing something obvious here -- is that CallContext is used by all sorts of services including transactions, log4net, some IoC frameworks...??

What's scary -- unless we're missing something obvious here -- is that CallContext is used by all sorts of services including transactions, log4net, some IoC frameworks...??

Yeah. That was my first thought as I read through all this. I've used CallContext as a persistence mechanism more times than I can remember based on a combination of "that's how HttpContext works" and "that's how all these IoC frameworks do it."

Any further thoughts on this? Obviously if this were a huge in-the-field problem we'd be seeing alternate approaches. Has anyone seen one?

The idea is that all conection objects are the same in all threads but I need them to be thread static to make sure that the connection object is not use simultaneous for opening two datareaders for example.

So I don´t care in which thread a request is procesed since all thread have a smilar connection. What is important to me is that the same connection is not use to do two or more task at the same time.

Primarily, leaving your connection instances floating about like this is a resource drain. Expensive resources like this should be released as early as possible, not just left open for the duration. Ok, so it's only 20 connections, but imagine if everyone did that.

You've also got to be very careful about keeping the connection instances 'unsullied': avoid using SQL context, certain security schemes etc... I've seen this thread-connection-reuse scheme combine with a transaction leak to bring down an entire system (rather than just a certain request always bombing out, which would have been easier to diagnose).

And you've still got the issue that the connection instance you start with in the HTTPHandlers isn't neccesarily the one you use in your Page_Load etc... so woe betide you if you try and use a transaction that spans the two (though, for various reasons, this would be a bad thing to do anyway).

piers7, am trying to reproduce the same scenario (Thread switching), which is discussed. I am using Asp.net 2.0 (AS.net development server). From your log it is clear that the issue is reproducible. But i could not able to do that in 2.0? Can you please confirm that it is not the problem in 2.0. Also it would be helpful if you can post your code with which you able to reproduce.

Thanks for your article, and solution which i will now use, really appreciate it and can't see much in the way of MS docs on this. I think it is stupid that asp.net even allows the "static" keyword - why would you ever need it? Really it should be configurable so you can change the meaning of your statics to what you need, with the default being per-request. The only reason i can think of for not doing this is too hard to implement, which is a crappy reason for making life difficult for everyone! This is a hangover from c++ single threaded pre web apps, and should have been brought into the 21st century by now.

I was also looking at Thread.CurrentPrincipal, but I shyed away due to SQL Server Windows auth and CAS issues. For simplicity's sake I'd rather developers know their code is running in a single configurable service account, with a business-domain-specific CurrentRequest.

I guess you could use the IPrincipal as your user-singleton if you wanted to, but it would tie your business objects to knowing about a particular IPrincipal (or at least a particular interface), so I'm not convinced its a great solution.

One alternative option for ThreadStatic type behavior that only lasts the length of a request is a follows:

- Create Custom IPrinciple and IIdentity Classes. - Add a property to one of these that contains a Dictionary of objects - Each request at the PostAuthenticateRequest stage, replace the existing Principle with the custom Principle (you can easily just wrap the existing one to retain the values) (Set Thread.CurrentPrincipal AND the HttpContext.Current.User to the new principal object)

After this, you can access your dictionary in a non webspecific way by: ((CustomPrincipal)Thread.CurrentPrincipal).MyDictionary

It seems like a bit of work up front, but it really is not too bad, and performs well. The advantage of this method is that: - The dictionary will only last the length of the ThreadContext (i.e. the length of a request). There is no need to manually clean at the end - ASP.Net takes care of transferring the Principal to any new threads if the thread changes for a request - .Net takes care of transferring a reference to the principal to all child threads, this is available from child threads also.

I have used this method successfully before. Once the initial pain of implementing this pattern, I really never had to think about much again - it just seems to work.

Very informative and unleashed article Thanks and the all the comments.I am a newbie in this type of problem. kindly someone explain me How can I do it with .net framework 4 in WebApplication.I want to store my EntityFrameWork context I want to ensure it one context per request not share context..Any Help would be appreciable.. Thanks

I've conducted my own test using the same slow/fast page concept in .NET 4.0 but wasn't able to see any thread switching behavior. The only time I can make it happen it's when I mark the page as Async and use AddOnPreRenderCompleteAsync to add my own being/end request handler. In summary, thread switching happens in an async operation is perfectly understandable and the author needs to publish how he made it happen in a synchronous operation.

中華民國虛擬總統: 7 years later, given you're almost certainally not using a single core PC, it's going to be hard to reproduce under the same level of trival load. Try ramping it up a bit (a lot). Also, the threshold for request queuing / handoff has probably been tuned a fair bit between ASP.Net 2 and 4.

If anyone is looking to try to reproduce it, make sure you're using IIS or IIS Express and not Cassini (the built in web server) because Cassini is extremely different.

For me, a thread context variable is set during BeginRequest, and the thread sometimes changes between there and the handling of the request. My options appear to be (a) don't do it, or (b) try to make sure it happens right before the request is processed so there's more chance that it will stay.