But I Really Need To Open 500 PSTs!

Today we’re revisiting an issue I first encountered in Outlook 2003. We had a customer, a law firm, who kept all the mail associated with a particular case in a PST. Their users would pass these PSTs around, and depending on their case load, a single user might have over 100 PSTs open at the same time. This worked fine when they had Outlook 2000, but when they installed Outlook 2003 they found if they tried to open more than 20 or so PSTs, they started getting errors such as “Outlook failed to add the personal store to the session” and “Out of memory or system resources. Close some windows or programs and try again.”

Although we don’t recommend opening so many PSTs at the same time, we investigated to find out what had changed, and whether we could help them out. In order to understand what had changed, it helps to know a bit about how the PST provider, and by proxy, Outlook, manages open PSTs.

There are two mechanisms involved in the PST provider’s management of PST files to consider. The first is the per-PST cache. Any time you’re dealing with random reads/writes to a file, maintaining an in memory cache is critical to performance. The PST provider uses an algorithm based on the file size and other internal details about the PST to determine how large of an in-memory cache to allocate. In Outlook 2002, we greatly increased the size of the caches we use for all types of PST files. Larger blocks of memory mean we consume more of the address space available to the process. In practical terms, this meant we could open about 20 average sized PSTs before we were no longer able to allocate a block of memory large enough to hold the cache for the next file. As a test, I tried to open brand new PSTs and was able to get 85 of them before we ran out of memory.

Based on our investigation, and the needs of this and other customers, we decided to go in and fine tune the cache sizes to allow more PSTs to be opened at once. This involved a tradeoff in performance, so we placed this change behind a registry key. The KB article 907651 documents the hotfix we generated for Outlook 2003, as well as the UseLegacyCacheSize registry key. Outlook 2007 includes the fine tuned cache as well, but the path to the registry key changes slightly. Whereas in Outlook 2003 the key is under “HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\PST”, in Outlook 2007 you’ll set it under “HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Outlook\PST”. (Incidentally, this fix was backported to Outlook XP too. There, the reg key will have a 10.0 in it.)

Now, once this registry key is set, if we try and stress test how many PSTs we can open, we’ll see we hit a practical limit of just under 300, but, unlike before, our address space isn’t exhausted. And where before, if your MAPI based process could only open 20 PSTs, you could always run another process and open 20 more, this new upper limit is spread across all MAPI processes on the box. This is due to the second mechanism the PST provider uses, shared memory.

As you might recall from my discussion of PST files in multiple profiles, the PST provider uses shared memory to track and coordinate access to PSTs across multiple processes. Each PST opened incurs a tax on this shared memory, and after enough PSTs are opened, there is no longer space in this block to track any more PSTs. As part of the work we did in introducing the UseLegacyCacheSize, we also increased the size of this shared memory block, which is what will allow you to reach nearly 300 open PSTs across multiple processes.

Now – just because you can open nearly 300 PSTs doesn’t mean you should. You’ll start to see noticeable performance issues when you get over 100 PSTs. In fact, if shouldn’t set UseLegacyCacheSize at all if you don’t need it. The smaller cache has a performance penalty. If you have an application which for some reason needs to open a large number of PSTs, I would consider designing it not to need all of them open at the same time. Only open the PST when you need it and unload it immediately after.

Hi – I added the PST key as described; but am still getting the error. I’m using Outlook 2003 on a newly-created account, and trying to add another .pst file; but it’s falling over when I try to create the file, before I try to import anything to go into it. Can’t believe it’s really a problem of size!

Hi – thanks for your reply. To explain more fully: I originally had a POP3 account with a single .pst file of 700-odd M (so presumably well within limits) which suddenly started giving "unknown error" 0x80040119 and stopped sending and receiving. I tried scanpst to no avail; so bought a proprietary .pst fixer and ran it against the file. I then re-named this version and the original to .old (so Outlook wouldn’t try to pick them up when I replaced the account), and deleted the account and its mailbox using the Virgin (my IP) website’s mailbox and account creation facility. This is where the description I originally posted above starts: I created a new mailbox and account using the Virgin website; and opened it in Outlook, forcing it to create a new (tiny) .pst file for the account. I then intended to add my newly-repaired (hopefully) .pst file to this account; but got no further than using File–>New–>Outlook Data File to try to create a new one that I could import this into – got the famous memory error, even though I hadn’t yet tried to import the file. Only Outlook was open on my machine; and as far as I know, all the .pst files it might have picked up had had their names changed to .old. There’s another user on the same home machine which has continued to work perfectly throughout all this (that user wasn’t logged in while I was trying to fix the "broken" user’s account).

As said above, I tried adding a new PST registry key as described on the Microsoft website (the nearest thing to a likely-sounding fix that my Googling turned up), but this didn’t seem to make a difference. However, if this really isn’t the problem you described, that might explain why! I would, though, love to know what the problem here is, and how to fix it, if you can suggest anything. Happy to provide any further information required.

What if you just have two (only 2!) mail stores you want to open simultaneously yet Outlook (2007) still runs out of memory? My primary mail store is server-based ~235MB in size and I have a PST-based archive (mail older than 6 months) that is much larger – ~950MB in size. Using two different profiles (one with just the primary store, the other with just the archive store), Outlook works just fine. If I try to open both my primary and archive stores simultaneously however, I receive the "Out of memory" error. I suspect the root cause of the problem here is that I am running Outlook with Business Contact Manager 2007 installed.

My machine configuration is currently Vista-64 with 8GB RAM although I have also tried XP with 4GB RAM and Windows 7 RC with 8GB RAM – same problem.

Lloyd – whatever you’re running in to, it’s not this problem. And I doubt anything’s running out of memory. Failures to create objects tend to get bubbled up as memory failures even if memory had nothing to do with the cause.

If you think BCM may be part of the problem, try uninstalling it to determine if the behavior changes.