Blog Stats

Meta

Posts Tagged ‘Hang’

Not too long ago I was asked to start looking into then recent reports of Outlook hanging on first launch. The first launch hang symptom happened as mail items were writing into the OST (cached mode) as user’s came in each morning and opened Outlook. Forcefully closing and reopening Outlook was a workaround but the cause needed to be identified as the impact was potentially the entire firm.

After a couple days I began to see a common theme. I was using Resource Monitor (resmon.exe – also accessible from the Task ManagerPerformance tab) in an attempt to identify the culprit. For hang scenarios, simply recreate the issue, open Resource Monitor, go to the Overview tab and locate the hung process. Typically a hung process will be displayed in red:

Right-click the hung process and select Analyze Wait Chain to get a list of the thread IDs. One of these threads will contain the module(s) that is causing the process to become unresponsive.

To look inside the threads, you can use Process Explorer. Just right-click on the target process, select Properties and go to the Threads tab. From here, use the TID column to locate the threads from the wait chain provided by resmon. Double-click the thread or select the Stack button to view the loaded modules in the thread. In this case, I saw a consistent stack in each of the hung Outlook processes on the various workstations:

There is a single 3rd party module appearing here, everything else is Microsoft. This made sense as this particular module was part of a recent Outlook add-in upgrade. The other threads were also examined but the only 3rd party modules they were referencing were proven addins that had never caused issues. A POC was tested on with several users over the course of a couple weeks by disabling the add-in via load behavior in the registry and issue did not resurface. The issue was raised with the vendor and a newer client for the add-in was provided that resolved this problem.

Another way to identify the cause assuming you have Windows Debugging Tools installed is to generate a dump of the hung process from the Task Manager. A quick !analyze –v –hang pointed to the same culprit (this is a 32bit process from a 64bit OS so you will need to get the 32bit stacks. Use .load wow64exts and .effmach x86 commands before !analyze):

After a recent security update for our XP workstations, a couple complaints came in where user’s were having difficulty opening Microsoft Office files across the network. In some cases, the file would open, but only after a delay of a few minutes. In other cases, the file would not open at all, causing the Office application (Word, Excel) to become unresponsive and hung up. The files were not ridiculously large, and opening the same files locally did not present a problem. Identifying the cause was a simple matter of turning to Process Explorer and examining the stack of the working program thread:

You can see the stack growing with a couple dozen calls to some component named GKExcel.dll. Turning on the Lower Pane to view DLLs (View > Lower Pane View > DLLs), I can see it is described as Microsoft Component, but the description is too generic to make out the purpose:

After you install MS11-021 and the Office File Validation (OFV) Add-in for Microsoft Office 2003 (KB 2501584), workbooks stored in a network location open more slowly over the network in Excel 2003 than they did without the OFV installed. The decrease in performance depends on the size of the workbook and bandwidth of the network, and in some scenarios, can seem to cause Excel to crash.

The issue is not specific to Excel, however. Word files were taking several minutes to open as well. Resolving is a simple matter of uninstalling the Microsoft Office File Validation Add-in or modifying the registry to make an exception for the application opening the file. To do this:

Go to HKCU\Software\Policies\Microsoft\Office\<ver>\<application>\Security.

I recently put together a quad core system from parts my brother was retiring from his home system. Soon after getting everything up and running, the new system would sometimes hard hang while working via VPN. A hard hang is when the OS becomes completely unresponsive but is still running. I had earlier prepared myself for the next instance of this encounter by enabling CrashOnCtrlScroll in the Windows registry so that the next time it happened I could manually crash the system from the keyboard and examine the memory dump with WinDbg for signs of the responsible culprit. You can read about how to enable this option in this earlier blog Forcing a System Crash on an Unresponsive PC.

Upon opening a kernel dump, the analyze –v command is a clickable hyperlink1.

A couple weeks ago a co-worker asked me if it was possible to recover text from a hung application. The user had spent a good amount of time typing into text field and upon trying to submit the information, IE became hung and would not recover. The user didn’t want to have to retype everything from scratch again. The short answer to this is yes. The long answer is “Yes, if you can be patient.” I have never actually needed to perform this myself because it’s not always that practical and, to be honest, the task can be somewhat tedious. But if you really need to recover text and Word didn’t auto-recover or Outlook lost the draft after you clicked send, or IE is in the process of a “GUI crash”, then turn to the power of the dump.

The idea here is if the application is still running but stalled, it still resides in memory, along with anything you typed into it. When you dump the process, you are dumping its presence in memory to a file that you can pick apart. Dumping a hung process is simple enough. On an XP system, open Process Explorer, right-click the process, and choose Create Dump. In Vista & Windows 7, this option is now built into the Task Manager. You can then copy the dump to a system where WinDbg is installed to open it.

The problem starts after the user has had their Windows profile recreated but continues to experience some Outlook performance issues. Each time she clicks the New button in Outlook to create a new message, there is a 5 to 10 second delay before the blank message opens. There is also a 10+ second delay after she clicks send and Outlook becomes unresponsive for that time. I connect to her and quickly look at the Office Outlook registry for any unapproved 3rd party add-ins and see nothing out of the ordinary. I decide to dump the Outlook process when the hang is encounter and turn to Process Explorer. This is a simple matter of right-clicking the process and choosing Create Dump when it is in an unresponsive state. Actually, I collect two dumps, the other with Procdump in hang mode. I do this to ensure that the dumps are consistent; if the dumps are not similar then I may need to collect more until a pattern emerges.

I copied the dumps to my workstation and opened them with WinDbg and run the !analyze –v –hang command. The stack for both dumps are similar. With the exception of the NRTExchn component (our DMS add-in for Outlook), these are all MS modules. Normally, this wouldn’t excite me but something stands out like a sore thumb as I move up the stack, the presence of msi.dll in the stack of the current examine thread.

One of the most common complaints your going to encounter in the desktop support role is when Outlook becomes unresponsive while a user is performing any random task in Outlook, whether it be switching between folders, going into a delegated mailbox, or sorting messages. Often, the knee-jerk reaction by the front line technical support is to assume something is wrong with Outlook and begin the gamut of what I term “blind-troubleshooting”. This usually involves running an range of scripted fixes (that is meant to address issues with some of our in-house or 3rd party add-ins), followed by a repair, reinstall, and/or creation of a new Outlook profile. Over the lifetime of this incident(s), the issue can drag out to several days because usually the issue cannot be reproduced immediately after the first fix, so each additional fix is tried at some point later when the user calls back. In some cases, this drags out for a couple weeks to where the user’s workstation is replaced or Windows profile is recreated.

The other day while trying to remove some add-ons from Internet Explorer, Windows DEP was kicking in and closing the IE Setting control panel box (rundll32.exe process) for whatever reason:
Or at least it was trying to. Read the rest of this entry »

After loading up Word today, I noticed it would become hung each time I went to the Insert tab. I was able to reproduce the behavior after killing the process and starting it up again. To troubleshoot, I started by going to the Windows 7 Task Manager, right-clicking, and selecting Create Dump File from the context menu:Read the rest of this entry »

The initial issue I was asked to look at started with Internet Explorer failing to gracefully exit. Instead, it would just hang and required manual intervention via the Task Manager to kill the iexplore.exe process. I connected remotely to the workstation and ran Process Explorer so I could examine IE’s threads for anything out of the ordinary. Sure enough, I saw the presence of a .tmp file:Read the rest of this entry »

Sometimes Microsoft applications do a a good job at identifying the problem module when they crash. For example, in the Windows Vista/7 OS IE crashes with “Internet Explorer has stopped working”. Simply click the Details chevron to and look at the Fault Module Name:Read the rest of this entry »