Blog Stats

Meta

Posts Tagged ‘Process Explorer’

I recently began encountering a problem where one of our application packages was getting hung during the install process. I could see the program directory correctly populated with application files but it seemed to get hung at end, and msiexec processes continued to run indefinitely. After manually killing each process, I checked to see if the application actually completed installing. When trying to run one of the applications that has a dependency on one of the failed MSI’s APIs, I encountered the following:

Further, Outlook was crashing when trying to load the application’s main component.

Afterwards, the application with the dependency launched without error and Outlook was no longer crashing. However, the root issue still needed to be identified as I had no idea what else in the MSI was failing to complete. After manually deleting the components from the file system (it was missing from Programs and Features), I ran the MSI again and this time used Process Explorer to look inside the hung msiexec processes. There were multiple processes running, but I could see one process, though, eating CPU resources, and this was likely where I might find the culprit:

Opening the process details, I select the Threads tab. The thread that is doing all the work (or in this case getting hung up) is indicated by the Cycles tab. From here, I opened the thread by selecting it and clicking Stack:

Stacks are read from the bottom up. You can see a 3rd party component here (HCApi – McAfee HIPS) at play. Note, this might not be entirely abnormal as you can always expect any anti-virus suite to be hooking itself into any number of processes. So, to confirm what I was seeing, I used the Task Manager to dump the msiexec process that was using the CPU time by right-clicking it and select Create dump file (which can also be done via Process Explorer):

Dumps can be analyzed with WinDbg or a tool like DebugDiag 2.0. I have grown increasingly lazy and forgetful over the years with WinDbg. It has a very high learning curve and if you don’t use it much, its easy to forget everything except the old faithful !analyze –v or !analyze –v –hang. I used DebugDiag instead. When it is installed, simply right-click the dump file and select Analyze Crash/hang Issue from the context menu and point it to the crash file.

It will do its best to figure out the issue in the most vague way and you almost always need to do some interpretation of your own. For the most part, the analysis summary can be ignored.

A little bit down in the report points me to what I was seeing in Process Explorer with the problem thread that was using all the CPU time:

You can click the Thread ID like a hyperlink to follow it down in the report, expanding the thread. It reveals a familiar site. However, there is a bit more insight as the entry point is revealed and I can see the problem has something to do with the CustomAction table of the MSI itself.

Through a quick process of elimination using Orca to remove some of the rows in the CustomAction table, I narrow down the cause specifically down to action ISSelfRegisterCosting.

Without that row, the MSI install completes normally; or so at least it seems. I have no idea what removing this action might have elsewhere so this merely a hack. To further confirm the problem is being caused by McAfee, I perform the install on a virtual machine where McAfee is not installed and it proceeds normally. I then reach out to our McAfee enterprise admin and ask him to disable HIPS on one of my workstation. After doing so, the MSI runs and the application installs as expected. He inherits the problem.

Update

This has since been corrected by MacAfee with a HIPS update. One of the DLLs trying to get registered was getting blocked.

After a recent security update for our XP workstations, a couple complaints came in where user’s were having difficulty opening Microsoft Office files across the network. In some cases, the file would open, but only after a delay of a few minutes. In other cases, the file would not open at all, causing the Office application (Word, Excel) to become unresponsive and hung up. The files were not ridiculously large, and opening the same files locally did not present a problem. Identifying the cause was a simple matter of turning to Process Explorer and examining the stack of the working program thread:

You can see the stack growing with a couple dozen calls to some component named GKExcel.dll. Turning on the Lower Pane to view DLLs (View > Lower Pane View > DLLs), I can see it is described as Microsoft Component, but the description is too generic to make out the purpose:

After you install MS11-021 and the Office File Validation (OFV) Add-in for Microsoft Office 2003 (KB 2501584), workbooks stored in a network location open more slowly over the network in Excel 2003 than they did without the OFV installed. The decrease in performance depends on the size of the workbook and bandwidth of the network, and in some scenarios, can seem to cause Excel to crash.

The issue is not specific to Excel, however. Word files were taking several minutes to open as well. Resolving is a simple matter of uninstalling the Microsoft Office File Validation Add-in or modifying the registry to make an exception for the application opening the file. To do this:

Go to HKCU\Software\Policies\Microsoft\Office\<ver>\<application>\Security.

This is an XP workstation so likely the post-mortem default debugger is capturing the exception. I UNC-navigate to \\computername\Documents and Settings\All Users\Application Data\Microsoft\DrWatson. I grab both the drwtsn32.log and user.dmp files. They have recent time stamps of the day before which means that they were likely created as a result of the issue the user was experiencing. I start by examining the log file, starting from the bottom working my way up. The user’s initial complaint was the IE was crashing when going to various websites. I expected to find iexplore.exe process crashing in the log. A few searches in the text file later, I find IE crashing on that day:

I often find myself running Process Monitor and Process Explorer on user workstations. But to get the most of either of these tools, you really need to configure symbols so you can accurately read thread and stack information when doing a deeper analysis of a process. This can often be a nuisance because I am a cut and paste type of guy and even after doing it numerous times, I still have trouble recalling the Microsoft symbol path. On top of that, many times I run these tools with the user connected so speed is a necessity. To work around that, I decided to write a small script that I can run from a file server that will do it for me. Run the script before your start either tool.

There is one prerequisite, however: you need the full dbghelp.dll from the Windows Debugging Tools as the debug help DLL in system32 is not sufficient. If you are running on both 32 and 64 bit systems, you will need to get both the 64 bit and 32 bit versions. Store them away on network share and modify the script below to look to that share. In my case, we are still a mixed environment so I renamed the 32 bit dbghelp to dbghelp32.dll while the 64 bit version remains unchanged and created two different scripts. The respective DLL will copied into a folder C:\DbgHelp on the local system .

I also added a 5 second duration for new open and close processes and threads.

The problem starts after the user has had their Windows profile recreated but continues to experience some Outlook performance issues. Each time she clicks the New button in Outlook to create a new message, there is a 5 to 10 second delay before the blank message opens. There is also a 10+ second delay after she clicks send and Outlook becomes unresponsive for that time. I connect to her and quickly look at the Office Outlook registry for any unapproved 3rd party add-ins and see nothing out of the ordinary. I decide to dump the Outlook process when the hang is encounter and turn to Process Explorer. This is a simple matter of right-clicking the process and choosing Create Dump when it is in an unresponsive state. Actually, I collect two dumps, the other with Procdump in hang mode. I do this to ensure that the dumps are consistent; if the dumps are not similar then I may need to collect more until a pattern emerges.

I copied the dumps to my workstation and opened them with WinDbg and run the !analyze –v –hang command. The stack for both dumps are similar. With the exception of the NRTExchn component (our DMS add-in for Outlook), these are all MS modules. Normally, this wouldn’t excite me but something stands out like a sore thumb as I move up the stack, the presence of msi.dll in the stack of the current examine thread.

In addition to local processes locking up files and preventing their usage by other processes, files can also become locked by remote processes, too. Finding the remote system that has a handle(s) on the file can be a little bit more more involved. I mean this literally: a little a bit more involved. Knowing which tools to use or where to look can make this task just as simple as isolating it to a local process as outlined in part I.

This example is a recreation of an issue I encountered while working on a workstation remotely. It manifested itself as a failure to logon as the user: “Windows cannot load the locally stored profile…”

Every now and then some app somewhere is going to hook into some user file and prevent it from loading when its host application is opening up, resulting in some kind of error. A common one is when opening Outlook: “The file C:\Documents and Settings\username\Local Settings\Application Data\Microsoft\Outlook\outlook.ost is in use and could not be accessed. Close any application that is using this file, and then try again.”

Sometimes when you are troubleshooting with Process Explorer, it’s helpful to be able to view functions in threads to isolate a problem. The same goes for Process Monitor when viewing the Stack tab in the properties of an operation. By default, Process Explorer and Process Monitor will point to the dbghelp.dll in the windows\system32 folder, but this is a stripped down version and doesn’t support symbol server functionality. Instead, you need the dbghelp.dll from the Windows Debugging Tools to properly resolve symbols, otherwise you will encounter the Process Explorer Warning box when you click the Threads tab in process properties:Read the rest of this entry »

The initial issue I was asked to look at started with Internet Explorer failing to gracefully exit. Instead, it would just hang and required manual intervention via the Task Manager to kill the iexplore.exe process. I connected remotely to the workstation and ran Process Explorer so I could examine IE’s threads for anything out of the ordinary. Sure enough, I saw the presence of a .tmp file:Read the rest of this entry »

If you had a chance to review the Diagnostics-Performance logs, you may have encountered warning events coming from event ID 500 “The Desktop Window Manager is experiencing heavy resource contention. Video memory resources are over-utilized and there is thrashing happening as a result…” or 501 “…Graphics subsystem resources are over-utilized.”Read the rest of this entry »