Since the 1960’s, hard disks have always used a block size of 512 bytes for the default read/write block size.Recently drive manufacturers have been moving toward a larger block size to improve performance and reliability.Currently there are two types of disks available with a 4KB sector size: 512 byte emulated, and 4KB block sized disks.

Disks with 4KB block size and 512 bytes per sector emulation

For performance reasons, drive manufacturers have already produced disks with 4KB native block size, which use firmware to emulate 512 bytes per sector.Because of the emulated 512 byte sector size, the file system and most disk utilities will be blissfully unaware that they are running on a 4KB disk.As a result, the on-disk structures will be completely unaffected by the underlying 4KB block size.This allows for improved performance without altering the bytes per sector presented to the file system.These disks are referred to as 512e (pronounced “five-twelve-eee”) disks.

Disks with 4KB block size without emulation

When the logical bytes per sector value is extended to 4KB without emulation, the actual file system will have to adjust to this new environment.Actually, NTFS is already capable of functioning in this environment provided that no attached FS filter drivers make false assumptions about sector size.Below are the highlights of what you should expect to see on a disk with a 4KB logical sector size.

1. It will not be possible to format with a cluster size that is smaller than the 4KB native block size.This is because cluster size is defined as a multiple of sector size.This multiple will always be expressed as 2n .

2. File records will assume the size of the logical block size of 4KB, rather than the previous size of 1KB. This actually improves scalability to some degree, but the down-side is that each NTFS file record will require 4KB or more in the MFT.

3. Sparse and compressed files will continue to have 16 clusters per compression unit.

4. Since file records are 4 times their normal size, it will be possible to encode more mapping pairs per file record. As a result, larger files can be compressed with NTFS compression without running into file system limitations.

5. Since the smallest allowable cluster size is 4KB, NTFS compression will only work on volumes with a 4KB cluster size.

6. Bytes per index record will be unaffected by the 4K block size since all index records are 4KB in size.The on-disk folder directory structures will be completely unaffected by the new block size, but a performance increase may be seen while accessing folder structure metadata.

7. The BIOS Parameter Block (BPB) will continue to have the same format as before, but the only positive value for clusters per File Record Segment (FRS) will be 1. In the case where clusters per FRS is 1, the FRS byte size is computed by the following equation:

NTFS BIOS Parameter Block Information

BytesPerSector : 4096

Sectors Per Cluster : 1

ReservedSectors : 0

Fats : 0

RootEntries : 0

Small Sectors : 0 ( 0 MB )

Media Type : 248 ( 0xf8 )

SectorsPerFat : 0

SectorsPerTrack : 63

Heads : 255

Hidden Sectors : 64

Large Sectors : 0 ( 0 MB )

ClustersPerFRS : 1

Clust/IndxAllocBuf : 1

NumberSectors : 50431 ( 196.996 MB )

MftStartLcn : 16810

Mft2StartLcn : 2

SerialNumber : 8406742282501311868

Checksum : 0 (0x0)

If the cluster size is larger than the FRS size, then ClustersPerFrs will be a negative number as shown in the example below (0xf4 is -12 decimal).In this case, the record size is computed with the equation:

In short, NTFS will always force a 4096 byte cluster size on disk with a 4KB sector size regardless of the cluster size.

NTFS BIOS Parameter Block Information

BytesPerSector : 4096

Sectors Per Cluster : 4

ReservedSectors : 0

Fats : 0

RootEntries : 0

Small Sectors : 0 ( 0 MB )

Media Type : 248 ( 0xf8 )

SectorsPerFat : 0

SectorsPerTrack : 63

Heads : 255

Hidden Sectors : 64

Large Sectors : 0 ( 0 MB )

ClustersPerFRS : f4

Clust/IndxAllocBuf : f4

NumberSectors : 50431 ( 196.996 MB )

MftStartLcn : 4202

Mft2StartLcn : 1

SerialNumber : 7270585088516976380

Checksum : 0 (0x0)

8. Aside from the 4KB file record size, there are a few other things to know about 4KB drives. The code for implementing update sequence arrays (USA’s) has always worked on a 512 byte assumed sector size and it will continue to do so. Since file records are 4 times their normal size, the update sequence arrays for file records now contain 9 entries instead of 3. One array entry is required for the sequence number (blue) and eight array entries for the trailing bytes (red).The original purpose of USA is to allow NTFS to detect torn writes.Since the file record size is now equal to the block size, the hardware is capable of writing the entire file record at once, rather than in two parts.

I don’t actually own a 4KB disk, but I was able to give you this preview thanks to a nifty tool called VStorControl.Vstor is a tool which allows you to create virtualized SCSI disks with arbitrary block sizes and is available for download with the Windows 7 SDK.

Hello all, East here. I wanted to give you a hint on how to use a RHS dump to find what thread was part of the Windows Server 2008 R2 cluster RHS recovery deadlock.

First let me start off with letting you know that Windows Server 2008 R2 will create two types of user-mode dumps:

1 - A heap dump (.hdmp) - which is an extended mini-dump that contains additional data such as the process memory.

2 - A mini dump (.mdmp) - which will only contain mainly stack information and not enough info to help you get further.

When a Windows Server 2008 R2 cluster has a resource that does not respond to one of the main cluster queries in a timely fashion, RHS will try to recover that resource by restarting it in an isolated process as well as creating a WER report and a user-mode dump of the deadlock.

Now if you want to get adventurous and review the dump file to see the call stacks of RHS and what the resource was doing, you will look at the WER dump with the extension “.hdmp”.Since we are looking at a process with multiple threads, we need to find the one that is showing the WER reporting calls and the one showing the RHS deadlock.

Run the following command after opening the dump under your debugger “~*k”. In the below example, thread 3 is the one we are interested in. You can tell this because the process and thread ID matches the cluster log above. If you don’t have a cluster log, this is the thread with wer.dll on the stack.

We will switch context to thread 3 by using the “ ~3s ” command, and follow that with “ kn “ to show the stack with frame numbers.Debug Tip: Since we already know that its thread id is “aac” we can also switch directly to thread 3 by using its thread id in the following command syntax:“ ~~[aac]s ”

Here is where we get into tricky territory about where to look. I know from experience that the RDI register from above is going to end up containing my information for this resource. Another way to verify you are in the right area, dumping RDI will show the CallType.

In most cases the cluster log can provide enough information to help you narrow down possible cluster resource issues. If not, the user-mode dump may be able to help verify additional data, especially if you are debugging a third party resource dll that is deadlocking RHS.

Greetings fellow debuggers, today I will be blogging about Event ID 129 messages.These warning events are logged to the system event log with the storage adapter (HBA) driver’s name as the source.Windows’ STORPORT.SYS driver logs this message when it detects that a request has timed out, the HBA driver’s name is used in the error because it is the miniport associated with storport.

Below is an example 129 event:

Event Type:Warning

Event Source:<HBA_Name>

Event Category:None

Event ID:129

Date:4/9/2009

Time:1:15:45 AM

User:N/A

Computer:<Computer_Name>

Description:

Reset to device, \Device\RaidPort1, was issued.

So what does this mean? Let’s discuss the Windows I/O stack architecture to answer this.

Windows I/O uses a layered architecture where device drivers are on a “device stack.”In a basic model, the top of the stack is the file system. Next is the volume manager, followed by the disk driver. At the bottom of the device stack are the port and miniport drivers. When an I/O request reaches the file system, it takes the block number of the file and translates it to an offset in the volume. The volume manager then translates the volume offset to a block number on the disk and passes the request to the disk driver.When the request reaches the disk driver it will create a Command Descriptor Block (CDB) that will be sent to the SCSI device.The disk driver imbeds the CDB into a structure called the SCSI_REQUEST_BLOCK (SRB). This SRB is sent to the port driver as part of the I/O request packet (IRP).

The port driver does most of the request processing. There are different port drivers depending on the architecture. For example, ATAPORT.SYS is the port driver for ATA devices, and STORPORT.SYS is the port driver for SCSI devices.Some of the responsibilities for a port driver are:

Providing timing services for requests.

Enforcing queue depth (making sure that a device does not get more requests that it can handle).

The port driver interfaces with a driver called the “miniport”. The miniport driver is designed by the hardware manufacturer to work with a specific adapter and is responsible for taking requests from the port driver and sending them to the target LUN.The port driver calls the HwStorStartIo() function to send requests to the miniport, and the miniport will send the requests to the HBA so they can be sent over the physical medium (fibre, ethernet, etc) to the LUN. When the request is complete, the miniport will call StorPortNotification() with the NotificationType parameter value set to RequestComplete, along with a pointer to the completed SRB.

When a request is sent to the miniport, STORPORT will put the request in a pending queue. When the request is completed, it is removed from this queue. While requests are in the pending queue they are timed.

The timing mechanism is simple. There is one timer per logical unit and it is initialized to -1. When the first request is sent to the miniport the timer is set to the timeout value in the SRB. The disk timeout value is a tunable parameter in the registry at: HKLM\System\CurrentControlSet\Services\Disk\TimeOutValue. Some vendors will tune this value to best match their hardware, we do not advise changing this value without guidance from your storage vendor.

The timer is decremented once per second.When a request completes, the timer is refreshed with the timeout value of the head request in the pending queue. So, as long as requests complete the timer will never go to zero. If the timer does go to zero, it means the device has stopped responding.That is when the STORPORT driver logs the Event ID 129 error. STORPORT then has to take corrective action by trying to reset the unit. When the unit is reset, all outstanding requests are completed with an error and they are retried. When the pending queue empties, the timer is set to -1 which is its initial value.

Each SRB has a timer value set. As requests are completed the queue timer is refreshed with the timeout value of the SRB at the head of the list.

The most common causes of the Event ID 129 errors are unresponsive LUNs or a dropped request.Dropped requests can be caused by faulty routers or other hardware problems on the SAN.

I have never seen software cause an Event ID 129 error. If you are seeing Event ID 129 errors in your event logs, then you should start investigating the storage and fibre network.

Hi All, this is Karim Elsaid and I’m a Support Escalation Engineer working with the Dubai platforms support team.Recently I was working on a very challenging and interesting case, and I wanted to share that experience with you.

One of our customers was experiencing a problem on all his Domain Controllers (Running x86 Windows Server 2003), where intermittently the DCs ran out of resources. When one of the servers ran out of resources it had to be restarted to restore its functionality. The following event was logged on the DC when the problem occurred:

This event gets logged when the Local Security Authority Subsystem (lsass.exe) process is unable to allocate memory from the heap.

When this problem occurs, users are not able to authenticate and Exchange servers cease functioning until the DC gets rebooted.

By the time we got the call, the DC that was exhibiting the issue had been rebooted, hence when we checked the lsass.exe memory usage it was very low, but when we checked another DC on the same site, the lsass.exe memory usage was around 1.7GB.Well, that’s pretty high but it is worth mentioning that by default lsass.exe may utilize a large amount of virtual memory for the ESENT Cache (Active Directory database), nevertheless this should not cause any issues on DCs as it should back-off when needed.

The issue here is that the memory usage for lsass.exe kept growing and never got released back again - this is called a process memory leak.

To confirm our theory we obtained a Perfmon Log from the DC and checked how Lsass.exe is doing with memory, which proved the theory that lsass.exe is actually leaking memory.

We can see from the graph above a steady increase in lsass.exe private bytes.The DC was booted without /3GB and hence the limit of the user mode process address space is 2GB.

Now, we need to explore the options on how to track and address such leaks. We could obtain an lsass.exe process dump and check the heap, or obtain a Full memory dump of the server while in the failed state, to check the lsass.exe heap usage and overall system state, but the thing is none of the above may immediately reveal what’s going on.We would need to be very lucky to get the dump while the actual memory call that happens inside lsass.exe process is executing, which is very slim.

What we are looking for is a “pattern”, the pattern which is causing the actual leak inside the lsass.exe process.

For that we will turn to a tool that I see many of the customers are not aware of, but in fact it is very handy and easy to troubleshoot such a memory leak problem. We will use the “User Mode Dump Heap” or UMDH. By using this tool it will do the job for us, it will monitor the all heap calls that happen in the lsass.exe process.All we need to do is to download the “Debugging Tools For Windows”, enable user mode stack acquisition from the command line, download the symbols from Microsoft public symbol servers, reboot the server and we are good to go.

So now, all heap blocks that the lsass.exe process allocates will get monitored by UMDH, and the tool will tell us information about that specific call stack.For example, the number of allocations using this stack, number of bytes consumed, etc…

Back to our main problem: We have setup the DC for UMDH and started monitoring the lsass.exe process.We verified UMDH is working after the reboot by generating a sample log - it is very critical to generate this simple log to make sure that you are able to resolve the symbols correctly.

We kept generating UMDH logs daily and then the interesting pattern started to appear, which shows:

+ 123097760 ( 123098480 -720) 1538731 allocsBackTrace4BA0

+ 1538722 ( 1538731 -9)BackTrace4BA0allocations

ntdll!RtlAllocateHeapSlowly+00000041

ntdll!RtlAllocateHeap+00000E9F

msvcrt!malloc+0000006C

NTDSA!ABBind_local+0000031C

NTDSA!NspiBind+0000013D

RPCRT4!Invoke+00000030

RPCRT4!NdrStubCall2+00000299

RPCRT4!NdrServerCall2+00000019

RPCRT4!DispatchToStubInCNoAvrf+00000038

RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0000011F

RPCRT4!RPC_INTERFACE::DispatchToStub+000000A3

RPCRT4!OSF_SCALL::DispatchHelper+00000149

RPCRT4!OSF_SCALL::DispatchRPCCall+0000010D

RPCRT4!OSF_SCALL::ProcessReceivedPDU+0000057F

RPCRT4!OSF_SCALL::BeginRpcCall+00000194

RPCRT4!OSF_SCONNECTION::ProcessReceiveComplete+00000435

RPCRT4!ProcessConnectionServerReceivedEvent+00000021

RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+000001B8

RPCRT4!ProcessIOEventsWrapper+0000000D

RPCRT4!BaseCachedThreadRoutine+0000009D

RPCRT4!ThreadStartRoutine+0000001B

kernel32!BaseThreadStart+00000034

From the backtrace 4BA0 we can see that there are ~1.5 heap allocation calls that have occurred without being freed from the first day we started to monitor lsass using UMDH.

There is also another call stack with exactly the same symptoms:

+ 51525199 ( 51525506 -307) 1538802 allocsBackTrace4B92

+ 1538793 ( 1538802 -9)BackTrace4B92allocations

ntdll!RtlAllocateHeapSlowly+00000041

ntdll!RtlAllocateHeap+00000E9F

LSASRV!LsapAllocatePrivateHeapNoZero+0000004F

RPCRT4!operator new+0000000D

RPCRT4!W2AAttachHelper+00000032

RPCRT4!CNlDelUnicodeAnsi::Convert+00000014

NTDSA!ABBind_local+0000005D

NTDSA!NspiBind+0000013D

RPCRT4!Invoke+00000030

RPCRT4!NdrStubCall2+00000299

RPCRT4!NdrServerCall2+00000019

RPCRT4!DispatchToStubInCNoAvrf+00000038

RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0000011F

RPCRT4!RPC_INTERFACE::DispatchToStub+000000A3

RPCRT4!OSF_SCALL::DispatchHelper+00000149

RPCRT4!OSF_SCALL::DispatchRPCCall+0000010D

RPCRT4!OSF_SCALL::ProcessReceivedPDU+0000057F

RPCRT4!OSF_SCALL::BeginRpcCall+00000194

RPCRT4!OSF_SCONNECTION::ProcessReceiveComplete+00000435

RPCRT4!ProcessConnectionServerReceivedEvent+00000021

RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+000001B8

RPCRT4!ProcessIOEventsWrapper+0000000D

RPCRT4!BaseCachedThreadRoutine+0000009D

RPCRT4!ThreadStartRoutine+0000001B

kernel32!BaseThreadStart+00000034

From both call stacks above we can see that the originating function that leads to the heap allocation is “NTDSA!NspiBind”

NSPI is the interface which the Exchange Address Book provider uses to talk to the Global Catalogs. Typically when Outlook clients want to talk to the directory for address book, they talk to the Exchange server directly and then the Exchange server will look up the appropriate GC, forward the request to it and finally send the result back to the originating clients.This method is called the DSproxy method. Outlook clients can also send referral requests (RFR) to the Exchange server to get a referral to a GC and then start talking to this GC directly. You can find more information about this operation “Here”

With this new information in hand, we turned our attention to the Exchange server for a while. We did so by obtaininga netmon trace from the Exchange server, and started examining it to reveal a massive number of RFR requests originating from outlook client machines toward the Exchange server:

In the netmon capture we saw an excessive number of NspiBind requests been sent from the Outlook client machines to the Exchange server and Exchange server in turn sending these NspiBind requests to the GC, in the netmon capture we didn’t find any frames for NspiUnBind requests, which should be used to destroy that connection.

When we checked the Exchange server security logs we found a massive number of logon events that are occurring for every single client:

Event Type:Success Audit

Event Source:Security

Event Category:(2)

Event ID:540

Date:5/26/2010

Time:3:49:09 PM

User:

Computer:EXCHSERVER

Description:

Successful Network Logon:

User Name:User_Name

Domain:CONTOSO.COM

Logon ID:(0x0,0x3DA1F42)

Logon Type:3

Logon Process:Kerberos

Authentication Package:Kerberos

Workstation Name:

Logon GUID:{b03f9ad2-3150-e1be-f50a-b3abb9d2c09d}

Caller User Name:-

Caller Domain:-

Caller Logon ID:-

Caller Process ID: -

Transited Services: -

Source Network Address:10.11.12.13

Source Port:0

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

So, now we know the problem of lsass.exe memory consumption on the DCs is actually occurring because of a massive number of NspiBind requests that are being sent to the DCs without proper NspiUnbind requests.

So we turned our attention to the client machines that have Outlook installed and started to list all 3rd party add-ins that are installed on them.

We found that all affected client machines are actually sharing 3rd party add-ins for email content archival. Further investigation on this software revealed a strange MAPI Wrapper DLL library, and our customer confirmed that the start time of the problem approximately matches the time they have rolled out the content archival software on the client machines.

We uninstalled this content archival add-in from a couple of machines and these machines stopped hammering the Exchange Server and GC with NspiBind requests.

Later we uninstalled it from the rest of the client machines and the lsass.exe process on the DCs has finally stabilized and we no longer have any issues with its memory usage.

I’d like to mention that starting with Windows Server 2008 we have implemented a “NSPI Max Sessions Per User” where you can limit the number of NSPI requests from a client to the server.

That’s it! I hope you enjoyed reading and hopefully this should help you to troubleshoot any memory leaks you may encounter.The information contained here can be applied to other processes when you suspect heap leaks.

A customer recently reported a rather peculiar problem.They were working on a roll out of Windows 7, and one of the policies they employed on their domain was the “Show only specified control panel applets” setting.As its name implies, this policy allows an administrator to specify the names of any Control Panel icons they want their users to see, and all others are not shown.This company chose to allow a few of the Control Panel items included with Windows as well as the Mail icon that is added when Microsoft Office (specifically Outlook) is installed, which allows users to configure their Outlook profiles.

The problem the customer noticed was that, with the policy was in place, the first time a user opened the Control Panel on a computer, all of the allowed icons appeared except for Mail.If the user closed Control Panel and reopened it, the icon suddenly appeared.In fact, from that point on, the Mail icon would appear for that user on that computer.However, if that user logged on to a different computer, or a different user logged in to the first user’s computer, the icon was missing until Control Panel was opened for a second time.

Figure 1: Policy setting on our test system

Figure 2: Result from first open

Figure 3: Result on subsequent opens (desired)

Control Panel isn’t something we get many cases on, so I had to figure out how it worked, and what would cause it to fail only the first time.One of the first things I discovered as I debugged the issue was that the Control Panel is actually loaded in two passes – a “fast” pass and a “slow” pass.This is done so icons we know will be loaded quickly can be displayed first so a user doesn’t get stuck with an empty window waiting for some of the slower icons to load.What makes an icon fast or slow?It turns out shell maintains a cache of Control Panel items, so it can display them quickly without having to query the actual item for its name, description, icon, etc.If we have to ask the item for its information, that will take longer, so it is loaded during the second, slow phase:

shell32!CControlPanelDataWorkItem::_LoadSlowData

shell32!CControlPanelDataWorkItem::DoWork

shell32!CFrameTask::InternalResumeRT

shell32!CRunnableTask::Run

shell32!CShellTask::TT_Run

shell32!CShellTaskThread::ThreadProc

shell32!CShellTaskThread::s_ThreadProc

shlwapi!ExecuteWorkItemThreadProc

ntdll!RtlpTpWorkCallback

ntdll!TppWorkerThread

kernel32!BaseThreadInitThunk

ntdll!__RtlUserThreadStart

ntdll!_RtlUserThreadStart

Luckily for these icons, they won’t stay relegated to slow loading status forever.During the slow loading phase, one of the things we do is add the item’s information to the Control Panel cache, so we can load it during the fast phase in the future.

shell32!CPLD_AddControlToReg

shell32!CControlPanelEnum::_NextNonCachedCpl

shell32!CControlPanelEnum::Next

shell32!CRegFolderEnum::Next

shell32!CControlPanelAppletList::_AddAppletsToCategories

shell32!CControlPanelAppletList::LoadSlowApplets

shell32!CControlPanelDataWorkItem::_LoadSlowData

shell32!CControlPanelDataWorkItem::DoWork

shell32!CFrameTask::InternalResumeRT

shell32!CRunnableTask::Run

shell32!CShellTask::TT_Run

shell32!CShellTaskThread::ThreadProc

shell32!CShellTaskThread::s_ThreadProc

shlwapi!ExecuteWorkItemThreadProc

ntdll!RtlpTpWorkCallback

ntdll!TppWorkerThread

kernel32!BaseThreadInitThunk

ntdll!__RtlUserThreadStart

ntdll!_RtlUserThreadStart

This explains why the item was being loaded on subsequent opens of Control Panel – the item was added to the user’s cache on the first run, so the next time, it was loaded during the fast pass:

But that only answers half the question – while we now know why the icon appears every other time, we still don’t know why it doesn’t show up the first time.To figure this out, I debugged a system with the “Show only specified control panel applets” policy applied and one without the policy set, both as they opened Control Panel for the first time, and compared where the return codes from the functions diverged.

Starting with CControlPanelEnum::_NextNonCachedCpl, in the non-working case, we can see that it returns 0:

So we know something in this function is causing us to return FALSE in the failing case and TRUE in the working case.An easy way to see where the return result of function calls is the wt (watch trace) debugger command with the -oR switch to see the return values.I set the level (-l) to 1 so I’d only see functions directly called from CControlPanelEnum::_NextNonCachedCpl.Here were the results.

Here, we can see that the function CControlPanelEnum::_CanEnumerateApplet returns true in the working case, but false in the failing case.This is what is causing the return value from _NextNonCachedCpl to differ, which is ultimately what causes the icon to load/not load.So let’s take a look at _CanEnumerateApplet.Using wt inside this function showed something rather interesting:

Looking at the two, we can see that they both call GetDetailsEx, but in the failing case we seem to get an error result: 80070002, and in the working case we get 0, and then proceed to call CPL_DoesPolicyAllow.CPL_DoesPolicyAllow is not called on the machine with the policy applied.A quick code review of CPL_DoesPolicyAllow showed that this function checks if a policy prevents or allows an item from being displayed.So if we can just get to that function, the icon will load.Now we need to know why GetDetailsEx fails, preventing us from calling CPL_DoesPolicyAllow.If we look up the error code, it is pretty generic:

Next I stepped through and into GetDetailsEx.I’ll save you the tedious steps and output, since the procedure is similar to what I’ve already shown above.Anyway, I was able to trace the error 2 as being returned by a call to the registry a few functions deep into GetDetailsEx:

kernel32!RegOpenKeyExW

shell32!CControlPanelFolder::_GetExtPropRegKey

shell32!CControlPanelFolder::_InitExtPropRegValNameCache

shell32!CControlPanelFolder::_PropertyFromPidl

shell32!CControlPanelFolder::GetDetailsEx

shell32!CRegFolder::GetDetailsEx

shell32!CControlPanelEnum::_CanEnumerateApplet

shell32!CControlPanelEnum::_NextNonCachedCpl

This is good news, because it means we can find out what we’re looking for that we aren’t finding.First I stepped out to get back to CControlPanelFolder::_InitExtPropRegValNameCache.Sure enough, eax was 2, which is our ‘cannot find the file’ error.

Remember that function parameters are passed in in reverse order, so the second parameter is the second to last value passed to the function.In this case, we can see that it was stored in eax.Of course, eax has been overwritten by our return value of 2, but we can see that just before pushing eax, we get the value from ebp-0x270.Dumping that out as a Unicode string, we get the key we were looking for:

0:020> du @ebp-270h

14c5ed54"Software\Microsoft\Windows\Curre"

14c5ed94"ntVersion\Control Panel\Extended"

14c5edd4" Properties\System.ApplicationNa"

14c5ee14"me"

The first parameter is the root key, which was:

0:020> dc @ebp-278h L1

14c5ed4c80000002

We can look up this value in the Windows SDK header files and see that it is for HKLM:

#define HKEY_LOCAL_MACHINE(( HKEY ) (ULONG_PTR)((LONG)0x80000002) )

Aha!So now we know we were trying to access “HKLM\ Software\Microsoft\Windows\CurrentVersion\Control Panel\Extended

Properties\System.ApplicationName” which didn’t exist, so we were unable to proceed with loading the icon.

What’s that key for, you ask?MSDN tells us that it is used to store the Canonical Name of the Control Panel item.Canonical Names were added to the Control Panel in Windows Vista as a better way to organize and name the items in the Control Panel.By convention, the canonical name of “Mail” is “Microsoft.Office.Outlook.Profiles.”Because the “Show only specified control panel applets” policy can now accept either the legacy name (“Mail”) or the canonical name, we are checking the registry for this string.Since the key is not present, GetDetailsEx cannot determine the canonical name, and the icon does not load.

As a test, I created the key HKLM\ Software\Microsoft\Windows\CurrentVersion\Control Panel\Extended

Properties\System.ApplicationName and added a REG_SZ to it with a name of the file location of the Mail CPL file ("C:\Program Files\Common Files\SYSTEM\MSMAPI\1033\MLCFG32.CPL") and a value of "Microsoft.Office.Outlook.Profiles".Sure enough, the icon loads up on the first try for new users.

While new Control Panel items should implement a canonical name, that doesn’t work for existing CPLs, like Mail from Outlook 2003.Since the canonical name isn’t actually required (CPL_DoesPolicyAllow works with both name formats), we’re following up with the development team to allow CPL_DoesPolicyAllow to be called even if a canonical name is not found.

Hello All, Ron Riddle here again to share another mystery relating to the XP Mode Auto Publish feature for Windows 7 that had an unexpected root cause.As you may know, the Auto Publish feature allows applications installed within the guest to be automagically available from the start menu of the Windows 7 host.Of course, one requirement for this to work properly is that the feature must be enabled; however, I have encountered two such instances where the feature is listed as ‘Not Available’ within the Virtual PC Settings UI once the guest is started.

Debugging the Issue

The Virtual PC Integration Components Services Application (1-vmsrvc) service, which runs within a vmsrvc.exe instance on the guest, decides whether the Auto Publish feature should be enabled.For non-Windows 7 guests, the first thing 1-vmsrvc does is issue a WMI query to determine whether KB961742 (for an XP guest) or KB961741 (for a Vista guest), which provide Remote Applications Integrated Locally (RAIL) support, have been applied.

Root Cause Analysis

Before we issue the WMI query, we must first activate the CLSID_WbemLevel1Login component.Here’s an excerpt below:

However, I noticed that the activation attempt failed with WBEM_E_CRITICAL_ERROR(0x8004100a).

0:009> r eax

eax=8004100a

So, I proceeded to debug the activation attempt from within the Windows Management Instrumentation(winmgmt) service, since it provides the class factory for this component.Notice the call to LoadLibraryExW passing a relative path for the lpFileName parameter.This means that a search strategy must be applied which leverages the PATH environment variable.

0:002> k

ChildEBP RetAddr

0086f438 594976e2 kernel32!LoadLibraryExW

0086f494 7751d8a7 wmisvc!CForwardFactory::CreateInstance+0xf8

0086f4b8 7751daac ole32!GetInstanceHelperMulti+0x20

0086f578 77e799f4 ole32!CObjServer::CreateInstance+0x251

0086f59c 77ef421a RPCRT4!Invoke+0x30

0086f9a8 77ef4bf3 RPCRT4!NdrStubCall2+0x297

0086fa00 77600c15 RPCRT4!CStdStubBuffer_Invoke+0xc6

0086fa40 77600bbf ole32!SyncStubInvoke+0x33

0086fa88 7752ad31 ole32!StubInvoke+0xa7

0086fb60 7752ac56 ole32!CCtxComChnl::ContextInvoke+0xe3

0086fb7c 776007f5 ole32!MTAInvoke+0x1a

0086fbac 77602df3 ole32!AppInvoke+0x9c

0086fc80 77600715 ole32!ComInvokeWithLockAndIPID+0x2c2

0086fccc 77e794bd ole32!ThreadInvoke+0x1cd

0086fd00 77e79422 RPCRT4!DispatchToStubInC+0x38

0086fd54 77e7934e RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x113

0086fd78 77e8a384 RPCRT4!RPC_INTERFACE::DispatchToStub+0x84

0086fdb8 77e8a3c5 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xc0

0086fdf8 77e7bcc1 RPCRT4!LRPC_SCALL::DealWithRequestMessage+0x2cd

0086fe1c 77e7bc05 RPCRT4!LRPC_ADDRESS::DealWithLRPCRequest+0x16d

0:002> du poi(esp+4)

59491668"wbemcore.dll"

Surprisingly, I found that the requested library could not be found using the standard search strategy.

0:002> !gle

LastErrorValue: (Win32) 0x7e (126) - The specified module could not be found.

LastStatusValue: (NTSTATUS) 0xc0000135 - {Unable To Locate Component}This application has failed to start because %hs was not found. Re-installing the application may fix this problem.

I then decided to enable boot logging within the Process Monitor tool to catch a glimpse into why the load for wbemcore.dll was failing.Here’s an excerpt from the Process Monitor log:

8:18:11.1652951 PMsvchost2.exe1628QueryOpenC:\WINDOWS\system32\%SystemRoot%\system32\wbemcore.dllPATH NOT FOUND

8:18:11.1653627 PMsvchost2.exe1628QueryOpenC:\WINDOWS\system32\%SystemRoot%\wbemcore.dllPATH NOT FOUND

8:18:11.1654161 PMsvchost2.exe1628QueryOpenC:\WINDOWS\system32\%SystemRoot%\System32\Wbem\wbemcore.dllPATH NOT FOUND

At this point, it became clear to me that something must be wrong with the PATH environment variable configuration because the SystemRoot environment variable was not being properly expanded.Sure enough, the registry showed that the Path value was of type REG_SZ rather than REG_EXPAND_SZ.

Once I saved off the data for the Path value and recreated it specifying the proper type(REG_EXPAND_SZ), the issue was resolved!

Sidenote on the Debugging Strategy

I chose to debug the services by attaching ntsd.exe and redirecting the session to the kernel debugger.Alternatively, I could have chosen to use a remote debugging session since it’s much more natural than redirecting to the kernel debugger, in my opinion.However, this issue was further complicated by the fact that introducing a debugger in the mix had potential to change the timing enough such that I struggled to reproduce the issue using a remote session.The reason for this is that if we slow down the 1-vmsrvc execution enough, the winmgmt service itself will attempt to load wbemcore.dll, albeit through an activation request where an absolute path is specified, thereby avoiding application of a search strategy, which will succeed.Thus, when 1-vmsrvc attempts to activate CLSID_WbemLevel1Login, it will now succeed because wbemcore.dll is already loaded and the Auto Publish feature will now be ‘Enabled’!

Configuration

I leveraged the Image File Execution Options key, creating a new entry for vmsrvc.exe and configuring the Debugger value with the following command line:

I also isolated the winmgmt service into its own svchost.exe, copied %systemroot%\system32\svchost.exe to %systemroot%\system32\svchost2.exe, and then created a new key for svchost2.exe with the following command line:

Although redirecting ntsd.exe to the kernel debugger was cumbersome from a usability perspective, I found the ability to debug both processes from a central facility very appealing and worthwhile.

Conclusion

For issues like these that ultimately resolve to a misconfiguration of the OS, I can’t help but think how unfortunate it was that I didn’t stumble onto root cause sooner via the routine task of launching some executable from a command shell.Surely this would have been a red flag and could have saved me a lot of time debugging!

While this was a rather extreme example of how a misconfiguration of the OS can affect other seemingly unrelated parts such as the Auto Publish feature of XP Mode, the take-away here is when you detect that the Auto Publish feature is ‘Not Available’, you should begin with standard WMI troubleshooting; and, as we’ve just seen here, a quick sanity check of the environment might not be a bad idea either! :)

I recently worked on an interesting system hang issue with a deadlocked work queue and wanted to share some information about how we resolved the issue.

In this example, we will demonstrate how a cascaded work item can deadlock a work queue. As you can see from the illustration, we have three worker threads. Most work queues have 16 or more threads servicing them, but to simplify this example, three worker threads will be used. In the first illustration, all worker threads are waiting for the queue’s event to be signaled (indicating that a work item was inserted into the queue).

Thread ~ac0.001 is a process or service related thread that is currently in a running state (indicated by the color blue). This thread needs some work to be accomplished in an arbitrary context. To do this, the thread constructs an IO_WORK_ITEM and specifies a callback routine to be performed by the worker thread(s).

Fig. 1 – All worker threads available

Once the required work items have been defined, our thread would call IoQueueWorkItem for each item. This would signal the queue’s notification KEVENT that there is work to be done. In figure 2, we can see the work items inserted into the queue (Fig. 2).

Fig. 2 – Thread inserts work items into the queue

At this point, the work queue’s event is signaled to wake up the worker(s). Once running, each of the threads removes a work item from the queue and runs the function associated with the work item.

Since each work item has a completion event, this event will be signaled when the worker is finished processing the item. Because of this, thread ~ac0.001 would normally call KeWaitForMultipleObjects to wait for the work item completion events to be signaled (Fig. 3).

Fig. 3 – Requesting thread waits for completion

Upon completion of a work item, each of the worker threads would normally signal the corresponding work item event(s) and then go back into a wait state for new work items and thread ~ac0.001 would continue execution.

Now let’s look at what happens when workers need to enqueue one or more additional work items to the same queue. This is not a problem as long as there is at least one available worker thread to service these cascaded work items. The available worker(s) would simply handle these and eventually this situation would resolve itself (Fig. 4).

Fig. 4 – Work items 4, 5, and 6 will be handled by Thread ~008.004

If there are no more available workers, work items 4, 5, and 6 will never be handled. This results in an unresolvable situation (Fig. 5). No more items can be removed from the queue.

Fig. 5 – No available worker threads to handle items 4, 5, and 6

The result is a deadlock on the queue (Fig. 6). To avoid this type of bug, a developer must take cascaded work items into consideration. It is not ideal for a worker routine to enqueue work to its own queue, but it is a common practice in the software industry. To avoid deadlocks, a developer may want to consider using a separate work queue for these cascaded items. If this is not possible, the developer should ensure that there are sufficient worker threads to handle any reasonable load scenario.

Fig. 6 – Deadlocked work queue

In summary, exercise caution when writing your worker routines and be sure to do plenty of stress testing to ensure your driver’s design is feasible in real world scenarios.

Example

A few weeks ago, I received a memory dump from a customer who was reporting a daily system hang.

One of the first things I do with a hang dump is to do a quick assessment with !VM, !Locks, and !exqueue 6. If there's something obvious going on, those three debugger commands will usually get things moving in the right direction.

In this case, the output of !exqueue 6 caught my eye. In the Critical Work Queue, all critical system worker threads were blocking on a wait operation issued by backyard_driver. It had called KeWaitForSingleObject and was waiting for notification events. I also noticed a large number of pending work items for backyard_driver and there were two different worker routines in these items.

PENDING: IoWorkItem backyard_driver+0x830768F2 (0001e228)

IoObject (8823e148) Context (b40e4384)

PENDING: IoWorkItem backyard_driver+0x830767da (0001e110)

IoObject (8823e148) Context (b40be9f0)

From looking at the thread stacks, I could tell that the waiting worker routines were from the worker routine at +0x1e110, and the pending items were for a worker routine starting at offset +0x1e228.

This means that one worker routine was calling IoQueueWorkItem for the same queue and then waiting for completion. This is fine as long as there are one or more worker threads available to process the cascaded work items, but in this case all threads were busy waiting for other work items in the same queue to complete. The queue was essentially deadlocked.

It turns out that this issue was triggered by a sudden surge of IO related to a PNP arrival. To prevent the issue from occurring, we added a registry value to allow for additional critical worker threads to be created.

HKLM\SYSTEM\CurrentControlSet\Control\SessionManager\Executive

AdditionalCriticalWorkerThreads : 0x14

For more information about this registry key, refer to http://msdn.microsoft.com/en-us/library/ee377058(BTS.10).aspx

Howdy fellow debuggers! This is Graham McIntyre, I am an Escalation Engineer in Platforms Global Escalation Services.We get questions from time to time from customers who experience a WHEA bugcheck 0x124, or system event, for help in interpreting the error record. The information applies to Windows Server 2008 / Vista SP1 and Windows 2008 R2 / Windows 7.

I thought I would go through an example error record, point out some commonly asked questions, and show you how to find specific information on the error.In many cases, the information is specific to a particular processor / hardware vendor, the customer will need to follow up with them. But, we can help to some extent to parse the data.

For an initial primer on WHEA and hardware error reporting, I suggest reading this whitepaper:

I’ll provide some further links to some specific WHEA information along the way.

Getting Started:

A WHEA bug check 0x124, WHEA_UNCORRECTABLE_ERROR, indicates that a fatal hardware error has occurred.The bug check parameters give you further information on the WHEA error record generated.

In this example case, the first parameter was 0 so this indicates that this is a Machine Check Exception (MCE).An MCE is generated by certain classes of processors, such as Intel and AMD 64-bit processors.

As you can see from the output, a WHEA error record is made of several sections.Each section is actually a sub-section of the one above it. The sections go from most generic, to most specific, based on the exact type of error which occurred.

CPER / WHEA record – this is defined in Appendix N of the UEFI spec version 2.2 (these can be obtained from www.uefi.org)

The format of most of the sections is defined in the UEFI Spec version 2.2 as part of the Common Platform Error Record (CPER) definition.The last section decribes a Machine Check Archtecture (MCA) which is defined by the processor manufacturer.In this case, it is an Intel processor

The CPER record header is a WHEA_ERROR_PACKET_V2, and describes the severity and type of error. In this case it is a fatal Machine Check Exception (MCE)

Section 0 is a Generic Processor error. This error record section contains processor error data that is not specific to a particular processor architecture. The data that is contained in this section is described by the WHEA_PROCESSOR_GENERIC_ERROR_SECTION structure.

Section 1 is an x86/x64 Processor Error. This error record section contains processor error data that is specific to the x86 or x64 processor architecture. The data that is contained in this section is described by the WHEA_XPF_PROCESSOR_ERROR_SECTION structure.

Section 2 is of type WHEA_XPF_MCA_SECTION and contains the machine check and other machine-specific register information. The actual structure which holds the MCA data is a Microsoft specific extension of the CPER specification. We build this record by reading the Machine Specific Registers (MSRs) which are processor specific, and filling in the fields. These (and many of the above) are defined in the header file cper.h in the SDK.

Some of the questions which I was asked about this record, and their answers:

1.Why is the processor number (31) listed in the MCA record (Section 2) different than the processor id / APIC ID (37) in sections 0 and 1?

The answer to this one is that the numbers have different meanings, and different sources.The one in sections 0 and 1 is the initial APIC ID of the CPU which reported the machine check.The APIC ID for a logical CPU is set by the hardware on boot.The processor number in Section 2 is the logical processor number (the value returned from KeGetCurrentProcessorNumberEx) of the processor which is creating the WHEA error record. This may or may not be the same processor which reported the machine check error, depending on the IRQL which the processor generating the error was running.If the IRQL was < DISPATCH_LEVEL, then it is scheduled to run on the reporting processor.Otherwise, it may be logged on a different processor.

How do you map APIC IDs to logical IDs? One way is using the !smt debugger extension.This shows the APIC IDs and logical CPU number for all CPUs.

Table B-1 in Appendix B of the Intel guide says that this Family and Model is a “Intel Xeon Processor 7500 Series”

b. What is the MCA Error code?

In order to find this out, we need to parse the MCi_STATUS structure.The ‘i’ is used in the Intel guides as a placeholder for the bank number.An error bank is a processor specific set of MSRs.Some banks are publically documented in what the type of error represents, and some are not.If the bank is not documented, then you will need to check with the processor manufacturer.

Now that we know the processor family and model, we can look up the meaning of specific bank of registers.These are listed in this form: MSR_MCi_STATUS.So since we know the bank number is 5, we can find the meaning of MSR_MC5_STATUS.Here’s what the Intel guide shows:

This is why the error shows as “Internal Unclassified”.Since this is not a publicly documented code, the next step would be to contact Intel for further information.But, at least now you have verified the information and will have good data to send to the hardware manufacturer.In other cases, the bank and MCA code may be more clearly documented and further action could be taken.

Further Reading:

There is more information regarding WHEA on MSDN and in several WinHEC conference presentations on the Microsoft site:

Hi All, my name is Ron Riddle and I’m an Escalation Engineer on the core Windows team.I worked an issue recently wherein a svchost.exe was crashing due to heap corruption; so, after enabling Page Heap and breaking out the services as needed, I received a user-mode dump that would show me the culprit.I was expecting to find a legitimate bug either in our code or a third-party module; but, much to my surprise, I found that malware had caused a buffer overrun and the subsequent crash.With that, I would like to share the simple approach I took in identifying the malware within the dump file.

1. I start by dumping out the offending call stack.Notice that the debugger wasn’t able to map the code addresses to a loaded or unloaded module.

0:003> kbn

# ChildEBP RetAddr Args to Child

WARNING: Frame IP not in any known module. Following frames may be wrong.

2. Next, I try to learn more about the mystery address, such as what larger allocation it was a part of.

0:003> !address 0x2b685b0

Usage:<unclassified>

Allocation Base:02b60000

Base Address:02b61000

End Address:02b81000

Region Size:00020000

Type:00020000MEM_PRIVATE

State:00001000MEM_COMMIT

Protect:00000040PAGE_EXECUTE_READWRITE

3. By now, I am suspicious of a rogue module, so I proceed in searching the aforementioned address range for a DOS Signature(i.e. 0x5A4D or “MZ”) that I know any Portable Executable file must contain.I start with the Base Address from the above output and use the Region Size to specify my range.

4. Now that I have some hits, I’ll start with the first one and verify whether it’s a valid module.Bingo!

0:003> !dh -a 02b615d8

File Type: DLL

FILE HEADER VALUES

14C machine (i386)

5 number of sections

37304740 time date stamp Wed May 05 08:27:28 1999

0 file pointer to symbol table

0 number of symbols

E0 size of optional header

2102 characteristics

Executable

32 bit word machine

DLL

OPTIONAL HEADER VALUES

10B magic #

7.00 linker version

600 size of code

600 size of initialized data

0 size of uninitialized data

10B0 address of entry point

1000 base of code

----- new -----

10000000 image base

1000 section alignment

200 file alignment

1 subsystem (Native)

4.00 operating system version

0.00 image version

4.00 subsystem version

6000 size of image

400 size of headers

41AE checksum

00100000 size of stack reserve

00001000 size of stack commit

00100000 size of heap reserve

00001000 size of heap commit

0DLL characteristics

0 [0] address [size] of Export Directory

4000 [28] address [size] of Import Directory

0 [0] address [size] of Resource Directory

0 [0] address [size] of Exception Directory

0 [0] address [size] of Security Directory

5000 [4C] address [size] of Base Relocation Directory

0 [0] address [size] of Debug Directory

0 [0] address [size] of Description Directory

0 [0] address [size] of Special Directory

0 [0] address [size] of Thread Storage Directory

0 [0] address [size] of Load Configuration Directory

0 [0] address [size] of Bound Import Directory

2000 [44] address [size] of Import Address Table Directory

0 [0] address [size] of Delay Import Directory

0 [0] address [size] of COR20 Header Directory

0 [0] address [size] of Reserved Directory

SECTION HEADER #1

.text name

3CC virtual size

1000 virtual address

400 size of raw data

400 file pointer to raw data

0 file pointer to relocation table

0 file pointer to line numbers

0 number of relocations

0 number of line numbers

68000020 flags

Code

Not Paged

(no align specified)

Execute Read

SECTION HEADER #2

.rdata name

68 virtual size

2000 virtual address

200 size of raw data

800 file pointer to raw data

0 file pointer to relocation table

0 file pointer to line numbers

0 number of relocations

0 number of line numbers

48000040 flags

Initialized Data

Not Paged

(no align specified)

Read Only

SECTION HEADER #3

.data name

56 virtual size

3000 virtual address

200 size of raw data

A00 file pointer to raw data

0 file pointer to relocation table

0 file pointer to line numbers

0 number of relocations

0 number of line numbers

C8000040 flags

Initialized Data

Not Paged

(no align specified)

Read Write

SECTION HEADER #4

INIT name

1D4 virtual size

4000 virtual address

200 size of raw data

C00 file pointer to raw data

0 file pointer to relocation table

0 file pointer to line numbers

0 number of relocations

0 number of line numbers

E2000020 flags

Code

Discardable

(no align specified)

Execute Read Write

SECTION HEADER #5

.reloc name

82 virtual size

5000 virtual address

200 size of raw data

E00 file pointer to raw data

0 file pointer to relocation table

0 file pointer to line numbers

0 number of relocations

0 number of line numbers

42000040 flags

Initialized Data

Discardable

(no align specified)

Read Only

5. Because I’m not sure which sections might contain identifying characteristics, I decide to go spelunking through all the sections (except for the relocation section) looking for said characteristics that might help me to identify the rogue module.I start with the relative virtual address of the .text section @ 0x1000 and continue through the INIT section @ 0x4000.

0:003> dc 02b615d8+0x1000 l4000/4

…

02b63c5800000065 646c6977 73737265 72756365e...wilderssecur

02b63c6800797469 65726874 78657461 74726570ity.threatexpert

02b63c7800000000 74736163 6f63656c 00007370....castlecops..

02b63c886d617073 73756168 00000000 65737063spamhaus....cpse

02b63c9865727563 00000000 61637261 00746962cure....arcabit.

02b63ca869736d65 74666f73 00000000 626e7573emsisoft....sunb

02b63cb800746c65 75636573 6f636572 7475706delt.securecomput

02b63cc800676e69 69736972 0000676e 76657270ing.rising..prev

02b63cd800000078 6f746370 00736c6f 6d726f6ex...pctools.norm

02b63ce800006e61 6f63376b 7475706d 00676e69an..k7computing.

02b63cf872616b69 00007375 72756168 00000069ikarus..hauri...

02b63d086b636168 74666f73 00000000 74616467hacksoft....gdat

02b63d1800000061 74726f66 74656e69 00000000a...fortinet....

02b63d2864697765 0000006f 6d616c63 00007661ewido...clamav..

02b63d386f6d6f63 00006f64 63697571 6165686bcomodo..quickhea

02b63d480000006c 72697661 00000061 73617661l...avira...avas

02b63d5800000074 66617365 00000065 6c6e6861t...esafe...ahnl

02b63d6800006261 746e6563 636c6172 616d6d6fab..centralcomma

02b63d780000646e 65777264 00000062 73697267nd..drweb...gris

02b63d880074666f 74657365 00000000 33646f6eoft.eset....nod3

02b63d9800000032 72702d66 0000746f 74746f6a2...f-prot..jott

02b63da800000069 7073616b 6b737265 00000079i...kaspersky...

02b63db865732d66 65727563 00000000 706d6f63f-secure....comp

02b63dc872657475 6f737361 74616963 00007365uterassociates..

02b63dd87774656e 616b726f 636f7373 65746169networkassociate

02b63de800000073 75727465 00007473 646e6170s...etrust..pand

02b63df800000061 68706f73 0000736f 6e657274a...sophos..tren

02b63e0863696d64 00006f72 6661636d 00006565dmicro..mcafee..

02b63e1874726f6e 00006e6f 616d7973 6365746enorton..symantec

02b63e2800000000 7263696d 666f736f 00000074....microsoft...

02b63e3865666564 7265646e 00000000 746f6f72defender....root

02b63e480074696b 776c616d 00657261 77797073kit.malware.spyw

02b63e5800657261 75726976 00000073 304ce942are.virus...B.L0

…

02b6434854464f53 45524157 63694d5c 6f736f72SOFTWARE\Microso

02b64358575c7466 6f646e69 435c7377 65727275ft\Windows\Curre

02b643686556746e 6f697372 78655c6e 726f6c70ntVersion\explor

02b64378415c7265 6e617664 5c646563 646c6f46er\Advanced\Fold

02b64388485c7265 65646469 48535c6e 4c41574fer\Hidden\SHOWAL

02b643980000004c 63656843 5664656b 65756c61L...CheckedValue

…

02b63ee8ffffffff 02b6a44f 02b6a453 70747468....O...S...http

02b63ef8772f2f3a 672e7777 796d7465 6f2e7069://www.getmyip.o

02b63f0800006772 70747468 772f2f3a 772e7777rg..http://www.w

02b63f1873746168 7069796d 72646461 2e737365hatsmyipaddress.

02b63f28006d6f63 70747468 772f2f3a 772e7777com.http://www.w

02b63f3869746168 69796d73 726f2e70 00000067hatismyip.org...

02b63f4870747468 632f2f3a 6b636568 642e7069http://checkip.d

02b63f586e646e79 726f2e73 00000067 61207069yndns.org...ip a

02b63f6865726464 00007373 ffffffff 02b6a55eddress......^...

…

02b6485800000020 74666f53 65726177 63694d5c...Software\Mic

02b648686f736f72 575c7466 6f646e69 435c7377rosoft\Windows\C

02b6487865727275 6556746e 6f697372 75525c6eurrentVersion\Ru

02b648880000006e 646e7572 32336c6c 6578652en...rundll32.exe

02b6489873252220 73252c22 00000000 0065006e"%s",%s....n.e.

02b648a800730074 00630076 00000073 00000020t.s.v.c.s... ...

6. The list of anti-malware software vendors was a dead give-away that I was dealing with malware.Finally, I conducted a Bing search using various artifacts from the preceding spew.In the end, I was able to confirm that the rogue module was, in fact, the Conficker worm by simply running a full scan of the system using a signature-based scanner.

I hope this walk-through provided you with techniques that you can leverage to identify rogue modules within your dump files, should that become necessary.Until next time, happy bug-hunting and watch out for the worms!