The User-Mode WMI_LOGGER_CONTEXT

Right from its introduction for Windows 2000, before it even had its current
name, Event Tracing for Windows (ETW) has parallel implementations in kernel and
user modes. The practical difference is not just with where the code is but with
where the trace buffers are managed. One aim of ETW is that events are written quickly
into trace buffers which are then serviced independently from a logger thread, e.g.,
to flush full buffers to an Event Trace Log (ETL) file. An event provider that executes
as a user-mode process can choose the user-mode ETW implementation so that the trace
buffers are managed in user mode and writing an event (to a trace buffer) avoids
the delay of going to and from kernel mode. The user-mode WMI_LOGGER_CONTEXT
structure is what NTDLL—and before it, ADVAPI32—keeps for managing an event logger,
known more formally as an event tracing session, that has user-mode trace buffers.

Documentation Status

Microsoft seems never to have disclosed even a name for this structure. It is
here named WMI_LOGGER_CONTEXT for its correspondence
with a kernel-mode structure for which Microsoft’s names (not only of the structure
but also of its members) are known from public symbol files, starting with Windows
XP SP3 and Windows Server 2003 SP2. Work backwards through the kernel binaries to
the kernel-mode structure’s origin in Windows 2000, and it is abundantly plain that
whatever the user-mode structure that’s described in this article has ever been
named, it and the kernel-mode
WMI_LOGGER_CONTEXT started with a shared definition: what members they
have in common are laid out in the same order. Though the layouts have since diverged,
such that they seem unlikely to be differentiated just by preprocessor directives
for conditional compilation, they surely still are developed each with one eye on
the other.

The wonder is that Microsoft’s names and types for this structure haven’t slipped
out long before. After all, its kernel-mode correspondent seems to be no more secret
than many another internal detail, if only to be disclosed as an aid to kernel-mode
debugging. Perhaps I’ve just missed the publication or not thought what obvious
names to search for.

Or not. User-mode event tracing has much about it that Microsoft demonstrably
doesn’t document. This applies especially to functionality that is implemented for
user-mode loggers but not (yet) for tracing through the kernel—and happens often
enough to suggest a pattern of meaning very much not to document it. Consider, for
instance, that Windows 8 and higher provide for user-mode loggers to compress what
they flush to ETL files, but this useful trick isn't implemented for kernel-mode
tracing until later releases of Windows 10. Any programmer or administrator who
has ever worried how big their user-mode tracing session’s ETL files may get will
plausibly think that the EVENT_TRACE_COMPRESSED_MODE
flag for the logging mode might be a helpful thing to know. Microsoft surely agrees,
else why does the Start-EtwTraceSession command in
PowerShell have a -Compress switch? Yet, if only according
to Google, today, 18th December 2018, the whole of Microsoft’s website has just
one page that mentions this flag that programmers would need to know for starting
an event tracing session that compresses its own ETL file—and this page is not programmer
documentation but PowerShell user documentation. For programmers,
EVENT_TRACE_COMPRESSED_MODE didn’t even get defined
in EVNTRACE.H until the Software Development Kit (SDK) for the 1607 edition of Windows
10 and it still isn’t listed among the
Logging Mode Constants. Such lack of disclosure is not credibly by accident
or oversight—except, perhaps, that one page slipped through.

Layout

Curiously, the user-mode WMI_LOGGER_CONTEXT has been
far more stable than its kernel-mode counterpart. It has grown, of course, and members
have come and gone, but there has been only one large-scale rearrangement (for version
6.1) and the move from ADVAPI32 to NTDLL in version 5.2 didn’t change the structure
at all. The following changes of size are known:

Versions

Size (x86)

Size (x64)

5.0

0xD0

5.1 to 5.2

0xD8

0x0120

6.0

0xF8

0x0150

6.1

0x0120

0x01A0

6.2

0x0160

0x0210

6.3 to 10.0

0x0170

0x0220

These sizes, and the offsets, types and names in the detailed layout below, come
from inspection of the binaries for ADVAPI32 (in its versions 5.0 and 5.1) and NTDLL
(in its versions 5.2 and higher), and comparison both with similar code in contemporaneous
versions of the kernel and with the offsets, types and names that are known for
the kernel-mode structure from public symbol files for the kernel. Where correspondence
seems close, it seems reasonable to assume that types and names are the same for
both the kernel-mode and user-mode structures. Such analysis is inevitably inexact
and prone to oversight. Editorial decisions for cases where correspondence is not
so close are explained after the table.

As noted in the introduction, user-mode loggers can compress event data in version
6.2 and higher but there is no kernel-mode correspondent until the 1607 release
of Windows 10. The implementations are very different. The only known name that
seems safe to take as common to both implementations is that of the buffer that’s
used for the algorithm’s workspace.

A user-mode logger compresses successive trace buffers ever deeper into a compression
buffer that’s the size of two trace buffers. The first is flushed whenever it fills.
Any overflow is then moved down to become a partial trace buffer at the start of
the compression buffer. See that any one flush from the compression buffer is of
one trace buffer’s worth of data that has been compressed from potentially many
trace buffers. Version 6.2 counts them altogether as one buffer written or lost.
Version 6.3 tracks their number. If flushing fails, version 6.2 does not reset its
compression buffer. In later versions, all whole trace buffers that were in the
compression buffer are explicitly lost. If the compression buffer began with the
overflow data of a partly flushed trace buffer, then this overflow is retained in
the compression buffer so that any subsequent flush that does succeed will complete
the partial trace buffer that is already in the ETL file.

Offset (x86)

Offset (x64)

Definition

Versions

0x013C (6.2);
0x014C

0x01C8 (6.2);
0x01D8

LIST_ENTRY ProviderBinaryList;

6.2 and higher

0x0144 (6.2);
0x0154

0x01D8 (6.2);
0x01E8

LIST_ENTRY WinRtProviderBinaryList;

6.2 and higher

It seems strange now, but not until version 6.2 do ETL files record which providers
are enabled for the session.

Offset (x86)

Offset (x64)

Definition

Versions

0x014C (6.2);
0x015C

0x01F8

unknown dword

6.2 and higher

0x0150 (6.2);
0x0160

0x0200

unknown pointer to array of pointers

6.2 and higher

0x0154 (6.2);
0x0164

0x0208

unknown pointer to array of structures

6.2 and higher

0x0158 (6.2);
0x0168

0x0200 (6.2);
0x0210

unknown pointer to processor stream index map

6.2 and higher

There is something to write here!

Offset (x86)

Offset (x64)

Definition

Versions

Remarks

0x011C (6.1);
0x015C (6.2);
0x016C

0x0198 (6.1);
0x0208 (6.2);
0x0218

WMI_BUFFER_HEADER *ProcessorBuffers [ANYSIZE_ARRAY];

6.1 and higher

last member in 6.1 and higher

Where the kernel-mode WMI_LOGGER_CONTEXT has objects
such as semaphores and events, and pointers to objects such as threads, a user-mode
structure can only have handles. Although the applicable members must change type,
they needn’t change names and it is assumed above that they don’t.

Each logger creates its own thread for flushing trace buffers. All versions keep
the thread ID and disclose it, when queried, by copying it to the
LoggerThreadId member of a WMI_LOGGER_INFORMATION.
It is here proposed that the thread ID as kept in the logger context likely has
the same name (and type). The early versions, implemented in ADVAPI32, also keep
a handle to the logger thread. This stops with version 5.2, but it is assumed above
that the LoggerThread remained defined (no reuse of
its space being known). Version 6.0 seems then to have removed it, only to have
version 6.1 go back to keeping the handle.

The unknown CONDITION_VARIABLE supports
EVENT_TRACE_BLOCKING_MODE. If tracing an event would
exhaust its processor’s current trace buffer but all other trace buffers are in
use (including because they are yet to be flushed) and no more can be created, then
the event would ordinarily be lost. Waiting for a trace buffer to become available
isn’t much of an option. After all, much of the point to tracing an event is that
it is done without disturbing whatever activity is being traced. For some loggers,
however, the intended activity may be relatively infrequent and take time anyway,
such that time spent waiting for a buffer to trace an event to is less a penalty
than would be the event’s loss. Blocking mode allows the wait. It’s another ETW
feature that’s particular to user-mode tracing sessions (and is not documented).

See that in all versions, a user-mode logger reserves address space sufficient
for whatever it ends up adopting as MaximumBuffers.
It immediately commits memory for whatever it ends up adopting as
MinimumBuffers, and thereafter commits more only when
more buffers actually are needed for whatever flow of events are received. For this
purpose of reserving and committing, each trace buffer is a whole number of pages,
rounded up from BufferSize bytes. Neither the page-aligned
buffer size nor the address of the reservation has any correspondent in the kernel-mode
structure.

Versions 6.1 and higher end the structure with a variable-size array of pointers,
one per processor, each to the trace buffer that is currently in use for that processor.
This too has no correspondence with the kernel-mode structure. It does, however,
supersede what had been a pointer to just such an array, and in such a way that
keeping the name would require no change to the source code. The pointer’s name,
ProcessorBuffers, is known from correspondence with
the kernel-mode structure.

This page was created on 15th
December 2018 and was last
modified on 31st December 2018.