Real-time Windows performance monitoring

Performance monitoring is an important part of the Windows administrator's toolkit. Windows generates a lot of data about a system's health. Proper analysis of that data can make the difference between a healthy, well functioning system, and one that suffers many bouts of downtime.

Splunk supports the monitoring of all Windows performance counters available to the system in real time, and includes support for both local and remote collection of performance data.

Splunk's performance monitoring utility gives you the abilities of Performance Monitor in a web or command-line interface. Splunk uses the Performance Data Helper (PDH) API for performance counter queries on local machines.

Both full instances of Splunk and universal forwarders support local collection of performance metrics. Remote performance monitoring is available through WMI (Windows Management Instrumentation) and requires that Splunk runs as a user with appropriate Active Directory credentials.

The performance monitor input runs as a process called splunk-perfmon.exe. This process will run once for every input defined, at the interval specified in the input. You can configure performance monitoring using Splunk Web, or either perfmon.conf (for getting local performance data) or wmi.conf (for getting performance data from a remote machine).

Security and remote access considerations

Splunk gets data from remote machines using either WMI or a forwarder. Splunk recommends using a universal forwarder to send performance data from remote machines to an indexer. Review "Introducing the universal forwarder" in the Distributed Deployment Manual for information about how to install, configure and use the forwarder to collect performance metrics.

If you choose to install forwarders on your remote machines to collect performance data, then you can install the forwarder as the Local System user on those machines. The Local System user has access to all data on the local machine, but not to remote machines.

If you want Splunk to use WMI to get performance data from remote machines, then you must ensure that your network and Splunk instances are properly configured. You cannot install Splunk as the Local System user, and the user you install with determines the set of performance metrics Splunk will see. Review "Security and remote access considerations" in the "Monitor WMI Data" topic in this manual for additional information on the requirements you must satisfy in order for Splunk to collect remote data properly using WMI.

After you install Splunk with a valid user, add that user to the following groups before enabling local performance monitor inputs:

Performance Monitor Users (domain group)

Performance Log Users (domain group)

Enable local Windows performance monitoring

You can configure local performance monitoring either in Splunk Web, or by using configuration files.

Splunk Web is the preferred way to add performance monitoring data inputs. This is because you can make typos when using configuration files, and it's important to specify performance monitor objects exactly as they are defined in the Performance Monitor API. See "Important information about specifying performance monitor objects in perfmon.conf" below for a full explanation.

Configure local Windows performance monitoring with Splunk Web

1. Click Manager in the upper right-hand corner of Splunk Web.

2. Under Data, click Data Inputs.

3. Click Local performance monitoring.

4. Click New to add an input.

5. Enter a unique, memorable name for this input.

6. Under Available objects, choose the performance object whose counters you wish to display.

Splunk loads the available performance counters for the selected object.

Note: You can only add one performance object per data input. This is due to how Microsoft handles performance monitor objects. Many objects enumerate classes that describe themselves dynamically upon selection. This can lead to confusion as to which performance counters and instances belong to which object, as defined in the input. If you need to monitor multiple objects, create additional data inputs for each object.

7. Under Counters, choose the counters in the Available counters list box that you want Splunk to monitor by clicking once on them.

The selected counter moves from the Available counters list box to the Selected counters list box.

8. Under Instances, select the instances you want Splunk to monitor by clicking on those instances in the Available instances list.

The selected instance moves from the Available instances list box to the Selected instances list box.

Note: The "_Total" instance is a special instance, and is present for many types of performance counters. This instance is defined as the average of any associated instances under the same counter. Data collected for this instance can be significantly different than for individual instances under the same counter.

For example, when monitoring performance data for the "Disk Bytes/Sec" performance counter under the "PhysicalDisk" object on a system with two disks installed, the available instances displayed include one for each physical disk - "0 C:" and "1 D:" - as well as the "_Total" instance. In this case, the "_Total" instance is the average of the two physical disk instances.

Performance monitoring configurations are controlled by perfmon.conf. To set up performance monitoring using configuration files, create and/or edit perfmon.conf in %SPLUNK_HOME%\etc\system\local. If you haven't worked with Splunk's configuration files before, be sure to read "About configuration files" before you begin.

perfmon.conf contains one stanza, where you specify:

Attribute

Required?

Description

interval

Yes

How often, in seconds, to poll for new data. If this attribute is not present and defined, the input will not run, as there is no default.

object

Yes

The performance object that you wish to capture. If this attribute is not present and defined, the input will not run, as there is no default.

counters

Yes

One or more valid performance counters that are associated with the object specified in object. Multiple counters are separated by semicolons. You can also use an asterisk (*) to specify all available counters under a given object. If this attribute is not present and defined, the input will not run, as there is no default.

instances

Yes, at least one

One or more valid instances associated with the performance counter specified in counters. Multiple instances are separated by semicolons. You can specify all instances by using an asterisk (*).

index

No

The desired index to route performance counter data to. If not present, the default index is used.

disabled

No

Whether or not to gather the performance data defined in this input. Set to 1 to disable this stanza, and 0 to enable it. If not present, it defaults to 0 (enabled).

The following example of perfmon.conf collects performance data from the local disk on the system and places it into the 'perfmon' index:

Important information about specifying performance monitor objects in perfmon.conf

When specifying values for the object, counters and instances attributes in perfmon.conf stanzas, be sure that those values exactly match those defined in the Performance Monitor API, including case, or the input will return incorrect data, or no data at all. If Splunk is unable to match a performance object, counter or instance value that you've specified in perfmon.conf, it will log that failure to splunkd.log. For example:

The best way to ensure that you specify the correct objects, counters, and instances is to use Splunk Web to add performance monitor data inputs.

Enable remote Windows performance monitoring over WMI

You can configure remote performance monitoring either in Splunk Web or by using configuration files.

When collecting performance metrics over WMI, you must configure Splunk to run as an AD user with appropriate access for remote collection of performance metrics. You must do this before attempting to collect those metrics. Both the machine running Splunk and the machine(s) Splunk collects performance data from must reside in the same AD domain or forest.

Note: WMI self-throttles by design to prevent denial of service attacks. Splunk will also throttle WMI calls it makes as an additional precautionary measure if these calls return an error. Depending on the size, configuration, and security profile of your network, installing a local forwarder on the system from which you want to collect performance metrics might be a better choice. Consult "Considerations for deciding how to monitor remote Windows data" in this manual for additional information.

Important information regarding WMI-based performance metrics

When gathering remote performance metrics through WMI, you might notice that some metrics return zero values, or values that are not in line with values returned by Performance Monitor. This is because of a limitation in the implementation of WMI for performance monitor counters, and is not an issue with Splunk or how it retrieves WMI-based data..

The data structures within these classes are defined as either 32- or 64-bit unsigned integers, depending on the version of Windows you are running. Performance Monitor objects, meanwhile, are defined as floating-point variables. This means that you might see WMI-based metrics that appear anomalous, due to rounding factors.

For example, if you collect data on the "Average Disk Queue Length" Performance Monitor counter at the same time you collect the Win32_PerfFormattedData_PerfDisk_PhysicalDisk\AvgDiskQueueLength metric through WMI, the WMI based metric might return zero values even though the Performance Monitor metric is returning values greater than zero (but less than 0.5). This is because WMI rounds the value down before displaying it.

If you require additional granularity in your performance metrics, it's better to configure the performance monitoring inputs on a universal forwarder on each machine from which you wish to collect performance data. You can then forward that data to an indexer. Data retrieved using this method is more reliable than data gathered remotely using WMI-based inputs.

Configure remote Windows performance monitoring with Splunk Web

1. Click Manager in the upper right-hand corner of Splunk Web.

2. Under Data, click Data Inputs.

3. Click Remote Performance monitoring.

4. Click New to add an input.

5. Enter a unique name for this collection.

6. Under Select target host, enter the name of a valid Windows host to query performance monitor objects from, then click "Query..."

Splunk connects to the host and gets the available performance objects.

7. In the "Available objects" drop-down, select the performance object whose counters you wish to display.

Splunk loads the available performance counters for the selected object.

Note: You can only add one performance object per data input. This is due to how Microsoft handles performance monitor objects. Many objects enumerate classes that describe themselves dynamically upon selection. This can lead to confusion as to which performance counters and instances belong to which object, as defined in the input. If you need to monitor multiple objects, create additional data inputs for each object.

8. Under Counters, choose the counters in the "Available counters" list box that you want Splunk to monitor by clicking once on them.

The selected counter moves from the "Available counters" list box to the "Selected counters" list box.

9. Next, under Instances, select the instances you want Splunk to monitor by clicking on those instances in the Available instances list.

The selected instance moves from the "Available instances" list box to the "Selected instances" list box.

Note: The "_Total" instance is a special instance, and is present for many types of performance counters. This instance is defined as the average of any associated instances under the same counter. Data collected for this instance can be - and oftentimes is - significantly different than for individual instances under the same counter.

For example, when monitoring performance data for the "Disk Bytes/Sec" performance counter under the "PhysicalDisk" object on a system with two disks installed, the available instances displayed include one for each physical disk - "0 C:" and "1 D:" - as well as the "_Total" instance. In this case, the "_Total" instance is the average of the two physical disk instances.

10. You can optionally tell Splunk to collect the same set of metrics from additional hosts by specifying those hosts, separated by commas, in the field provided.

11. Specify an interval, in seconds, between polls.

12. Optionally, choose the destination index for this collection.

By default, the "default" index is selected.

13. Click Save.

The input is added and enabled.

Note: Win32_PerfFormattedData_* classes will not show up as available objects. If you wish to monitor Win32_PerfFormattedData_* it needs to be added directly in wmi.conf

Remote performance monitoring configurations are controlled by wmi.conf. To set up remote performance monitoring using configuration files, create and/or edit wmi.conf in %SPLUNK_HOME%\etc\system\local. If you haven't worked with Splunk's configuration files before, be sure to read "About configuration files" before you begin.

Caution: Splunk strongly recommends that you use Splunk Web to create remote performance monitor inputs. This is because the names of performance monitor objects, counters, and instances must exactly match what is defined in the Performance Monitor API, including case. Splunk Web uses WMI to get the properly-formatted names, eliminating this problem.

wmi.conf contains one stanza for each remote performance monitor object that you wish to monitor. In each stanza, you specify:

Global settings

Attribute

Required?

Description

Default

initial_backoff

No

How long, in seconds, to wait before retrying a connection to a WMI provider when an error occurs. If Splunk continues to have problems connecting to the provider, then it will double the wait time between connection attempts until either it can connect, or until the wait time is greater than or equal to the integer specified in max_backoff.

5

max_backoff

No

The maximum amount of time, in seconds to attempt to reconnect to a WMI provider.

20

max_retries_at_max_backoff

No

How many times, after Splunk has reached max_backoff seconds between reconnection attempts with a WMI provider, to continue to attempt to reconnect to that provider.

2

checkpoint_sync_interval

No

How long, in seconds, to wait for state data to be flushed to disk.

2

Input-specific settings

Attribute

Required?

Description

Default

interval

Yes

How often, in seconds, to poll for new data. If this attribute is not present, the input will not run, as there is no default.

N/A

server

No

One or more valid servers against which you wish to monitor performance. Multiple entries are separated by commas.

The local machine

event_log_file

No

The names of one or more Windows event log channels to poll. This attribute tells Splunk that the incoming data is in event log format.

Note: Do not use the event_log_file attribute in a stanza that already contains the wql attribute.

N/A

wql

No

A valid Windows Query Language (WQL) statement that specifies the performance object(s), counter(s), and instance(s) you wish to poll remotely. This attribute tells Splunk to expect data from a WMI provider.

Note: Do not use the wql attribute in a stanza that already contains the event_log_file attribute.

N/A

namespace

No

The namespace in which the WMI provider you want to query resides. The value for this attribute can be either relative (Root\CIMV2) or absolute (\\SERVER\Root\CIMV2), but must be relative if you specify the server attribute.

Note: Only use the namespace attribute in a stanza that contains the wql attribute.

Root\CIMV2

index

No

The desired index to route performance counter data to.

default

current_only

No

The characteristics and interaction of WMI-based event collections.

if wql is defined, this attribute tells Splunk whether or not it should expect an event notification query. Set to 1 to tell Splunk to expect an event notification query, and 0 to tell it expect a standard query. See below for additional requirements on WQL and event notification queries.

if event_log_file is defined, tells Splunk whether or not to only capture events that occur when Splunk is running. Set to 1 to tell Splunk to only capture events that occur when Splunk is running, and 0 to gather events from the last checkpoint or, if no checkpoint exists, the oldest events available.

N/A

disabled

No

Tells Splunk whether or not to gather the performance data defined in this input. Set this to 1 to disable performance monitoring for this stanza, and 0 to enable it.

0

The following example of wmi.conf gathers local disk and memory performance metrics and places them into the 'wmi_perfmon' index:

Additional information on WQL query statements

When building WQL queries, make sure that the queries are structurally and syntactically correct. If you don't, you might get undesirable results, or no results at all. In particular, when writing event notification queries (by specifying current_only=1 in the stanza in which a WQL query resides), your WQL statement must contain one of the clauses that specify such a query (WITHIN, GROUP, and/or HAVING). Review this MSDN article on Querying with WQL for additional information.

Splunk Web eliminates problems with WQL syntax by generating the appropriate WQL queries when it is used to create performance monitor inputs.

Increased memory usage during collection of performance metrics

When collecting data on some performance objects, such as the "Thread" object and its associated counters, you might notice increased memory usage in Splunk. This is normal, as certain performance objects consume more memory than others during the collection process.

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »