Common issues with Splunk and WMI

This topic discusses common issues encountered when getting WMI-based data into Splunk. It offers solutions for problems such as the following:

Splunk can't get data from remote machines.

Splunk can't get local data through WMI.

Splunk sometimes crashes when getting remote data.

Splunk connects to WMI differently depending on product version.

Splunk can't get data from remote machines

When Splunk can index events on the local machine, but can't get data from remote machines using WMI, authentication or network connectivity is often the reason. Splunk requires a user account with valid credentials for the Active Directory (AD) domain or forest in which it's installed in order to collect data remotely. It also requires a clear network path to the machine from which it gets data, unblocked by firewalls on either the source or target machines.

Determine that Splunk has been installed as a domain user

The first thing to do is to make sure that Splunk is installed as a domain user. If this requirement isn't met, Splunk won't be able to get data remotely even if the network is functioning.

1. Open a command prompt.

2. Run the SC command to query the Services Command Manager about the splunkd and splunkweb services.

The SERVICE_START_NAME field tells you the user that Splunk is configured to run as. If this field shows LocalSystem, then Splunk is not configured to run as a domain user. Uninstall Splunk, then reinstall it and make sure to specify "Other user" during the setup process.

Note: You can also determine which user Splunk is configured to run as by using the Services control panel.

Review the splunkd.log file

If Splunk is correctly configured as a domain user, the next step is to investigate why Splunk is having problems connecting to WMI providers.

Open the %SPLUNK_HOME%\var\log\splunk\splunkd.log file and search for wmi.

When Splunk encounters an error attempting to connect to a WMI provider, it logs errors in splunkd.log as follows:

The following table shows the most common errors encountered when connecting to WMI providers:

Error code

Description

80070005

Access is denied. (due to an incorrect login)

80041064

User credentials cannot be used for local connections.

800706BA

The RPC server is unavailable.

80041003

Access Denied. (due to explicit access restrictions)

If you see lines within the log file that contain HRESULT error then Splunk is unable to complete the WMI operation due to a network connectivity or authentication problem. You can use the WBEMTEST utility to corroborate what is shown in Splunk's log file.

Enable debug logging

You can get even more detailed information about what is causing the errors by enabling debug logging in Splunk's logging engine.

Note: After you have confirmed the cause of the error, be sure to turn debug logging off.

To enable debugging for WMI-based inputs, you must set two parameters:

1. Edit log.cfg in %SPLUNK_HOME\etc. Add the following parameter:

[splunkd]
category.ExecProcessor=DEBUG

2. Edit log-cmdline.cfg, also in %SPLUNK_HOME%\etc. Add the following parameter:

category.WMI=DEBUG

Note: You can place this attribute/value pair anywhere in the file, as long as it is on its own line. log-cmdline.cfg does not use stanzas.

3. Restart Splunk:

C:\Program Files\Splunk\bin> splunk restart

4. Once Splunk has restarted, let it run for a few minutes until you see debug log events coming into Splunk.

Note: You can search Splunk's logfiles within Splunk by supplying index="_internal" as part of your search string. Review "What Splunk logs about itself" in the Troubleshooting Manual for additional information.

5. In the Namespace field of the Connect window, type in the namespace of the server that is experiencing errors.

Note: You must type in the full path of the namespace. For example, if the server you are attempting to connect to is called ADLDBS01, you must type in \\ADLDBS01\root\cimv2 (including the backslashes).

6. Click Connect.

Note: You should be able to connect to the server without needing to supply credentials. If you are prompted for credentials, then the Splunk user is not correctly configured to access WMI.

7. Once you are connected to the server, set your WMI connection mode by selecting one of the radio buttons in Method Invocation Options the lower right corner of the WBEMTEST window:

For Splunk 3.4.9 and earlier, choose Asynchronous.

For versions of Splunk after 3.4.9, choose Semisynchronous.

8. Click "Query…"

The Query window appears.

9. In the Query window, type in a valid Windows Query Language (WQL) statement, such as the one supplied below, then click Apply.

Check Windows Firewall

If Windows Firewall (or any other firewall software) is running on either the source or target machine, Splunk might be blocked from getting data through WMI providers. Make sure that you explicitly allow WMI through on the firewalls on both machines. You can also disable Windows Firewall, but this is not recommended by Splunk or Microsoft.

Splunk is unable to get local data through WMI

When Splunk is unable to get data from the local machine through WMI providers, this might be because WMI is experiencing issues under load. When this happens, try restarting the Windows Management Instrumentation (wmimgmt) service from within the Services control panel, or by using the sc command-line utility.

Splunk sometimes crashes when collecting data over WMI

WMI can occasionally cause the splunk-wmi.exe process to crash. Splunk will spawn a new process when this happens (you can tell by the changed process ID).

While there is no guaranteed fix for this issue, you can reduce the number of crashes by reducing the number of servers you are monitoring through WMI with any given Splunk instance. Limit the number of WMI-based inputs per instance to 80 or fewer.

If you monitor the same subset of WMI providers on large numbers of machines, you can run into WMI memory constraints on the monitoring server. This can also cause crashes. Limit the number of WMI-based data inputs per server monitored through WMI. It's best to reduce the total number of WMI connections per instance to 120 or fewer on 32-bit Windows servers, and 240 or fewer on 64-bit Windows servers.

Consider using universal forwarders to get your data. You can either install universal forwarders on a few machines and get data from other machines through WMI, or you can put universal forwarders on all remote machines.

Splunk connects to WMI differently based on product version

When Splunk makes requests to WMI, it does so in one of three ways: Synchronous, asynchronous and semisynchronous.

Splunk makes what are known as semisynchronous calls to WMI providers. This means that when Splunk makes a call to WMI, it continues running while WMI deals with the request.

Semisynchronous mode offers the best balance of resource usage and security on the computer making the request. It differs from the faster asynchronous mode, but is more secure due to the way that the system handles retrieval of the WMI objects. Both of these modes are faster than synchronous mode, which forces programs making that kind of WMI request to wait until WMI returns the data.

When WMI is dealing with a large number of requests, you might notice a slower response because memory usage on the system increases until the retrieved WMI objects are no longer needed by Splunk (after indexing).

Note: Versions of Splunk prior to 3.4.10 make asynchronous connections to WMI providers.

Manually verify that WMI is working

To test WMI, you can run the splunk-wmi.exe command manually with a desired query and/or namespace to see the output that it produces.

Caution: When running this command, be sure to temporarily change Splunk's data store directory (the location that SPLUNK_DB points to), so that you do not miss any WMI events. To change Splunk's database store, refer to "Test access to WMI providers" in the Getting Data In Manual.

Comments

Could you explain why evt_resolve_ad_obj attribute is not working for WMI? It works for Splunk UF if reading events locally. <br /><br />Since this attribute is not working, what is the right way to select if we want GUID translated or not?

Enter your email address, and someone from the documentation team will respond to you:

Send me a copy of this feedback

Please provide your comments here. Ask a question or make a suggestion.

Feedback submitted, thanks!

You must be logged into splunk.com in order to post comments.
Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic.
If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk,
consider posting a question to Splunkbase Answers.

0
out of 1000 Characters

Your Comment Has Been Posted Above

We use our own and third-party cookies to provide you with a great online experience. We also use these cookies to improve our products and services, support our marketing campaigns, and advertise to you on our website and other websites. Some cookies may continue to collect information after you have left our website.
Learn more (including how to update your settings) here »