Disable SIOC IO Metrics Collection For Auto Tiering Storage Systems

Most VMware admins know by now that auto tiering storage systems should have SIOC configured differently than if they have storage systems that don’t do auto tiering. Especially if it’s auto tiering at block level. It’s also recommended to have IO metric collection disabled because it will get different results based on where the blocks on the datastore physically reside. What you may not know is that VMware has enabled IO metric collection for every datastore by default with ESXi 6.0. This can cause some weird latency spikes on your storage system that can be very hard to isolate if you don’t know what you’re looking for or what might be causing it. So here is how you fix the problem.

If like a lot of people you still primarily use the C# client for vSphere then you will have to switch to a web browser to fix this problem. This will require you log into the vSphere Web Client. Once you are logged in select Storage -> <Datastore> -> Manage. You should see something similar to the following image:

Click on the Edit button and then click the check box next to Disable Storage I/O statistics collection as in the image below:

Click Ok and repeat the process for your other datastores.

If you would like to do the above more automatically by using a PowerCLI Script, please check out this great blog post by Burke Azbill here.

Final Word

Hopefully the above helps you reduce any unexpected latency spikes on your datastores and storage systems you use with ESXi 6.0. As the number of storage systems with auto tiering and multiple storage types increases over time this becomes even more relevant. Given the trends in the industry it is a wonder VMware made this a default setting in the first place. As storage systems become more intelligent there is less and less need for features such as Storage IO Control, which means admins should be able to focus on other more important things, rather than worrying about noisy neighbor problems. FYI, If you are a Nutanix customer this also applies to you. I found this little gem while running an NCC check on a cluster that I was getting periodic unexplained latency spikes. Needless to say the problem has now been resolved after making this change.

Share this:

Like this:

Related

About the Author

Michael is Technical Director, Business Critical Applications Engineering at Nutanix. He has been using VMware products since 1998 and has been deploying ESX solutions since 2002. He specializes in designing virtualization solutions for Unix to Linux migrations, business critical applications, disaster avoidance, and mergers and acquisitions. Michael has been in the IT industry since 1995 and consulting since 2001. Michael is Nutanix Platform Expert (NPX) #007. In addition to VMware Certified Design Expert (VCDX) he holds VCP, VCP-Cloud, VCAP-DCD, VCAP-DCA, VCAP-CID, VCAP-CIA, ITIL Foundation, MCP I, and MCSE (NT4 – 2K3).

Maybe I’m misunderstanding things but are you saying having the stats on affects performance of running vms or just gives false alerts?

The way I read this option it’s just collecting statistics (used for SDRS I guess) but doesn’t actually do anything to affect performance since Storage I/O control is off. At worst it’s reporting false positives and feeding false data to SDRS (so the latter might make bad decisions). If you don’t use SDRS, is there any harm in having this on?

[…] Yes, Vmware has SIOC and SDRS, but these don’t really work with modern flash arrays or HCI environments. In fact if you are using Vmware with Nutanix or any other platform with advanced storage auto-tiering capabilities, it is advisable to turn SIOC off […]

I’m using autotiering with SIOC manual threshold reasonably high – depending on tiers, but usually 30ms works pretty well. In this configuration you can still benefit from SIOC to some extend – till autotiering kicks in and migrates extends to the higher tier.

AFAIK Storage spikes are caused by SIOC injector which is used if you have Percentage of peak throughput configured with SIOC, which is unfortunately default settings nowadays.

Sponsors

Featured Virtualization Book

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Email

Disclaimer

The views expressed anywhere on this site are strictly mine and not the opinions and views of VMware or anyone else. All content is provided without any form or warranty explicit or implied, for informational purposes and for use at your own risk.