VM Memory Usage Stats: vCenter vs. Windows

Although there is all kinds of information out there regarding this very topic, I had a hard time getting all of this straight in my head a couple months back. There just didn’t really seem to be a definitive guide or article about this topic to help anyone. Granted I am not saying that this will be The Definitive Guide by any means, but it certainly doesn’t hurt to give it a shot. Also it seems like a pretty alright first article to get myself out on those interwebs

My confusion came when I was recommending to my colleague and supervisor that we lower the allocated memory to several VMs. We were approaching being very overcommited on our hosts in terms of memory and there were several guests that had not even used more 19% of their allocation in the last several months.

Their first instinct is to look at what the guest is reporting in terms of memory usage. This I knew wasn’t really the best idea since the guest is generally unaware of the host on which it is running. Their first stop was the guest Task Manager.

Task Manager Performance Tab

Also in the Guest we took a look at Resource Monitor

ResourceMonitor

Taking a look at these, it can be very, Very misleading as to what the guest OS is using in terms of memory. From this picture it seems as though the guest is using 73% of its memory, which is much higher than the less than 19% I had been reporting from vCenter.

Memory Usage in vCenter

Zooming Up

What exactly is going on here?!?

To understand what is happening, we have to first understand what these terms mean

Inside the Guest OS

Commit (KB) =”Amount of virtual memory that is reserved for use by a process.”
In other words, this is the amount of memory that a guest process has reserved, which doesn’t necessarily mean it is actively using it.

Working Set (KB)=”Amount of memory in the private working set plus the amount of memory the process is using that can be shared by other processes.”
This is obviously a pretty vague description, also defined/described as “set of pages in the virtual address space of the process that are currently resident in physical memory”. This second description is MUCH better. What this is saying is that the working set is memory from the process that has been assigned a location on the memory. Again this isn’t memory necessarily being actively used/accessed, but just memory that has a residence/location. An analogy that just came to me(so be prepared)…Think of it as a hotel room that has been assigned to you. You have that whole room if you want it, but its possible you may only Really use the room for sleeping. Which means even though that location is assigned to you, you aren’t actively using it for a majority of the time you have it. You can be so greedy sometimes!!!
Similarly some processes like to acquire and hold on to memory even though they are not actively using it.

Shared (KB)= Don’t need a quote for this one. Essentially the memory that is being shared with another process

Private (KB)= This is the amount of memory that is assigned only to you.

“ESX uses a statistical sampling approach to estimate the aggregate virtual machine working set size without any guest involvement. At the beginning of each sampling period, the hypervisor intentionally invalidates several randomly selected guest physical pages and starts to monitor the guest accesses to them. At the end of the sampling period, the fraction of actively used memory can be estimated

as the fraction of the invalidated pages that are re-accessed by the guest during the epoch. ESX uses a statistical sampling approach to estimate the aggregate virtual machine working set size without any guest involvement. At the beginning of each sampling period, the hypervisor intentionally invalidates several randomly selected guest physical pages and starts to monitor the guest accesses to them. At the end of the sampling period, the fraction of actively used memory can be estimated as the fraction of the invalidated pages that are re-accessed by the guest during the epoch.”

In layman’s terms, vCenter and Esx/Esxi have a really complicated algorithm for calculating how much of the memory is actively, yes actively being used by the guest.

Well, that’s a great article, and I also really like and use the “Active” metric a lot.

the main issue that my organization has is that how do you explain this to your Cloud Costumers..? I mean, you can arge with them all the day and tell them that our “Active” Metric is the correct, and they we’ll say that thay are seein 90% use of RAM.

That is a great question. Generally with those types of customers “seeing is believing” and it may be hard for them to just take your word for it.

I would suggest a couple of things.

First if it shows them with a 90% memory usage, I would use the memory performance tester. That is linked on my Resource Loading Page. Using that you can show them that if you active use another 10% of the memory Task Manager still shows there being some left over, meaning task manager was not accurate in the first place.

Secondly I would show them perfmon. If you run perfmon inside of the VM it has vm specific counters which will give you an accurate reading of the resources being used as it uses VMware Tools.

[…] been one of my first ones regarding understanding memory usage and statistics inside of the VM. Found Here. That either tells me that I am going under appreciated, or I am not achieving the same quality […]

Correct, you can use perfmon with the VM metrics to see a more accurate picture. This utilizes vmware tools and the actual host for its metrics.
I had not really went into perfmon, but it is a another viable alternative. However it pulls its metrics from the vcenter, so you can look in vcenter for the same stuff.