Re: memory utilization is increasing

T Jose,

You showed us a single snapshot with no time indication. Please gives us an indication what specific numbers suggested to you that there might be a problem.And why not toss in the platform, OS version, Filesystem (version)used, and essential Patch information such that we are not temped to wildly suggest to add more patches.

What is 'gradually'. Hours? Days? Weeks?What specific component in the kmeminfo is increasing, by how much?

That M_TEMP is high, but is that the component which was growing? By giving us more numbers the folks who lack the clairvoyance skills that Sooraj apparently has may be able to try to help better.

Can you perhaps show output this week and last week... at a similar time of day? Comparing with data right after the reboot is probably not useful, as applications still have to get up and get settled.

The apparent leak might just be goodness, as caches becoming more and more useful.For example for 11.11 and JFS 3.3 M_TEMP is used for inode cache according to the elusive document ftp://eh:spear9@hprc.external.hp.com/memory.htm

Re: memory utilization is increasing

Hello,

We've recently gotten to the bottom of a very similar problem, having to do with the way in which the kernel's superpage pool - which is used to supply pages of memory to various kernel memory arenas by the arena alocator - is responding to certain patterns of allocation and freeing.

When the kernel needs more memory to satisfy a request from a kernel subsystem, and it does not have any blocks already available of sufficient size, it pulls in a new fixed-size region of memory, called a "superpage," from the system-wide free memory pool. The size of the region depends on the amount of physical memory.

On our 32GB systems, a new superpage is 256MB, so that may be about the size your 28.5GB system is using. Our 64GB systems use a 1GB superpage.

Then the superpage is broken in half enough times to supply a memory region of sufficient size for the arena's request. So if a kernel driver or the like requests 64MB of kernel memory, your 265MB superpage will be broken in half once, then once again to a 64MB block, and two blocks - a 128MB and a 64MB - will be placed on the superpage free list.

Things break down, though, if the requests are not a power of two. If the request is 129MB, for example, you'll have a 128.5MB superpage with one 129MB region, plus a free list consisting of 128MB, 64MB, 32MB, 16MB, 8MB, 4MB, 2MB, 1MB, and a 512k regions. The kernel pulled 256MB from free memory to satisfy a 128.5MB allocation. If there's another 128.5MB allocation, it can't fit in any of the free spaces left in the previous superpage, so the kernel has to fetch another 256MB out of the system free pool and divvy it up in the same way.

Then, if some other kernel arena needs even one page of memory (4k) it may take it out of the 49.9% free space in one of these superpages. And until every last allocation in a superpage is freed, that 256MB can't be returned to the system free pool.

And as more and more allocations come along, the chances of any one superpage being freed dwindle to nearly zero, leaving many gigabytes stuck in the "system" category. One of our superpages had ONE page allocated out of a full gigabyte.

You can also see this behavior during backups if the inode cache table is set too large - the VxFS system during a backup winds up needing a huge amount of kernel memory for the vx_inode_cache arena to cache inodes which will only be touched once during a backup, so more and more superpages are pulled in.

But while the inodes are cached sequentially by the backup and push the superpage pool to a high water mark, they're unlikely to be expired sequentially, so you end up with a lot of one-page vx_inode_cache allocations scattered across many nearly-empty superpages.

Judging by the superpage pool free number you show of only 74.7 megabytes, that's evidently not your problem - most of the allocations from the pool are still reserved by the arenas that are using them.

You can get a better sense of this by looking at the kmeminfo -arena output - you'll probably see one of the dynamic arenas such as M_IOSYS overwhelmingly responsible for the memory utilization in that list. The M_TEMP arena would also merit some scrutiny.

You may also be able to do "kmeminfo -kas analyze" - better to do it on a dump image than a live system, though - to see how each of the superpages is broken out to the arenas. Warning: it takes a very, very long time to run.

It's possible that if you push your freemem number below lotsfree, you'll spur some memory reclamation activity and free up some unused space from arenas back into the superpage pool, but that won't be guaranteed to free up an entire superpage for return to the system free pool.

Re: memory utilization is increasing

2 things:1) kmeminfo is a private support tool which should be used on support request only as indicated in the output headers you cut.

2) you probably have a memory leak on M_TEMP index 0, now find what is the root cause you have 2 ways: a) guess work, and blind patching - looking at patch texts which indicate MEMORY LEAK- b) have it analyzed by hp support, with proper tools, and tools interpretation.