Re: High Page Fault

Not necessarily a problem... do you fully understand what page faults are?

It would be interesting to see if you also have high values in the at (address translation) column in a vmstat output.

If I execute a new process for the first time on a system, the text pages (program code) of that process will not be in physcial memory so this will generate a page fault, and then (as the process has not been run before) a page-in from physical disk (i.e. loading the program code into physical memory).

When the process terminates, we don't need to write out for the static program code, so we don't get a page out from that and the pages will stay in physical memory (although not referenced by any processes) unless there is memory pressure.

Now if I execute the same process again - the kernel virtual memory subsystem will know that those pages are already in physical memory and instead of having to do another page in, it can just do an address translation (which you can see from vmstat)

The upshot of all this is that you have a lot of processes that run and then terminate very often you can see very high page faults without a high number of page-ins, as these are just address translations.

There may be other reasons too (I'm not sure if memory mapped files play in this space as well), but we'll need a VM wizard such as Don Morris to answer that sort of question (and probably correct a few errors in my explanation above as well!)

Re: High Page Fault

Well you certainly don't seem to have a lot og page-ins going on there, but then neither do you appear to have a lot of address translations...

A %sys of 90% is not particularly normal - what application is running on this host? Can you identify waht processes are using all the system CPU time? (from your original output it looks like you have glance)

Re: High Page Fault

most of CPU is in use by system process(es)it is slightly normal %90 percent CPU ( my customer server sometime > %90 CPU but not all day long!)I wanted to state, if this situation happens temporarily not prolonged it is not too bad. But if this happens due to bottlenecks OR got stuck processes then need to be solved.

Re: High Page Fault

So this java program - who wrote it and what does it do? Has it *always* behaved like this, or just started to recently - who setup the startup parameters for the java program? have they been changed recently?

Perf tuning for java apps is quite a specialist area about which I know very little... you probably need to go back to the devlopers of the application and discuss with them... if you feel up to it, you could do worse than start reading this manual:

Re: High Page Fault

Well, I had a much longer reply and the forum submittal ate it. Here's the quicker version:

Assuming the vmstat output is also from the high Page Fault period, the low at field and higher system call field along with the original high Faults but low KB Paged In stats makes me suspect that the java process may be doing some sort of page release/invalidation and re-instantiation activity (perhaps as part of Java GC work? And the change may be in the object creation/utilization in the workload?). The data as given certainly indicates that the application is frequently calling down to the kernel and that the pages generated are Demand fill pages, not anything to do with paging in or out.

tusc might give some indication if you sample the application over a period to see what syscalls it is making.

Another alternative would be to pick up Caliper (http://www.hp.com/go/caliper) and point it at the application, being sure to profile the kernel as well as the application to get an idea of what's going on in the system faults. I think this would be more likely to produce results.

Re: High Page Fault

> Well, I had a much longer reply and the forum submittal ate it. Here's the quicker version

Yes that regularly happens to me - many times my beautifully crafted essays are eaten by the forum submission monster and my follow up post is just "please do xyz" without any explantion why!

Still from my point of view, at least your (no doubt stinging) critique of my horribly vague ramblings on the inner workings of VM kernel voodoo will never see the light of day and I can go on pretending I know what I'm talking about!