And, yet another followup to my own message...
Using a guesstimator value of 1 mb of bitmaps per 4 gb of CNFS buffer
storage, there are roughly 175 mbs worth of CNFS bitmaps on the machine.
Since the bitmaps are being mmap()'d, we assume that the 175 mbs are shared
accross all the nnrpd processes. This is a correct assumption! However, we
thought this wasn't the case because of the output of ps's RSS column.
After exchanging a few emails to/from Matt Dillon (thanks Matt!), I was
made aware that the RSS column also includes resident pages from the
mmap()'d regions of the CNFS bitmaps, which are infact shared amongst all
the nnrpds. So the nnrpds show up as using more than they really are.
Also, to determine where the RSS value in the ps output is coming from under
FreeBSD (not sure about others), you can do:
cat /proc/<pid>/map (or dd if=/proc/<pid>/map bs=64k)
Where <pid> is the process id of your nnrpd, innd or whatever. The output
of which should be similar to (without the headers):
+--------------------+--------+---------+---+---------------------+----+
| mapped regions |resident|kern VM- | | | |
| start end |pages |obj. ptr |flg| |type|
+--------------------+--------+---------+---+---------------------+----+
0x08048000 0x080a6000 62 64 0xe1654a20 r-x 11 1 0x4 COW NC vnode
0x080a6000 0x080a8000 2 0 0xe167a2a0 rw- 1 0 0x2180 COW NNC vnode
0x080a8000 0x080bb000 6 0 0xe1869840 rw- 1 0 0x2180 COW NNC swap
0x080bb000 0x080c2000 7 0 0xe137a240 rwx 1 0 0x2180 COW NNC swap
0x080c2000 0x08286000 443 0 0xe15be3c0 rwx 1 0 0x2180 NCOW NNC default
...
where each page is 4kbytes. And, the 'vnode' type represent file mmap()s
while 'swap' and 'default' represent mmap()s of anonymous memory.
The rapidly growing RSS values of the nnrpds during LIST, LISTGROUP and
XOVER commands to mention a few, are attributed to the pages being made
resident from the mmap()'d region of the CNFS bitmaps which get accessed.
If however, you were to leave your nnrpd idle for a few seconds, and
check its RSS again, you will notice that the value goes down slowly, as
the pages get reclaimed due to non-use.
So, do not trust the output of top or ps under FreeBSD especially for
applications that do a whole lot of mmap'ing and leave their maps cached.
Hope this helps those who have been searching for an explanation to
all this.
-mohan