My (current) system (cause this very error happens with more than one system, but all have Geforce 8xxx GPUs):
3 GB of system RAM
GeForce 8400 GS 512 MB RAM
AMD Athlon 64 X2 Dual Core Processor 4200+

Now I read (parts) of the article on wikipedia about the Geforce 8 series, and the bug therein, where it says that at least according to the article, the issue not got solved by any new nvidia drivers. The bug is describes as so: More and more data gets loaded into the graphics cards RAM, but nothing of the oldest stuff gets unloaded, and so the GPU performance gets slower and slower.
According to the wikipedia article, the bug especially occurs with Nvidia 8xxx cards will low memory, mine just has 512 MB, which seems to be somewhere in the middle, not the smallest of RAM, and not the largest...

When I just exit X and restart X, but keep the virtual text consoles and the running Porteus as it was, all runs smoothly...

So, it seems the freezing X issue could really be because of the Nvidia 8xxx series bug. Especially, because when I exit X and restart X, but keep everything else running (I mean: the virtual consoles of Porteus), all runs quick and fine again.

Any ideas how to further examine this?
So far, I got no info in any of the logs, and probably the NVidia drivers won't even tell the system about what's going on. So far, I was not able (by searching the web) to find anything about that bug and how it can be undone running x86-64 Linux...

Well, I think you are depending on Nvidia.If this is a Hardware construction bug,
and a Driver update can not correct it...
A free Linux-driver will not help (imho), because Nvidia do not provide their specs,
so they are always some time (and features) behind.

....how to examine this?....
I do not know any special testing tool,but I would play a game which includes
huge textures to fill-up the graphic cards RAM and observe if the performance
goes down after some Levels.
and if so I would sell this damn card on Ebay..
Good Luck

To be true, I can give no recomendation, because I do/can not play games
on my box.(Geforce 2mx)
You should take a relatively new Game to fill 512mb, and i never thought
that i would say this in public, but maybe you can borrow a Game from your
neighbors/friends and test your card (and only for testing) in a win-installation..