The V$ views (with the exception of V$OSSTAT) will show you only what Oracle thinks is happening, not the reality what really happens in a process.

So, the UGA/PGA numbers show you close to real memory usage only when Oracle itself maintains these properly. If due to a bug or some external factor memory is allocated without updating these counters, then you'll see discrepancy between what Oracle thinks it's using vs real memory usage. So snapper showed no memory growth as these "pga/uga memory bytes" counters were not updated by Oracle properly.

But when standard OS tools say there is a memory shortage, I would believe the OS tools as opposed to Oracle which is just another application on top of the OS.

You correctly did the stack profiling, so that it should be evident roughly where the looping and memory leak is happening.

I would first have ran pmap -x on the process (after TOP showed the huge memory usage). I reproduced the problem with your script and ran pmap:

I did a metalink search with the top functions reported in the stack traces ... didn't find an exact match (you don't always get lucky), but here's one with similar stack trace, causing a session to hang. No memory issues reported though, but you may have a variation of this bug (or some completely different bug in the same code):

Of all the bugs logged for this issue,Bug 9004844 - query using regexp_replace spins in lxregmatch
and Bug 9134741 - ora-07445 [kghfrmrg()+1260] create table as select
both of which are marked as duplicates of bug 8579113,
both say FIX BY 12.1 and 11.2.0.4, so that it when there will be a solution.