[Sbcl-devel] Re: [Sbcl-help] threads, streams, memory leak?

On Monday 04 April 2005 18:00, G=C3=A1bor Melis wrote:
> I've found a message describing the exact same problem:
> http://article.gmane.org/gmane.lisp.steel-bank.devel/1879
>
> SBCL allocates two buffers on the wrong thread (accept), and never calls
>
> deallocate-system-memory when the thread exits, so not even restarting
> the
> server can discard the accumulated buffers.
>
> Some measurements were conducted on sbcl 0.8.21 with the attached
> program
> against apache bench. GC was turned off.
>
> /usr/sbin/ab -n 10000 -c 64 http://localhost:2002/
>
> concurrency | 1 8 64
> ------------------------------------------------
> without patches | 6.988660 6.779214 6.736636
> B Downing's patch | 6.530214 6.319840 6.491603
>
> Performance degradation is still not a problem at 64 threads as there is
> not
> much lock contention.
>
> The 'connection reset by peer' problem is still there, though, but it's
> hard to trigger (seems to happen more after startup).
I'd hate to monopolize this thread, but I've found the cause of the connect=
ion=20
reset by peer problem: the test program has not read the request.
With that bug out of the way the test ran for a long time producing only th=
e=20
occasional "The value NIL is not of type WEAK-POINTER." message. The attach=
ed=20
patch makes the finalizer thread safe. With this and the above patch applie=
d=20
the test server survives the apache bench onslaught with 60 threads seeming=
ly=20
indefinitely.
>
> Gabor

Thread view

On Monday 04 April 2005 18:00, G=C3=A1bor Melis wrote:
> I've found a message describing the exact same problem:
> http://article.gmane.org/gmane.lisp.steel-bank.devel/1879
>
> SBCL allocates two buffers on the wrong thread (accept), and never calls
>
> deallocate-system-memory when the thread exits, so not even restarting
> the
> server can discard the accumulated buffers.
>
> Some measurements were conducted on sbcl 0.8.21 with the attached
> program
> against apache bench. GC was turned off.
>
> /usr/sbin/ab -n 10000 -c 64 http://localhost:2002/
>
> concurrency | 1 8 64
> ------------------------------------------------
> without patches | 6.988660 6.779214 6.736636
> B Downing's patch | 6.530214 6.319840 6.491603
>
> Performance degradation is still not a problem at 64 threads as there is
> not
> much lock contention.
>
> The 'connection reset by peer' problem is still there, though, but it's
> hard to trigger (seems to happen more after startup).
I'd hate to monopolize this thread, but I've found the cause of the connect=
ion=20
reset by peer problem: the test program has not read the request.
With that bug out of the way the test ran for a long time producing only th=
e=20
occasional "The value NIL is not of type WEAK-POINTER." message. The attach=
ed=20
patch makes the finalizer thread safe. With this and the above patch applie=
d=20
the test server survives the apache bench onslaught with 60 threads seeming=
ly=20
indefinitely.
>
> Gabor

Quoting Nikodemus Siivola (nikodemus@...):
> Adding WITHOUT-GCING around calls to individual finalizers helps some,
> but as you say does not solve the real issue, and in absence of (AFAIK)
> an existing body of work on such GC protocols I'm inclined to punt and
> say "know thy finalizers".
A more robust finalization mechanism running finalizers in a dedicated
thread could be built as a contrib.
The current implementation would then be left as a low-level alternative
for use my SBCL itself and anyone unfortunate enough to lack sb-thread.
d.

On Apr 10, 2005, at 8:29 AM, David Lichteblau wrote:
> Quoting Nikodemus Siivola (nikodemus@...):
>> Adding WITHOUT-GCING around calls to individual finalizers helps some,
>> but as you say does not solve the real issue, and in absence of
>> (AFAIK)
>> an existing body of work on such GC protocols I'm inclined to punt and
>> say "know thy finalizers".
>
> A more robust finalization mechanism running finalizers in a dedicated
> thread could be built as a contrib.
>
> The current implementation would then be left as a low-level
> alternative
> for use my SBCL itself and anyone unfortunate enough to lack sb-thread.
... which would be most of the ports. I don't see why a contrib for
finalizers couldn't run them in a dedicated thread on sb-thread and run
them "normally" on non-sb-thread. Admittedly the behavior of specials
might be different.
--
Brian Mastenbrook
brian@...
http://www.iscblog.info/

On Sun, 10 Apr 2005, David Lichteblau wrote:
> A more robust finalization mechanism running finalizers in a dedicated
> thread could be built as a contrib.
I'm all for such a contrib, but I claim that it is not a full solution
even on threaded platforms: finalizers must still be re-entrant. Unless
I'm mistaken it would be effectively the same as running individual
finalizers in unithreaded SBCL inside a WITHOUT-GCING: re-entry by the way
of FINALIZER -> GC -> FINALIZER can no longer happen, but FOO -> GC ->
FINALIZER -> FOO can still occur.
I'm not particularly opposed to running finalizers in a separate thread on
threaded platforms, though, but I cannot claim to have really thought
about it.
This is besides the point, but I think that finalizers from eg. WITH-ALIEN
could be probably implemented more efficiently by the way of hooking
directly to GC: push dynamic aliens to a list reserved for that purpose,
and deallocate them from UNSAFE-CLEAR-ROOTS.
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious."
Lispnik: "Buddha is big, has hairy armpits, and laughs."

On Sunday 10 April 2005 15:53, Nikodemus Siivola wrote:
> On Sun, 10 Apr 2005, David Lichteblau wrote:
> > A more robust finalization mechanism running finalizers in a dedicated
> > thread could be built as a contrib.
>
> I'm all for such a contrib, but I claim that it is not a full solution
> even on threaded platforms: finalizers must still be re-entrant.
I would say that with the threaded version the individual finalizers need only
be thread safe (as opposed to reentrant).
> Unless
> I'm mistaken it would be effectively the same as running individual
> finalizers in unithreaded SBCL inside a WITHOUT-GCING: re-entry by the way
> of FINALIZER -> GC -> FINALIZER can no longer happen, but FOO -> GC ->
> FINALIZER -> FOO can still occur.
Still in the threaded scenario: I cannot see how FOO -> GC -> FINALIZER -> FOO
can occur. Sure, there is limited machinery on the finalizer thread used to
invoke the finalizers and that must be reentrant. In a unithread sbcl this is
not enough.
Gabor

On Tue, 12 Apr 2005, [iso-8859-1] G=E1bor Melis wrote:
> I would say that with the threaded version the individual finalizers=20
> need only be thread safe (as opposed to reentrant).
Good point. Provided that the second call occurs in another thread this is=
=20
indeed the case; and indeed I think this was something David tried to=20
bring home to me a couple of days back on #lisp. I seem to be a bit slow=20
on the pickup...
One question for those of you running threaded systems: could someone=20
outline pros and cons for a separate finalizer thread?
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious."
Lispnik: "Buddha is big, has hairy armpits, and laughs."

On Tue, 5 Apr 2005, G=C3=A1bor Melis wrote:
> With that bug out of the way the test ran for a long time producing only =
the
> occasional "The value NIL is not of type WEAK-POINTER." message. The atta=
ched
> patch makes the finalizer thread safe. With this and the above patch appl=
ied
Thank you. I've committed something along the lines of your finalizer=20
patch, plus additional GC sanitation as 0.8.21.23.
As I'm not running SBCL on Linux, I cannot fully wouch for workingness of=
=20
my changes with threads -- testing would be most appreciated.
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious."
Lispnik: "Buddha is big, has hairy armpits, and laughs."

On Friday 08 April 2005 14:33, Nikodemus Siivola wrote:
> On Tue, 5 Apr 2005, G=C3=A1bor Melis wrote:
> > With that bug out of the way the test ran for a long time producing only
> > the occasional "The value NIL is not of type WEAK-POINTER." message. The
> > attached patch makes the finalizer thread safe. With this and the above
> > patch applied
>
> Thank you. I've committed something along the lines of your finalizer
> patch, plus additional GC sanitation as 0.8.21.23.
>
> As I'm not running SBCL on Linux, I cannot fully wouch for workingness of
> my changes with threads -- testing would be most appreciated.
Works great. At least most of the time. If gc hits while holding the finali=
zer=20
store mutex, say in run-pending-finalizers, this kind of code gets into=20
trouble:
(finalize "hello"
(lambda ()
(finalize "world"
(lambda ()))))
(gc :full t))
WARNING:
recursive lock attempt #S(SB-THREAD:MUTEX
:NAME "Finalizer store lock."
:LOCK 0
:DATA NIL
:VALUE 6764)
In a slightly more realistic example the second call to finalize may come f=
rom=20
an innocent looking open. A recursive lock or without-gc around the mutex=20
would be the easy way out in this case, but the finalizer can be run while=
=20
holding any mutex so that would not be enough. Maybe it is a case for a=20
separate finalizer thread. I just cannot imagine a general solution to this=
=20
in a single threaded environment.
Gabor

On Sun, 10 Apr 2005, G=C3=A1bor Melis wrote:
> In a slightly more realistic example the second call to finalize may come=
from
> an innocent looking open. A recursive lock or without-gc around the mutex
> would be the easy way out in this case, but the finalizer can be run whil=
e
> holding any mutex so that would not be enough.
I was about to say that "that's not a problem, since we tell users that
their finalizers may run in any thread they should know better then
to try and acquire locks", but then I realized that would effective say
"run only standard CL and code you've written yourself in finalizers",
which is just a bit too nasty even for SBCL. ,-)
> Maybe it is a case for a separate finalizer thread. I just cannot=20
> imagine a general solution to this in a single threaded environment.
I think you're right about the separate thread. I assume you mean=20
stuff like the following in single-threaded environment?
(defvar *rec* nil)
(defun oops ()
(when *rec*
(error "recursive oops"))
(let ((*rec* t))
(gc))) ; or just cons enough to cause one
(finalize "oops" #'oops)
(oops)
I think the deeper issue here is that GC without finalizers and other=20
interesting stuff can be pretty magic in the sense that users don't have
to know or think about it. However, as soon as we expose parts of it --=20
like providing finalizers -- we're effectively engaging in a protocol,
and need to think about the responsibilities involved.
Adding WITHOUT-GCING around calls to individual finalizers helps some,
but as you say does not solve the real issue, and in absence of (AFAIK)
an existing body of work on such GC protocols I'm inclined to punt and
say "know thy finalizers".
Cheers,
-- Nikodemus Schemer: "Buddha is small, clean, and serious."
Lispnik: "Buddha is big, has hairy armpits, and laughs."