thread/socket behvior

Jeff Polakow wrote:
> Don Stewart <dons at galois.com> wrote on 10/09/2008 02:56:02 PM:
>> > jeff.polakow:
> > > We have a server that accepts messages over a socket, spawning
> threads to
> > > process them. Processing these messages may cause other, outgoing
> > > connections, to be spawned. Under sufficient load, the main
> server loop
> > > (i.e. the call to accept, followed by a forkIO), becomes
> nonresponsive.
> > >
> > > A smaller distilled testcase reveals that when sufficient socket
> activity
> > > is occurring, an incoming connection may not be responded to
> until other
> > > connections have been cleared out of the way, despite the fact
> that these
> > > other connections are being handled by separate threads. One
> issue that
> > > we've been trying to figure out is where this behavior arises
> from-- the
> > > GHC rts, the Network library, the underlying C libraries.
> > >
> > > Have other GHC users doing applications with large amounts of
> > socket usage
> > > observed similar behavior and managed to trace back where it
> originates
> > > from? Are there any particular architectural solutions that
> people have
> > > found to work well for these situations?
> >
> > Hey Jeff,
> >
> > Can you say which GHC you used, and whether you used the threaded
> > runtime or non-threaded runtime?
> >
> Oops, forgot about that...
>> We used both ghc-6.8.3 and ghc-6.10.rc1 and we used the threaded
> runtime. We are running on a 64 bit linux machine using openSUSE 10.
The scheduler doesn't have a concept of priorities, so the accepting thread
will get the same share of the CPU as the other threads. Another issue is
that the accepting thread has to be woken up by the IO manager thread when
a new connection is available, so we might have to wait for the IO manager
thread to run too. But I wouldn't expect to see overly long delays. Maybe
you could try network-alt which does its own IO multiplexing.
If you have multiple cores, you might want to try fixing the thread
affinity - e.g. put all the worker threads on one core, and the accepting
thread on the other core. You can do this using GHC.Conc.forkOnIO, with
the +RTS -qm -qw options.
Other than that, I'm not sure what to try right now. We're hoping to get
some better profiling for parallel/concurrent programs in the future, but
it's not ready yet.
Cheers,
Simon