Jay Lan wrote:> Shailabh Nagar wrote:> >>Hi Andrew,>>>>Two developments on the tgid overhead issue:>>>>1. The latest results show that overhead is significant>>only when the exit rate exceeds roughly 1000 threads/second.>> > > > I worked with Shailabh this week to run various testing and> debugging as he requested. I was pulled off to some urgent> task yesterday and surprising saw this coming this morning...

Sorry...didn't mean to surprise. I sent you the data last nightprivately with request for comments.

Your testing and help has been very valuable and helped uncovertwo issues: the locking patch (sent separately) and also adependency between taskstats and delay accounting (for which anotherpatch is being sent out shortly).

True, but my point is that the overhead is at an extremelyhigh exit rate. I think the test in which you saw 109% overheadran 5000 iterations of 1000 threads and had an elapsed time of294 seconds (with tgid turned off) giving an exit rate of roughly8500 exits/second, right ?

My results confirm the high overhead at these exit rates. In fact,on the system I used, I see the 649% overhead for the 2200 exits/second caseeven higher than yours) but the point is whether that exit rateis a valid design criteria.

> And, the per-thread group processing also increase the rate of ENOBUFS> at the receiver.

> I need to check with other guys to find out if 1000 threads/sec> indeed unrealistic at our customers' environments. A good> design should allow a mechanism to turn off the penalty due to> a feature that is not common to everybody. I do not understand> your objection.

Only objection is that design shouldn't cater to a case that isextremely unlikely in practice. In most situations, there is noor insignificant penalty.

Perhaps others on the list can also chip in whether this kind of exitrate is realistic in some scenarios and where the peformancepenalty matters (i.e. not system shutdown etc.)

Please note that the exits have to be for multithreaded apps, notsingle-threaded ones for which tgid sending is already turned off.

Thanks,Shailabh

> > Regards,> - jay> > >>2. A new patch that modifies the locking used within taskstats,>>brings down the overhead of the extreme case quite a bit.>>I'll submit the patch along shortly in a separate mail.>> >>To get back to the effect of exit rate, I modified the fork+exit>>benchmark to vary the rate at which exits happened and>>ran tests on a 4-way 1.4 GHz x86_64 box. The kernel was 2.6.17,>>uses the delay accounting/taskstat patches in 2.6.17-mm1 + the new>>locking patch mentioned in 2. above.>>>>The results show that differential between tgid on and off>>starts becoming significant once the exit rate crosses roughly 1000>>threads/second. Below that exit rate, the difference is negligible.>>Above it, the difference starts climbing rapidly.>>>>So I guess the question is whether this rate of exit is representative>>enough of real life to warrant making any more changes to the existing>>patchset, beyond the locking changes in 2. above.>>>>>From my limited experience, I think this is too high an exit rate>>to be worrying about overhead.>>>>>> %ovhd of tgid on over off>> (higher is worse)>>>>Exit User Sys Elapsed>>Rate Time Time Time>>>>2283 25.76 649.41 -0.14>>1193 -10.53 88.81 -0.12>>963 -11.90 3.28 -0.10>>806 -8.54 -0.84 0.16>>694 -4.41 2.38 0.03>>>>Exit Rate: units are threads exiting per second.>>Calculated by (#threads_forked+exited)/(elapsed_time)/2>>Since app pretty much does only thread create and exit for 10000>>threads (1000 threads, 10 iterations), this is a good measure>>for exit rate.>>>>%diff in user, sys, elapsed times calculated using>>(tgid_on - tgid_off)/tgid_off * 100>>where tgid_on/off times are reported by /usr/bin/time as before.>>>>Each data point for tgid_on and tgid_off was an average>>of 10 runs of the fork+exit benchmark.>>The rate of exits was controlled by delaying the individual>>threads through a usleep before being allowed to exit.>>>>Machine was 4-way 1.6GHz x86_64 Opteron.>>>>"exit_recv -w", the user program consuming the stats, was running>>on the side, reading the stats but not writing to a file or>>printing to screen.>> > >