On Thu, Jun 16, 2011 at 1:14 PM, Andi Kleen wrote:> >> > I haven't analyzed it in detail, but I suspect it's some cache line bounce,> > which> > can slow things down quite a lot. Also the total number of invocations> > is quite high (hundreds of messages per core * 32 cores)

The fact is, glibc is just total crap.

I tried to send uli a patch to just add caching. No go. I sent*another* patch to at least make glibc use a sane interface (and thecache if it needs to fall back on /proc/stat for some legacy reason).We'll see what happens.

Paul Eggbert suggested "caching for one second" - by just calling"gettimtofday()" to see how old the cache is. That would work too.

The point I'm making is that it really is a glibc problem. Glibc isdoing stupid expensive things, and not trying to correct for the factthat it's expensive.

> > I did, but I gave up fully following that code path because it's so> > convoluted :-/I do agree that glibc sources are incomprehensible, with multiplelayers of abstraction (sysdeps, "posix", helper functions etc etc).

In this case it was really trivial to find the culprit with a simple

git grep /proc/stat

though. The code is crap. It's insane. It's using/sys/devices/system/cpu for _SC_NPROCESSORS_CONF, which is at least areasonable interface to use. But it does it in odd ways, and actuallycounts the CPU's by doing a readdir call. And it doesn't cache theresult, even though that particular result had better be 100% stable -it has nothing to do with "online" vs "offline" etc.

But then for _SC_NPROCESSORS_ONLN, it doesn't actually use/sys/devices/system/cpu at all, but the /proc/stat interface. Which isslow, mostly because it has all the crazy interrupt stuff in it, butalso because it has lots of legacy stuff.

I wrote a _much_ cleaner routine (loosely based on what we do intools/prof) to just parse /sys/devices/system/cpu/online. I didn'teven time it, but I can almost guarantee that it's an order ofmagnitude faster than /proc/stat. And if that doesn't work, you canfall back on a cached version of the /proc/stat parsing, since ifthose files don't exist, you can forget about CPU hotplug.

> > So you mean caching it at startup time? Otherwise the parent would> > need to do sysconf() at least , which it doesn't do (the exim source doesn't> > really know anything about libdb internals)Even if you do it in the children, it will help. At least it would berun just _once_ per fork.

But actually looking at glibc just shows that they are simply doingstupid things. And I absolutely _refuse_ to add new interfaces to thekernel only because glibc is being a moron.