Rusty Russell wrote:> On Sat, 2005-05-07 at 23:57 +0530, Srivatsa Vaddagiri wrote:> >>Two solutions have been proposed so far:>>>> A. As per Nick's suggestion, impose a max limit (say some 100 ms or>> say a second, Nick?) on how long a idle CPU can avoid taking

Yeah probably something around that order of magnitude. I suspectthere will fast be a point where either you'll get other timersgoing off more frequently, and / or you simply get very quicklydiminishing returns on the amount of power saving gained fromincreasing the period.

>> local-timer ticks. As a result, the load imbalance could exist only>> for this max duration, after which the sleeping CPU will wake up>> and balance itself. If there is no imbalance, it can go and sleep>> again for the max duration.>>>> For ex, lets say a idle CPU found that it doesn't have any near timer>> for the next 1 minute. Instead of letting it sleep for 1 minute in>> a single stretch, we let it sleep in bursts of 100 msec (or whatever>> is the max. duration chosen). This still is better than having>> the idle CPU take HZ ticks a second.>>>> As a special case, when all the CPUs of an image go idle, we>> could consider completely shutting off local timer ticks>> across all CPUs (till the next non-timer interrupt).>>>>>> B. Don't impose any max limit on how long a idle CPU can sleep.>> Here we let the idle CPU sleep as long as it wants. It is>> woken up by a "busy" CPU when it detects an imbalance. The>> busy CPU acts as a watchdog here. If there are no such>> busy CPUs, then it means that nobody will acts as watchdogs>> and idle CPUs sleep as long as they want. A possible watchdog>> implementation has been discussed at:>>>> http://marc.theaimsgroup.com/?l=linux-kernel&m=111287808905764&w=2> > > My preference would be the second: fix the scheduler so it doesn't rely> on regular polling.

It is not so much a matter of "fixing" the scheduler as just addingmore heuristics. When are we too busy? When should we wake anotherCPU? What if that CPU is an SMT sibling? What if it is across theother side of the topology, and other CPUs closer to it are busyas well? What if they're busy but not as busy as we are? etc.

We've already got that covered in the existing periodic pull balancing,so instead of duplicating this logic and moving this extra work to busyCPUs, we can just use the existing framework.

At least we should try method A first, and if that isn't good enough(though I suspect it will be), then think about adding more complexityto the scheduler.

> However, as long as the UP case runs with no timer> interrupts when idle, many people will be happy (eg. most embedded).>

Well in the UP case, both A and B should basically degenerate to thesame thing.

Probably the more important case for the scheduler is to be able toturn off idle SMP hypervisor clients, Srivatsa?