>> >> > To make double sure I have no task on my nohz cpuset CPU, I've booted> > the system with the isolcpus command line isolating the same cpu I've> > assigned to the nohz set. This shouldn't be needed of course, but just> > in case.>> Ah I haven't tested with that isolcpus especially as it's headed toward> removal.>

I had the cpuisol option in the boot loader config but I did set up aproper cpuset as well, so I believe it made no difference.

I added the cpuisol option after noticing how many tasks I was unableto move from the root cpuset to the system cpuset due to them beingbound per CPU and hoped (for no good reason, I admit) that cpuisolwill somehow help with the task isolation I wanted to get to test thenohz task.

That's a different obstacle to workload of the kind where nohz cpusetwould be useful, but should probably be discussed in another thread:-)

>> >> > I then ran a silly program I've written that basically eats CPU cycles> > (https://github.com/gby/cpueat) and assigned it to the nohz set and> > monitored the number of interrupts using /proc/interrupts> >> > Now, for the things I've noticed -> >> > 1. Before I turn adaptive_nohz to 1, when no task is running on the> > nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very> > few function call IPIs are seen. However, when I turn adaptive_nohz to> > 1 (but still with no task running on the CPU), the tick remains idle,> > but I get an IPI function call interrupt almost in the rate the tick> > would have been.>> Yeah I believe this is due to RCU that tries to wake up our nohz CPU.> I need to have a deeper look there.

I believe you are right with the reason for the IPI.

Before setting adaptive_nohz for the cpuset I did not get the IPI onan idle CPU. After setting it I started getting the IPI regularlyeven when the CPU was idle.

> > 2. When I run my little cpueat program on the nohz CPU, the tick does> > not actually goes off. Instead it ticks away as usual. I know it is> > the only legible task to run, since as soon as I kill it the tick> > turns off (regular nohz mode again). I've tinkered around and found> > out that what stops the tick going away is the check for rcu_pending()> > in cpuset_nohz_can_stop_tick(). It seems to always be true. When I> > removed that check experimentally and repeat the test, the tick indeed> > stops with my cpueat task running. Of course, I don't suggest this is> > the sane thing to do - I just wondered if that what stopped the tick> > going away and it seems that it is.>> Are you sure the tick never goes off?

Yes, I put debug code in the cpuset_nohz_can_stop_tick(). Every timethe function was called for that CPU rcu_pending() returned 1.

> But yeah may be there is something that constantly requires RCU grace> periods to complete in your system. I should drop the rcu_pending()> check as long as we want to stop the tick from userspace because> there we are off the RCU state machine.

I added debug code to rcu_pending() and noticed that the rcu_bh wasthe one pending each time.

I found that odd - my VM didn't even have a netowrk interfaceconfigured (except maybe for lo), let alone saw any network traffic. Ithought rcu_bh was mostly used for networking code (Paul?)

> > 3. My little cpueat program tries to fork a child process after 100k> > iteration of some CPU bound loop. It usually takes a few seconds to> > happen. The idea is to make sure that the tick resumes when nr_running> > > 1. In my case, I got a kernel panic. Since it happened with some> > debug code I added and with aforementioned experimental removal of> > rcu_pending check, I'm assuming for now it's all my fault but will> > look into verifying it further and will send panic logs if it proves> > useful.>> I got some panic too but haven't seen any for some time. I made a> lot of changes since then though so I thought the condition to trigger> it just went away.>> IIRC, it was a locking inversion against the rq lock and some other lock.> Very nice condition for a cool lockup ;)

It certainly sounds exciting :-)

Let me know if I an help test anything else.

Thanks,Gilad

--Gilad Ben-YossefChief Coffee Drinkergilad@benyossef.comIsrael Cell: +972-52-8260388US Cell: +1-973-8260388http://benyossef.com"Dance like no one is watching, love like you'll never be hurt, singlike no one is listening... but for BEEP sake you better code likeyou're going to maintain it for years!"--To unsubscribe from this list: send the line "unsubscribe linux-kernel" inthe body of a message to majordomo@vger.kernel.orgMore majordomo info at http://vger.kernel.org/majordomo-info.htmlPlease read the FAQ at http://www.tux.org/lkml/