A week has passed since the last major update to BFS and -ck was posted, allowing me to concentrate on receiving and responding to any bug reports. As it turns out, there were very few apart from the recurring local_softirq_pending warning/stalls. This is nice because it means MuQSS is mostly ~stable now. Mainline has even had more "stable" releases in the same time as MuQSS for 4.8, moving to 4.8.6 in the interim.

In this version I've added aggressive handling of pending softirqs in the hope the warnings and stalls all go away. The true reason the handling of softirqs are being dropped still escapes me but is likely related to the fact that MuQSS does a lot of lockless rescheduling across CPUs to decrease overhead but this does not give guarantees that locking would.

Additionally, I've added a number of APIs to the kernel to do specified millisecond schedule timeouts which use the highres timers which are mandatory now for MuQSS. The reason for doing this is there are many timeouts in the kernel that specify values below 10ms and the timer resolution at 100Hz only guarantees timeouts under 20ms.

I've also added a code sweep across the entire kernel looking for timeout calls under 50ms and use the new interface in its place. Additionally there are numerous places where schedule_timeout(1) are used in the kernel where a "minimum timeout" is expected, yet this is entirely Hz dependent, again being up to 20ms in duration. I've replaced all these with a 1ms timeout, emulating what would happen on a 1000Hz kernel, but without the overhead of running the higher Hz kernel. I'm not entirely sure this will equate to any real world improvements but the fact it's used in things like audio drivers worries me that it might.

Finally I've replaced the standard msleep call from userspace to use highres timers, in case there are userspace applications that expects msleep to actually give some kind of sleep that resembles what's asked of it, instead of something Hz limited, in case this is leading to slowdowns in userspace due to assumptions on the userspace coders' part. Calls to msleep() from userspace now give 100us accuracy at 100Hz instead of 20ms.

All these timing changes add overhead since they're trying to emulate the timing accuracy of running at 1000Hz but in a latency-focused scheduler I believe they're appropriate, and they do not incur the overhead that actually changing Hz would incur. Additionally they add accuracy to timers and timeouts that 1000Hz does not afford.

In the -ck tarball of broken-out patches, I've kept these timer changes separate to allow the muqss scheduler to be applied by itself should they prove problematic, and they will make merging with future kernels easier.

What do you mean by "hangs up?" Do you mean it hangs completely needing hard reset? Also does JACK run realtime audio? If so, then there is no way to prevent it from hanging up the machine should something go wrong.

I just restarted the machine and tried again, osu! audio now appear to be glitched: a part of the audio got repeated until the game was quit. After it was quit, it turned into 'Z' state with a child (which is itself) in 'R' state. Neither the parent nor the child can be killed with SIGKILL

Would the audio be trying to get realtime priority somehow? If so it might be getting SCHED_ISO instead. Try disabling sched iso by setting its percentage to zero to see if that helps?echo 0 > /proc/sys/kernel/iso_cpu

I managed to run osu! without system lockup by luck :P. But after a few second osu! audio start to get repeated again and again, if left like that then osu! will cause the whole system to hang after a few second, if quit, osu! turns into a zombie process with a child, which is itself stuck in "Running" state and could not the killed by any means (even SIGKILL).

Must be some magic combination of that software and your hardware/drivers that recreate this issue since I can't reproduce it anywhere here... Sorry I have no other leads. Try recording top output in batch mode until it goes wrong to see what is happening there.

I hard crash everytime I open osu! now, but I have been able to delay these hangs by changing the sample format, sample rate, resample method, and realtime scheduling with pulseaudio.

It hard crashes on all settings, without fail, when the sound distorts on note clicks for this song: https://osu.ppy.sh/s/157896. Some songs play okay on default settings, but crash when the map becomes complex.

Also, the first time i try to record the log, the system stutter, mouse pointer jumps around, dmesg doesn't show anything unusual, but rebooting stuck forever at remount RO state, so sysrq is used to shutdown. Here is the log of that time: http://pastebin.com/EqdwGPuZ

Very weird. The only thing I can see unusual going on in that toplog is a realtek usb card reader that appears to be stuck... which unless you're running off it somehow during this workload is a completely unrelated issue.

Only other thing I can suggest is to try muqss without the timer changes and see if the problem is in the main muqss patch or the timer changes. They're split out for -ck and muqss alone is here:MuQSS135

It seems that the hrtimer changes subtly broke task accounting: with 0.135 I see individual processes (esp. burners like longer-running C++ compilers) accounted with >100% CPU time (usually ~105-110%), which is clearly impossible.

Actually the >100% predates 135. It happens only on tasks that fork and do work along the way, like a compile, since both forked processes/threads are being accounted while they're running concurrently which does actually add up to 100%. You won't be able to reproduce it on long running single processes/threads.

@ck:Regarding the 4.7 kernel 135 MuQSS patch and if you're willing to support it for another little while: I've needed to change one hunk for kernel/sched/idle.c to eliminate a build failure:@@ -213,7 +219,10 @@ static void cpu_idle_loop(void)

where -----+ if (unlikely(softirq_pending(cpu)))should be -----+ if (unlikely(softirq_pending(smp_processor_id())))

if I understood this correctly.

And additionally for kernels from 4.7.7 upwards the hunk for kernel/sched/sched.h should be taken from the 4.8 MuQSS patch.Then, kernel compiles and works fine :-) Thank you Con!(I hope the above cited patch lines are understandable.)

@ck:Just want to leave a positive experience, that with this v0.135 my previously reported interactivity problems with 100Hz gone down below the 1000Hz level. Meaning, it's o.k. now for my machine, on my old 4.7.10 kernel (that I've run at 1000Hz all time before).BR, Manuel Krause

I've run the usual benchmarks. I used my desktop this time (Intel Haswell 4770k). The fan on my laptop is starting to fail. So no comparison with older MuQSS, sorry.

I've put the results in a new spreadsheet:https://docs.google.com/spreadsheets/d/163U3H-gnVeGopMrHiJLeEY1b7XlvND2yoceKbOvQRm4/edit?usp=sharing

Nothing new in terms of throughput. MuQSS is roughly on par with CFS, except under partial load (make j2 and j4).

I also ran interbench -L 8 on CFS@300Hz, CFS@1000Hz and MuQSS135@100Hz, with intel_pstate+powersave frequency governor.It doesn't show differences between all the kernels, so I wonder if I did things right. I used interbench from your git repo.

Thanks. The first time you run interbench it benchmarks the speed of your CPU and uses it every time after that. If you run it the first time without the performance governor, it will create useless results from then after. If you add -b on startup it will perform the benchmark again.

Thanks, I forgot about that.I've re-done the tests. The first time I used performance governor with CFS@300Hz, and then for every other test powersave (the default).The numbers of loops per ms goes from ~800,000 to ~4,000,000, that's better.The results are more like the ones expected. I can't look at them in detail right now, but they are on the spreadsheet.

I had a quick look at the interbench results. I have the impression that overall:- CFS@1000Hz is not that bad- the difference between Interactive values is rather smallBut you understand theses results better than I do. Am I correct ?

I also ran the same tests you did with the Phoronix test suite.The results are here:http://openbenchmarking.org/result/1611092-LO-CFSVSMUQS19

The results shows the same things as the ones you got with MuQSS112.I'll think I'll redo the pgbench test with 'in buffer' scaling instead of the default 'on disk' because it seems the SSD is a bottleneck in the last test.

I can confirm that with my old hardware periodic timer ticks result in a lower latency with 100 HZ. Performance is better. I can't confirm that periodic timer ticks are only for reducing power consume.

You might be right, but fact is that my Core2 Duo machine consumes less CPU (about 10%!) whith periodic timers when using vapoursynth video filters. I compared this with different kernels (your ck and liquorix) - it's always the same!

But an old desktop system on a power cable does not need to spare little bit of electric power but wants to have a little more throughput: Having an old y2009 Core2Duo I think oftimeout>1ms rr_interval>6none CONFIG_NO_HZ_IDLE???

If you don't care at all about power savings then sure periodic timer tick is fine compared to tickless idle. You should also set your cpu frequency governor to performance as well then since no matter how good the governor is, it will never work as well as simply setting it to performance. The only reason to set it to anything else is to save power, reduce heat, and decrease fan noise. They do still use less power when idle with the performance governor, but not as much as with cpu frequency scaling enabled.

My laptop didn't completely turned off (keyboard and screen light still on, screen blanked, laptop get hot) after suspend. I was still able to wake the system up using the power button, and was able to retrieve this log: http://pastebin.com/zJifGCWm

I'm using your patches since a lot of time to give some life to an old atom Z520 with great satisfaction.

Since MuQSS (now i'm using version 0.135 with HZ 100) i'm experiencing random panics at boot.When the boot process doesn't hangs, the system seems to work very well (except hybernation - but i'm not sure it is related)

Thanks for looking into it. During the night i have tried the same build with patches 0001,0011,0012,0013 into the laptop with Atom Z520 and the situation is even worst since it boots only 1/10 with panics about "update_process_times.. tick_periodic.. tick_handle_periodic.. local_apic_timer.. smp_apic_timer... EIP: scheduler_tick"

Boot again with 4.7.6 + bfs works still great without any issue.I know that look back seems a bad way to go ahead, but... do you plan prepare a set of bfs patches also for kernel 4.8.x?

To run it in muqss, is it: schedtool -I -e virtualbox ?VirtualBox worked with bfs for years. Have been following all the blogs since this one to current, and read up on stubs but have found scant information on creating stubs. Have read several wiki's on cgroups (incl Arch & Redhat), and scheduling and looking at the example for amarok, is best I can comprehend of it.I'm currently arch 4.8.11-1 stock since vbox quit working in ck.Thank you.