Monday, 9 September 2013

BFS 0.441, 3.11-ck1

Apart from the usual resync to keep up with the mainline churn, there are a few additions from BFS 0.440. A number of changes dealing with wake lists as done by mainline were added that were missing from the previous code. There is a good chance that these were responsible for a large proportion of the suspend/resume issues people were having with BFS post linux 3.8. Of course I can't guarantee that all issues have been resolved, but it has been far more stable in my testing so far.

The other significant change is to check for throttled CPUs when choosing an idle CPU to move a process to, which should impact the behaviour and possibly throughput when using a scaling CPU governor, such as ondemand.

Those of you still using the evil proprietary Nvidia binary driver (as I still do) will encounter some issues and will need to use a patched pre-release driver from them if you build it yourself, until they release a new driver.

Sleep/suspend still doesn't work well on my machine. How could I debug?I'm running Archlinux with graysky binaries. When the sleep procedure is initiated, the disks power down, but the PC remains with the fans on and a blinking white cursor on the screen.

Same here. At least the machine does not hang anymore, it simply does not suspend neither using systemctl suspend nor kde (which uses pm-utils I think). Using Archlinux and graysky's repo binaries for nehalem processors.

Odd, all of my suspend resume issues have gone away, but I'm not using systemd so perhaps that's related? Try adding the no_console_suspend option to your kernel boot command and see if there's any meaningful information there.

This solution didn't worked all the time... I get random crashes on resume. So I reverted to lower HZ and tried debug the problem again.This time the backtraces look a bit more helpful:http://paste.fedoraproject.org/45174/38125077/

I'll be sticking to 3.10 for a while. And I guess quite a few other people too. Though of course I understand that maintaining two different trees isn't everyone's idea of fun. Fortunately, BFS for 3.10 works without issues for me (unless a minor kernel update breaks it.)

Dear graysky,i noticed that if i use the newest linux-ck (Sandybridge) my external USB Mouse stopped working after about 5 sec.With linux-3.11.1-1 all seams right.Anybody else noticed that? Or is it just me (Lenovo E530 i7)

Sounds like a powersaving option, but if you're saying the non-ck kernel does not exhibit this behavior and that is the only change, that does raise my level of suspicion. The real test would be for you to compile linux-ck from the AUR twice:

1) Untouched PKGBUILD, compile, install reboot. Does this happen?

2) Since the ck1 patchset is preapplied to the configs in the linux-ck package, you will need to grab the official ARCH config (or config.x86_64) and overwrite the ones included in the linux-ck.tar.gz source package. Then update the sha256sums in the PKGBUILD, and then comment out the line that patches the source with CK1 (line 120 in 3.11.1-2-ck). Now compile, install, reboot, recheck. Does it still happen? If so, it it highly likely to be something in the ck1 patchset.

Strange thinks happend, it seams that if i had the Powercable of my Laptop plugged in the mouse continue to work. But as soon i deplug the Powercable the Mouse switches of.So it might be a patchset problem. Any tipps for powersetting configuration?All the best.

Dear graysky, it all seams that it is a problem the patchset, i did all the thinks you said and still the same problem. But: before the update to 3.11.1-2 all things worked right. So maybe something wrong with that?

@Ralph UlrichThanks for the review and comment. As my understanding, sched_domains_mutex is not for one cpu, there is existed code which make sure sched_domains_mutex is held then run into for_each_cpu loop, the online code is at http://lxr.free-electrons.com/source/kernel/sched/core.c#L5398

I finally managed to at least find a workaround for the suspend crash.It seems there is a problem with the process migration in the cpu offline code.simply run this command before suspending:ps -eo pid | xargs -I'{}' taskset -pc 0 {}this sets all processes on cpu 0 so no need for migrationafter resuming this: (you have to change this to your core count this is for a 4 core i7 ):ps -eo pid | xargs -I'{}' taskset -pc 0-7 {}

Thank you for encouraging to apply these three patches! Currently running @3.11.5+BFS+BFQ with integrated Intel graphics GM45. Seems to work well and I hope we won't face any latency or power consumption related regressions. I'm looking forward to PHORONIX testing the latter..

BTW, in my kernel from the openSUSE repo the only difference is #define DEF_FREQUENCY_UP_THRESHOLD (63) -- vs. -- (80). There is no #define MAX_SAMPLING_DOWN_FACTOR (10) in BFS/ck1.

@CK...Would be interesting, though, what you say about that 63 vs. 80 difference.

ATM, I can tell that my notebooks fan is running @ higher rpm than before, what is o.k. when also running the BOINC client and not on batteries. And if I correctly interprete the PHORONIX diagrams, this should be the intended result of the patches: That we don't have idle cycles if they are not indicated by the actual (CPUs) load.

Currently running 3.11.6 with BFS, BFQ, the three aforementioned "Radeon faster" patches and the manual enablement of TICK_CPU_ACCOUNTING=y instead of IRQ...

Works well and seems to be a bit snappier when browsing with latest Firefox ESR while watching .avi videos in parallel. Also, editing >this here< is better with this setup when having the normal use case up.

Manuel, sorry, i can´t say anything about desktop interactivity and responsiveness. I only try to figure out what config-options will make the perf. drop. I am not sure with anything at moment and i will do some more benchmarks. What i can say:TICK_CPU_ACCOUNTING seems to be faster than IRQ_TIME_ACCOUNTINGCONFIG_NO_HZ_IDLE seems to be faster than CONFIG_HZ_PERIODIC

Please be patient (for a few days (or weeks)), i will post my cognitions.

I don't know how much RAM throughput affects performance, but in Graysky's compile benchmark in 3.11 BFS was clearly the winner: http://grayskysblog.blogspot.it/2013/09/bfs-v0441-vs-cfs-in-linux-3110.html

CONFIG_NO_HZ_IDLE=yCONFIG_IRQ_TIME_ACCOUNTING=y# CONFIG_COMPACTION is not setCONFIG_MIGRATION=y# CONFIG_TRANSPARENT_HUGEPAGE is not set# CONFIG_CLEANCACHE is not set# CONFIG_CMA is not set# CONFIG_KSM is not set# CONFIG_FRONTSWAP is not set# CONFIG_ZSWAP is not setCONFIG_ZSMALLOC=y# CONFIG_ZRAM is not set# CONFIG_ZRAM_DEBUG is not set

Just noticed that while I am launching a compile job with -j num_core_in_my_system, UI responsiveness is impacted (ie: mouse cursor severely micro-blocking, web browser taking a very long time to load pages).

I am wondering if this behavior is surprising and if using schedtool -I on the Xserver would be a good idea to workaround this issue.

> I am wondering if this behavior is surprising and if using schedtool -I on the Xserver would be a good idea to workaround this issue.

not surprising if all processes run in same scheduling class. You should start your compiler task with schedtool -D. In fact, I start every xterm that way by adding the following to my .profile and .bashrc (the latter sources the former):

I use for the make job the toolsched from Con, you can find it on this blog, but 1 year or so in the past. It does automatically use the schedtool for make (and other user defined apps). But if there are big compile jobs with heavy io, it could be a problem with "vm.dirty_ratio", as described by me below. If the cache is to big, the write back of data to the disk take to long and blocks the other apps.

Btw. Con, I do have a question for toolsched. If I use 'sudo make install' than I got a error, that make is not found. Ok, I can use 'su -c "make install" ', but thats not sudo. Any suggestions?Thanks.

I have been experiencing random shutdowns since some time ago. I cannot say for sure after which upgrade, but I'm positive it only occurs in the 3.11 series.My laptop simply shuts down (not a reboot!), the battery led lighting up for a second. After this, pressing the power button doesn't "wake up" the screen, which remains black and without any power (i.e. not even the backlight turns on). I press the power button once again, for a hard shutdown, and then everything works again. I cannot see a pattern -- sometimes this happens as soons as I boot into X, other times it can be after hours, even days.Booting with the vanilla ARCH kernel never elicits this behaviour, hence my suspicion that the kernel might be culprit. .confighttp://pastebin.com/qZFgkiFB

XXX=CONFIG_NO_HZ_IDLE=yCONFIG_IRQ_TIME_ACCOUNTING=y# CONFIG_COMPACTION is not setCONFIG_MIGRATION=y# CONFIG_TRANSPARENT_HUGEPAGE is not set# CONFIG_CLEANCACHE is not set# CONFIG_CMA is not set# CONFIG_KSM is not set# CONFIG_FRONTSWAP is not set# CONFIG_ZSWAP is not set# CONFIG_ZSMALLOC is not set# CONFIG_ZRAM is not set# CONFIG_ZRAM_DEBUG is not set# CONFIG_CROSS_MEMORY_ATTACH is not set# CONFIG_CONTEXT_TRACKING is not set# CONFIG_RESOURCE_COUNTERS is not set# CONFIG_RCU_FAST_NO_HZ is not set# CONFIG_USER_NS is not set

Another time again a question to you latency addicted people:Does someone of you know any knobs, hints or web-links on how to ease the pain of heavy swapping I/O?(I've ported my setup to the newer machine with Core2duo, 4GB RAM, 4GB /dev/shm & 10GB of swap on the 2nd disk. Often using the /dev/shm as a RAMDISK to decode files to and re-code them back to disk from there.) Now with 3.11.7+BFQ+BFS as of ck1. During swapping, avi video replay will stutter in frames for video and sound.

had here similiar problems (but could be different causes!) with newer kernels and BFS and BFQ, writing to some slow devices (USB Thumb drive and NFS share over my wlan) leads to a unusable system. The whole systems freeze for 10 second and more. (See it on the jumping clock applet). Using here a i7 with 8GB Ram and that should not happen on Linux ;). So I found a solution for my problem, described here http://www.novell.com/support/kb/doc.php?id=7010287. Changed the default 40 to "vm.dirty_ratio = 4" and "vm.dirty_background_ratio = 2". Now its really snappier, no more drop outs.

"Set sampling_down_factor to greater than 1 it acts as a multiplier for the scheduling interval for reevaluating load when the CPU is at its top speed due to high load. This improves performance by reducing the overhead of load evaluation and helping the CPU stay at its top speed when truly busy, rather than shifting back and forth in speed. Valid values are from 1-100000"cat /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factorecho 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

"The value is in milliseconds, and the default value is set to 6ms. Valid values are from 1 to 1000. Decreasing the value will decrease latencies at the cost of decreasing throughput, while increasing it will improve throughput, but at the cost of worsening latencies"cat /proc/sys/kernel/rr_intervalecho 4 > /proc/sys/kernel/rr_interval

and adding -fno-defer-pop to cflags and cxxflags:http://ovekarlsen.com/Blog/turning-ubuntu-12-04-into-a-professional-low-jitter-os/

if you use a Intel Core2Duo, than I would prefere X86_INTEL_PSTATE and switch from ondemand (at least on 3.11, 3.12 should be better with ondemand) to powersave (Laptop) or performance (PC) governor. I had some trouble with it, already described on this blog, with some detailed links.

Thank you very much for these hints, fo far! Needs some time for testing. :-)

Regarding the cflags & cxxflags: I'm not sure where to add the -fno-defer-pop. I've added them at the end of HOSTCFLAGS & HOSTCXXFLAGS, but reading /usr/src/linux/kernel/bounds.s this is not persistent, -fdefer-pop remains active.

Thank you, Anonymous @10 November 2013 12:25, that made the -fno-defer-pop persistent! -- And it seems to have a positive effect.

I have to say, that I already had applied the ondemand-governor patches when they came up in this blog's thread. Setting /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor to sth. > 1 doesn't help. Regarding the vm tunables, I've now:8 > /proc/sys/vm/dirty_ratio (default openSUSE 12.3 @ 20)3 > /proc/sys/vm/dirty_background_ratio (default -"- @ 10)70 > /proc/sys/vm/swappiness (default -"- @ 60)Going lower with these values I'd get more stuttering in A/V playback during swapping. But it's very hard to subjectively distinguish interactivity with each variation of these three tunables.Additionally, setting the BFQ runtime tunable "low_latency" to 0 (i.e. off instead of 1 == on == default) results in worse interactivity. Just for the records.And... then I've set CONFIG_RCU_BOOST_PRIO=94 (results in -95 in htop) letting me to manually adjust the prios of kswapd, kwin, Xorg and pulseaudio via schedtool higher or lower than RCU_BOOST_PRIO. But at the latest from then I'm currently flying blindly, so far.Adjusting rr_interval slightly higher / lower than 6 (BFS default) both decreased interactivity, on here.

How much does -fno-defer-pop persistent affect performance? Also, I saw you and Mike suggesting different tweaks and was wondering if there is any consensus about which values to use for the vm tunables on a desktop (activities like web browsing, watching movies, compiling...)

Sorry, guys'n gals, I'm quite new with linux benchmarking.I want to deliver the above questioned performance-diff in the way of the previously posted ones and installed the most recent PTS and miss a GUI or a value in the console version to select "point 3 'complex system test'" as mentioned in "Anonymous 19 October 2013 06:33" in this thread.

OMG! Sorry! It obviously was to late last night and I was too blind to find that "/usr/bin/phoronix-test-suite interactive" would lead to the selections menu. Some follow-up questions: Do you run it with X desktop up and in a console window? As root? Or booted into single user session without X?

as I already wrote, if you are on OpenSuse (like me) or any other non Ubuntu distro, you don't need the ondemand governor patches. Ubuntu only forgot to reenable the X86_INTEL_PSTATE, which you should use for your Intel Core 2 Duo. It is much better than the other governors and handles the CPU clock speed for its own. But maybe Con could say more, if and how BFS would influence there.

PS: As you can read in Phoronix comments to the benchmark test of the better ondemand code in Linux 3.12, these is only in Ubuntu distros without X86_INTEL_PSTATE.

Mmh, I'm not sure about the X86_INTEL_PSTATE, it is activated in my kernel .config since months. But how can I be sure, that it's active or see the difference, if not? The active driver is acpi-cpufreq, the actual governor is ondemand.

Regarding the consensus about the vm tunables, I have to say that I had ported my best-effort settings from -unicore- to my new system -dualcore-: Now seeing that Mike's (sysitos') settings are quite better for multicore systems.4 > /proc/sys/vm/dirty_ratio (default openSUSE 12.3 @ 20)2 > /proc/sys/vm/dirty_background_ratio (default -"- @ 10)These seem to eliminate most of the A/V hickups I had with heavy swapping.I still keep my opinion, that it'll remain a per system, per user job to adjust these for his/her own, what we don't want for linux future.

a "ls /sys/devices/system/cpu/intel_pstate" should output something like that: "max_perf_pct min_perf_pct no_turbo". Than PSTATE is active.

For my vm settings I even must lower it, because writing on an USB thumb drive with ntfs3g still blocks the system.So my /etc/sysctl.conf contains now:vm.dirty_bytes = 209715200vm.dirty_background_bytes = 104857600(Btw. you can use either bytes or ratio values there)

Hi, Mike,the INTEL_PSTATE appears to be not supported for my "Penryn" Core2Duo P8400, neither by hardware nor by the kernel pstate driver? Kernel doesn't tell anything even if booting with commandline added "intel_pstate=enable". But I only did a short, i.e. too long, web search that didn't provide results stating at which generation of Intel cpus this feature was implemented. I assume going from Core i3 to newer.

Btw., lowering the dirty_background_* and dirty_* in the vm would follow CK's approach until 3.7-ck1, setting both *_ratio to 1 in 'mm-decrease_default_dirty_ratio-1.patch', what he dropped since 3.8. (For my old unicore system this "=1" setup was unusable at that former times.)

I, personally, will wait some weeks for openSUSE 13.1 and the repos to mature. I had too many bad experiences in the past when upgrading openSUSEs shortly after release date.

maybe I miss something, but could you please describe the advantages of your circle dead-lock patch and why it should be used.As I see in the patch, you set the mutex_lock one time for the whole systems and not like Con for every CPU (which does mean for my i7 8x lock and unlock).