A development blog of what Con Kolivas is doing with code at the moment with the emphasis on linux kernel, BFS and -ck.

Friday, 4 July 2014

BFS 0.449

Hot on the heels of the BFS448 release, I was doing some experimenting for some ideas I had (nothing productive so far) when I discovered the long-standing "CPU locality" code which determines the relationship between CPUs (eg. if they're SMT siblings or separate physical CPUs etc) was broken. So I've fixed the code that determines that, along with printing out what BFS believes to be the relationship (called locality) in dmesg on startup. An example output from a 2 thread, 2 core CPU would be:

Me? I just omitted/ reverted the above mentioned two patches from Alfred Chen and then patched 3.15.3 with CK's incremental BFS one: 448-449. The rest of AC's patches stayed in my kernel setup and do work.Sorry, that I can't say anything to ath9k. Not in use on here.

Don't use the "Use prefered raid6 gen function", it's hardcore to optimize on my cpu(core2) to reduce kernel boot time. PHC is another well-know patch to under-volt cpu, but not everyone use it these days. None of them is bfs related.

For other patches I work on bfs, I will sync them up with 0449 and post back soon.

But, I'd like to ask: What do I need?: So far, I had both SCHED_MC and SCHED_SMT enabled in my .config. After little research I found out that my Intel P8400 Core 2 Duo doesn't ship Hyper-Threading, while /proc/cpuinfo does very well export the "ht" flag. And Con's new LOCALITY code exports only "LOCALITY CPU 0 to 1: 2". Somewhere I've read that keeping the SMT way alive in ht-flagged systems could benefit code execution that relies on that extension, even if not supported by the hardware. I'm in doubt in what of that all may be a fairy-tale or what setting is the only "right" one.

@ManuelWell, it is the first time I found out ht flag for Core2Duo cpus. I guess that means cpu can support ht but physically it just has one ht per core. CONFIG_SCHED_SMT should be only used in scheduler code, if you enable it while you don't have it, it will waste cpu instructions to consider it.

@ck There are 3 more patches for bfs, pls review if you have time and comments will be welcome

#1 [BFS] Sync up try_to_wake_up* functions https://bitbucket.org/alfredchen/linux-gc/commits/5d55545095a235728a60666198df54f9566bfc19?at=linux-3.15.y-gc-- Sync try_to_wake_up* functions with mainline, so it can make the sync up work easier from release to release

#2 [BFS] Add WARN_ON_ONCE if rq is dereferenced in wait_task_inactive() https://bitbucket.org/alfredchen/linux-gc/commits/caea825efc73d77e66d9f022e902683c1f2ac02f?at=linux-3.15.y-gc-- Add WARN_ON_ONCE, if it doesn't show up in 1~2 release, the !rq check can be safe remove

#3 [BFS] Remove runqueue wake_list. https://bitbucket.org/alfredchen/linux-gc/commits/0e4d67b21a28aaed10cd9836e5487bb7717016ba?at=linux-3.15.y-gc-- The runqueue wake_list is used in ttwu_queue_remote(). In BFS,tasks are just put into grq, so wake_list is no needed.

1.The try_to_wake_up functions have the extra function in mainline for dealing with other flags which BFS does not use so there is no point adding the extra function. 2. While the idea of removing the check for rq over time is sound, the problem is it will be different on different architectures so there is no guarantee if the warning doesn't show up on x86* that it is safe to remove.3. A valid cleanup.

Also note to others using Alfred's patches, I still disagree on the other patch [2] in his previous list. I will try to put pending accepted patches into my bfs directory.

@ckThanks for the review.1. I am not adding the extra function, mainly just sync-up the function name changes like ttwu_post_activation -> ttwu_do_wakeup. As for the flags, I will check if they are used by other functions and try to clean them up.2. After expand the percpu macro, basically it is var_address + per_cpu_offset, despite the different ways to determining per_cpu_offset for different arch, it is very unlikely it returns an zero pointer.

I have 2 more minor changes these two weeks, will post out after testing.

What's the percpu macro in reference to? Do you mean the optimisation you removed in your first set of patches? Checking to see if SMT siblings exist is CPU architecture dependent action and not one based on hard code. Running an SMT kernel on an AMD cpu will only find no siblings once it is running.

Then how are you going to know if there's a codepath during the unlocked IRQs context switch on other architectures that doesn't dereference rq briefly enough for it to be NULL, which is all that is testing for?

Let's explore it a little detaila. expand the task_rq() macro, task_rq(p)=>cpu_rq(task_cpu(p)),task_cpu(p)=>task_thread_info(p)->cpu,task_thread_info(task)=>((struct thread_info *)(task)->stack);cpu_rq() is the final step and it is a per_cpu variable which basiclly var_address + per_cpu_offset. So rq is looked up by cpu index in task thread info.b. If we consider all the possibility that we call task_rq() without any lock or irq, that turns out it may crash the kernel or rq could be any value

thanks for the new version of BFS. I already know, that the problem with hibernate on ath9k still persist, but you should know, that the problem is on 3.15 more worse. Now even the suspend doesn't work anymore without a kernel hung. No problem so far with CFS.

Ok, can't verify this. Tested it only with the actual 3.12.24 kernel (btw, even the best kernel for my Dell Vostro 3750 ;) ) with BFS, BFQ and exFAT and here all is working fine (hibernate and suspend). My tests with 3.14 were already written down here, without the ath9k module all is working. And with CFS all is working. So maybe the loading of the ath9k does trigger the bug, maybe some new network algorithm, but I am not an expert ;(. My test with 3.15 shows, that suspend leads to an kernel hung nearly instantly, if there are some network monitors are running. No network activity and there is no hung. But hibernate always leads to a hung. So are my studies.

Btw. pf, which kernel do you support with your pf-kernel, some long time kernels too or as the zen-kernel (which I use at the moment) only the newest one? Thanks.

Good investigating so far guys. One of PF's screenshots shows a possible recursive lock around the sched_init_smp function and that did change in mainline between 3.12 and 3.13 so perhaps those changes were not portable to BFS.

tried the sisrevert patch against vanilla 3.15.5 and bfs449, but my system still freezes after suspend.

It's odd that with 3.15+bfs449 I can suspend and resume _once_ without any issues, but do another suspend+resume after that and system freezes. with 3.13 (and 3.14 if I remember correctly) system would freeze right after the first suspend+resume.

So switched my minipci WLAN card again. Hopefully the last one, because its a nightmare to do this on a Dell Vostro 3750. With other words, can't test anymore the hibernate bug with the ath9k wifi card and bfs.

Btw. Using now an Intel 7260 wifi card with bluetooth, good/excelent performance on 2,4GHz and even in 5GHz area its great and this with only 2 antennas. (Wasn't aware, that this does work at all).And hibernate/suspend does work with BFS ;)

PS: But remember, there are different models with different specs on the market!

I am just testing these patches:bfs449-fix_get_nohz_tt.patchbfs449-fix_idle_warn.patchbfs449-fixnr_cpu_ids.patchbfs449-pstate_scaling.patchbfs449-remove_wlist.patchbfs449-sisrevert.patchbfs449-sis-test.patch

Suspend to ram is OK, Hibernate I dont use and have disabled in .config . My system as ever: MacMini Core2Duo

Thanks guys. @PF I've been staring at the lock debugging photos you took as they seem the key to finding this issue though I'm still stumped as to what it is about that particular module that makes it happen. I'm still looking whenever I have time and we'll nail this sooner or later..

Hi Con,tested only with the new patch. For me it seems to be better, but the CPU hung still remains (for suspend and hibernate). But now after hibernating the WLAN does reconnect, was not the case without it. Dmesg shows multiple times, that CPU0 hung for min. 23 sec, top shows a kernel threads eating the CPU ;) and goes to values of 20 and more. System isn't usable anymore.But fine to see, that you are working on it. Thanks again.