This patch fixes a bug located by Vegard Nossum with the aid ofkmemcheck, updated based on review comments from Nick Piggin,Ingo Molnar, and Andrew Morton.

The boot CPU runs in the context of its idle thread during boot-up.During this time, idle_cpu(0) will always return nonzero, which willfool Classic and Hierarchical RCU into deciding that a large chunk ofthe boot-up sequence is a big long quiescent state. This in turn causesRCU to prematurely end grace periods during this time.

This patch changes the rcutree.c and rcuclassic.c rcu_check_callbacks()function to ignore the idle task as a quiescent state until thesystem has started up the scheduler in rest_init(), introducing anew non-API function rcu_idle_now_means_idle() to inform RCU of thistransition. RCU maintains an internal rcu_idle_cpu_truthful variableto track this state, which is then used by rcu_check_callback() todetermine if it should believe idle_cpu().

Because this patch has the effect of disallowing RCU grace periodsduring long stretches of the boot-up sequence, this patch also introducesJosh Triplett's UP-only optimization that makes synchronize_rcu() be ano-op if num_online_cpus() returns 1. This allows boot-time code thatcalls synchronize_rcu() to proceed normally. Note, however, that RCUcallbacks registered by call_rcu() will likely queue up until later inthe boot sequence. Although rcuclassic and rcutree can also use thissame optimization after boot completes, rcupreempt must restrict itsuse of this optimization to the portion of the boot sequence before thescheduler starts up, given that an rcupreempt RCU read-side criticalsection may be preeempted.

In addition, this patch takes Nick Piggin's suggestion to make thesystem_state global variable be __read_mostly.

Changes since v3:

o WARN_ON(nr_context_switches() > 0) to verify that RCU switches out of boot-time mode before the first context switch, as suggested by Nick Piggin.Changes since v2:

o Created rcu_blocking_is_gp() internal-to-RCU API that determines whether a call to synchronize_rcu() is itself a grace period.

o The definition of rcu_blocking_is_gp() for rcuclassic and rcutree checks to see if but a single CPU is online.

o The definition of rcu_blocking_is_gp() for rcupreempt checks to see both if but a single CPU is online and if the system is still in early boot.

This allows rcupreempt to again work correctly if running on a single CPU after booting is complete.

o Added check to rcupreempt's synchronize_sched() for there being but one online CPU.

Tested all three variants both SMP and !SMP, booted fine, passed a shortrcutorture test on both x86 and Power.

+/*+ * A context switch is a grace period for rcupreempt synchronize_rcu()+ * only during early boot, before the scheduler has been initialized.+ * So, how the heck do we get a context switch? Well, if the caller+ * invokes synchronize_rcu(), they are willing to accept a context+ * switch, so we simply pretend that one happened.+ *+ * After boot, there might be a blocked or preempted task in an RCU+ * read-side critical section, so we cannot then take the fastpath.+ */+static inline int rcu_blocking_is_gp(void)+{+ return num_online_cpus() == 1 && !rcu_idle_cpu_truthful;+}+ #endif /* __LINUX_RCUPREEMPT_H */diff --git a/include/linux/rcutree.h b/include/linux/rcutree.hindex d4368b7..a722fb6 100644--- a/include/linux/rcutree.h+++ b/include/linux/rcutree.h@@ -326,4 +326,10 @@ static inline void rcu_exit_nohz(void) } #endif /* CONFIG_NO_HZ */