Re: rcu self-detected stall messages on OMAP3, 4 boards
[Posted September 26, 2012 by corbet]
| From: |
| "Paul E. McKenney" <paulmck-AT-linux.vnet.ibm.com> |
| To: |
| Paul Walmsley <paul-AT-pwsan.com> |
| Subject: |
| Re: rcu self-detected stall messages on OMAP3, 4 boards |
| Date: |
| Sat, 22 Sep 2012 16:17:33 -0700 |
| Message-ID: |
| <20120922231733.GI2934@linux.vnet.ibm.com> |
| Cc: |
| "Bruce, Becky" <bbruce-AT-ti.com>,
"Paul E. McKenney" <paul.mckenney-AT-linaro.org>,
"<linux-kernel-AT-vger.kernel.org>" <linux-kernel-AT-vger.kernel.org>,
"<linux-omap-AT-vger.kernel.org>" <linux-omap-AT-vger.kernel.org>,
"<linux-arm-kernel-AT-lists.infradead.org>"
<linux-arm-kernel-AT-lists.infradead.org>,
"Hilman, Kevin" <khilman-AT-ti.com>,
"Shilimkar, Santosh" <santosh.shilimkar-AT-ti.com>,
"Hunter, Jon" <jon-hunter-AT-ti.com>,
"<snijsure-AT-grid-net.com>" <snijsure-AT-grid-net.com> |
| Archive‑link: | |
Article |
On Sat, Sep 22, 2012 at 10:20:19PM +0000, Paul Walmsley wrote:
> Hi Paul
>
> On Sat, 22 Sep 2012, Paul E. McKenney wrote:
>
> > Strangely enough, I believe that I have inadvertently fixed this in
> > my -rcu tree:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/next
> >
> > Nevertheless, if you get a chance to try it, I would be interested to
> > hear if my guess is correct.
>
> Yes, good news: the stall warnings go away with that branch.
Very good!
> > The trick is that a kthread drives the grace period in -rcu, regardless
> > of whether or not there are callbacks.
>
> This is "rcu: Move quiescent-state forcing into kthread" ?
Yep, plus the preceding commits moving grace-period initialization and
cleanup into that same kthread. This was motivated by a bug report
last February complaining about 200-microsecond latency spikes from
RCU grace-period initialization. On systems with 4096 CPUs.
Real-time response. It is far bigger than I thought. ;-)
> Added some debugging into rcu_gp_kthread() after that commit and can
> confirm that the quiescent-state forcing loop does start a few times when
> there are zero callbacks pending (modulo any races in my measurement
> code).
Cool, thank you! Assuming it works, that indicates that there is long-term
value to the fix for this problem. On larger systems, extra grace periods
are not what you want, as their expense increases with the number of CPUs.
> > However, the backport would not be something that -stable would be happy
> > with, so I will be putting together a fix for mainline. This thing
> > has been in the kernel since about 2004, not sure why you didn't hit
> > it earlier.
>
> One other data point in that regard - noticed the warnings don't appear
> when the board is booted with:
>
> commit 4fa3b6cb1bc8c14b81b4c8ffdfd3f2500a7e9367
> Author: Paul E. McKenney <paul.mckenney@linaro.org>
> Date: Tue Jun 5 15:53:53 2012 -0700
>
> rcu: Fix qlen_lazy breakage
You lost me on this one. This is already in mainline, so if you were
using (say) 3.6-rc6, you would already have this commit applied.
Thanx, Paul
