Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?

Paul E. McKenney wrote:
On Wed, Jan 30, 2008 at 09:18:49AM +0100, Wolfgang Grandegger wrote:
Paul E. McKenney wrote:
On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
Luotao Fu wrote:
Hi,

Wolfgang Grandegger wrote:
..........
Do you still get high latencies with:

CONFIG_PREEMPT_RCU_BOOST=y
CONFIG_RCU_TRACE=y
CONFIG_NO_HZ is not set

With this setting I have not yet realized latencies > 150us. Could you
please give it a try? If I change one of the parameters above, latency
increases in short time.
I played through some combination of the RCU options and can back your
observation this time: With the rcu Tracer or the priority boost turned
off I also could measure reliably extraordinory high latencies. If they
are both turned on, no high latencies could be measured. Turning on the
dynamic ticker however doesn't seem to cause high latencies during my
test runs. Seemed like an rcu issue here.
I'm just making a long test run on my TQM5200 module with my good
settings. After more than 4.5 hours under load, cyclictest shows a
maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.

Further such results only appear if the target board is booted with
nfsrootfs. (As I already have mentioned several times before), which
leads my suspection to rcu usage in nfs implementation. In this case
this problem might even be platformindependent. I'd have to do some
tests on one of our arm boards later to test this. Since there're no
reports like this for other architecture as powerpc till now, I doubt
quite if this is verifiable.
It's also my suspicion that the high latencies are related to the RCU
usage in the network layer, where it's heavily used. What is really
wired is that switching off CONFIG_RCU_TRACE has a negative impact on
the latency. As I see it, it just adds some trace points, but I might
have missed something.
I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
would have low latencies rather than high latencies. So I am quite
surprised by your result. I will dig into this more.
Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
"CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.

OK, thank you for the confirmation.

The large latencies were from cyclictest, correct? Did other tests
also show these latencies? In either case, could you please send the
exact command line you used to run the test?

Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
far, I have not done other tests. Any recommendation?
As no-rt load I used "while ls; do ls /bin; done" in one telnet window
and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
in another. But already "while ls; do ls /bin; done" is enough to
trigger the high latencies quickly. Note also, that I work on a root
files-ystem mounted via NFS resulting in a lot of network traffic and
utilization.

Thanks,

Wolfgang.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/