Re: Kernel panic: Route cache, RCU, possibly FIB trie.

Jesper Dangaard Brouer a écrit :

On Tue, 21 Mar 2006, Robert Olsson wrote:

Jesper Dangaard Brouer writes:

> I have tried to track down the problem, and I think I have narrowed it
> a bit down. My theory is that it is related to the route cache
> (ip_dst_cache) or FIB, which cannot dealloacate route cache slab
> elements (maybe RCU related). (I have seen my route cache increase to
> around 520k entries using rtstat, before dying).
>
> I'm using the FIB trie system/algorithm (CONFIG_IP_FIB_TRIE). Think
> that the error might be cause by the "fib_trie" code. See the syslog,
> output below.

> Syslog#1 (indicating a problem with the fib trie)
> --------
> Mar 20 18:00:04 hostname kernel: Debug: sleeping function called from invalid context at mm/slab.c:2472
> Mar 20 18:00:04 hostname kernel: in_atomic():1, irqs_disabled():0
> Mar 20 18:00:04 hostname kernel: [<c0103d9f>] dump_stack+0x1e/0x22
> Mar 20 18:00:04 hostname kernel: [<c011cbe1>] __might_sleep+0xa6/0xae
> Mar 20 18:00:04 hostname kernel: [<c014f3e9>] __kmalloc+0xd9/0xf3
> Mar 20 18:00:04 hostname kernel: [<c014f5a4>] kzalloc+0x23/0x50
> Mar 20 18:00:04 hostname kernel: [<c030ecd1>] tnode_alloc+0x3c/0x82
> Mar 20 18:00:04 hostname kernel: [<c030edf6>] tnode_new+0x26/0x91
> Mar 20 18:00:04 hostname kernel: [<c030f757>] halve+0x43/0x31d
> Mar 20 18:00:04 hostname kernel: [<c030f090>] resize+0x118/0x27e

Hello!

Out of memory?
One of the crashed was caused by out of memory, but all the memory was allocated through slab. More specifically to ip_dst_cache.

Running BGP with full routing?
No, running OSPF with around 760 subnets.

And large number of flows.
Yes, very large number of flows.

Whats your normal number of entries route cache?
On this machine, rigth now, between 14000 to 60000 entries in the route cache. On other machines, rigth now, I have a max of 151560 entries.

And how much memory do you have?
On this machine 1Gb memory (and 4 others), most of the machines have 2Gb.


From your report problems seems to related to flushing either rt_cache_flush
or fib_flush (before there was dev_close()?) so all associated entries should
freed. All the entries are freed via RCU which due to the deferred delete
can give a very high transient memory pressure. If we believe it's memory problem
we can try something out...

There is definitly high memory pressure on this machine!
Slab memory usage, range from 39Mb to 205Mb (at the moment on the production servers).


Did you tried 2.6.16 ?

It contains changes in kernel/rcupdate.c so that not too many RCU elems are queued (force_quiescent_state()). So in the case a rt_cache_flush is done, you have the guarantee all entries are not pushed into rcu at once.

Eric
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: Kernel panic: Route cache, RCU, possibly FIB trie.
    ... My theory is that it is related to the route cache ... One of the crashed was caused by out of memory, but all the memory was allocated through slab. ... On this machine, rigth now, between 14000 to 60000 entries in the route cache. ...
    (Linux-Kernel)
  • Re: [PATCH, RFC] RCU : OOM avoidance and lower latency
    ... >>>The underlying problem of RCU and memory usage could be solved more ... >>>safely by making sure that the sleeping memory allocator path always ... >>not yet freed due to a grace period not having elapsed. ... >>If my reading of the IP route cache code is correct, ...
    (Linux-Kernel)
  • Kernel panic: Route cache, RCU, possibly FIB trie.
    ... My theory is that it is related to the route cache ... around 520k entries using rtstat, ... And how much memory do ... From your report problems seems to related to flushing either rt_cache_flush ...
    (Linux-Kernel)
  • Re: [PATCH, RFC] RCU : OOM avoidance and lower latency
    ... This might sounds as a violation of RCU rules, but seems quite reasonable. ... The underlying problem of RCU and memory usage could be solved more ... first time and when the RCU grace period ends, ... It still sees too many entries above), ...
    (Linux-Kernel)
  • Re: [PATCH, RFC] RCU : OOM avoidance and lower latency
    ... >> of RCU rules, but seems quite reasonable. ... > safely by making sure that the sleeping memory allocator path always ... If my reading of the IP route cache code is correct, ... it would be possible to end up in a steady state with most of the entries ...
    (Linux-Kernel)