Re: irqbalance mandatory on SMP kernels?

Theodore Ts'o wrote:
On Mon, Apr 17, 2006 at 11:01:33AM -0400, Lee Revell wrote:

There is an in-kernel IRQ balancer. Redhat just choose to turn it
off, and do it in userspace instead. You can re-enable it if you
compile your own kernel.

Round-robin IRQ balancing is inefficient anyway. You'd get better cache
utilization letting one CPU take them all.


IIRC, Van Jacobsen at his Linux.conf.au presentation made a pretty
strong argument that irq balancing was never a good idea, describing
them as a George Bush-like policy. "Ooh, interrupts are hurting one
CPU --- let's hurt them **all** and trash everybody's cache!"

Nothing nowadays does round-robin of interrupts, either the in-kernel
or userspace balancers ... but we do migrate them occasionally (in the
order of 1s or so)

Which brings up an interesting question --- why do we have an IRQ
balancer in the kernel at all? Maybe the scheduler's load balancer
should take this into account so that processes that have the
misfortune of getting assigned to the wrong CPU don't get hurt too
badly (or maybe if we have enough cores/CPU's we can afford to
dedicate one or two CPU's to doing nothing but handling interrupts);
but spreading IRQ's across all of the CPU's doesn't seem like it's
ever the right answer.

Because *something* has to be balanced, and moving processes around
is expensive too. Personally I find the process model cleaner, but
maybe it's less efficient - you'd also add extra overhead for accounting
to each interrupt, which we don't do now.

I'm not claiming that moving irqs is worse or better than moving
processes - just that it's a tradeoff, both suck. Perhaps the
real answer is that we shouldn't be getting that many interrupts
anyway - technologies like NAPI and simpler device interrupt collation
should reduce the load, and most of the work should be done in the
back-ends anyhow (though those are often locally bonded to the CPU
the irq arrived on).

M.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: [parisc-linux] [patch 15/23] Add cmpxchg_local to parisc
    ... non-SMP-safe counter that protects updates against interrupts. ... could be vastely used in the kernel. ... "Local atomic operations only guarantee variable modification atomicity ... that only one CPU writes to the local_t data. ...
    (Linux-Kernel)
  • Re: Interrupt using all the CPU
    ... In cases where interrupts reach 25%, the load is very easily noticable, ... is false, you wouldn't be able to log in to the box, the kernel itself ... would have trouble doing something, ... Display either cpu or io statistics. ...
    (freebsd-questions)
  • Re: Although CONFIG_IRQBALANCE is enabled IRQs dont seem to be balanced very well
    ... Probably kernel is optimizing this even with CONFIG option enabled. ... For a UDP bandwidth lower than 23 Mbps the interrupt rate at the receiver interface was lower than 2000 interrupts per second. ... In this case all interrupts were distributed to CPU 0. ...
    (Linux-Kernel)
  • Re: Interrupt using all the CPU
    ... Yes i suspected a bug when i first saw this because of what you say. ... 100% CPU usage should be noticed. ... In cases where interrupts reach 25%, the load is very easily noticable, ... is false, you wouldn't be able to log in to the box, the kernel itself would have trouble doing something, but interrupt handling. ...
    (freebsd-questions)
  • Re: 2.6.27-rc6 xen soft lockup
    ... device interrupts are bound to vcpu 0 and there's nothing much else to ... During the time of the lockup the cpu load, as measured with cacti, was about 4% ... So I would say mostly idle ... # Device Drivers ...
    (Linux-Kernel)