Re: 2.6.24-rt1 IRQ routing anomaly

Steven Rostedt wrote:
[CC'd Thomas and Jon]

Thomas, Jon, looks like the someone has the funny interrupt controller.

On Thu, 21 Feb 2008, Mark Hounschell wrote:

According to /proc/interrupts, every interrupt received by eth1 is also
being received by the sound card EMU10K1. The problem showed itself
first with this. The sound system was quiet BTW.

It does not happen with 2.6.24 vanilla.

kernel: irq 19: nobody cared (try booting with the "irqpoll" option)
kernel: Pid: 1832, comm: IRQ-19 Not tainted 2.6.24.2-crt #2
kernel: [<c013d6da>] __report_bad_irq+0x36/0x75
kernel: [<c013d910>] note_interrupt+0x1f7/0x227
kernel: [<c013ce85>] thread_simple_irq+0x61/0x74
kernel: [<c013d455>] do_irqd+0x0/0x22f
kernel: [<c013d507>] do_irqd+0xb2/0x22f
kernel: [<c013d455>] do_irqd+0x0/0x22f
kernel: [<c012b137>] kthread+0x38/0x5d
kernel: [<c012b0ff>] kthread+0x0/0x5d
kernel: [<c0104c13>] kernel_thread_helper+0x7/0x10
kernel: =======================
kernel: ---------------------------
kernel: | preempt count: 00000001 ]
kernel: | 1-level deep critical section nesting:
kernel: ----------------------------------------
kernel: .. [<c02b03b3>] .... __spin_lock_irq+0xe/0x1e
kernel: .....[<00000000>] .. ( <= _stext+0x3feff000/0x14)
kernel:
kernel: handlers:
kernel: [<f4d16544>] (snd_emu10k1_interrupt+0x0/0x42c [snd_emu10k1])
kernel: turning off IO-APIC fast mode.
kernel: irq 19: nobody cared (try booting with the "irqpoll" option)
kernel: Pid: 1832, comm: IRQ-19 Not tainted 2.6.24.2-crt #2
kernel: [<c013d6da>] __report_bad_irq+0x36/0x75
kernel: [<c013d910>] note_interrupt+0x1f7/0x227
kernel: [<c013ce85>] thread_simple_irq+0x61/0x74
kernel: [<c013d455>] do_irqd+0x0/0x22f
kernel: [<c013d507>] do_irqd+0xb2/0x22f
kernel: [<c013d455>] do_irqd+0x0/0x22f
kernel: [<c012b137>] kthread+0x38/0x5d
kernel: [<c012b0ff>] kthread+0x0/0x5d
kernel: [<c0104c13>] kernel_thread_helper+0x7/0x10
kernel: =======================
kernel: ---------------------------
kernel: | preempt count: 00000001 ]
kernel: | 1-level deep critical section nesting:
kernel: ----------------------------------------
kernel: .. [<c02b03b3>] .... __spin_lock_irq+0xe/0x1e
kernel: .....[<00000000>] .. ( <= _stext+0x3feff000/0x14)
kernel:
kernel: handlers:
kernel: [<f4d16544>] (snd_emu10k1_interrupt+0x0/0x42c [snd_emu10k1])

Looking at /proc/interrupts I could see the the EMU10K1 interrupt was
going to town. I was busy busy on eth1 at the time.

So a simple externall ping test with a quiet system at run level-3 revealed:

# lspci cat before.ping
CPU0 CPU1
0: 85 0 IO-APIC-edge timer
1: 396 420 IO-APIC-edge i8042
3: 4 2 IO-APIC-edge
4: 5 1 IO-APIC-edge
6: 1 4 IO-APIC-edge floppy
7: 0 0 IO-APIC-edge parport0
8: 2 0 IO-APIC-edge rtc
9: 0 1 IO-APIC-fasteoi acpi
12: 21 84 IO-APIC-edge i8042
14: 8457 8179 IO-APIC-edge libata
15: 1016 1519 IO-APIC-edge libata
16: 60 60 IO-APIC-fasteoi aic7xxx
17: 113 96 IO-APIC-fasteoi eth1
18: 44 47 IO-APIC-fasteoi
19: 99 114 IO-APIC-fasteoi EMU10K1
NMI: 0 0 Non-maskable interrupts
LOC: 93895 94157 Local timer interrupts
RES: 8831 8188 Rescheduling interrupts
CAL: 4176 5267 function call interrupts
TLB: 271 235 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0


Then from an external machine: ping -c10 10.10.10.200


# cat after.ping
CPU0 CPU1
0: 85 0 IO-APIC-edge timer
1: 464 432 IO-APIC-edge i8042
3: 4 2 IO-APIC-edge
4: 5 1 IO-APIC-edge
6: 1 4 IO-APIC-edge floppy
7: 0 0 IO-APIC-edge parport0
8: 2 0 IO-APIC-edge rtc
9: 0 1 IO-APIC-fasteoi acpi
12: 21 84 IO-APIC-edge i8042
14: 8460 8198 IO-APIC-edge libata
15: 1360 1549 IO-APIC-edge libata
16: 60 60 IO-APIC-fasteoi aic7xxx
17: 129 102 IO-APIC-fasteoi eth1
18: 44 47 IO-APIC-fasteoi
19: 105 130 IO-APIC-fasteoi EMU10K1
NMI: 0 0 Non-maskable interrupts
LOC: 104387 104637 Local timer interrupts
RES: 8890 8214 Rescheduling interrupts
CAL: 4176 5267 function call interrupts
TLB: 271 236 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0


44 interrupts added to both eth1 and EMU10K1

This is a known problem with this. Some interrupt controlers are funny
and do funny things when an interrupt is masked, but interrupts enabled.
They route the interrupt to the wrong interrupt line. The only reason that
vanilla doesn't show it, is that vanilla does the interrupt handler
when the interrupt is triggered, so it has no need to mask. RT on the
other hand, runs interrupts in threaded context, which triggers this
little quirk because we mask the interrupt. For some strange reason, the
interrupt controller will trigger the interrupt for another interrupt, if
that interrupt line is masked.

To prove this is the problem, boot with noapic in the kernel command line.
1) the problem should disappear.
2) (I'm betting) you see that the eth and EMU10K1 share the same
interrupt line.


Yep, you were right. They do share the same IRQ and the problem does go away.
Unfortunately I can't run this machine with noapic. I need irq affinity.

I see from the back trace that this is i386. We have a workaround for this
on x86_64. Jon Masters has been working on better solutions too.


Yes this is i386. Is it just a certain interrupt controller that acts this way or are there more?
I guess I have at least one of "those funny interrupt controllers" Hmm



#lspci

00:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P]
System Controller (rev 20)
00:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P]
AGP Bridge
00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ISA (rev 05)
00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE
(rev 04)
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI (rev 03)
00:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado]
(rev 6c)
00:09.0 Class Class ff00: Compro Computer Services, Inc. Unknown device
4610 (rev 03)
00:10.0 PCI bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] PCI (rev 05)
01:05.0 VGA compatible controller: nVidia Corporation NV25 [GeForce4 Ti
4400] (rev a2)
02:04.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 (rev 04)
02:04.1 Input device controller: Creative Labs SB Live! Game Port (rev 01)
02:05.0 Communication controller: National Instruments PCI-GPIB (rev 01)
02:06.0 SCSI storage controller: Adaptec AHA-2930CU (rev 03)
02:07.0 Communication controller: National Instruments PCI-GPIB (rev 01)
02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado]
(rev 78)

Again this does not happen with 2.6.24 vanilla. I'm not sure about
earlier RT kernels.

Regards
Mark



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • [PATCH] ppc32: CPM2 PIC cleanup
    ... Cleaned up the CPM2 interrupt controller code: ... Added the ability to offset the IRQs ... interrupt controllers ... extern int cpm2_get_irq; ...
    (Linux-Kernel)
  • Re: [PATCH] uio: User IRQ Mode
    ... In this mode the user space driver ... If I take a level triggered interrupt then the IRQ handler code must ... -> interrupt signal to interrupt controller gets asserted ... -> interrupt source is still masked by interrupt controller ...
    (Linux-Kernel)
  • Re: 2.6.5 yenta_socket irq 10: nobody cared!
    ... On Thursday 08 April 2004 09:37, Kitt Tientanopajai wrote: ... my patch also changes the interrupt assignment for the USB controller at 00:1d.1 ... i looked at the spec of the o2micro controller. ... the serial interrupt controller in the 82801CAM ...
    (Linux-Kernel)
  • RE: spurious 8259A interrupt
    ... On Mon, 22 Mar 2004, Guennadi Liakhovetski wrote: ... > interrupt can produce several spurious ones. ... Given that the CPU needs to have a vector ... interrupt controller says; ...
    (Linux-Kernel)
  • Re: Interrupt context...
    ... > gone through most of the posts on interrupt in usenet. ... > kernel stack and ISR is executed. ... More may be saved depending on the architecture. ... Here the kernel have assembler code to save all general ...
    (comp.os.linux.development.system)