CONFIG_PACKET_MMAP revisited

odain2_at_mindspring.com
Date: 10/29/03

  • Next message: Ernst Herzberg: "2.6.0-test9 can't kill process in stat 'R'" 
    To: linux-kernel@vger.kernel.org
Date:	Tue, 28 Oct 2003 23:09:13 -0500

    I've been looking into faster ways to do packet captures and I stumbled on
    the following discussion on the Linux Kernel mailing list:

    http://www.ussg.iu.edu/hypermail/linux/kernel/0202.2/1173.html

    In that discussion Jamie Lokier suggested having a memory buffer that's
    shared between user and kernel space and having the NIC do DMA transfers
    directly to that buffer as an alternative to using Alexy's shared ring
    buffer stuff. The argument was that this would avoid the memory copy that
    the kernel does from the DMA buffer to the memory mapped ring buffer.
    However, Alan Cox pointed out that the main cost of the memory copy is
    getting the data from system memory (where the NIC put it via DMA) into the
    L1 cache (DMA doesn't do any cache coherence so it can't go there
    directly). The memory copy (presumably from L1 cache to L1 cache) is
    insignificant compared to this cost and since you'll need to get the data
    into L1 cache to use it anyway, the memory copy is virtually free.

    I'm wondering if this takes all of the costs into account. If I understand
    how this works the user space application can't get at the packet without a
    context switch so that the kernel can first copy the packet to the shared
    buffer. The cost of the context switch is pretty high and this seems to me
    to be the main bottleneck. I believe that in normal operation each packet
    (or with NICs that do interrupt coalescing, every n packets) causes an
    interrupt which causes a context switch, the kernel then copies the data
    from the DMA buffer to the shared buffer and does a RETI. That's fairly
    expensive. If, on the other hand, data could be copied directly to
    user-space accessible memory the NIC wouldn't need to generate any
    interrupts and the kernel wouldn't need to get involved at all (this
    assumes a NIC that can be configured not to generate any interrupts). The
    user space application could then poll the shared buffer and process
    packets as fast as possible (some synchronization mechanism is clearly
    needed here, but I think some of the high-end programmable NICs could do
    this). Would this not be significantly more efficient then the current
    implementation?

    Thank,
    Oliver

    PS: I'm not a mailing list subscriber so CCs on responses would be
    appreciated.

    --------------------------------------------------------------------
    mail2web - Check your email from the web at
    http://mail2web.com/ .

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  • Next message: Ernst Herzberg: "2.6.0-test9 can't kill process in stat 'R'"

    Relevant Pages

    • Re: Library design for downloading an unknown amount of data?
      ... Multiple manufacturers just adds to the fun! ... Some devices send everything in a single packet, ... you could malloc the correct sized buffer. ... devices that only have 32KB of memory. ...
      (comp.lang.c)
    • Re: Library design for downloading an unknown amount of data?
      ... A potential problem with this approach is that the buffer size will be ... Some devices send everything in a single packet, ... In some protocols it is simply fixed, and for others you can choose it yourself, but you should use the maximum allowable size for maximum speed. ... Right now the device with the highest memory capacity has 2MB of memory. ...
      (comp.lang.c)
    • Re: contigmalloc() and mmap()
      ... there seems no big differences between the kernel ... > to the card on another node, it will be DMAed to memory too. ... The buffer is mmaped to user process space, ... > mmap driver's buffer (allocated by contigmalloc()) and is killed, ...
      (freebsd-hackers)
    • Re: [2.4] heavy-load under swap space shortage
      ... > This file controls the operation of the bdflush kernel ... > -to a clean buffer, which can just be forgotten about). ... > -value can lead to memory shortage when bdflush isn't woken ... > Kswapd is the kernel swapout daemon. ...
      (Linux-Kernel)
    • Re: contigmalloc() and mmap()
      ... there seems no big differences between the kernel ... > to the card on another node, it will be DMAed to memory too. ... The buffer is mmaped to user process space, ... > mmap driver's buffer (allocated by contigmalloc()) and is killed, ...
      (freebsd-hackers)