The ultimate TOE design

From: Jeff Garzik (jgarzik_at_pobox.com)
Date: 09/15/04

  • Next message: Ingo Molnar: "Re: [patch] remove the BKL (Big Kernel Lock), this time for real" 
    Date:	Wed, 15 Sep 2004 15:33:47 -0400
To: netdev@oss.sgi.com

    (reply-to set to netdev)

    Every now and then people ask on the lists about TOE, TCP assist, and
    that sort of thing. Ignoring the issue of TCP hardware assist, I wanted
    to describe what I feel is an optimal method to _fully offload_ the
    Linux TCP stack.

    Put simply, the "ultimate TOE card" would be a card with network ports,
    a generic CPU (arm, mips, whatever.), some RAM, and some flash. This
    card's "firmware" is the Linux kernel, configured to run as a _totally
    indepenent network node_, with IP address(es) all its own.

    Then, your host system OS will communicate with the Linux kernel running
    on the card across the PCI bus, using IP packets (64K fixed MTU).

    This effectively:

    1) fragment processing, IPsec, and other services onto the card.

    2) You can use huge card<->host MTUs, which makes sendfile(2) faster
    with _zero_ kernel changes

    3) You can let the PCI card do 100% of the checksum
    processing/generation, and treat the network connection connection
    across the PCI bus as CHECKSUM_UNNECESSARY.

    2) With enough RAM and cpu cycles, you can even offload complex services
    like Web services: the PCI card runs Apache, and fetches files across
    the network (your PCI bus!) from the host system.

    3) Does not require _any_ modification of Linux network stack.
    Interfacing with the card merely requires a simple DMA interface to copy
    IP (not ethernet) packets across the PCI bus, and that fits within the
    existing Linux net driver API.

    4) ensures that the TOE "firmware" [the Linux kernel] can be easily
    updated in the event of new features or (more importantly) security
    problems.

    5) Linux is the most RFC-compliant net stack in the world. Why
    re-create (or license) an inferior one?

    6) Long-term maintenance of TOE firmware is a BIG problem with existing
    full-TOE systems. Under this design, sysadmins would update and patch
    their PCI card with security updates just like any other system on their
    network. This is added work, yes, but it's a known quantity and a task
    they are already doing for other systems.

    7) The design is both portable [tons of embedded CPUs, with and without
    MMUs, can run Linux] and scalable.

    My dream is that some vendor will come along and implement such a
    design, and sell it in enough volume that it's US$100 or less. There
    are a few cards on the market already where implementing this design
    _may_ be possible, but they are all fairly expensive. Just need enough
    resources on the PCI to be able to Linux as a
    router/firewall/iSCSI/web-proxy gadget.

    And I'm not aware of anybody doing a direct IP-over-PCI thing, either.

    But I'll keep on dreaming... ;-)

            Jeff

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  • Next message: Ingo Molnar: "Re: [patch] remove the BKL (Big Kernel Lock), this time for real"

    Relevant Pages

    • Re: [opensuse] Re: router DHCP suddenly not talking to one machine (10.1)
      ... into Windows the DHCP servers hands out the usual IP address ... It is only in Linux, after it worked with this network ... card for 2 days, that the problem happens. ...
      (SuSE)
    • [opensuse] [SLE] Slow transfers from Linux Server
      ... I am having performance issues with my file server running SUSE Linux ... cat5e via a PCI Netgear GA302T Gigabit card. ... reports that the card is connected at 1000FD and no network errors are ... The Windows box is connected to the same gigabit switch using a 3com ...
      (SuSE)
    • RE: [opensuse] router DHCP suddenly not talking to one machine (1 0.1)
      ... into Windows the DHCP servers hands out the usual IP address ... It is only in Linux, after it worked with this network ... network is down and that the DHCP client is still waiting. ... assign this card a static IP address. ...
      (SuSE)
    • Re: Getting D-Link DFE-670TXT PC card to run
      ... Welp...I'm now posting this messge to you from within Linux! ... The card was working ... even though my Linksys router was using DHCP. ... So I went back to "netconfig" and reconfigured the network paramaters ...
      (comp.os.linux.portable)
    • Re: e100 PCI bridge problem
      ... computers by making sure the card is installed on the primary PCI bus. ... The special card that has these 4 ports onboard contains a bridge chip itself which explains the issues. ... irq routing in linux may not be the same as in windows. ...
      (Linux-Kernel)