Re: [RFC][PATCH 0/9] Network receive deadlock prevention for NBD

On Wed, 2006-08-09 at 17:07 +0400, Evgeniy Polyakov wrote:
On Wed, Aug 09, 2006 at 02:37:20PM +0200, Peter Zijlstra (a.p.zijlstra@xxxxxxxxx) wrote:
On Wed, 2006-08-09 at 09:46 +0400, Evgeniy Polyakov wrote:
On Tue, Aug 08, 2006 at 09:33:25PM +0200, Peter Zijlstra (a.p.zijlstra@xxxxxxxxx) wrote:
http://lwn.net/Articles/144273/
"Kernel Summit 2005: Convergence of network and storage paths"

We believe that an approach very much like today's patch set is
necessary for NBD, iSCSI, AoE or the like ever to work reliably.
We further believe that a properly working version of at least one of
these subsystems is critical to the viability of Linux as a modern
storage platform.

There is another approach for that - do not use slab allocator for
network dataflow at all. It automatically has all you pros amd if
implemented correctly can have a lot of additional usefull and
high-performance features like full zero-copy and total fragmentation
avoidance.

On your site where you explain the Network Tree Allocator:

http://tservice.net.ru/~s0mbre/blog/devel/networking/nta/index.html

You only test the fragmentation scenario with the full scale of sizes.
Fragmentation will look different if you use a limited number of sizes
that share no factors (other than the block size); try 19, 37 and 79
blocks with 1:1:1 ratio.
^^^^^^

19, 37 and 79 will be rounded by SLAB to 32, 64 and 128 bytes, with NTA it
will be 32, 64 and 96 bytes. NTA wins in each allocation which is not
power-of-two (I use 32 bytes alignemnt, as the smallest one which SLAB
uses). And as you saw in the blog, network tree allocator is faster
than SLAB one, although it can have different side effects which are not
yet 100% discovered.

So that would end up being 19*32 = 608 bytes, etc..
As for speed, sure.

Also, I have yet to see how you will do full zero-copy receives; full
zero-copy would mean getting the data from driver DMA to user-space
without
a single copy. The to user-space part almost requires that each packet
live
on its own page.

Each page can easily have several packets inside.

For sure, the problem is: do you know for which user-space process a
packet
is going to be before you receive it?

As for the VM deadlock avoidance; I see no zero overhead allocation path
- you do not want to deadlock your allocator. I see no critical resource
isolation (our SOCK_MEMALLOC). Without these things your allocator might
improve the status quo but it will not aid in avoiding the deadlock we
try to tackle here.

Because such reservation is not needed at all.
SLAB OOM can be handled by reserving pool using SOCK_MEMALLOC and
similar hacks, and different allocator, which obviously work with own
pool of pages, can not suffer from SLAB problems.

You say "critical resource isolation", but it is not the case - consider
NFS over UDP - remote side will not stop sending just because receiving
socket code drops data due to OOM, or IPsec or compression, which can
requires reallocation. There is no "critical resource isolation", since
reserved pool _must_ be used by everyone in the kernel network stack.

The idea is to drop all !NFS packets (or even more specific only keep
those
NFS packets that belong to the critical mount), and everybody doing
critical
IO over layered networks like IPSec or other tunnel constructs asks for
trouble - Just DON'T do that.

Dropping these non-essential packets makes sure the reserve memory
doesn't
get stuck in some random blocked user-space process, hence you can make
progress.

And as you saw fragmentation issues are handled very good in NTA, just
consider usual packet with data with 1500 MTU - 500 bytes are wasted.
If you use jumbo frames... it is posible to end up with 32k allocation
for 9k jumbo frame with some hardware.

Sure, SLAB does suck at some things, and I don't argue that NTA will
not
improve. Its just that 'total fragmentation avoidance' it too strong
and
this deadlock avoidance needs more.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: [PATCH 1/1] network memory allocator.
    ... I'm not understanding, if you have a page A with two packets, a and b; ... Kevent based network AIO already handle it. ... Kevent network AIO is completely different from network tree allocator. ... it only provides a memory. ...
    (Linux-Kernel)
  • Re: [RFC][PATCH 0/9] Network receive deadlock prevention for NBD
    ... Convergence of network and storage paths" ... On your site where you explain the Network Tree Allocator: ... power-of-two (I use 32 bytes alignemnt, as the smallest one which SLAB ... The to user-space part almost requires that each packet ...
    (Linux-Kernel)
  • Re: [RFC][PATCH 0/9] Network receive deadlock prevention for NBD
    ... Convergence of network and storage paths" ... On your site where you explain the Network Tree Allocator: ... You only test the fragmentation scenario with the full scale of sizes. ... power-of-two (I use 32 bytes alignemnt, as the smallest one which SLAB ...
    (Linux-Kernel)
  • Re: [PATCH 1/1] network memory allocator.
    ... I'm not understanding, if you have a page A with two packets, a and b; ... Kevent based network AIO already handle it. ... Each socket has it's limit, so if allocator got enough memory, blocked ... round up to nearest order page alloc per object; SLAB is build to avoid ...
    (Linux-Kernel)
  • Re: [PATCH 1/1] network memory allocator.
    ... I'm not understanding, if you have a page A with two packets, a and b; ... Each socket has it's limit, so if allocator got enough memory, blocked ... round up to nearest order page alloc per object; SLAB is build to avoid ... So the actual internal fragmentation of the current kmalloc/SLAB ...
    (Linux-Kernel)