Re: [RFC 0/3] Recursive reclaim (on __PF_MEMALLOC)

On Wed, 2007-09-12 at 15:47 -0700, Christoph Lameter wrote:
On Wed, 12 Sep 2007, Peter Zijlstra wrote:

assumes single critical user of memory. There are other consumers of
memory and if you have a load that depends on other things than networking
then you should not kill the other things that want memory.

The VM is a _critical_ user of memory. And I dare say it is the _most_
important user.

The users of memory are various subsystems. The VM itself of course also
uses memory to manage memory but the important thing is that the VM
provides services to other subsystems

Exactly, and because it services every other subsystem and userspace,
its the most important one, if it doesn't work, nothing else will.

Every user of memory relies on the VM, and we only get into trouble if
the VM in turn relies on one of these users. Traditionally that has only
been the block layer, and we special cased that using mempools and
PF_MEMALLOC.

Why do you object to me doing a similar thing for networking?

I have not seen you using mempools for the networking layer. I would not
object to such a solution. It already exists for other subsystems.

Dude, listen, how often do I have to say this: I cannot use mempools for
the network subsystem because its build on kmalloc! What I've done is
build a replacement for mempools - a reserve system - that does work
similar to mempools but also provides the flexibility of kmalloc.

That is all, no more, no less.

The problem of circular dependancies on and with the VM is rather
limited to kernel IO subsystems, and we only have a limited amount of
them.

The kernel has to use the filesystems and other subsystems for I/O. These
subsystems compete for memory in order to make progress. I would not
consider strictly them part of the VM. The kernel reclaim may trigger I/O
in multiple I/O subsystems simultaneously.

I'm confused by this, I've never claimed part of, or such a thing. All
I'm saying is that because of the circular dependency between the VM and
the IO subsystem used for swap (not file backed paging [*], just swap)
you have to do something special to avoid deadlocks.

[*] the dirty limit along with 'atomic' swap ensures that file backed
paging does not get into this tight spot.

You talk about something generic, do you mean an approach that is
generic across all these subsystems?

Yes an approach that is fair and does not allow one single subsystem to
hog all of memory.

I do no such thing! My reserve system works much like mempools, you
reserve a certain amount of pages and use no more.

If so, my approach would be it, I can replace mempools as we have them
with the reserve system I introduce.

Replacing the mempools for the block layer sounds pretty good. But how do
these various subsystems that may live in different portions of the system
for various devices avoid global serialization and livelock through your
system?

The reserves are spread over all kernel mapped zones, the slab allocator
is still per cpu, the page allocator tries to get pages from the nearest
node.

And how is fairness addresses? I may want to run a fileserver on
some nodes and a HPC application that relies on a fiberchannel connection
on other nodes. How do we guarantee that the HPC application is not
impacted if the network services of the fileserver flood the system with
messages and exhaust memory?

The network system reserves A pages, the block layer reserves B pages,
once they start getting pages from the reserves they go bean counting,
once they reach their respective limit they stop.

The serialisation impact of the bean counting depends on how
fine-grained you place them, currently I only have a machine wide
network bean counter because the network subsystem is machine wide -
initially I tried to do something per net-device but that doesn't work
out. If someone more skilled in this area comes along and sees a better
way to place the bean counters they are free to do so.

But do notice that the bean counting is only done once we hit the
reserves, the normal mode of operation is not penalised by the extra
overhead thereof.

Also note that mempools also serialise their access once the backing
allocator fails, so I don't differ from them in that respect either.

Attachment: signature.asc
Description: This is a digitally signed message part

Relevant Pages

  • [RFC][PATCH 0/9] Network receive deadlock prevention for NBD
    ... Convergence of network and storage paths" ... "Net vm deadlock fix " ... Maybe it is another memory deadlock, ... dipping into the memalloc reserve if it must. ...
    (Linux-Kernel)
  • Re: [PATCH] [Request for inclusion] Filesystem in Userspace
    ... > to allocate network buffers down to the PF_MEMALLOC boundary. ... > the network stack is only allowed to allocate memory down to the ... A lot of people think that mempool solves this problem, ... normal allocation, but that fails, so they start eating into the mempool ...
    (Linux-Kernel)
  • Re: Odd performance problems after upgrade from 4.11 to 6.0-Stable
    ... After the upgrade, the system performs poorly. ... job produced minimal CPU utilization and little progress. ... My network link only ran at 3.4 Mbps (yes, that's bits, not ... runs out of memory and starts to use the swap file. ...
    (freebsd-stable)
  • [PATCH 02/20] net: vm deadlock avoidance core
    ... the block device must never stall because of ... that it must always find enough memory to build/send packets over the network ... Memory pressure will add to that; what if there is simply no memory left to ... goto nodata; ...
    (Linux-Kernel)
  • Re: Broadcom 440x NIC not recognized on boot
    ... I am attempting to add support for the built-in network device in my ... dmesg output now indicates the card is recognized, ... And it ain't getting the register memory it needs, ... The source code also refers to PCI assigning the memory block, ...
    (freebsd-questions)