allocation failures with CBQ bandwidth limiting & high net use (was Re: Filesystem kernel hangup, 2.6.3 (bad: scheduling while atomic!))
From: Brad Allen (Ulmo-Usenet_at_Usenet.Q.Net)
Date: Tue, 27 Apr 2004 21:39:19 -0700 (PDT) To: linux-kernel@VGER.KERNEL.ORG
> Just tried it without HyperThreading, with JFS, the filesystem
> hanged without any kernel oops after about 10 minutes of 1Gbit/s FTP
> What I got was: [...]
> mserv4 kernel: swapper: page allocation failure. order:0, mode:0x20 [...]
> mserv4 kernel: [<c02ad599>] e1000_alloc_rx_buffers+0x59/0x100
I just corresponded a problem I originally thought yours might be
related to, so now I think it's even less likely they're related.
After a due-diligence check for latest version before post, I found
and upgraded to kernel version 2.6.6-rc3, and during my tests of that
new version inadvertently finally found the test that revealed a
correspondence with these errors: running CBQ network bandwidth
limiting, queuing and prioritization, *and* high network use (using
that bandwidth limiting). The CBQ stuff is done at boot. As soon as
I started the high network use, the problem started (see below
"full_kernel_output" file for an idea of what kernel messages I was
seeing en masse), and then during this high network use, I turned off
CBQ and everything was normal again.
This is what I wrote before the above discovery, which gives
I'm also seeing "page allocation failure" errors in a 2.6 series
kernel with e1000 driver with lots of filesystem access and lots of
outbound network, which I think are our common themes. I'm seeing the
call trace with e1000 driver routines in nearly every list.
Otherwise, much else is different:
* Linux kernel 2.6.6-rc2 used for this particular run
as documented, but same allocation failure errors
also seen in 2.6.6-rc3, 2.6.6-rc1, 2.6.5,
2.6.5+kraxel263-1, 2.6.4+kraxel263-1 (+kraxel is
drivers/media/video stuff ported in sooner than 2.6
series got it).
* I was doing relatively heavy client NFSv3, not XFS
or JFS like you: ~2MiB/s steady write, ~2MiB/s
steady read. (Test ran for ~2 hours. Crashed
with only 37s to go before filling up remote disk!
Coincidence, because it had no idea the remote disk
was getting full, I think ... but never did finish
filling it, which was my planned test end point.)
* I am using experimental IVTV driver, which is
itself having trouble and undergoing active
development, to record and play TV using Hauppauge
990 (WinTV PVR 350). This card does all the heavy
compute intensive stuff: MPEG2 encode & decode, as
well as TV stuff. The computer merely has to
bitshuffle & stuff, which is *not* that hard. The
bandwidth of the system ought to allow more like four
of these cards recording & at least one playing, if
things were tuned right, I think.
* My IRQ is shared with that above TV card, e1000
(Intel GbE) card, and video output card; the first
two get used extensively (the last one -- the local
console video -- almost never gets used, except for
* I had network output going almost full-bore, as
a stress test of the above, but I still get the
errors when this isn't the case. 61,840B/s inside
* I am using CBQ to limit network output speed to
Intenret, and priority of packets. 62,464B/s including
IP packets (headers, etc.). Link measured max at a
rough ~66,560B/s inside TCP/IP, so I stay away from it.
So, this is basically definitely hitting output
speed buffering on occasion.
* It's bridging between two ethernet cards (sender
of through-IP traffic (~61,840B/s) in from the
e1000 GbE card, being sent out an on-board 100MbE).
NFSv3 (from & to server) isn't bridged, but also
is going via that e1000 GbE card.
* The computer I'm seeing this on is relatively old
and moderately low resource for today's memory &
CPU use standards:
1999 Dell Optiplex GX1 PIII 450MHz 128MiB ram
I'm not sure if this is related to your problem. I searched
linux-kernel by using the two quoted phrases "page allocation failure"
"e1000_alloc_rx_buffers" in Google, and your above message is the only
one that came out.
I am worried that perhaps page allocation failures are very vague
diagnostic tools. I take it this simply means I ran out of (perhaps a
certain kind of, e.g. and what I suspect, kernel) memory.
Here is the kernel message output (one sample excerpt below):
and .config options:
Here's my CBQ script:
Note that I *do* use LVM (everyone I talk to seems to -- although I
think it's called DM now; btw, I do use and like EVMS on more modern
hosts -- and I'm sure I'd like it on old ones, too, but just haven't
bothered to do it there), but since not much of that computer's access
is via local disk, I doubt that matters that much.
My MTU for GbE (e1000) is 9000, and NFS block size 8192 bytes.
That GbE is a consumer grade Intel model.
Here is a sample of the kernel output; note please that these errors
are not all exactly identical, so if you are reading them please check
the above full_kernel_output for any nuances and big differences (all
kernel messages for about two hours from boot to crash).
swapper: page allocation failure. order:3, mode:0x20
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to firstname.lastname@example.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/