Kernel Panic - 2.6.23-rc4-mm1 ia64 - was Re: Update: [Automatic] NUMA replicated pagecache ...

On Wed, 2007-09-12 at 19:38 +0530, Balbir Singh wrote:
Lee Schermerhorn wrote:
On Wed, 2007-09-12 at 07:22 +0530, Balbir Singh wrote:
Lee Schermerhorn wrote:
[Balbir: see notes re: replication and memory controller below]

A quick update: I have rebased the automatic/lazy page migration and
replication patches to 23-rc4-mm1. If interested, you can find the
entire series that I push in the '070911' tarball at:

http://free.linux.hp.com/~lts/Patches/Replication/

I haven't gotten around to some of the things you suggested to address
the soft lockups. etc. I just wanted to keep the patches up to date.

In the process of doing a quick sanity test, I encountered an issue with
replication and the new memory controller patches. I had built the
kernel with the memory controller enabled. I encountered a panic in
reclaim, while attempting to "drop caches", because replication was not
"charging" the replicated pages and reclaim tried to deref a null
"page_container" pointer. [!!! new member in page struct !!!]

I added code to try_to_create_replica(), __remove_replicated_page() and
release_pcache_desc() to charge/uncharge where I thought appropriate
[replication patch # 02]. That seemed to solve the panic during drop
caches triggered reclaim. However, when I tried a more stressful load,
I hit another panic ["NaT Consumption" == ia64-ese for invalid pointer
deref, I think] in shrink_active_list() called from direct reclaim.
Still to be investigated. I wanted to give you and Balbir a heads up
about the interaction of memory controllers with page replication.

Hi, Lee,

Thanks for testing the memory controller with page replication. I do
have some questions on the problem you are seeing

Did you see the problem with direct reclaim or container reclaim?
drop_caches calls remove_mapping(), which should eventually call
the uncharge routine. We have some sanity checks in there.

Sorry. This one wasn't in reclaim. It was from the fault path, via
activate page. The bug in reclaim occurred after I "fixed" page
replication to charge for replicated pages, thus adding the
page_container. The second panic resulted from bad pointer ref in
shrink_active_list() from direct reclaim.

[abbreviated] stack traces attached below.

I took a look at an assembly language objdump and it appears that the
bad pointer deref occurred in the "while (!list_empty(&l_inactive))"
loop. I see that there is also a mem_container_move_lists() call there.
I will try to rerun the workload on an unpatched 23-rc4-mm1 today to see
if it's reproducible there. I can believe that this is a race between
replication [possibly "unreplicate"] and vmscan. I don't know what type
of protection, if any, we have against that.



Thanks, the stack trace makes sense now. So basically, we have a case
where a page is on the zone LRU, but does not belong to any container,
which is why we do indeed need your first fix (to charge/uncharge) the
pages on replication/removal.

We do try to see at several places if the page->page_container is NULL
and check for it. I'll look at your patches to see if there are any
changes to the reclaim logic. I tried looking for the oops you
mentioned, but could not find it in your directory, I saw the soft
lockup logs though. Do you still have the oops saved somewhere?

I think the fix you have is correct and makes things works, but it
worries me that in direct reclaim we dereference the page_container
pointer without the page belonging to a container? What are the
properties of replicated pages? Are they assumed to be exact
replicas (struct page mappings, page_container expected to be the
same for all replicated pages) of the replicated page?

Before "fix"

Running spol+lpm+repl patches on 23-rc4-mm1. kernel build test
echo 1 >/proc/sys/vm/drop_caches
Then [perhaps a coincidence]:

Unable to handle kernel NULL pointer dereference (address 0000000000000008)
cc1[23366]: Oops 11003706212352 [1]
Modules linked in: sunrpc binfmt_misc fan dock sg thermal processor container button sr_mod scsi_wait_scan ehci_hcd ohci_hcd uhci_hcd usbcore

Pid: 23366, CPU 6, comm: cc1
<snip>
[<a000000100191a30>] __mem_container_move_lists+0x50/0x100
sp=e0000720449a7d60 bsp=e0000720449a1040
[<a000000100192570>] mem_container_move_lists+0x50/0x80
sp=e0000720449a7d60 bsp=e0000720449a1010
[<a0000001001382b0>] activate_page+0x1d0/0x220
sp=e0000720449a7d60 bsp=e0000720449a0fd0
[<a0000001001389c0>] mark_page_accessed+0xe0/0x160
sp=e0000720449a7d60 bsp=e0000720449a0fb0
[<a000000100125f30>] filemap_fault+0x390/0x840
sp=e0000720449a7d60 bsp=e0000720449a0f10
[<a000000100146870>] __do_fault+0xd0/0xbc0
sp=e0000720449a7d60 bsp=e0000720449a0e90
[<a00000010014b8e0>] handle_mm_fault+0x280/0x1540
sp=e0000720449a7d90 bsp=e0000720449a0e00
[<a000000100071940>] ia64_do_page_fault+0x600/0xa80
sp=e0000720449a7da0 bsp=e0000720449a0da0
[<a00000010000b5c0>] ia64_leave_kernel+0x0/0x270
sp=e0000720449a7e30 bsp=e0000720449a0da0


After "fix:"

Running "usex" [unix systems exerciser] load, with kernel build, io tests,
vm tests, memtoy "lock" tests, ...


Wow! thats a real stress, thanks for putting the controller through
this. How long is it before the system panics? BTW, is NaT NULL Address
Translation? Does this problem go away with the memory controller
disabled?

System panics within a few seconds of starting the test.

NaT == Not a Thing. Kernel reports null pointer deref as such. I
believe that NaT Consumption errors come from attempting to deref a
non-NULL pointer that points at non-existent memory.

I tried the workload again with an "unpatched kernel" -- i.e., no
automatic page migration nor replication, nor any other of my
experimental patches. Still happens with memory controller configured
-- same stack trace.

Then I tried an unpatched 23-rc4-mm1 with memory controller NOT
configured, still panic'ed, but with a different symptom: first a soft
lockup, then a NULL pointer deref--apparently in soft lockup detection
code. Panics because it OOPses in interrupt handler.

Tried again, same kernel--mem controller unconfig'd: this time I got
the original stack trace--NaT Consumption in shrink_active_list().
Then, softlockup with NULL pointer deref therein. It's the null pointer
deref that causes the panic: "Aiee, killing interrupt handler!"

So, maybe memory controller is "off the hook".

I guess I need to check the lists for 23-rc4-mm1 hot fixes, and try to
bisect rc4-mm1.


as[15608]: NaT consumption 2216203124768 [1]
Modules linked in: sunrpc binfmt_misc fan dock sg container thermal button processor sr_mod scsi_wait_scan ehci_hcd ohci_hcd uhci_hcd usbcore

Pid: 15608, CPU 8, comm: as
<snip>
[<a00000010000b5c0>] ia64_leave_kernel+0x0/0x270
sp=e00007401f53fab0 bsp=e00007401f539238
[<a00000010013b4a0>] shrink_active_list+0x160/0xe80
sp=e00007401f53fc80 bsp=e00007401f539158
[<a00000010013e780>] shrink_zone+0x240/0x280
sp=e00007401f53fd40 bsp=e00007401f539100
[<a00000010013fec0>] zone_reclaim+0x3c0/0x580
sp=e00007401f53fd40 bsp=e00007401f539098
[<a000000100130950>] get_page_from_freelist+0xb30/0x1360
sp=e00007401f53fd80 bsp=e00007401f538f08
[<a000000100131310>] __alloc_pages+0xd0/0x620
sp=e00007401f53fd80 bsp=e00007401f538e38
[<a000000100173240>] alloc_page_pol+0x100/0x180
sp=e00007401f53fd90 bsp=e00007401f538e08
[<a0000001001733b0>] alloc_page_vma+0xf0/0x120
sp=e00007401f53fd90 bsp=e00007401f538dc8
[<a00000010014bda0>] handle_mm_fault+0x740/0x1540
sp=e00007401f53fd90 bsp=e00007401f538d38
[<a000000100071940>] ia64_do_page_fault+0x600/0xa80
sp=e00007401f53fda0 bsp=e00007401f538ce0
[<a00000010000b5c0>] ia64_leave_kernel+0x0/0x270
sp=e00007401f53fe30 bsp=e00007401f538ce0



Interesting, I don't see a memory controller function in the stack
trace, but I'll double check to see if I can find some silly race
condition in there.

right. I noticed that after I sent the mail.

Also, config available at:
http://free.linux.hp.com/~lts/Temp/config-2.6.23-rc4-mm1-gwydyr-nomemcont



Later,
Lee

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/