Re: 2.6.26-rc5-mm1

Hello,

Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> wrote on 10.06.2008 07:01:49:
On Tue, 10 Jun 2008 06:57:02 +0200 Mariusz Kozlowski <m.
kozlowski@xxxxxxxxxx> wrote:

Witam,

On Mon, 9 Jun 2008 21:14:54 +0200
Mariusz Kozlowski <m.kozlowski@xxxxxxxxxx> wrote:

Hello Balbir,

Andrew Morton wrote:
Temporarily at

http://userweb.kernel.org/~akpm/2.6.26-rc5-mm1/


I've hit a segfault, the last few lines on my console are


Testing -fstack-protector-all feature
registered taskstats version 1
debug: unmapping init memory ffffffff80c03000..ffffffff80dd8000
init[1]: segfault at 7fff701fe880 ip 7fff701fee5e sp
7fff7006e6d0 error 7

With absolutely no stack trace. I'll dig deeper.

Hey, I see something similar and I actually have a stack
trace. Here it goes:

bash[498] segfault at ffffffff80868b58 ip ffffffffff600412 sp
7fffa3d010f0 error 7
init[1] segfault at ffffffff80868b58 ip ffffffffff600412 sp
7fff9e97f640 error 7
init[1] segfault at ffffffff80868b58 ip ffffffffff600412 sp
7fff9e97eed0 error 7
Kernel panic - not syncing: Attemted to kill init!
Pid 1, comm: init Not tainted 2.6.26-rc5-mm1 #1

Call Trace:
[<ffffffff80254632>] panic+0xe2/0x260
[<ffffffff802fa8ba>] ? __slab_free+0x10a/0x630
[<ffffffff80265a8e>] ? __sigqueue_free+0x5e/0x70
[<ffffffff802851eb>] ? trace_hardirqs_off+0x1b/0x30
[<ffffffff802851eb>] ? trace_hardirqs_off+0x1b/0x30
[<ffffffff80259b54>] do_exit+0xb84/0xc30
[<ffffffff80259c5a>] do_group_exit+0x5a/0x110
[<ffffffff8026a3b5>] get_signal_to_deliver+0x2c5/0x620
[<ffffffff8020bb3b>] do_notify_resume+0x11b/0xd10
[<ffffffff8028da5b>] ? trace_hardirqs_on+0x1b/0x30
[<ffffffff805cd0f3>] ? _spin_unlock_irqrestore+0x93/0x130
[<ffffffff8026865c>] ? force_sig_info+0x10c/0x130
[<ffffffff8022fb9c>] ? force_sig_info_fault+0x2c/0x40
[<ffffffff802dd7dd>] ? print_vma_addr+0x10d/0x1d0
[<ffffffff805cbb67>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff8028d8da>] ? trace_hardirqs_on_caller+0x15a/0x2c0
[<ffffffff8020d4c9>] retint_signal+0x46/0x8d

This was copied manually so typos are possible.


Thanks. Could someone send a config please? Or a bisection result
;)

In my case it turns out to be gcov patches - in which I'm interested
in to see (and play with) the tests coverage.

#
# gcov
#
kernel-call-constructors.patch
kernel-introduce-gcc_version_lower-macro.patch
seq_file-add-function-to-write-binary-data.patch
GOOD
gcov-add-gcov-profiling-infrastructure.patch
GOOD
gcov-create-links-to-gcda-files-in-build-directory.patch
gcov-architecture-specific-compile-flag-adjustments.patch
BAD

I can not bisect between the last two due to build error. Config
is attached.


(cc Peter)

Sorry for the delay. Unfortunately I don't have as much time for this
as I'd like to.

Thanks for the report. These look like the "known architecture problems"
that I've hinted at in the gcov announcement post (I'm assuming this is
x86_64 as I've seem similar reports in the past).

Possible reasons:

1) initrd overwrites kernel: When kernel and initrd are loaded to static
addresses, the oversized gcov kernel may overlap with the initrd load
address. Solution: move initrd loading address.

Not using initrd.

2) out-of-memory: Kernel plus profiling code may just not fit into a
minimal memory configuration any more. Solution: add memory.

2G is ok I assume.

3) write-protection of kernel code: gcc keeps profiling code and data
close together in the .text section. Solution: any mechanism that
protects .text against writes should be disabled when running a
profiled kernel.

Not using it.

4) as of yet undiscovered incompatibilities between arch-dependent code
and gcc's -fprofile-arcs option. Examples would be:

* code which is run before memory access preparations were made
* hard coded section sizes
* relative address displacements which are out of range

Unfortunately I neither have access to a machine nor the skill to debug
4) myself, so if 1)-3) can be ruled out,

Yes they can be ruled out.

I'd like to ask for more help
on this one:

First off, someone needs to track down the offending file(s). This is
done by putting a line containing "GCOV := n" in all Makefiles below
arch/x86_64 (or go one step further back and set
CONFIG_GCOV_PROFILE_ALL=n). If my assumption is correct, then the
kernel should boot fine afterwards. In that case, remove the lines
again one-by-one, while compiling and booting after each change. If the
problem can be narrowed down to a single Makefile, replace the single
"GCOV := n" line with multiple "GCOV_file.o := n" lines, one for each
generated object file. Then again, same approach as before: remove
those lines, compile and boot until it breaks. Finally post your
results.

After a few hours and tons of reboots I narrowed it down to
arch/x86/kernel/Makefile:

a) works
obj-y += tsc_$(BITS).o io_delay.o rtc.o
GCOV_tsc_$(BITS).o := n
#GCOV_io_delay.o := n
#GCOV_rtc.o := n

b) doesn't work
obj-y += tsc_$(BITS).o io_delay.o rtc.o
#GCOV_tsc_$(BITS).o := n
#GCOV_io_delay.o := n
#GCOV_rtc.o := n

and that points to arch/x86/kernel/tsc_64.c

At this point we would need someone with x86_64 arch skills to look at
the file and find out why this code is broken with "-fprofile-arcs"
enabled (on s390 we discovered at least one actual code bug this way,
so the analysis might just be of general use). Alternatively we can
just keep these files from being profiled.


Mariusz
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: BOOT_CS
    ... > Keep initrd few Mb after kernel because I do not know where exactly ... Linux (+ initrd) from a DOS floppy or from operating system dated 98 ... at the place you want to load Linux. ... XMS allocated memory so the physical address is ...
    (Linux-Kernel)
  • Re: 2.6.26-rc5-mm1
    ... In my case it turns out to be gcov patches - in which I'm interested ... initrd overwrites kernel: When kernel and initrd are loaded to static ... gcc keeps profiling code and data ...
    (Linux-Kernel)
  • Re: Crash while mapping memory in pagetable_init() (Was: Re: .config)
    ... head.S maps the kernel size plus INIT_MAP_BEYOND_END, which is currently set to 128K. ... all memory into the kernel address space. ... It tries to map kernelsize+initial_pagetables+128k ... The initrd is supposed to be loaded as far away from the kernel as possible. ...
    (Linux-Kernel)
  • Re: initrd not loaded with > 4 GB and PAE kernel
    ... Diego Santa Cruz wrote: ... I have installed the FC6 kernel 2.6.18-1.2849.fc6PAE on a machine ... The kernel is not booting since it does not find the initrd and thus ... will let you pass the correct memory map to the kernel. ...
    (Fedora)
  • Re: [Fastboot] [PATCH] free initrds boot option
    ... this at all (kexec provides a new kernel image, ... initrd image to be loaded along with kernel. ... hence we can save time when the memory loader is slow. ...
    (Linux-Kernel)