Re: 2.6.12.2 dies after 24 hours

From: Chris Mason (mason_at_suse.com)
Date: 07/13/05

  • Next message: Nathan Scott: "Re: RT and XFS" 
    To: "Rob Mueller" <robm@fastmail.fm>, akpm@osdl.org
Date:	Tue, 12 Jul 2005 20:42:09 -0400

    On Tuesday 12 July 2005 20:27, Rob Mueller wrote:
    > > > We're also applying the attached patch. There's a bug in reiserfs that
    > > > gets tickled by our huge MMAP usage (it's amazing what really busy
    > > > Cyrus daemons can do to a server, ouch). It's fixed in generic_write,
    > > > so we take the few percent performance hit for something that doesn't
    > > > break!
    > >
    > > Interesting - When I got the problem it was on mail servers under high
    > > load (handling 60.000 emails pr. hour) with reiserfs as file system. I
    > > have seen this problem on 5 different servers so I am confident that
    > > it is not hardware failure.
    > >
    > > Sometimes the server load just rises and then the server dies other
    > > times the load rises but the kernel manages to get it back alive
    > > filling up syslog with messages like this
    >
    > Sounds like a different issue. The patch Bron included before fixes (or at
    > least reduces to the point where it fixes it for us) a problem where
    > processes get stuck in D state and are unkillable. A reboot is required to
    > remove them. Apparently this is a known bug in ReiserFS (see messages
    > below). As noted, the same bug exists in ext3. There appears to have been
    > some patches to try and fix it for both reiserfs and ext3, but I'm not sure
    > if they're in the mainline kernel yet.
    >
    > http://www.ussg.iu.edu/hypermail/linux/kernel/0409.0/2056.html
    > http://hulllug.principalhosting.net/archive/index.php/t-22774.html
    >

    There is a much less complex solution that I've just recently gotten working
    in the SUSE kernel. If reiser3/ext3 don't log the inode during atime
    updates, the problem goes away.

    You can solve this now by mounting with -o noatime (although that might not
    play well with cyrus, not sure). My current patch works around this in ugly
    ways, what I plan on doing during OLS is finding out why ext3 is still
    logging the inode all the time.

    For reiser3, this was to avoid kswapd having to log a bunch of inodes in
    response to memory pressure, but that was back in 2.4 when things were
    different. We shouldn't need to do it anymore...

    -chris
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  • Next message: Nathan Scott: "Re: RT and XFS"

    Relevant Pages

    • Re: How many people are using 2.6.16?
      ... avoid tail packing if an inode was ever mmapped ... This patch fixes a confusion reiserfs has for a long time. ...
      (Linux-Kernel)
    • Re: Finding hardlinks
      ... ReiserFS terminology) is free before assigning it to new file/directory? ... Inode free space can have at most 2^31 extents --- if inode numbers ... In the worst case, when free/used extents are small, some free oids are ... was a patch for reiserfs v3 to store this map in special hidden file but ...
      (Linux-Kernel)
    • [PATCH 0/4] Allow reiserfs to work with selinux
      ... I sent out these patches in December, but they were dropped somewhere along the ... internal inode loop needs IS_PRIVATE test ... - reiserfs: ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: 2.6.20-rc4: known unfixed regressions (v2)
      ... on file close reiserfs tries to "pack" content of last incomplete page of file into metadata blocks. ... It does not actually truncate, it calls the same function which does truncate, but file size does not change. ... The patch fixes the confusion by making reiserfs to avoid tail packing if an inode was ever mmapped. ... reiserfs_mmap and reiserfs_file_release are serialized with mutex in reiserfs specific inode. ...
      (Linux-Kernel)