Re: ext3/ext4 directories don't shrink after deleting lots of files

On Fri, May 15, 2009 at 01:29:04PM -0400, Timo Sirainen wrote:
On Fri, 2009-05-15 at 06:58 -0400, Theodore Tso wrote:
I was rather thinking something that I could run while the system was
fully operational. Otherwise just moving the files to a temp directory +
rmdir() + rename() would have been fine too.

I just tested that xfs, jfs and reiserfs all shrink the directories
immediately. Is it more difficult to implement for ext* or has no one
else found this to be a problem?

It's probably fairest to say no one has thought it worth the effort.

My problem is with mail servers and Maildir format where it's possible
that a user has tons of emails and wants to delete them. The mailbox
maybe slowly grows back to the huge size, but in the meantime it's
slower than necessary.

The problem is that unless the user is deleting a *huge* number of
files, it's rare that the directory entry block goes completely empty.
If you shrink from 15,000 messages to 12,000 messages, say, because of
the fact that we use a hashed b-tree as our data structure, the leaf
blocks in the btree generally still contain some directory entries.
So to fix this we need to actually coalesce directory leaf blocks on
the fly, on top of everything else that I had mentioned. It's
certianly doable, but again, someone would have to submit a patch. We
might get around to it one of these days, but plates of those of us
who are doing ext4 are pretty full with higher priority items at
present.

There is an off-line fix that works quite well -- e2fsck -fD, but
obviously that requires scheduling downtime.

How big of a deal is this for you? I use a local maildir myself, and
they can get quite large:

% ls /home/tytso/isync/mit
total 2132
1412 cur/ 716 new/ 4 tmp/

But once they are in cache, it's no longer a major problem. I suppose
on a mail server where you have a very large number of users, caching
2 megs of directory data per user could get ugly; and it does take
time the first time you pull their directory entry into the cache.
What sort of performance degredation are you measuring, and what are
the impacts operationally at the moment for you? Is this just a
theoretical concern, or are you measuring a significant slowdown as a result?

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: [PATCH] xen: core dom0 support
    ... 1f4f931501e9270c156d05ee76b7b872de486304) to improve pvops ... Well it's the L2 cache references which are being measured here, ... instructions, but we know that there's a lot more going on), ... It's measuring kernel stats too - and i very much saw the ...
    (Linux-Kernel)
  • Re: Thread Question - When do they die?
    ... If you're measuring the total run time of an application, there can easily be a large delay the first time running it that you don't see in subsequent times, as a variety of things get loaded for the first run but then remain cached in memory, ready to go for subsequent runs. ... As the name suggests, it's a cache of some sort, but I don't know everything it does. ... If you've got a timer on a specific section of code, then depending on what that code actually does, again it could be a simple matter of disk caching, or if your application is interacting with external processes then there could be caching going on there. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: [PATCH/RFC] Simplified Readahead
    ... Right the first time the next_size is decremented ... >since the pages are already in the page cache the next_size keeps incrementing. ... >>in the page cache the readahead turns off. ... What I do now for page cache hits is count how many pages in a row are ...
    (Linux-Kernel)
  • RE: [SLE] OO: If you can make it, I can break it!
    ... any file read from disk is cached the first time it is loaded. ... developer;-) Once in cache it stays there unless the memory is needed ... > the memory hits. ...
    (SuSE)
  • Re: [patch 02/12] Immediate Values - Architecture Independent Code
    ... still in dcache. ... You're assuming that the CPU's cache LRU is perfect. ... right now is that not enough users are there, so measuring ... at high frequency by any workload. ...
    (Linux-Kernel)