A different aproach to archiving files

From: Kasper Dupont (kasperd_at_daimi.au.dk)
Date: 02/25/04 

Date: Wed, 25 Feb 2004 20:55:06 +0100

I have found that tar is a nice tool for creating full
backups of my filesystem. But of course keeping full
backups from every day will take a considerable amount
of diskspace.

Of course I could make backups of changes only, but I
like having full backups.

So I got the idea that there must be some way to store
all those tar files with full backups in an efficient
way that will take advantage of the redundancy to use
less space for the storage.

Conventinal compression doesn't help here. Sure gzip
is nice and all, and it means I use about half the
space. But with 10 different full backups, I still use
more space than necesarry.

Neither gzip nor bzip2 will take advantage of identical
data in different tar files. And it will not even take
advantage of identical files stored in the same tar
file if they are too far apart.

So I came up with a different aproach. Imagine a special
archive format, that will notice every time two
identical blocks apear in the same or in different
files being stored.

This is what I have written. A simple archive format,
where data is stored in 512 bytes blocks (like tar) and
a header block for each stored file points to a four
level indirect block to locate it's contents.

I think it would even be simple to write a filesystem
driver, that would allow mounting the archive and
access its contents.

Right now the code doesn't perform very well, and there
are bugs that can cause corruption of the archive. But
the format is fairly robust, and I can always repair
any damages by truncating the archive to the size it
had before the failing operation.

I'd like comments about the idea and ways to improve it.

The code is here. It will probably fail when the size
of the archive aproaches 1GB, but when those bugs have
been fixed it should support archives up to 2TB.

http://www.daimi.au.dk/~kasperd/tstore-0.1.2.tgz 

-- 
Kasper Dupont -- der bruger for meget tid paa usenet.
For sending spam use mailto:aaarep@daimi.au.dk
/* Would you like fries with that? */

Relevant Pages