Re: Compression to a fit a space...

From: jack (not_at_all.org)
Date: 09/03/04 

Date: Fri, 03 Sep 2004 23:47:34 +0200

Fredderic wrote:
> Greetings peoples...
>
> I'm trying to write a program which needs to fill a particular space
> with a tar.gz (or even better, a tar.bz2) of a collection of files. The
> files it needs to pack in are log mostly files, and the space it needs to
> fill is usually about 20-30mb left by a CD image that's being built by
> another package.
>
> Most importantly, I need to be able to fill as close to but no more than a
> given space with a tar.gz containing as many of a set of files as I can
> get to fit.
>
> My current implementation simply builds a tar one file at a time,
> compressing it after each file and checking the size of the resultant
> compressed version. However these logs compress quite substantially, so
> the uncompressed tar could conceivably grow quite large if I were to feed
> it an empty CD instead of one that was mostly used (the important logs get
> put on first, followed by as many others as will fit -- the others are
> included just for convenience and don't really matter if they don't make
> it in). Not to mention how incredibly slow the process is.
>
> I was thinking of building a tar archive myself, one file at a time;
> generate the necessary tar header, attach the file, then compress the
> whole chunk. If it will fit, tag it onto the end of the archive. gunzip
> will extract each part as it encounters it in the file, outputting them
> all as a single stream which can be fed to tar for extraction. But a
> collection of individual gz'd files concatenated together won't be as
> small as a gz's tar, and may in fact be bigger in the case of a large
> number of small files.
>
> Archive programs don't really have that problem, because they just split
> the last file across as many disks as it needs. I however, need each
> tar.gz to be complete on its own. So I need to know, as I'm building it,
> how big it will be if I add a given file. And the best way to do that is
> if I can "rewind" the archival process any time I add a "too big" file.
> Does anyone know of a compression library where this is possible, or of a
> better way of doing what I need to do...

I'd suggest tar's "-L" option in the first place.

Then, You may want to try the "-r" option.

Cheers, Jack. 

-- 
----------------------------------------------------------------------
My personal reading of the string "MicroSoft" expands to "NanoWeak"...

Relevant Pages

  • Re: FTPing a folder across to remote webspace
    ... > lot of repeated test it will compress a lot. ... > gzip only processes single files, so yes you should tar first. ... > In your message you mentioned databases, but not which sort of database. ... MySQL backup. ...
    (uk.comp.os.linux)
  • Compression to a fit a space...
    ... get to fit. ... My current implementation simply builds a tar one file at a time, ... However these logs compress quite substantially, ...
    (comp.os.linux.development.apps)
  • Re: FTPing a folder across to remote webspace
    ... >How do I find the size of the main folder, sub folders and files before and ... lot of repeated test it will compress a lot. ... so yes you should tar first. ... backup of a database. ...
    (uk.comp.os.linux)
  • Re: Compressing folders/sub-folders with TAR : how to ?
    ... > What's the syntax to compress all folders and sub-folders found with TAR? ... that's an archiver. ... bzip2 whose role is to take an arbitrary stream of data as input ...
    (comp.unix.questions)
  • Re: TAR Question
    ... Create a new tar file of just those files, then compress it. ... Uncompress and extract the small tar files one at a time. ... You might also be able to use zcat: compress the original tar file then use ...
    (AIX-L)