Re: gzip / compression problems

On Tue, 27 Feb 2007 06:47:11 -0800, Andy Savidge wrote:

Hi,

We have a White Box EL 3.0 box that we're using as a samba file server
and a CVS server.

We were using the

tar -cvz <directory> | ssh backup_user@backup_server '(cat >
backup.tgz)'

method to backup certain shares overnight, but upon trying to recover
some files we discovered that the backup.tgz would neither tar -xvzf,
nor gunzip (gzip -d). At first we thought this might be corruption
during the ssh transfer of data, but eventually we narrowed it down to
compression problems on the file server.

In short when I gzip a file it will complete without error. Even with
the -v option it just states the compression rate.

When I gzip -d or gunzip the .gz file I get the following output:

# ls -lh tech.tar
-rw------- 1 root root 3.9G Feb 27 10:44 tech.tar
# ls -l tech.tar
-rw------- 1 root root 4179896320 Feb 27 10:44 tech.tar

(The tar checks out fine, btw)

# gzip -v tech.tar
tech.tar: 26.6% -- replaced with tech.tar.gz

# ls -l tech.tar.gz
-rw------- 1 root root 3106771787 Feb 27 12:39 tech.tar.gz

# gzip -t tech.tar.gz

gzip: tech.tar.gz: invalid compressed data--crc error

gzip: tech.tar.gz: invalid compressed data--length error

Is the entire file bad?


This tends to happen with larger files, but also fails on some smaller
files, for example one failure is a file of 175MB.

<snip>

Any suggestions or help would be very gratefully received.

Cheers,

Andy

Note: comment inline.

First: Are the files closed which are being backed up? For example, I
have seen an error issued by rsync which states, "file changed as we read
it," or something to that effect. If your files are open, then you may
need to use a device mapper (snapshot-origin) solution to freeze your
backup.

Also, I have seen some Microsoft Access databases not be backed up because
of a restrictive file lock (that is, they are simply omitted from the
backup, but not corrupted.) This is something which must also be watched
for, lest the backup be incomplete.

Another possibility:
Since you are using a recent GNU/Linux distribution, file sizes greater
than 4G should be okay. However, it might be worth checking that the
corruption doesn't occur when a certain file size is reached.

--
Douglas Mayne
.

Relevant Pages

  • Re: Backups: rsync, software RAID, other strategies?
    ... I use the following configuration on my file server. ... typical rsync that will mirror the source to the ... You can also purge the backup by removing ... >of the drives as a backup for the other. ...
    (freebsd-questions)
  • Re: LAN access while VPN is up
    ... > only do a nightly backup - mainly a bandwidth problem. ... The scheme consumes around 10GB per windoze box on the file server, ... > Is there not a Canadian law against computer tampering? ... The music industry here managed to shoot itself in the foot rather ...
    (comp.security.firewalls)
  • Re: Maxtor external hard drive no longer recognized by My Computer
    ... backup of my drive hourly to my file server, ... hardly a reason or budget for RAID. ... I got the SCSI RAID controller free with the drives ...
    (microsoft.public.windowsxp.hardware)
  • Re: Backing up my settings and files - noob alert :)
    ... >find that I need to know how to backup my data in case it happens again. ... we simply boot using a floppy, connect to the file server, and pour the ... you'll have to run both those commands as root. ... Access: Sun Jan 30 18:38:32 2005 ...
    (alt.os.linux.redhat)
  • Re: Backing up my settings and files - noob alert :)
    ... >find that I need to know how to backup my data in case it happens again. ... we simply boot using a floppy, connect to the file server, and pour the ... you'll have to run both those commands as root. ... Access: Sun Jan 30 18:38:32 2005 ...
    (linux.redhat)