Rsync fails on local transfers

I've got a script I wrote up which backs up my computer using rsync. A
short overview is that I created a file system image using dd, and now I
mount the image and rsync from the server's HD to the image file.

Periodically during large backups, I get the following error:

rsync: connection unexpectedly closed (16931 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(453) [sender=2.6.9]

When I Google for the problem, I get many results about a SSH connection
timing out because of inactivity during the delete stage, but I'm not
using SSH, so I don't get why I would have this problem: both the source
and destination are local.

Any ideas on how to debug this would be great. I've thought of using
strace, but it'd result in a huge file which would have to be sorted
through (as rsync isn't technically crashing, rsync would continue with
a shutdown procedure and the logging wouldn't stop right at the point of
failure).

Attached is the backup script I use (the configuration file it loads in
is very simple, so I'm not including that).

Justin W

#!/bin/bash

# User where the reports of the backup are sent to
MAILTO=root
# Path to backup's configuration file
CONF_PATH=/root

# Overview of backup process
# 1) Read and parse configuration file
# 2) Loop through each backup
# 3) Create a FS snapshot using LVM
# 4) Mount the snapshot FS
# 5) Mount our backup file in a loop filesystem (we can guarantee the space
# will be present this way, and it allows selective recovery of files if
# need be)
# 6) rsync the differences between the snapshot FS and the backup FS
# 7) Unmount the backup FS
# 8) Unmount the snapshot FS
# 9) Delete the snapshot LVM (leaving it will degrade performance)
# 10) Repeat 3-9 if necessary

# Give us a large enough path to execute commands
PATH=/bin:/sbin:/usr/bin:/usr/sbin

# Save the old IFS variable so that we can restore it when we're done
ORIGIFS=$IFS
IFS=$(echo -en "\n\b")

[ -f "$CONF_PATH/backup.conf" ] || {
echo "Configuration file not found ($CONF_PATH/backup.conf). Exiting.";
exit 1; }

# Read from the configuration file any exclude filters for the rsync output
LOG_EXCLUDE=$(cat "$CONF_PATH/backup.conf" | egrep "^[ \t]*LOG_EXCLUDE=" |
cut -d'=' -f 2-)
LOG_FILE=/tmp/backup$$.log

# Get each line from the configuration file one at a time, ignoring anything
# that starts with a pound (#) sign
for i in $(cat "$CONF_PATH/backup.conf" | egrep -v "^[ \t]*#.*|^[ \t]*LOG_EXCLUDE=.*"); do
# Parse out the locations we need
VOL=$(echo $i | sed "s@^\([^:]*\):\(.*\)@\1@")
TARG=$(echo $i | sed "s@^\([^:]*\):\(.*\)@\2@")

# Check to make sure the paths are valid
[ -b $VOL ] || { echo "Source volume not valid: $VOL. Skipping";
continue; }
[ -f $TARG ] || { echo "Target image not valid: $TARG. Skipping";
continue; }

# Get necessary information about the source volume.
# Get only the necessary line, and not the headers, and then compress the
# info so it is only separated by a single space (makes finding fields
# easier in the next steps). There will be an extra space at the beginning,
# so fields are 1 greater than what they seem.
#
# Make sure the volume is a LVM Logical Volume
lvs -a $VOL 2>/dev/null 1>/dev/null || {
echo "$VOL not an LVM Logical Volume ($?). Skipping."; continue; }
# Now we actually collect the info
INFO=$(lvs -a $VOL | tail -n 1 | sed 's/[ ][ ]*/ /g')
# Field 1: Logical Volume Name
VOL_NAME=$(echo $INFO | cut -d' ' -f 2)
# Field 2: Volume Group Name
GROUP_NAME=$(echo $INFO | cut -d' ' -f 3)
# Field 3: Attributes - Not needed right now, but may come in handy
ATTRS=$(echo $INFO | cut -d' ' -f 4)
# Field 4: Volume Size
VOL_SIZE=$(echo $INFO | cut -d' ' -f 5)

# Now we just create a name for the snapshot volume, and then we can
# create it. Use a unique name guaranteed to be the same through multiple
# runs (or else blkid.tab fills with unneccessary partition info)
SNAP_NAME=backup_$(echo "$GROUP_NAME/$VOL_NAME" | md5sum | cut -c-32)
SNAP_PATH=/dev/${GROUP_NAME}/${SNAP_NAME}
lvcreate -s -L${VOL_SIZE} -n $SNAP_NAME $VOL >/dev/null || {
echo "Snapshot of '${GROUP_NAME}/${SNAP_NAME}' failed ($?). Skipping.";
continue; }

# Mount our filesystems at temporary mount locations
SNAP_MOUNT=/tmp/$SNAP_NAME
TARG_MOUNT=/tmp/target-$SNAP_NAME
mkdir $SNAP_MOUNT || {
echo "Failed to create snapshot mount point ($?). Skipping.";
cleanup; continue; }
mkdir $TARG_MOUNT || {
echo "Failed to create target mount point ($?). Skipping.";
cleanup; continue; }
mount $SNAP_PATH $SNAP_MOUNT || {
echo "Mounting snapshot failed ($?). Skipping.";
cleanup; continue; }
mount -o loop $TARG $TARG_MOUNT || {
echo "Mounting target image failed ($?). Skipping.";
cleanup; continue; }

# Update the backup file system with files from the snapshot
# IMPORTANT!!: The trailing slash on the source is necessary for it to
# work in the expected way, or else all backups will end up within a
# subdirectory with the name $SNAP_MOUNT
strace -o /tmp/backup-fs \
rsync -aHEAXxv --stats --delete --ignore-errors $SNAP_MOUNT/ $TARG_MOUNT | \
egrep -v "$LOG_EXCLUDE" >> $LOG_FILE

# Cleanup stuff is put into a function so that it can be called before
# an exit call
function cleanup() {
# Unmount the backups and remove the temporary dirs
# Errors may occur here, but we don't really care. Just pipe the errors
# to /dev/null
umount $SNAP_MOUNT 2>/dev/null 1>/dev/null
umount $TARG_MOUNT 2>/dev/null 1>/dev/null
[ -d $SNAP_MOUNT ] && rmdir $SNAP_MOUNT
[ -d $TARG_MOUNT ] && rmdir $TARG_MOUNT

# Delete the snapshot
[ -b $SNAP_PATH ] && lvremove -f $SNAP_PATH >/dev/null
}
# Now call our cleanup routine on normal completions
cleanup
done

if [ -f $LOG_FILE ]; then
# Mail the log file, and then delete the file
mail -s "Backups on $(hostname)" $MAILTO < $LOG_FILE
rm $LOG_FILE
fi

# Restore the IFS
IFS=$ORIGIFS

--
fedora-list mailing list
fedora-list@xxxxxxxxxx
To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list

Relevant Pages

  • Re: RAID gotchas!
    ... will keep my backup up-to-date). ... # 3) Create a FS snapshot using LVM ... # 6) rsync the differences between the snapshot FS and the backup FS ... cleanup; continue;} ...
    (Fedora)
  • Re: Looking for advice on rsync
    ... > they were asking about was using rsync or some other mechanism to ... I explained that the backup can't really be ... never even notice the snapshot ... We can mount the snapshot as another filesystem, ...
    (comp.os.linux.misc)
  • Re: rsync and moving files [Re: backup w/ snapshots]
    ... The idea is that I have a main computer that I want to backup. ... So I thought that every day my backup-server could rsync my main computer, ... it takes a snapshot of the filesystem. ... and cover new sectors on disk. ...
    (freebsd-questions)
  • Re: RAID gotchas!
    ... will keep my backup up-to-date). ... Instead of mounting the file systems and using rsync between them, we want to copy the initial system, and conveniently we'll develop a file system large ... When that is done, confirm the file backed up the root file system, and then delete the snapshot file system. ...
    (Fedora)
  • Re: [opensuse] Suggestions for backup software?
    ... be something capable of bridging DVDs for a total backup, and probably CDs for incremental backup. ... At home I have an old file server running 10.0 and all you need is rsync that is already installed on your system. ... The way I set up rsync is to set up a primary script that calls rsync which then reads a second file to get a list of directories to back up. ...
    (SuSE)