Re: Looking for advice on rsync

From: Michael W Cocke (cocke_at_catherders.com)
Date: 10/16/04 

Date: Sat, 16 Oct 2004 09:23:39 -0400

On 16 Oct 2004 04:51:44 -0700, "Tony Lawrence" <pcunix@gmail.com>
wrote:

>Yesterday I was at a client site where they explained that they wanted
>to keep a stand-by server up to date and ready to take over in case of
>main system failure. Fine, lots of people do that, and currently they
>are doing it by restoring backups every morning. What they were asking
>about was using rsync or some other mechanism to keep the machines
>more current.
>
>My first reaction was to question them about their app: it's
>apparently a mess of Basic programs that work on hundreds of different
>data files 24 x 7. I asked if they shut down the users for backup.
>They don't. I explained that the backup can't really be guaranteed
>consistent if users are writing data to files because obviously files
>are going to be backed up while they are being written to. As the
>files are related to each other (A/R header and detail files, indexes,
>etc.) you can have inconsistent versions on the backup media.
>. Somebody told them (obviously incorrectly) that rsync could prevent
>this. There are databases that will let you replicate data while
>you run, but that's application level, and this app has no such
>ability. Rsync can't do any better than a backup for that.
>
>I talked about snapshots, and asked if they could shut down users for
>the very brief time it takes to do that. Nope. Can't ever stop the
>flow of data. I explained that rsync can't answer that problem any
>better than a tape backup can: files and indexes may be inconsistent
>with each other.
>
>I also wonder if rsync's rolling checksum might even make things
>better or worse. On the one hand we get less data transferred as
>opposed to just a rcp or whatever, but open files that may be getting
>writes during the checksum make things even more confusing.
>
>They can't change the app. They can't shut down users. Their bank says
>they have to have better disaster recovery. These things seem
>impossible to reconcile. My feeling is that if the banks demands have
>to be met, then the user community HAS to put up with periods where
>they can't use the app. If they use snapshots, that period can be
>brief, otherwise its going to be fairly long (13 GB of data to
>transfer).
>
>Comments?

I hate to admit this, but I had to deal with the exact same problem a
few years back. The most important question that you need to have
answered is - which BASIC, and exactly which mode are the files being
opened. The reason this is critical is that, unless the programmer
actually spent some effort on the code (unlikely in a business app
written in BASIC), the files aren't actually being opened in what a
linux programmer would call shared mode, they're likely opened in
'mode 3', which is not SHARED so much as it is DENY NONE.

That being the case, there's a reasonable chance that when rsync (or
BRU, but NOT tar) open the file, the application is not detecting that
the files are open, and the locks are not holding updates until the
file is closed again.

That means that the individual files are being backed up in an
indeterminate state AND the individual files are not all in the SAME
state.

In other words, they have no useful backup at all.

Rsync can't work miracles. They're going to have to learn to shutdown
long enough for an image snapshot.

Mike- 

--
If you can keep your head while those around you are losing theirs...
You may have a great career as a network administrator ahead!
--
Please note - Due to the intense volume of spam, we have installed 
site-wide spam filters at catherders.com.  If email from you bounces,
try non-HTML, non-encoded, non-attachments,