Re: backup throughput

Hactar wrote:


I've got a second hard drive, identical to the main one, onto which I
(nightly) duplicate the main hard drive, thus ensuring that I always
have a complete backup that is less than a day old.

I wrote a script to do this, and cron fires it off at 4am. I log the
results of that script, complete with a running account of how far it's
got [1] so I can see when the slowdown (if there is one) happens. A
sample is at http://royalty.no-ip.org:81/backup-throughput-2006-11-24.txt
.
Columns are time, GB_copied, MB/s. You can see that between 4:31:13 and
4:31:43 throughput dropped from40: MB/s to ~34 MB/s. Apparently
something started then that used up the IDE or PCI bandwidth (dd isn't
very CPU-bound), probably something invoked by cron, as nobody should be
on the computer at 4:30. I'm guessing the mystery process runs after
something else finishes (which might happen at a different time each
day), because the slowdown happens at a slightly different time each
day. Some days the backup doesn't happen at all, as if dd started then
immediately exited; I haven't figured that one out yet.

You would be well-advised to treat your backup process with a little more
care; start by running the backup live, and watch what it actually does.
Then I would also enable verbose reporting (or a debug mode) when running
dd, as that will allow you a chance to see what happens when it suddenly
dies.
Consider using something a little less "atomic" than dd; either rsync or an
actual backup program will be faster and more reliable.

As to what could be started around 4:30 every night - it is most likely one
of the other cron processes, like updatedb or logrotate.
Both of these will consume significant disk resources for a short period.

Also, if your system partitions are on the drives used for the backup, of
course you'll never get a constant throughput - on IDE, this is fantasy in
any case.


--
All your bits are belong to us.
.

Relevant Pages

  • Re: Cron message puzzle
    ... On Sunday 28 January 2007 23:19, Cameron Simpson wrote: ... There is a small backup job ... Is there a /var/log/maillog file tracing mail delivery? ... I can find no sign that the cron job ...
    (Fedora)
  • Re: Cron message puzzle
    ... | getting logwatch messages, but no cron ones. ... There is a small backup ... Is there a /var/log/maillog file tracing mail delivery? ... the cron job has actually run today - except that files created as late ...
    (Fedora)
  • Re: Changing Date Results in Cron Jobs not Running
    ... >> When our company turns the year, they want me to reset the date back ... >You can get them to run by restarting cron. ... If you have a rotating backup schedule (doing full backups ... Then run other backup and maintenance scripts... ...
    (comp.unix.sco.misc)
  • Re: Backup plan help....
    ... > run raid 55 but oh well) I need a once nightly backup, ... > the scripts to do it, but would rather not as my shell scripting is a ... > MySQL database, the rest of the files change rarley. ... tar up the web pages from cron ...
    (linux.redhat)
  • Re: Cron issue
    ... Bob Bailin wrote: ... When the backup ... >> use the mail command to see cron output I only see a small subset of the ... Replacing tar or cpio I do not believe will fix this problem. ...
    (comp.unix.sco.misc)