backup throughput


I've got a second hard drive, identical to the main one, onto which I
(nightly) duplicate the main hard drive, thus ensuring that I always
have a complete backup that is less than a day old.

I wrote a script to do this, and cron fires it off at 4am. I log the
results of that script, complete with a running account of how far it's
got [1] so I can see when the slowdown (if there is one) happens. A
sample is at http://royalty.no-ip.org:81/backup-throughput-2006-11-24.txt .
Columns are time, GB_copied, MB/s. You can see that between 4:31:13 and
4:31:43 throughput dropped from40: MB/s to ~34 MB/s. Apparently
something started then that used up the IDE or PCI bandwidth (dd isn't
very CPU-bound), probably something invoked by cron, as nobody should be
on the computer at 4:30. I'm guessing the mystery process runs after
something else finishes (which might happen at a different time each
day), because the slowdown happens at a slightly different time each
day. Some days the backup doesn't happen at all, as if dd started then
immediately exited; I haven't figured that one out yet.

Inside the loop that does "kill -USR1 ; sleep" until dd exits, I also
have it dump a process list to disk, sorted as top(1) does, but I see
nothing peculiar at that time.

So how do I figure out what my mystery process is?

Setup:

I have 2 250GB drives. They both are this kind:

/dev/hdc (the main drive):

Model=ST3250620A, FwRev=3.AAC, SerialNo=5QF03SXM
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
AdvancedPM=no WriteCache=enabled
Drive conforms to: device does not report version:

* signifies the current active mode

/dev/hde (the backup drive) is the same, only connected by this:

0000:00:0c.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) PCI0680 Ultra ATA-133 Host Controller (rev 02)
Subsystem: Silicon Image, Inc. (formerly CMD Technology Inc) PCI0680 Ultra ATA-133 Host Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, Cache Line Size: 0x01 (4 bytes)
Interrupt: pin A routed to IRQ 11
Region 0: I/O ports at b800 [size=8]
Region 1: I/O ports at b400 [size=4]
Region 2: I/O ports at b000 [size=8]
Region 3: I/O ports at a800 [size=4]
Region 4: I/O ports at a400 [size=16]
Region 5: Memory at d5000000 (32-bit, non-prefetchable) [size=256]
Expansion ROM at 50000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-

[1] from dd(1):
Note that sending a SIGUSR1 signal to a running `dd' process makes it print
to standard error the number of records read and written so far, then to
resume copying.

$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid; sleep 1; kill $pid

10899206+0 records in 10899206+0 records out

From two successive such readings I can compute the throughput in the
interval.

--
I firmly believed we should not march into Baghdad ...To occupy Iraq
would instantly shatter our coalition, turning the whole Arab world
against us and make ... a latter-day Arab hero assigning young soldiers
to a fruitless hunt for a securely entrenched dictator[.] -- GHWB

--
A well-lovd and corrctly traind domstc cnine is gnrlly slobbry, excitbl,
noisy, scatologically obsessed, xenophobic, pathetically unjudgmental,
embrrssngly uninhbtd, unreasnngly dvtd, hrtbrkngly dpndnt and wretchedly
craven. All othr knds of dog cmpre unfvrbly wth ths picture. - PB, AFCA
.

Relevant Pages

  • IDE throughput variance
    ... thus ensuring that I have a complete backup that is no more than 24 ... DMA modes: mdma0 mdma1 mdma2 ... The drives and interface card are new (from ... a latter-day Arab hero assigning young soldiers ...
    (comp.os.linux.hardware)
  • Re: OT-True Image Backup
    ... Make sure the USB drives file system is NTFS and not FAT32.. ... Jaymon is correct in that the disk image you create and save to the D: ... partition of your USB external HDD will have no effect on the other ... Step-by-Step Instructions for Using the Acronis True Image Program to Backup ...
    (microsoft.public.windowsxp.help_and_support)
  • Re: OT-True Image Backup
    ... Make sure the USB drives file system is NTFS and not FAT32.. ... Jaymon is correct in that the disk image you create and save to the D: ... partition of your USB external HDD will have no effect on the other ... Since the disk images you will be creating (at least the initial backup ...
    (microsoft.public.windowsxp.help_and_support)
  • Re: Back up data to USB?
    ... A flash drive is fine as a secondary means for storing backups of particular files, but I don't think it's suitable for storing drive images unless the amount of data is very small, nor would I rely on a USB flash drive as the only media for the backup of data. ... It also can do file backups and disk cloning. ... can take to back up the entire contents of one's day-to-day working HDD, ... With both hard drives connected, ...
    (microsoft.public.windowsxp.general)
  • Re: Need help backing up my C: drive
    ... I tried using Windows' backup and restore which works for important data ... consider a disk imaging program such as ... Acronis True Image to clone the contents of your day-to-day working HDD onto ... With both hard drives connected, ...
    (microsoft.public.windowsxp.general)