Re: rsync backup to ext3-formatted usb flash drive?

On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote:
On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:

Hi. I'm using svn-fast-backup (in the subversion-tools package) to
make
rsync backups of my subversion repository. The place I'm backing
up to
is on an ext3-formatted usb flash drive, and I'm wondering if this
is a
wise thing to do.

I know that flash drive blocks wear out after a certain number of
writes, and rsync uses hard links to avoid duplicate backup files. My
question is, does this mean that the inodes' reference counts will be
changed frequently enough that it might wear out that block of the
flash
drive any time soon? (I currently backup about 900 files one or two
times a day.) If so, is ext3 capable of using another block for the
inode, or will I lose the ability to read/write some files completely?

Thanks,
Brian


Here are some basic facts about USB flash drives:


The underlying NAND flash RAM is guaranteed for 100K writes (older
technology) or 1M (more recent). After that it's retention time
starts to decrease, sometimes to as little as a few weeks, and with
continued use eventually to complete failure. The retention time of
a new, fully-functional NAND flash RAM is hundreds of years.

USB flash drives have controller chips in the drive that, among other
things, talk USB protocol to the host. One of the other things they
do is try to distribute the writes over the available blocks,
extending the lifetime of the device beyond the raw lifetime of the
underlying NAND flash. But the firmware in the controller usually
assumes you are using a FAT file-system. If, for example, you use
EXT3 rather than FAT, you will probably confuse it. I don't know
what the effect will be on device life, but I'm guessing it will
probably be something like a return to the behavior of the raw NAND
flash.

Write blocks are large (4-8K or more). If you update one inode in a
block, you have to read the entire block, update the inode and write
the entire block back again. Each write to an inode causes an
effective write to all the inodes in the same block. So there's
roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode =
64 inodes/block) in the number of writes. Thus any given single
inode is good for roughly 16K updates before it wears out.

Let's suppose your rsync backup causes a small number of updates for
each inode it touches. Call it 10, conservatively (it very well
could be as low as 1 or 2 -- I don't know enough of the details of
how rsync works at the filesystem level). That gives you about 1600
rsync runs before you start to wear out the inodes. At twice a day,
you can safely use your USB flash drive for 800 days, or between 2
and 3 years.

So here's what I'd do. Use your USB flash drive for a year, then
retire it to the long-term archives and replace it with a new one.
Let that be one of those things you do on or near your birthday, like
schedule a checkup with your Doctor. Also: dismount the flash drive,
force an fsck, and remount it, every time you do a backup. This will
read and sanity-check every inode on the drive, without doing any
writes. If the fsck shows random errors, replace the drive because
it's probably starting to wear out.

Sound reasonable?


Rick



It does sound reasonable, with one sticking point: a lot of space in the
flash drive will never be used. I should have mentioned that; despite
having 900 files, the repository is currently only about 5 MB, and the
drive itself is 4 GB.

I'm looking at an alternative utility, svn-backup-dumps, in the same
package. I could do ~800 full backups, or many more incremental backups
with a full backup here and there, before running out of space. That
would only write most blocks once or twice, and once it filled up I
could erase old backups with only one write and reuse it, right? (If I
do this, should I reformat to FAT so the controllers won't be confused?)

Thanks,
Brian



--
To UNSUBSCRIBE, email to debian-user-REQUEST@xxxxxxxxxxxxxxxx
with a subject of "unsubscribe". Trouble? Contact listmaster@xxxxxxxxxxxxxxxx

Relevant Pages

  • Re: rsync backup to ext3-formatted usb flash drive?
    ... and rsync uses hard links to avoid duplicate backup files. ... USB flash drives have controller chips in the drive that, among other things, talk USB protocol to the host. ... If you update one inode in a block, you have to read the entire block, update the inode and write the entire block back again. ...
    (Debian-User)
  • usb scrambles screen
    ... I use usb flash drives for backup they are formated with and extended ... Today I created an ext3 file system on a new usb flash drive mounted it ...
    (Ubuntu)
  • Weird Stuff with VS2005
    ... I use one of those flash drives to take backups of key files on my ... It uses an XML file than can be edited by a form in the backup app to ... ALL of the controls on the form refuse to fire their events. ... Copy the new bin files to Program Files and it now works like ...
    (microsoft.public.dotnet.languages.vb)
  • Re: Backup question
    ... Melba's Jammin' wrote: ... The most recent backup protects you against machine failure. ... What I'm doing is using a pair of USB flash drives in combination with my DVD drive backing up in duplicate to the flash drives each day and watching the file size as it grows. ... You can of course do multisession burning with applicable software or your Utilities, but I have found those flash drives to be extremely stable and dependable as an interim step to burning that allows rewriting and editing of files with media that is OFF the system. ...
    (comp.sys.mac.apps)
  • Re: why would speed of ntfs do
    ... yes it is one of their usb flash drives. ... What would you need to know about the backup other than the 100 mb size ... I'm going to guess you're referring to a USB flash drive, since Sandisk ...
    (microsoft.public.windowsxp.perform_maintain)