secure transfer of huge files between two hosts?

From: peter pilsl (pilsl_at_goldfisch.at)
Date: 10/09/04 

Date: Sat, 09 Oct 2004 23:00:38 +0200

How do you transfer huge files (2-18Gigs) between two hosts?
It needs to be secure and it must offer a REGET-possibility (continue
download after the transfer was interrupted)

My first attempt was simply using https, which turned out to be
difficult, cause apache does not support big files. One can recompile it
to include big-file support, but there seem to be several problems and I
really dont want to go out to 10machines and recompile apache just for
this purpose. (and one would need to recompile all modules as well,
which always is a pain in the back)

Second attempt was scp, which does not support REGET.

Third attempt was ftp via a ssh-tunnel. I didnt get it to work.
(ssh -L 4021:localhost.21 user@remote.host -> passive -> dir ->
connection refused) and I remember always having troubles ftp via ssh.

Fourth attempt was psftp (there is a unix-port of putty), which supports
REGET (great tool : psftp) but does not support big files either. 2GB
ist the limit here as well.

Fifth attempt was rsync via ssh, which takes a *very* long time to
determine the needed delta-pieces and then fails anyway (there was
enough space on the targetdrive ...)
write failed on "/extra/monthly.imap.00-06-15.tgz": No space left on device
rsync error: error in file IO (code 11) at receiver.c(305)
rsync: connection unexpectedly closed (77 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)

I tried with different checksum-blocksizes, but always get the same result.
Rsync is an overkill anyway, cause it assumes possible changes all over
the file and not only at the end.

My last attempt will be to invoke dd or split to split the file to
chunks<2GB and transfer them seperately. On the other hand: we almost
have 2005 and I'm sure there is a solution to do it in once and I just
dont think about it by now.

thnx,
peter 

-- 
http://www2.goldfisch.at/know_list
http://leblogsportif.sportnation.at

Relevant Pages

  • Re: secure transfer of huge files between two hosts?
    ... peter pilsl wrote: ... cause apache does not support big files. ... One can recompile it ... > Rsync is an overkill anyway, cause it assumes possible changes all over ...
    (comp.os.linux.networking)
  • rsync error: error in file IO
    ... rsync: connection unexpectedly closed ... rsync error: error in rsync protocol data stream at ...
    (alt.linux)
  • rsync error: error in file IO
    ... rsync: connection unexpectedly closed ... rsync error: error in rsync protocol data stream at ...
    (alt.os.linux)
  • Re: rsync for OpenVMS?
    ... The current rsync() code assmes the presence of a UNIX shell, ... take a bit of a rewrite to move it to OpenVMS. ... A POSIX threads based port would probably get support from the ... A test server is needed to validate an implementation. ...
    (comp.os.vms)
  • using usb storage
    ... Using rsync to copy files to usb storage: ... Broken pipe ... rsync error: error in rsync protocol data stream at io.c ...
    (Linux-Kernel)