Re: secure transfer of huge files between two hosts?

From: peter pilsl (pilsl_at_goldfisch.at)
Date: 10/11/04 

Date: Mon, 11 Oct 2004 10:08:36 +0200

peter pilsl wrote:
>
> How do you transfer huge files (2-18Gigs) between two hosts?
> It needs to be secure and it must offer a REGET-possibility (continue
> download after the transfer was interrupted)
>

thnx a lot for all answers.

I finally came to two solutions:

A) if the file does not exist already then I use split as filter when
creating to get it into appropriate junks<2GB and then use https or
psftp to transfer it. Nevertheless I dont like split too much. I always
fear that one byte is lost when rejoining them with cat :)

B) if the file already exists I do the following: just download using
psftp also - which will stop after 2GB-limit is reached.
Additionally I use dd on the remote side (using the skip-option) to
create a file that contains the rest which is bigger than 2GB. (I use
bs=100000 here). I can download this file then and use dd on the local
side (using the seek-option) to add this rest to the already downloaded
first part.
This way is very handy cause:
i) I can start download immediately without needing to split first
ii) Splitting an existing 3GB-File into two parts needs 3GB emtpy space
and is timeconsuming after all (dealing with IDE here). Using seek and
split I need only one third of empty space and systemresources. Same on
local side.

conclusion:
*rsync is totally lost with such big files.
*support of big files is (and I fully understand this) not a
top-priority in many tools and ports.
* md5checksum is quite fine with big files for checking the transfer. On
my celeron700 with slow IDE it tooks about 7minutes to calculate the
checksum which is ok to me.
* unison looks great but its not a standardtool so I will not rely on
it. thnx.

peter 

-- 
http://www2.goldfisch.at/know_list
http://leblogsportif.sportnation.at