Re: why does scp not use all my cores on a multi-cpu machine?

At Wed, 17 Sep 2008 20:44:02 +0000 (UTC) Rahul <nospam@xxxxxxxxxxxxxx> wrote:


Robert Heller <heller@xxxxxxxxxxxx> wrote in
news:88WdnSKvocRFN1fVnZ2dnUVZ_qjinZ2d@xxxxxxxxxxxxxxx:

In the case of scp, it is really makes no sense to bother with
multithreading, even if multiple files are being transfered. Unless

Just found this though:

http://www.psc.edu/networking/projects/hpn-ssh/

Perhaps, there is some interest in parallelizing scp. I'm not sure I
understand what *exactly* they are trying to do though; is it the same
stuff that we discussed earlier in this thread?

Yes and no. They are mostly interested in increasing the size of the
internal (I/O) buffers, since that is a (big) part of the bottleneck.
The buffer size issue is actually part of the I/O bottleneck. They do
state the there is an ultimate bottleneck in the form of local I/O
throughput (disk speed) and network throughput. It is *possible* that
with large buffers and a fast enough I/O system, that then using
multiple processors/cores to speed up the cipler code *might* be a
useful performance improvement.

In the FAQ for hpn-ssh there is an important couple of Q&As
(http://www.psc.edu/networking/projects/hpn-ssh/faq.php):

"Q: Will HPN-SSH help me?
A: Maybe, it depends on a number of variables but the most important one
is the speed of your network connection to the Internet. It also depends
on the distance to the destination you are trying to reach. As a general
rule of thumb, the farther away the destination and the faster your
connection the greater the improvement will be. You can determine how
much HPN-SSH will help by multiplying the bandwidth to the destination
by the RTT (Round Trip Time). This is called the BDP (Bandwidth Delay
Product) and is expressed as BDP = BW(B/s) * RTT(s) and gives you the
number of bytes that can be in transit between any two hosts at one
time. If this value is less than the previous mention receive buffer on
the receiving host then the potential throughput of the connection will
be near line rates*. If the BDP is greater than the receive buffer the
throughput will be limited in direct proprotion to the difference
between the BDP and the receive buffer. As a rule of thumb you will
generally need at least a 10Mb/s connection to the internet to see a
benefit from HPN-SSH.

Q: I installed HPN-SSH but there is no improvement. Why not?
A: There are many possible answers to this but its important to
understand that HPN-SSH will not make every transfer faster. Transfers
in a local area network will not be improved by HPN-SSH and in some case
might even be slower (in these cases use the -o HPNDisabled=yes option).
You also have to make sure that your computer's network stacks are
properly tuned. This is especially critical on the reciever side of the
connection. Please see PSC's TCP Tuning page for details on how to do
this. You might also be limited by firewalls, packet loss, network
errors, and other problems that can affect network performance."

What these Q&As tell you is that for most people HPN-SSH is not going to
improve things. It is not useful on a LAN. It is mostly useful on a
WAN with a very high-speed network connection.





--
Robert Heller -- Get the Deepwoods Software FireFox Toolbar!
Deepwoods Software -- Linux Installation and Administration
http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
heller@xxxxxxxxxxxx -- Contract Programming: C/C++, Tcl/Tk

.

Relevant Pages

  • Re: Socket write behaviour is inconsistent?
    ... I enabled network tracing and in all cases the bytes written are the ... copy 1 byte to buffer, copy many bytes to buffer, copy one byte to ... "response" buffer already includes a termination byte. ... The server isnt supposed to close the connection; ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Windows buffering downloaded data?
    ... whenever I start to download a file or watch a video etc. from the net it takes really long time before anything starts to happen. ... When I test the network download speed with that computer, first it takes forever before the test starts to download the data and then it’s suddenly over and the test results shows that my network speed is something like 55M. ... It seems like the PC is buffering the data coming in and only then, when the buffer is full (or the connection is closed before the buffer getting full) the PC uses the data. ...
    (microsoft.public.windowsxp.network_web)
  • Re: Windows buffering downloaded data?
    ... whenever I start to download a file or watch a video ... When I test the network download speed with that computer, ... buffer is full (or the connection is closed before the buffer getting full) ...
    (microsoft.public.windowsxp.network_web)
  • Windows buffering downloaded data?
    ... whenever I start to download a file or watch a video ... When I test the network download speed with that computer, ... buffer is full (or the connection is closed before the buffer getting full) ...
    (microsoft.public.windowsxp.network_web)