[GLLUG] Transferring high volumes of data.
jjllmmss at googlemail.com
Wed Jun 11 19:09:31 UTC 2014
Sorry James, could not reply earlier.
I see almost 5ms when I ping between hosts (TCP or UDP are pretty similar).
I haven't checked the network cards, but I am pretty certain we are using
at least 1Gb ones, I'll review that tomorrow.
On 11 June 2014 13:23, James Courtier-Dutton <james.dutton at gmail.com> wrote:
> On 11 June 2014 06:52, James Courtier-Dutton <james.dutton at gmail.com>
> > On Jun 11, 2014 1:16 AM, "JLMS" <jjllmmss at googlemail.com> wrote:
> >> HI,
> >> I am wondering what are people out there doing to transfer high volumes
> >> data (100 GB or more every time) between geographically distant sites.
> >> I started using rsync (over ssh, including using a version of ssh
> >> optimized for performance during file transfers) and got very poor
> >> performance (3-7 MB/s).
> >> I started to play with sending data in parallel (going as far as
> >> some files) and although I improved speed by a factor of 3 or 4 times,
> >> time the transfers take is still unsatisfactory.
> >> I started by opening many instances of rsync and my bottleneck became
> >> amount of sessions ssh can handle before starting to drop connections,
> >> the end I had to settle for running around 10 rsync instances, this
> >> much better but would be considered still slow by the powers that be.
> >> I would appreciate any ideas, pointers, etc that may make possible to
> >> transfer such amounts of data in an efficient manner as possible. I am
> >> looking for expertise in the field rather than assisted googling
> >> if you find something very,very interesting I would of course love to
> >> about it) :-)
> > What is the latency of the link?
> > What is the bandwidth of the link?
> > You might need to use transfer protocols more suited to satellite links.
> > James
> Do you have answers to my questions?
> I am asking them because "latency" is a very important factor that
> limits file transfer speed when using TCP.
> You could have a 10Gig Bandwidth link, but if your latency is high you
> will get rather slow file transfers.
> In order to predict file transfer speeds you need "bandwidth" and
> "latency" as input to the calculations.
> Kind Regards
> GLLUG mailing list
> GLLUG at mailman.lug.org.uk
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the GLLUG