[GLLUG] Transferring high volumes of data.

[John Edwards]

Hi

On Wed, Jun 11, 2014 at 01:15:21AM +0100, JLMS wrote:
> I am wondering what are people out there doing to transfer high volumes of
> data (100 GB or more every time) between geographically distant sites.

I've used rsync over ssh to shift 100s of GB many times over both LAN
and Internet (leased line and ADSL). I have found rsync to be very
efficient in using all available bandwidth, sometimes so much so that
it interferes with other traffic and we use '--bwlimit' to limit the
bandwidth it will use.


> I started using rsync (over ssh, including using a version of ssh optimized
> for performance during file transfers) and got very poor performance (3-7
> MB/s).

Look at the Internet connections at both ends and test to a 3rd
location to see if there are bandwidth limits or throttling on the SSH
port you are using (default is 22). Also try running SSH over a
different port.


> I started to play with sending data in parallel (going as far as splitting
> some files) and although I improved speed by a factor of 3 or 4 times, the
> time the transfers take is still unsatisfactory.
<snip> 

Running multiple sessions of rsync over SSH in parallel should be
less efficient because of the multiple processes sitting in RAM, and
contending for CPU and I/O access.


-- 
#---------------------------------------------------------#
|    John Edwards   Email: john at cornerstonelinux.co.uk    |
#---------------------------------------------------------#
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mailman.lug.org.uk/pipermail/gllug/attachments/20140611/b4dc3fc2/attachment.pgp>