[Nottingham] Parallel rsync?

[Martin]

On 29/07/13 14:26, Jason Irwin wrote:
> On 29/07/13 13:01, Dan Caseley wrote:
>> I've no idea how to achieve this, but i thought I'd ask for my own
>> curiosity - what's the advantage to running parallel rather than series?
>> Isn't the pipe going to be the limiting factor here in any scenario?
> We're having something of a "file transfer war" in the office at the
> moment and one of .
> A normal Windows copy takes ~10 hours
> robocopy 15 mins
> rysnc ~8 mins!
> 
> robocopy can execute in a multi-threaded manner to alleviate the
> bottle-neck over what files to copy (this can take some time, especially
> with lots of files, even for rsync).  As we may start to copy very large
> folder structures, that is a concern.
> 
> I want rsync to win, dammit, as robocopy will suck over the VPN (no
> compression, file delta etc).

Something like?:

parallel -j +0 -m rsync args ::: source_folder\*


rsync will be faster than such as tar only if you can take advantage of
rsync updating only what has changed for files...

Beware also the Windows silliness whereby various office files can get
updated *without* the modified time being updated.


Have fun!

Cheers,
Martin


-- 
- ------------------ - ----------------------------------------
-    Martin Lomas    - OpenPGP (GPG/PGP) Public Key: 0xCEE1D3B7
- martin @ ml1 co uk - Import from   hkp://subkeys.pgp.net   or
- ------------------ - http:// ml1 .co .uk/martin_ml1_co_uk.gpg