[Watford] help replicating large amounts of data

[Steven Acreman]

I don't think there is anything better than rsync. It was written by a genius.

Just use the faster options like timestamps over md5.

Sent from my iPhone

On 13 Jun 2012, at 08:33, Rob Jefferis <rob at letchmore.co.uk> wrote:

> Hi all.
> 
> Has anyone got any ideas on the best / quickest way to replicate large volumes of data between servers.
> 
> We currently have 2 NAS boxes and would like to replicate 1 to the other. The underlying technology for the sync is rsync and I dont think this is really going to handle the volume of data.
> 
> I have just done a count up on one of the larger shares and it is currently around 13TB with 1.5 million folders and around 100 million files.
> 
> We have toyed with building some large Windows file servers and using DFSR but I dont think this would handle it either.
> 
> Anyone got any ideas on software / hardware that might help? I would estimate the total required storage at primary to be around 50TB so a guestimate would put that at around 350 million files.
> 
> The data needs to be available to Windows machines.
> 
> Thanks
> 
> Rob
> _______________________________________________
> Watford mailing list
> Watford at mailman.lug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/watford
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/watford/attachments/20120613/076acab3/attachment.htm>