[Durham] File sync, large scale

[Edmund Fisher]

Hey,

How about keeping it simple and running rsync as a daemon?

--
Ed

On Sun, Mar 1, 2020 at 9:28 PM Lucky Lee via Durham
<durham at mailman.lug.org.uk> wrote:
>
> Evenin' all.
>
> I have a need to stand up two Gluster clusters (likely multiple volumes) in different physical locations and sync data between them in both directions; effectively, they'll mirror each other at a file level.
>
> Latency of some minutes between a file appearing one side and being sync'd the other side is acceptable (by that I mean I don't need synchronous writes or updates within seconds).
> I've got around around 1Gbps of usable bandwidth between sites.
>
> I did consider Gluster geo-replication but that is one-way.
>
> In terms of scale, there's around 20 million files and approx 1Tb of data.  Some of the data (tens of thousands of files) will change at least daily, but the vast majority will be unchanged/static from day to day.
>
> I'm current considering syncthing & unison.  Any other recommendations?
>
> Thanks in advance,
> -Lee
>
>
>
>
> --
> Durham mailing list   -   Durham at mailman.lug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/durham
> http://www.nelug.org.uk/