[Nottingham] Copy without copying? (file copy deduplication)

[J J]

I need read/write access to the new folder.
Mucking around with OverlayFS I can effectively have a "Template" folder
and then a "Working" folder (plus a couple of others for reasons) and that
works well. Except for the fact that full file copies still happen and some
of these files are big.
So the situation is definitely better, but far from ideal.

I am going to see if I can grok XFS enough to do the similar thing, but do
some block-level magics.
LVM snapshots might also do it, but not sure if they would allow for
multiple, concurrent "Working" folders and AIUI that would mean mucking
around with how the system is deployed, not something I have control over.

The other wrinkle is that this needs t be done ad hoc.
But first things first...

On Fri, 7 Jun 2019 at 17:35, Martin via Nottingham <
nottingham at mailman.lug.org.uk> wrote:

> I similarly use btrfs with rsync for making deduplicated backups.
>
> The btrfs read-only snapshot feature also is ideal for backing up an
> active live system and being able to do a read-back check of the backup
> against the snapshot... All seamlessly.
>
> Note that the deduplication with rsync is at the file level. The
> copy-on-write for btrfs itself is at filesystem block level (16kBytes at
> the moment I believe for x86 systems).
>
> (For the sake of my paranoia, on the versioned backups themselves I use
> the rsync reflink to make a new snapshot rather than take a btrfs
> snapshot and then use resync --delete to update and clean up...)
>
> There are also experimental block level deduplicators for btrfs, both
> real time or offline. Not had the need to try those...
>
>
> I've also used LVM snapshots in the past and they work very well.
> However, my view is that LVM has now been long superseded by the
> features and operation of btrfs.
>
>
> All good fun.
>
> Good luck!
>
> Cheers,
> Martin
>
>
>
> On 07/06/2019 17:13, VM via Nottingham wrote:
> > well, a simple lvm snapshot would work that way, but you'd have to
> create a new volume group etc. and make sure its size is sufficient for
> syoring all changed blocks.
> >
> >
> > On June 7, 2019 3:47:19 PM UTC, J J via Nottingham <
> nottingham at mailman.lug.org.uk> wrote:
> >> Thanks for the tips. Overlay is deffo an improvement as it will only
> >> take a
> >> copy of a file if there is a change but it's still a full copy of the
> >> entire file.
> >> I am wondering if there's a way that XFS, ZFS or whatever can do
> >> something
> >> even deeper and maintain partial/block differences or something.
> >>
> >> On Fri, 7 Jun 2019 at 16:25, Andy Smith via Nottingham <
> >> nottingham at mailman.lug.org.uk> wrote:
> >>
> >>> Hello,
> >>>
> >>> On Fri, Jun 07, 2019 at 03:12:21PM +0100, J J via Nottingham wrote:
> >>>> With Ext4, is there some magic way I can create and ad hoc
> >> "differencing
>
>
> --
> Nottingham mailing list
> Nottingham at mailman.lug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/nottingham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/nottingham/attachments/20190610/8bc9c4bb/attachment.html>