[Nottingham] Copy without copying? (file copy deduplication)

[J J]

I must be misunderstanding something, I thought the snapshot retained the
original data as things continued to get written to the source. (This seems
to be what LVM does at any rate)
Which is great, but kinda the wrong way round for my use case.
I effectively want the changes written into the snapshot, leaving the
source untouched.
Thus I can have as many snapshots as I need (parallel tests etc) and then
chuck them.
Much more like the differencing you get with Hypervisors (and OverlayFS,
just at the block level to reduce space requirements).

If XFS or something can do that, awesome, and I'll start looking at it as
soon as Kubuntu gets itself out of whatever APT fankle it's managed to get
into!
Not sure how viable BTRFS would be as a choice, I thought the project was
slowly dying off (I know RedHat has dropped it) and any proposals will need
to be passed by Internal IT, so can't be too exotic.

On Mon, 10 Jun 2019 at 11:24, Martin via Nottingham <
nottingham at mailman.lug.org.uk> wrote:

> Jason,
>
> The obvious fix for that is to use btrfs whereby:
>
> You have your pristine folders in one btrfs subvolume (can even be
> mounted or set to be read-only);
>
> Then create a snapshot subvolume from that first subvolume;
>
> Your new snapshot subvolume can then be used or mounted read-write.
>
>
> Aside: Note that you don't even need to mount the subvolumes as separate
> mounts: they can even appear as 'normal' directories on a (single)
> higher mount point.
>
>
> The snapshot volume is created with zero copying of files and is very
> fast and low resource to do. You'll see no increase in disk usage!
>
> During use, the only file writing will be only for whatever new data is
> actually written. The magic of the btrfs CoW means that only new blocks
> of data get written. Unchanged file fragments remain unchanged and
> uncopied...
>
>
> Hope that fits your needs?
>
> Cheers,
> Martin
>
>
>
> On 10/06/2019 11:04, J J via Nottingham wrote:
> > I need read/write access to the new folder.
> > Mucking around with OverlayFS I can effectively have a "Template" folder
> > and then a "Working" folder (plus a couple of others for reasons) and
> > that works well. Except for the fact that full file copies still happen
> > and some of these files are big.
> > So the situation is definitely better, but far from ideal.
> >
> > I am going to see if I can grok XFS enough to do the similar thing, but
> > do some block-level magics.
> > LVM snapshots might also do it, but not sure if they would allow for
> > multiple, concurrent "Working" folders and AIUI that would mean mucking
> > around with how the system is deployed, not something I have control
> over.
> >
> > The other wrinkle is that this needs t be done ad hoc.
> > But first things first...
>
>
> --
> Nottingham mailing list
> Nottingham at mailman.lug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/nottingham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/nottingham/attachments/20190610/c568277e/attachment-0001.html>