[Nottingham] Copy without copying? (file copy deduplication)

[Martin]

I similarly use btrfs with rsync for making deduplicated backups.

The btrfs read-only snapshot feature also is ideal for backing up an
active live system and being able to do a read-back check of the backup
against the snapshot... All seamlessly.

Note that the deduplication with rsync is at the file level. The
copy-on-write for btrfs itself is at filesystem block level (16kBytes at
the moment I believe for x86 systems).

(For the sake of my paranoia, on the versioned backups themselves I use
the rsync reflink to make a new snapshot rather than take a btrfs
snapshot and then use resync --delete to update and clean up...)

There are also experimental block level deduplicators for btrfs, both
real time or offline. Not had the need to try those...


I've also used LVM snapshots in the past and they work very well.
However, my view is that LVM has now been long superseded by the
features and operation of btrfs.


All good fun.

Good luck!

Cheers,
Martin



On 07/06/2019 17:13, VM via Nottingham wrote:
> well, a simple lvm snapshot would work that way, but you'd have to create a new volume group etc. and make sure its size is sufficient for syoring all changed blocks.
> 
> 
> On June 7, 2019 3:47:19 PM UTC, J J via Nottingham <nottingham at mailman.lug.org.uk> wrote:
>> Thanks for the tips. Overlay is deffo an improvement as it will only
>> take a
>> copy of a file if there is a change but it's still a full copy of the
>> entire file.
>> I am wondering if there's a way that XFS, ZFS or whatever can do
>> something
>> even deeper and maintain partial/block differences or something.
>>
>> On Fri, 7 Jun 2019 at 16:25, Andy Smith via Nottingham <
>> nottingham at mailman.lug.org.uk> wrote:
>>
>>> Hello,
>>>
>>> On Fri, Jun 07, 2019 at 03:12:21PM +0100, J J via Nottingham wrote:
>>>> With Ext4, is there some magic way I can create and ad hoc
>> "differencing