[Gllug] Loopback mountable image file compression.

[Richard Jones]

On Sat, Aug 22, 2009 at 09:02:54AM +0100, general_email at technicalbloke.com wrote:
> I have a need to make and archive dd images of windows machines but
> naturally these are going to use a whole heap of space so I'd like to
> compress them. Wanting also to eat my cake it would be nice to be able
> to mount them as loopback devices without uncompressing the entire
> volume first so straight ahead gzip and other stream compression progs
> are out. I'm therefore wondering what block compression I could use:
> cloop, squashfs or something else I've not heard of. It would need to
> handle very large files, possibly multi terrabyte.

Have you looked at qcow2?  The compression offered by qcow2 is block
based, so it should perform much better than stream-based compression.
(cloop is also block-based, but qcow has other features).

  qemu-img convert -c -f raw input.img -O qcow2 output.qcow2

For examining and modifying the images later I'd naturally pimp
http://libguestfs.org/ It's not available in Ubuntu because Ubuntu's
debootstrap is broken, but you can use the Debian package linked from
the FAQ.  If that doesn't work, email me next week and I'll have a
proper go at fixing the Ubuntu build.

libguestfs can read qcow2 and cloop images, ext2/3/4, NTFS, and
ordinary partition tables and LVM.  (Basically anything that you could
mount on a Linux machine can be handled by libguestfs).

  $ guestfish -a my.img
  ><fs> run
  ><fs> lvs
  /dev/VolGroup00/LogVol00
  /dev/VolGroup00/LogVol01
  ><fs> mount /dev/VolGroup00/LogVol00 /
  ><fs> edit /etc/passwd

It's also fully scriptable, supporting many different languages.

> Can anyone offer some advice as to which might be best for speed /
> compression / capacity / reliability etc.

You need to test this.  Nothing will beat raw images for reliability,
mainly because they are huge and redundant.

> Also related, are there any CLI based imaging tools that are available
> in the stock Ubuntu repos which can synthesize a compacted image of an
> NTFS or EXT2/EXT3 partition based on the blocks/sectors actually used by
> that filesystem, as opposed to simply copying every block in the
> partition whether or not it is used by the filesystem? Note, this would
> have to have a CLI as I would need it to be scriptable and ideally it
> would be in the standard Ubuntu repos (Standard, Universe or Multiverse)
> to spare me the hassle of manually maintaining it.

You can possibly make ext2/3 images smaller by running zerofree on
them.  I say 'possibly' because zerofree looks like quite a dangerous
tool, and I wouldn't run it on my own filesystems unless they didn't
contain anything important or I had a very solid backup.  It's also
not clear to me how much you'd gain IRL.

libguestfs supports zerofree so you could use it to batch-compress
disk images (with a bit of scripting).

I've not heard of anything that can do the same for NTFS.  However the
NTFS format is reasonably well-understood so it should be possible to
write such a utility for it, for someone with a lot of time on their
hands ...

Rich.

-- 
Richard Jones
Red Hat
-- 
Gllug mailing list  -  Gllug at gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug