[Sussex] LVM and disk failure - how to recover?
Karl E. Jorgensen
karl at jorgensen.org.uk
Tue Jul 24 16:18:00 UTC 2007
On Tue, Jul 24, 2007 at 01:03:01PM +0000, Andy Smith wrote:
> Hi Karl,
>
> On Mon, Jul 23, 2007 at 10:54:03PM +0100, Karl E. Jorgensen wrote:
> > The bad news: I've got a volume group of about 800Gb, where one disk has
> > now failed miserably. No mirroring or striping. Just a bunch of disks.
>
> [...]
>
> > So, where do I go from here? Ideas are welcome and may be rewarded by
> > beverages (!)
>
> vgreduce to remove the dead LVs then set up linux software RAID?
Hadn't thought of that. D'oh.... But no luck...:
braun:~# vgreduce --removemissing --verbose --test vgbig
Test mode: Metadata will NOT be updated.
Finding volume group "vgbig"
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Wiping cache of LVM-capable devices
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda: read failed after 0 of 4096 at 500107771904: Input/output error
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 1024 at 500105150464: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 488552464384: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 118925230080: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find all physical volumes for volume group vgbig.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Test mode: Skipping archiving of volume group.
vgbig/mythtv has missing extents: removing (including dependencies)
vgbig/dar2 has missing extents: removing (including dependencies)
Removing PV with UUID lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3 from VG vgbig
Removing LV mythtv from VG vgbig
Removing LV dar2 from VG vgbig
Test mode: Skipping volume group backup.
Wrote out consistent volume group vgbig
Test mode: Wiping internal cache
Wiping internal VG cache
Since the only UUID listed belongs to the bad PV, I was undeterred and
continued without using --test:
Finding volume group "vgbig"
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Wiping cache of LVM-capable devices
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda: read failed after 0 of 4096 at 500107771904: Input/output error
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 1024 at 500105150464: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 488552464384: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 118925230080: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
...
[snip 50+ lines of i/o errors ad nauseum]
...
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Archiving volume group "vgbig" metadata (seqno 69).
vgbig/mythtv has missing extents: removing (including dependencies)
vgbig/dar2 has missing extents: removing (including dependencies)
Removing PV with UUID lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3 from VG vgbig
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
..
[snip another 35 lines of i/o errors]
...
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
/dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Found volume group "vgbig"
Found volume group "vgbig"
Loading vgbig-mythtv table
Suspending vgbig-mythtv (254:3)
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Couldn't find device with uuid 'lhGK56-NUIa-AJpQ-iUKL-4lHh-u5sF-TPaZA3'.
Volume group vgbig metadata is inconsistent
Found volume group "vgbig"
Found volume group "vgbig"
Loading vgbig-dar2 table
Suspending vgbig-dar2 (254:5)
Found volume group "vgbig"
Resuming vgbig-mythtv (254:3)
Found volume group "vgbig"
Resuming vgbig-dar2 (254:5)
Deactivating (if active) logical volume mythtv
Found volume group "vgbig"
Removing vgbig-mythtv (254:3)
Found volume group "vgbig"
Deactivating (if active) logical volume dar2
Found volume group "vgbig"
Removing vgbig-dar2 (254:5)
Found volume group "vgbig"
Removing LV mythtv from VG vgbig
Removing LV dar2 from VG vgbig
Creating volume group backup "/etc/lvm/backup/vgbig" (seqno 71).
Wrote out consistent volume group vgbig
And things look a *lot* healthier. I lost the mythtv and dar2 LVs (I'd
forgotten about dar2, but it was never mounted anyway), and the
Most commands still try to read from /dev/sda1, but aren't bothered
by it - I presume that's just lvm trying to find new PVs to play with.
I'm happy and no longer feel scared by the prospect of booting it.
What's your choice of beverage? It's well-deserved
--
Karl E. Jorgensen
karl at jorgensen.org.uk http://www.jorgensen.org.uk/
karl at jorgensen.com http://karl.jorgensen.com
==== Today's fortune:
Banacek's Eighteenth Polish Proverb:
The hippo has no sting, but the wise man would rather be sat upon
by the bee.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mailman.lug.org.uk/pipermail/sussex/attachments/20070724/a05035b6/attachment.pgp
More information about the Sussex
mailing list