[Sussex] LVM and disk failure - how to recover?

[Jon Fautley]

Karl E. Jorgensen wrote:
> I've got a problem. My MythBox is broken. Rather than being forced to 
> watch live TV (5 channels only) without the pause button, advert 
> flagging, fast-forward over adverts etc, I'm trying to fix it.

Eeep.. never good :/

It sounds like it's all fixed now, but I'll throw some information in
below just for the archives sake :)

> The machine trundles on OK (but without MythTV) and works fine.  The 
> existing mount points are all reachable.

That's normal. The VG is still activated, you'll just see badness
occuring if you try and access an LV on the dead PV. Lets hope your
metadata isn't stored on that one, too (doesn't sound like it is, phew!)

> Commands like "vgdisplay" reports 'Volume group "vgbig" not found' - 
> despite the fact I've got /usr as a LV in vgbig and /usr works fine. Go 
> figure...

You've just blown a big hole in it's head, forgive it for being confusing ;)

> Even a "vgchange -a y" reports the same thing - both after a slew of 
> /dev/sda errors and the message:
> 
>     Couldn't find all physical volumes for volume group vgbig.

Yup, normal. LVM won't activate with PVs missing.

> Other commands, like pvdisplay successfully reports all the other 
> physical volumes and the "good" logical volumes, but (obviously) still 
> complains about the lack of the disk, and lists the PV as "unknown 
> device"

Sounds like the disk is royally screwed.

> Annoyingly, I cannot remove the PV either:
>     braun:~# pvremove /dev/sda1
>       /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
>       No physical volume label read from /dev/sda1
>
>     braun:~# pvremove --force /dev/sda1
>       /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
>       No physical volume label read from /dev/sda1
>       Can't open /dev/sda1 exclusively.  Mounted filesystem?
> 
> It's lying!!! - or easily confused.

Nope, that's perfectly normal. You're asking LVM to remove the PV
signature from the disk. You're not asking it to remove it from the
volume group. There's a very big difference here.

Lets assume everything is healthy... if you run the above command on a
PV before using vgreduce, then you'll essentially end up in this
situation as you've just blown away a PV to the point that LVM can't
find it. (actually, you can recover from it, as you can re-add the
signature and manually specify the UUID, but that's for another day)


>     braun:~# pvremove --force --pretty-please /dev/sda1
>     pvremove: unrecognised option `--pretty-please'
>       Error during parsing of command line.
> 
> And utterly uncooperative!  Gah!

I'd file an RFE with the developer to add that option ;)

> Obviously, the failed disk has to go back (it's 3 months old and still 
> under warranty), but that means shutting the machine down. And with the 
> errors displayed, I'm not sure that LVM will like what it sees on a 
> fresh boot...

Nope, it'll fail to activate the VG and your system will explode in a
giant fireball(*)

* - Note: May not be true. But you won't be able to activate it, and
therefore not see any data.

> 
> PS: Random fortunes are really uncanny sometimes...

Ha, nice! :)

Good luck recovering all that data! :/

/j <thinking about getting a backup of his 1.8TB media partition...>
-- 
Jon Fautley RHCE, RHCX <jfautley at redhat.com> direct: +44 1252 362815
Technical Account Manager                    office: +44 1252 362700
Red Hat UK                                   mobile: +44 7841 558683
200 Fowler Avenue, Farnborough Business Park, Farnborough   GU14 7JP

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: OpenPGP digital signature
Url : http://mailman.lug.org.uk/pipermail/sussex/attachments/20070725/c310dcf2/attachment.pgp