[SWLUG] RAID Array data recovery

[Matthew Moore]

Hi all,

I've been having a few problems with my RAID 5 array this evening.  The
array is three 500gb drives in a RAID 5 config, using MDADM.  /dev/sdb.
/dev/sdc & /dev/sdd and the array is /dev/md0.  This morning in a bit of
a hurry I forced a power down, which seems to have corrupted the
superblock of the array.  I've been unable to create a backup superblock
(e2fsck -b 32768 /dev/md0) and /dev/sdd doesn't have a valid partition
table.  So while the array runs fine, it won't mount.  I've tried to
create a new Raid autodetect partition on /dev/sdd, but that doesn't
seem to work.  So in rough order I've done:

e2fsck /dev/md0 (doesn't work due to corrupt superblock)
e2fsck -b 32768 /dev/md0
cfdisk to create a new Linux raid autodetect partition

Creating the new Linux raid autodetect partition doesn't work, so I
guess that /dev/sdd is probably b0rked?

Some outputs:

root at hugh:/home/matt# dmesg | tail
EXT3-fs error (device md0): ext3_check_descriptors: Block bitmap for
group 880 not in group (block 0)!
EXT3-fs: group descriptors corrupted!

==================================================

root at hugh:/home/matt# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Mon Mar 17 17:17:09 2008
     Raid Level : raid5
     Array Size : 976772992 (931.52 GiB 1000.22 GB)
  Used Dev Size : 488386496 (465.76 GiB 500.11 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Sep 24 21:00:32 2009
          State : clean, degraded, recovering
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 43% complete

           UUID : e98ae76b:080944cc:619359b0:590c7d11
         Events : 0.20

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       3       8       48        2      spare rebuilding   /dev/sdd


(have removed and readded /dev/sdd, going to let it rebuild before
faffing further)

===================================================

root at hugh:/home/matt# fdisk -l /dev/sdd

Disk /dev/sdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdd doesn't contain a valid partition table

> root at hugh:/home/matt# fdisk -l /dev/sdc
> 
> Disk /dev/sdc: 500.1 GB, 500107862016 bytes
> 255 heads, 63 sectors/track, 60801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdc1               1       60801   488384001   fd  Linux raid autodetect

====================================================

root at hugh:/home/matt# e2fsck /dev/md0
e2fsck 1.39 (29-May-2006)
Group descriptors look bad... trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

The above happens when you try and mount the RAID array as well.

>From the above, is it possible to recover things?

If not on to plan B!  I can afford to buy a new 1tb hdd tomorrow if I
have too (would rather not, but if it can't be avoided).  I was then
thinking it would be possible to recover everything using dd.

My plan is:

Use dd to copy /dev/md0 to the 1tb hdd
Mount the img of /dev/md0
Backup the important data on /dev/mdo to my PC
Recreate the RAID 5 array from scratch (re-format and re-create)
Copy the img of /dev/md0 back to the newly created RAID array
Mount the RAID array and all my data is back where it needs to be?

Would that work?

Am I missing anything?

In the future, I'm giving up on RAID and just manually mirroring drives
using fsync once a week.  It'll cost about the same in hard drives and
then I won't have to worry about faffing with mdadm and software RAID
all the time.

Cheers,

Matt