[Nottingham] Growing a RAID 1

Thu Jan 26 11:05:21 UTC 2023

This is RAID 1, I am not doing 5/6 shenanigans.

On Thu, 26 Jan 2023 at 09:32, Michael Simms via Nottingham <
nottingham at mailman.lug.org.uk> wrote:

> Be really careful on BTRFS raid. Last I checked it was still "unfit for
> purpose" and the place I worked lost a lot of data due to raid 5 failures.
> On 26/01/2023 09:18, J I via Nottingham wrote:
>
> Thanks, Andy.
>
> I actually made the leap to BTRFS for the RAID, a couple of subvolumes and
> it all went very smoothly.
>
> Old (11.5 year old!) drives are out and I am now seriously considering
> flipping the other server.
>
> J.
>
> On Wed, 25 Jan 2023 at 17:08, Andy Smith via Nottingham <
> nottingham at mailman.lug.org.uk> wrote:
>
>> Hi,
>>
>> On Tue, Jan 17, 2023 at 11:58:01PM +0000, J I via Nottingham wrote:
>> > Can anyone point me at some instructions on how to do that?
>>
>> These will work (have done this myself many times):
>>
>>
>> https://raid.wiki.kernel.org/index.php/Growing#Extending_an_existing_RAID_array
>>
>> So, being a bit more verbose on what steps are relevant to you:
>>
>> 1. Make sure you have backups. Growing MD RAID-1 is pretty well
>>    tested but human error is a factor especially with tasks you are
>>    unfamiliar with.
>>
>> 2. Go back to step 1 after not taking it seriously enough the first
>>    time it was read.
>>
>> 3. Remove one of the devices from your RAID-1 by marking it failed.
>>
>>     # mdadm -f /dev/md0 /dev/sda
>>
>> 4. Physically remove sda from your computer and physically attach
>>    the replacement. If you can do hot swap then lucky you, you get
>>    to do all of this without switching your computer off. Otherwise
>>    clearly you'll be shutting it down to remove each device and add
>>    each new one. It should still boot fine and assemble the array
>>    (degraded).
>>
>> 5. I'm going to assume you have power cycled in which case the drive
>>    names may have changed. Your new drive will probably now be sda,
>>    BUT IT MIGHT NOT BE, SO CHECK THAT SDA IS WHAT YOU EXPECT IT TO
>>    BE. One other likely outcome is that your previous sdb is now sda
>>    and the new drive is sdb. You can do "smartctl -i /dev/sda" to
>>    get some vitals like model and serial number. If you had hotswap
>>    and didn't power cycle, your new drive will likely be /dev/sdc.
>>    It doesn't matter what it's called; just use the new name.
>>
>> 6. Partition your new drive how you want it. I see you have used the
>>    entirety of the old drive as the MD RAID member but current best
>>    practice is to create a single large partition rather than use
>>    the bare drive. There are various reasons for this which I won't
>>    go into at this stage. So let's assume you now have /dev/sda1, a
>>    partition on your first new drive.
>>
>> 7. Add your new drive to the (currently degraded) array:
>>
>>     # mdadm --add /dev/md0 /dev/sda1
>>
>>    The array will now be resyncing onto the new drive, though it
>>    will still be the same size. Check progress:
>>
>>     $ cat /proc/mdstat
>>     $ watch -d cat /proc/mdstat
>>
>> 8. Once that's done, repeat steps (3) to (7) with sdb. You should
>>    end up with a clean running array on both the new drives, but it
>>    will still be the old (small) size.
>>
>> 9. Tell the kernel to grow the array to as big as it can go:
>>
>>     # mdadm --grow /dev/md0 --size=max
>>
>>    I've never had an issue with the bitmap stuff it mentions but if
>>    concerned then you might want to do as it says.
>>
>>    After this your array should say it is the new size, though your
>>    LVM setup will not know of that.
>>
>> 10. Resize the LVM PV so LVM knows you have more to allocate from:
>>
>>     # pvresize /dev/md0
>>
>> That should be it. At no point should you have had to reboot into a
>> live environment or anything since the RAID should have continued
>> working in degraded state up until the end of step (8). If you DID
>> happen to encounter a hardware fault during the swap though, you
>> could be in for a bad time, hence backups.
>>
>> If you somehow DO end up with a non-working system and have to boot
>> into a live / rescue environment, don't panic. Most of them have
>> full mdadm tools and kernel support so you should be able to fix
>> things from them. If you hit a snag, ask on the linux-raid mailing
>> list. Don't be tempted to experiment unless you know exactly what
>> effect the various commands will have.
>>
>> I haven't discussed the bootloader implications since you said these
>> are not your boot drives. The page does a reasonable job of that.
>>
>> > I have a feeling it's going to be more complicated than just
>> > replacing the drives one at a time with a rebuild (and then some magic
>> to
>> > grow things to the full 4TB).
>>
>> That's really all it is.
>>
>> Note that you'll need a GPT partition table (not MBR) in order to
>> have a partition of ~4TB. That will be fine.
>>
>> > Do I need to be overly concerned with /etc/fstab which seems to be
>> using a
>> > UUID:
>> > /dev/disk/by-id/dm-uuid-LVM-47rf... /var/lib ext4 defaults 0 0
>>
>> This is a UUID to a device-mapper (LVM) device. Nothing will change
>> as far as your LVM configuration is concerned so that UUID will
>> remain the same.
>>
>> If you have other fstab entries you are concerned about, let us
>> know.
>>
>> Cheers,
>> Andy
>>
>> --
>> https://bitfolk.com/ -- No-nonsense VPS hosting
>>
>> --
>> Nottingham mailing list
>> Nottingham at mailman.lug.org.uk
>> https://mailman.lug.org.uk/mailman/listinfo/nottingham
>>
>
> --
> Nottingham mailing list
> Nottingham at mailman.lug.org.uk
> https://mailman.lug.org.uk/mailman/listinfo/nottingham
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.lug.org.uk/pipermail/nottingham/attachments/20230126/26ebd2e9/attachment.htm>