[Nottingham] I knew I'd forget something last night (filesystems and partitions)

[Martin]

Andy,

Thanks for good comment.

On 20/05/2020 15:41, Andy Smith via Nottingham wrote:
> Hello,
> 
> On Wed, May 20, 2020 at 01:56:47AM +0100, Martin via Nottingham wrote:
>> btrfs for everything else for both SSD and HDD;
> 
> In early 2014 I decided to evaluate btrfs by using it on my home
> file server.
> 
> On the whole it has been a mixed bag. The feature set is very nice.
> By far my most favourite thing is being able to repurpose mismatched
> types and capacities of storage from my dayjob into my home file
> server and trust that btrfs will mirror it across devices.
> 
> My main frustration with btrfs and the reason why I do not recommend
> its use and will not use it anywhere else — and at times regret
> using it in my home file server — is that its approach to
> availability absolutely sucks, and it's still got really bad bugs...


I started out on the following very adventurous stack:


ReiserFS 3.6
LVM
drbd
mdraid
(Spread across same sized HDDs but from x3 suppliers)


That steadily expanded from GBytes to finally 16TBytes of live data over
the years as HDDs and that particular client expanded.

Fortuitously, the technology available for the big HDDs expanded just
ahead of the data demands! One crunch came with another big blob of data
came /before/ the next planned shutdown and add-bigger-disks. The 'quick
fix' was to sacrifice the drbd raid across paired servers so as to
'instantly' double the available storage.

We usually had over a year between an actual disk failure despite
continuous 24/7 heavy usage. The LV/PV flexibility of LVM and in-situ
expandability of ReiserFS made near-live upgrades possible. Complicated
to manage but fantastic!

And it all proved remarkably robust. The entire stack was beautifully
flexible and bomb proof. We never lost any data.



> LVM on top of MD is rather unexciting but I will note that it's been
> years since I have seen any kind of software bug reported on
> linux-raid that actually caused loss of availability or data...

Agreed, and I've been in discussion with some of the developers years
ago. That software is very thoroughly done including for corner cases!

So much so:

I very strongly recommend using your own installed Linux mdraid that is
well proven, known, and robust, over any 'black box' hardware raid
solution where you are victim to the unknowns of rushed secret firmware
on proprietary secret formats tied to an expensive singular hardware
card that will fail on you...



Since the days of the "fs - LVM - mdraid" stack, there is now btrfs that
combines all those functions directly into the filesystem, plus lots more.

All dangerously bloated? Or beautifully flexible?

Comparing against mdraid, there is the one annoyance of btrfs using a
more cautious default in that any disk failure sends btrfs into
read-only mode. If you wish to maintain continuous running, then you can
use the mount option "degraded" to tell btrfs to keep running as far as
is possible.

The only other 'gotcha' I've hit has been using 'consumer' SATA HDDs
whereby you need to set their timeouts to be 7s:

	# Set timeout to 7.0s as is standard for raid arrays
	smartctl -q errorsonly -l scterc,70,70 "$dd"


Never had any problems for HDDs running on SAS links.



> For example, I did not know that apparently ASRock motherboards
> since at least 2015 like to REWRITE YOUR PARTITION TABLE AT EVERY
> BOOT if they don't see a valid one. Since you don't need a partition
> table to run RAID, that bites ASRock owners from time to time.
> 
>     https://marc.info/?l=linux-raid&m=158910882519940&w=2
> 
> With some help they got it back.

That's an Ouch!

As it happens, after some 'discussions' back in the early days, btrfs
keeps the start if its partition/device area unused just in case that
area is blindly overwritten with a partition table or whatever else...


Using btrfs, I tend not to bother with partition tables at all.

If you're on anything more recent than a 5.0 kernel, then I'd recommend
btrfs unless you have something strangely pathological!



Obviously, I've enjoyed some simply good very good results with btrfs
over recent years.

Hope there's some interesting comment there.

Good luck!
Martin