[Wolves] Storage advice.

[James Dutton]

On Mon, 15 Jan 2024 at 21:31, Simon Burke via Wolves
<wolves at mailman.lug.org.uk> wrote:
> Hi,
> Odd position I'm in at day job.
> I have a VM (VMWare) running Oracle Linux 9, with two 50tb VMDKs attached.
> The issue is that I need to span them, and it'll store a lot of tiny files (circa 80Tb in 2-3mb files), with a high daily rate of change.
> Choosing the right method to achieve this is proving a challenge, when I've got to get this going this week. So testing is limited.
> I can't have a single large VMDK due to a 62Tb limit in VMWare.
> I can just use ext4 and LVM, or Oracle Linux likes Btrfs. ZFS is provisionally out unless I get a third 50Tb disk, for ZRAID+1.
> Opinions?

Hi,

Have you heard of something called sharding?
This is where you get the application itself to distribute the data
across many disks and also many servers.
It ensures the system scales well horizontally.
An example of such an application is Elastic Search or an Object Store.
It allows you to have the application do the data distribution and
replicas as needed, and is generally more flexible than using many
disks joined together into single large virtual disks.
Does your application support sharding? If not, it might be worth
considering implementing it, as 80Tb is quite large and you seem to be
seeing some of the scaling challenges.

Kind Regards

James