An update on bcachefs

One confusing thing about RAID1/10 in btrfs is that it's not really what you think it is.

If you do RAID 1 on more than 2 devices (say n devices), btrfs only guarantees you have 2 copies of your data, not n.
https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_...

More over, you only read from one disk (IIRC I think they do load balancing with the PID or something like that so you really use RAID 1 / 10 for redundancy more than performance).

When you loose a device in RAID1, you can mount the FS but only once... https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_vol...

As for RAID10, it inherits the design features of RAID1. From the doc, it says "RAID-10 is built on top of these definitions. Every stripe is split across to exactly 2 RAID-1 sets and those RAID-1 sets are written to exactly 2 devices". The issue is you do not control on which devices your stripe is mirrored.

So a double disk failure on a btrfs RAID10 is more likely to end up with data loss data than on a traditional RAID-10.

There was a long discussion about that years ago: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/...

>The example below is probably a pathological case - but here goes. Let's say in this 4-disk example that chunks are >striped as d1,d2,d1,d2 where d1 is the first bit of data and d2 is the second:
>Chunk 1 might be striped across disks A,B,C,D d1,d2,d1,d2
>Chunk 2 might be striped across disks B,C,A,D d3,d4,d3,d4
>Chunk 3 might be striped across disks D,A,C,B d5,d6,d5,d6
>Chunk 4 might be striped across disks A,C,B,D d7,d8,d7,d8
>Chunk 5 might be striped across disks A,C,D,B d9,d10,d9,d10
>
>Lose any two disks and you have a 50% chance on *each* chunk to have lost that chunk. With traditional RAID10 you >have a 50% chance of losing the array entirely. With btrfs, the more data you have stored, the chances get closer to 100% >of losing *some* data in a 2-disk failure.
>
>In the above example, losing A and B means you lose d3, d6, and d7 (which ends up being 60% of all chunks).
>Losing A and C means you lose d1 (20% of all chunks).
>Losing A and D means you lose d9 (20% of all chunks).
>Losing B and C means you lose d10 (20% of all chunks).
>Losing B and D means you lose d2 (20% of all chunks).
>Losing C and D means you lose d4,d5, AND d8 (60% of all chunks)

>The above skewed example has an average of 40% of all chunks failed. As you add more data and randomise the >allocation, this will approach 50% - BUT, the chances of losing *some* data is already clearly shown to be very close to >100%.

The bottom line is RAID1 or 10 as defined by btrfs is not what you think they are if you haven't read the fine prints...