Filesystems, testing, and stable trees

This sounds like a problem that hit raid ... a problem sneaked in, the layout of an asymetric raid-10 was accidentally changed. (Asymetric as in different sized disks.)

It was really nasty in that you had to run a kernel the same side of the change as the one the raid was created by - both new kernel on old raid and old kernel on new raid would lead to data corruption. What do you do? You can't revert the change because any arrays created with a modified kernel would be inaccessible.

They ended up by adding a flag to the raid to identify old or new, and all new kernels now refuse to start an array if that flag is missing. It's easy to add, so new kernels are fine, people are unlikely to revert to old kernels, and asymetric raids aren't that common ...

Yes, upgrading does show up a completely different class of bug ... :-)

Cheers,
Wol