Changing filesystem resize patterns

> That isn't at all what I was suggesting - sorry that I wasn't clear.
> I was suggesting something like:
> First Gigabyte is 10 100MB AGs
> Next 9 Gigabytes are 9 1GB AGs
> Next 90GB are 9 10GB AGs
> Next 900GB are 9 100GB AGs
> etc

Yes, that's kind of what I assumed you were suggesting. It solves some of the "find the AG" corner cases, but it introduces a whole new set of problems that we'd have to solve. Compared to dumping the filesystem on dm-thin and changing a couple of hundred lines of accounting code, it's a massive undertaking.

> The AGs used for allocating files to device locations do not necessarily have to match the block groups used for placing inodes and other metadata.

XFS doesn't have "block groups". It has allocation groups - they are the units that hold inodes, metadata and data.

By default, XFS has no limitation on the AGs which can hold inodes and metadata within the filesystem. But there are user selectable allocation policies which place arbitrary constraints on where inodes and metadata can be placed. That's why I mentioned inode32 - the implicaitons of full 1TB sized AGs and inode32 are that only AG 0 can hold inodes. Hence the layout you suggest above will only provide 100MB of physical inode space w/ inode32 which equates to a hard cap of roughly 100k inodes no matter how large the fs is grown to. That is obviously not enough inodes for even a root disk on a modern system. Hence for inode32, we can't consider using a physical AG 0 size of less than 10GB for a filesystem with variable sized AGs that could be grown into the TBs range.

It also makes no sense if the filesystem starts large (e.g. we're starting with a capacity of TBs), and so we now have two completely different filesystem layouts - one for tiny filesystems that can grow, and one for anything that starts at over 500GB. That's not really sustainable from a maintenance and longevity POV - we need to reduce complexity and the size of the configuration matrix, not make it worse.

> In my above example on a 1TB device, XFS could treat the first 28 on-disk AGs as a single 100GB group for allocation, then each of the remaining AGs as other 100GB allocation groups.

Unfortunately, we can't do that easily. Locking, transaction reservations, etc all mean we have to treat AGs as independent entities.

For allocation policy purposes, we can group sequential AGs together (I was looking at this as "concatenated groups" to treat multiple TB of contiguous disk space as a single free space entity back in ~2007), but we still have to treat them as individual AGs when it comes to addressing, allocation, reflink and rmap tracking, etc. Hence it doesn't really solve any of the problems with having lots of small AGs in a large filesystem - it just hides them under another layer of abstraction.

There also performance implications of lots of AGs on spinning disks. At 4 AGs (default), the cost of seeking between data in different AGs is just on the edge of the cliff. We lose about 5% sustained performance on spinning disks at 4AGs vs 1AG (which is why ext4 is always a touch faster on single spinning disks than XFS). By 10 AGs the performance is halved because of the amount of seek thrashing between AGs that occurs as the data sets are spread across the internal address space. IOWs, from a performance perspective on spinning disks, we really want to avoid a large number of small AGs in a small filesystem if at all possible. To fix this we basically have to completely redesign the allocator algorithms to no treat all AGs equally...

> But as you suggest, there is a big difference between such abstract musings and the hard practical realities of changing code in a production filesystem that many bytes of data are depending on...

I wish that more people understood this.

-Dave.