LWN: Comments on "A mapping layer for filesystems"

A mapping layer for filesystems

zdzichu — Tue, 29 May 2018 16:12:55 +0000

That's why first thing you should do after getting such fake space is dd if=/dev/urandom of=/file. This would give you some guarantees that your space you need is there. And should avoid unpleasant surprises in the future.

A mapping layer for filesystems

fyrchik — Sat, 12 May 2018 17:41:17 +0000

Thin provisioning is useful in combo with deduplication: you can have 10TB of available space backed by only 5TB of storage (which looks reasonable for hundreds of similar VMs).

A mapping layer for filesystems

dgc — Fri, 11 May 2018 00:14:06 +0000

> I still think it would make much more sense to add a "Hey, can you increase
> your size?" request to the iSCSI/whatever block devices

Read the article again - that's essentially what the infrastructure I was talking about provides us with.

i.e. it provides us with the ability for the filesystem to request the amount of space it needs from the underlying storage before it starts doing operations that depend on that space being available. Hence if the device does not have space available, then the filesystem can back out and report ENOSPC to the application before getting into a state it can't sanely recover from if it receives device-based ENOSPC errors during IO....

-Dave.

A mapping layer for filesystems

Cyberax — Thu, 10 May 2018 18:20:23 +0000

Yes, I understand that. I still think it would make much more sense to add a "Hey, can you increase your size?" request to the iSCSI/whatever block devices rather than: "I pinky swear that I'm REALLY using this space".

This way you don't need to have untested failure paths that can happen at any moment.

A mapping layer for filesystems

farnz — Thu, 10 May 2018 10:38:06 +0000

Thin provisioning moves the management overhead from the thousands or tens of thousands of VMs using the storage system, to the single storage system.

Instead of having to ask for space in small chunks (say 10 GiB at most), and having to expand regularly to cope on each of the VMs, you can give the VMs virtual disks that appear big enough for (say) 5 years worth of predicted use. The server admin only has to check in and ask for more space when they have new projections showing that they need much more space than they have has assigned.

In the meantime, though, you now have tens of thousands of machines that have enough space for the worst case 5 year projection; they're not going to need that up front, so you really don't want to buy that much disk space today, only to leave hundreds of disks empty. Instead, you thin-provision; each virtual disk is only allocated as much space on the real drives as it really needs today, and you set up monitoring so that you know when you're going to need more real disks and can order them in before you need them.

This means that instead of tens of thousands of server admins allocating 10 GiB chunks every couple of days, you have a small number of storage admins buying and bringing online disks every few weeks or months as needed for your growth. Your servers think they have 500 PiB of space between them, but you've only bought 200 TiB so far, and bring another 10 TiB online whenever you run low on used disk.

If it all works as designed, thin provisioning is completely transparent to the clients - you always have more real disk than you're using, so the impact of thin provisioning is that you neither buy a huge amount of otherwise idle disk nor have to keep allocating space to the servers that need it. You just wait for the storage array to run low, then add another chunk.

A mapping layer for filesystems

dgm — Thu, 10 May 2018 09:31:57 +0000

First let me tell you that I don't have experience with large server pools.

That said, I fail to see how thin provisioning is better than allowing users to freely take more space as they need it.

By your explanation, it seems that it is used like some kind of pre-granted quota that you can use without explicit permission, but with the undesirable side effect of unexpected errors when resources that are supposedly there fail to materialize. That in turn forces complexity on the applications, because you cannot make assumptions.

If you want to have quotas, why don't you use quotas? Or is there some other reason?

A mapping layer for filesystems

ringerc — Thu, 10 May 2018 00:56:27 +0000

Thin provisioning is really useful in larger infrastructures and teams.

"Just add more storage when you need it" is easy when you're managing individual servers and you're responsible for the storage and the server.

It quickly becomes unfeasible when you're running an infrastructure where people can request (often virtual) servers and run them for $WHATEVER. The server is their problem. They may have one, they may be part of a team running hundreds or thousands of them. Numerous small sysadmin teams doesn't work well and leads to low server utilisation, excessive divergence in tooling and SOE between teams, overlapping software licensing spends, and more. So your organisation has moved toward centralising the underlying server infrastructure with virtualisation and SAN storage to improve utilisation, reduce hardware and power costs, and make staffing more efficient.

As a result you have this big SAN with, say, 50TB of storage. You don't want to upgrade to 200TB and allocate 4x or more the needed storage for all your servers just for slush free space, in case they might need it. Most of them won't. But you don't want server admins coming to you many times a day asking for another small space increase and doing constant maintenance work to resize file systems just-in-time either.

So you set the SAN up to lie about the real size of the LUNs it hands out. Sysadmin asks for 1TB for each of 10 servers? Sure, you can have 10TB, but you need to give me an estimate of likely actual growth for my capacity planning, because most of that storage won't exist when I give it to you. I'll add it progressively as it's actually needed. You don't have to resize your file systems all the time, I don't have to spend tons of power and money on empty disks. Their servers just have a FC HBA, 10G ethernet iSCSI or whatever to talk to the SAN.

Over larger farms of servers growth tends to even out and you get fairly predictable, plannable capacity needs and plannable costs. Things managers just love. SAN vendors love it too, because they can ship you powered down racks of extra disks and charge you to turn them on - no need for a site visit.

The support for thin provisioning on Linux can be used to implement similar schemes for things like a vm host that carries thousands or tens of thousands of VMs, most of which don't need much space. But where you can't easily predict which ones will, and when.

But everything's an estimate. If you fail to capacity plan properly or monitor properly you can run out of thin provisioned space entirely. That's quite bad. Most systems offer safeguards like reserved space pools for subsets of critical servers, so it won't bring down everything. But centralising storage does centralise failure.

That failure should still be limited to "oops, we can't write data to this volume" though. The problem is that right now, it isn't. Due to the fsync issues discussed on lwn recently and due to FS layer limitations, thin provisioning failures are not handled very gracefully, and write failures may turn into data loss.

A mapping layer for filesystems

Paf — Thu, 10 May 2018 00:31:21 +0000

Err, really? Thin provisioning of storage for VMs is incredibly powerful and flexible and not a kludge. Not everything is ext4 or XFS, and thin provisioning allows VM admins to not interact with the VMs themselves (for example, the VMs can stay up!). Also, I don’t necessarily have root on those VMs, and I sure as hell don’t want to have it.

Are you talking about some more specific instance of thin provisioning?

A mapping layer for filesystems

Cyberax — Wed, 09 May 2018 17:06:17 +0000

I never understood the need for thin provisioning. It's a freaking kludge with huge infrastructure costs across multiple layers.

We already have LVM. So just use it as a substrate for a filesystem, monitor its free space and extend the filesystem as needed. ext4 and xfs both support online expansion.

If free space/volumes in the LVM array are running low, simply add more. No need for any kernel-level features and can be implemented entirely in userspace.