perhaps running out of inodes could be taken "more seriously"?

Posted Jun 3, 2017 7:44 UTC (Sat) by MarcB (subscriber, #101804)
In reply to: perhaps running out of inodes could be taken "more seriously"? by Richard_J_Neill
Parent article: Improved block-layer error handling

I don't think running out of free inodes is conceptually different from running out of free space. Also, it is not a problem per se from the kernel's PoV: It cannot be prevented, it does not happen at random, it is not something exceptional at all.
It is just another resource exhaustion that user space has to deal with - and perhaps even is dealing with, so nothing is actually wrong.

Also, this used to be much more common in the past, when many filesystems allowed much fewer inodes by default. So, perhaps some administrators simply have forgotten (or never learned) that inode exhaustion is a real thing.

And diagnosing this - once you are aware that it can happen - is not harder than diagnosing "out of space" (in practice: even easier, as is is unlikely that large numbers of inodes are held by deleted yet opened files).
It can, and should, also be monitored just like free disk space.

perhaps running out of inodes could be taken "more seriously"?

Posted Jun 3, 2017 23:16 UTC (Sat) by Richard_J_Neill (subscriber, #23093) [Link] (5 responses)

Yes, you're right, excepting that the common "no space left on device" message is actually very misleading when there is in fact plenty of space.

Also, while the sysadmin can add extra monitoring and debugging, surely the point of a reliable system is to minimise the chance of human error.
We are used to the abstraction of a storage being "somewhere you can fill up with data"; the very existence of inodes should be no more the concern of the average programmer/sysadmin than the specifics of which pointer has which address... it should be "the computer's" problem, not "the operator's problem". If the computer is going to break that rule, and do so rarely, but catastrophically, the least it can do is to fail "noisily".

Anyway... in these days of LVM and resizeable volumes, why shouldn't the filesystem be able to automatically notice that it has lots of space but too few inodes, and automatically create more inodes as needed?

perhaps running out of inodes could be taken "more seriously"?

Posted Jun 4, 2017 1:39 UTC (Sun) by rossmohax (guest, #71829) [Link] (1 responses)

that is exactly what XFS is doing, inodes are allocated dynamically and you can never run out of them as long as you have free space. Try using XFS instead of ext4, it is awesome

perhaps running out of inodes could be taken "more seriously"?

Posted Jun 4, 2017 5:09 UTC (Sun) by matthias (subscriber, #94967) [Link]

Even XFS can run out of inodes. Inode numbers are mapped to blocks by a very simple mapping. It is roughly like every i-th block can have inodes. The possible inodes are just numbered starting by one. Once these blocks are filled (with inodes or data), XFS cannot create new inodes. Changing the mapping would change every single inode number.

We had once the following problem after growing a filesystem. Standard was at that time to only use 32-bit inode numbers. After growing the filesystem the 32-bit inode numbers where all in the already filled lower part of the filesystem.(*) Thus no new inodes could be created. Took a while to find that one only having the meaningful message "No space left on device.". Luckily it was a 64-bit system. Thus, we could just switch to 64-bit inode numbers. The other solution would have been to recreate the filesystem, not the quickest solution with a 56 TB filesystem.

That said the circumstances under which XFS runs out of inodes are very rare. So it would be very important to have meaningful error messages, to notice that one of these rare circumstances just happened.

(*) On fs creation XFS usually chooses the number i to be such that all possible inodes have 32-bit numbers. After growing this condition was not satisfied any more, as this number cannot be changed. On 32-bit systems, one would need to set this number i manually at fs creation time, if one wants to have the possibility to grow the filesystem.

perhaps running out of inodes could be taken "more seriously"?

Posted Jun 4, 2017 14:15 UTC (Sun) by MarcB (subscriber, #101804) [Link] (2 responses)

Remember that the possible error codes for syscalls were defined by POSIX, so simply adding an EOUTOFINODES would be non-compliant and could easily do more harm then good, because in practice, ENOSPC is a good fit for "out of inodes" and software might actually expect it to cover both cases:

If the software is some kind of cache, discarding the files that are least relevant is a proper course of action for both kinds of ENOSPC.
If the software is some kind of archival system, moving the oldest files to the next tier of storage will also help in both cases.

If the software can't freely discard or move data, all it can do, is scream for help, anyway.

Also, an ENOSPC due to lack of inodes will usually happen on open() while an ENOSPC due to lack of disk space will usually happen on write() or similar.
So applications could already translate this to proper error messages. It is common that the same error code has different meaning for different syscalls and developers should know this.

Of course, ideally filesystems would solve this problem completely. In fact, some do: btrfs has an upper limit of 2^64 inodes, as does XFS or ZFS (might be 2^48).
btrfs is fully dynamic, i.e. each btrfs, that is large enough to hold the inode information, can in fact contain 2^64 inodes. XFS is dynamic enough in practice (make sure to use "inode64", though. Otherwise inodes can only be stored in the lowest 1 TB, and that space can run out if also used for file data - been there, done that). Even NTFS allows 2^32 and is also fully dynamic

The ext-family is the big exception. Theoretically, the limit is also 2^32, but it cannot allocate space for inodes dynamically, and thus uses much lower limits by default. Otherwise, each inode would consume 256 bytes, even if unused.

perhaps running out of inodes could be taken "more seriously"?

Posted Jun 5, 2017 11:55 UTC (Mon) by nix (subscriber, #2304) [Link] (1 responses)

Remember that the possible error codes for syscalls were defined by POSIX, so simply adding an EOUTOFINODES would be non-compliant and could easily do more harm then good, because in practice, ENOSPC is a good fit for "out of inodes" and software might actually expect it to cover both cases

It might well do more harm than good, but the first part of your statement is just wrong. POSIX.1 2008 states (and all previous versions have similar wording):

Implementations may support additional errors not included in this list, may generate errors included in this list under circumstances other than those described here, or may contain extensions or limitations that prevent some errors from occurring.
The ERRORS section on each reference page specifies which error conditions shall be detected by all implementations (``shall fail") and which may be optionally detected by an implementation (``may fail"). If no error condition is detected, the action requested shall be successful. If an error condition is detected, the action requested may have been partially performed, unless otherwise stated.
Implementations may generate error numbers listed here under circumstances other than those described, if and only if all those error conditions can always be treated identically to the error conditions as described in this volume of POSIX.1-2008. Implementations shall not generate a different error number from one required by this volume of POSIX.1-2008 for an error condition described in this volume of POSIX.1-2008, but may generate additional errors unless explicitly disallowed for a particular function.

So adding more errors is not only not noncompliant, it is both explicitly permitted and very common.

perhaps running out of inodes could be taken "more seriously"?

Posted Jun 5, 2017 16:15 UTC (Mon) by nybble41 (subscriber, #55106) [Link]

> So adding more errors is not only not noncompliant, it is both explicitly permitted and very common.

Yes, for *new* error conditions not specified by POSIX. However:

> Implementations shall not generate a different error number from one required by this volume of POSIX.1-2008 for an error condition described in this volume of POSIX.1-2008, ...

The error list for the open() and openat() system calls specifies ENOSPC as follows:

> [ENOSPC]
> The directory or file system that would contain the new file cannot be expanded, the file does not exist, and O_CREAT is specified.

So if "the filesystem ... cannot be expanded" is read to include the "out of inodes" condition (a reasonable interpretation IMHO) then POSIX requires open() to return ENOSPC for this condition, and not some other error code.