Improved block-layer error handling

Ultimately, an fsync syscall returns whatever the filesystem's fsync operation returns, so if the filesystem wants to check for O_DIRECT and always return 0 without flushing, then it can do so today.

Now, that said...one wonders why an application would call fsync on an O_DIRECT fd?

If the software is some kind of cache, discarding the files that are least relevant is a proper course of action for both kinds of ENOSPC.
If the software is some kind of archival system, moving the oldest files to the next tier of storage will also help in both cases.

If the software can't freely discard or move data, all it can do, is scream for help, anyway.

Also, an ENOSPC due to lack of inodes will usually happen on open() while an ENOSPC due to lack of disk space will usually happen on write() or similar.
So applications could already translate this to proper error messages. It is common that the same error code has different meaning for different syscalls and developers should know this.

Of course, ideally filesystems would solve this problem completely. In fact, some do: btrfs has an upper limit of 2^64 inodes, as does XFS or ZFS (might be 2^48).
btrfs is fully dynamic, i.e. each btrfs, that is large enough to hold the inode information, can in fact contain 2^64 inodes. XFS is dynamic enough in practice (make sure to use "inode64", though. Otherwise inodes can only be stored in the lowest 1 TB, and that space can run out if also used for file data - been there, done that). Even NTFS allows 2^32 and is also fully dynamic

The ext-family is the big exception. Theoretically, the limit is also 2^32, but it cannot allocate space for inodes dynamically, and thus uses much lower limits by default. Otherwise, each inode would consume 256 bytes, even if unused.

Nor does it need one. POSIX should assume integrity by default unless applications say the opposite. One way applications can do that is by not checking any system call return values.

> POSIX also provides no way to say 'hey, fs, this file was written but failed integrity checks'

I don't think any changes to POSIX are required. We already have most of this in existing filesystems, just not in most existing filesystems.

In cases like compiles, where the writing application has completely disappeared before the block writes even start, there's no process to notify about the failure at the time the failure is detected. fsync() return behavior is irrelevant to this case--*every* system call, even _exit, returns before *any* error information is available. We want compiles to be fast, so we don't want to change this. A different solution is required. Note that reporting errors through fsync() is not wrong--it's just not applicable to this case.

For compiles we want to get the block-level error information passed from one producing process to another consuming process when the processes communicate through a filesystem. So let's do exactly that: If a block write fails, the filesystem should update its metadata to say "this data blocks were not written successfully and contain garbage now." Future reads of affected logical offsets of affected inodes should return EIO until the data is replaced by new (successful) writes, or the affected blocks are removed from the file by truncate, or the file is entirely deleted. If the filesystem metadata update fails too, move up the hierarchy (block > inode > subvol > planet > constellation > whatever) until the entire FS is set readonly and/or marked with errors for the next fsck to clean up by brute force.

Note that this scheme is different from block checksums. The behavior is similar, but block checksums are used to detect read errors (successful write followed by provably incorrect read), not write errors (where the write itself fails and the disk contents are unknown, possibly correct, possibly incorrect with hash collision). Checksums would not be an appropriate way to implement this. The existing prealloc mechanism in many filesystems could be extended to return EIO instead of zero data on reads. Prealloc already has most of the desired behavior wrt block allocation and overwrites.

> EIO is, ah, likely to be misinterpreted by essentially everything

I'm not sure how EIO could be misinterpreted in this context. The application is asking for data, and the filesystem is saying "you can't have that data because an IO-related failure occurred," so what part of EIO is misinterpreted exactly? What application (other than a general disk-health monitoring application, which could get detailed block error semantics through a dedicated kernel notification mechanism instead) would care about lower-level details, and which details would it use?

Also note EIO already happens in most filesystems, so we're not talking theoretically here. Most applications (even linkers), if they notice errors at all (**), notice EIO and do something sensible when they see it (*). This produces much, much more predictable results than just throwing random disk garbage into applications and betting they'll notice.

(*) interesting side note: linkers don't read all of the data in their input files, and will happily ignore EIO if it only occurs in the areas of files they don't read. Maybe not the best example case for a "data integrity" discussion. ;)

(**) for many years, GCC's as and ld didn't even notice ENOSPC, and would silently produce garbage binaries when the disk was full (maybe these would be detected by the linker later on...maybe not). Arguably we should also mark inodes with a persistent error bit if there is an ENOSPC while writing to them, but that *is* a major change which will surprise ordinary POSIX applications.

Why are nosql databases low value? Actually, nosql databases usually have a far higher signal/noise ratio - I converted a database from nosql to sql, and I think the size of the db DOUBLED.

Not only do nosql databases contain much more data per megabyte than relational, but they tend to be much faster - it's an old story but I remember stories about a company converting from UniVerse to (sn)Oracle, and it took SIX MONTHS for the consultants to get a Snoracle query (running on a twin Xeon) to outperform the old system running on a Pentium 90.

Or the "request for bids" put out by some University Astronomy department, that wanted a system to process 100K tpm. snOracle had to "cheat" to meet the target - delayed indexing, a couple of other tricks - while Cache had no trouble hitting 250K tpm.

RDBMSs are fundamentally inefficient, due to limitations in the relational model itself ... (just try and *store* a list in an rdbms).

Cheers,
Wol