Flags as a system call API design pattern

The problem with open() and O_TMPFILE was that the kernel treated unknown flags as optional, but an application using O_TMPFILE would want it as *mandatory*. Such an application would have no way to tell whether O_TMPFILE actually was honoured, because the open() would succeed, regardless of whether kernel recognised that flag (e.g. if you run the application on an older kernel). You can't test and retry, because open generally didn't treat unknown flags as a failure.

Generally, to be able to introduce new mandatory flags, while allowing optional flags, you need either to distinguish between mandatory flags and optional in the API in some way, or you need some other way to allow the application to feature-test at runtime (but what if it forgets to do this, and then gets run on an old kernel?).

Otherwise, you need to rely on being able to find an API useage-specific hack that happens to work, as was done for open/O_TMPFILE, by also setting some other unrelated flags that *would* together cause an error on older kernels. ;) These kind of API-specific hacks might not always be available.