LWN: Comments on "The shrinking role of ETXTBSY"

File version numbers à la OpenVMS?

Wol — Mon, 16 Sep 2024 19:12:41 +0000

Reading that again, I'm not sure if I made myself clear, but these were file system attributes. So you had "NR-1W" (either readers OR writer), "NR&1W" (as many readers as you liked, only one writer), and "NR&NW" (as many readers and writers as you liked).

So my accounts system had "NR&1W" set on all files, and only ever opened a file to write when it was doing a commit. There was also always an explicit open hierarchy (as in I only ever opened individual clients after opening the client summary, so I couldn't get a deadlock, same for other ledgers).

So it relied on programming discipline, but could be proven to work if the rules were followed.

Cheers,
Wol

File version numbers à la OpenVMS?

jch — Mon, 16 Sep 2024 17:38:11 +0000

> Again harping on Pr1mos, and I don't know how easy it would be to retrofit to Linux, but files had reader/writer access controls. I don't know whether it was set by the first application to open it, or more likely set in the file system, but you had a choice of "multiple readers (or one writer)", "multiple readers and one writer", and "multiple readers and multiple writers".

Aren't those just mandatory file locks taken at open time?

The shrinking role of ETXTBSY

MarcB — Mon, 16 Sep 2024 09:15:40 +0000

Reboot should be fine, the kernel will flush all buffers before unmount. It is only crashes or power losses that would be a problem.

Unix atomic actions, or how to replace an executable or library

nix — Sun, 12 Sep 2021 18:51:49 +0000

> Just out of curiosity, does the Linux run-time linker decrement the i_writecount in the library inode? Solaris did...

No. I'm fairly sure there is no way to do so: there was, but that was MAP_DENYWRITE, which, well, see this article... glibc is still careful to mmap its libraries with MAP_DENYWRITE, which the kernel then ignores :(

Interesting discussion on ETXTBSY

nix — Sun, 12 Sep 2021 18:45:54 +0000

> Currently only executables benefit from ETXTBSY. What if we added an O_DENYWRITE open flag.

What if we arranged for ETXTBSY to be emitted by default for files which have pages mapped executable in some process's address space? (The flag would obviously be flipped by the first mmap(MAP_EXEC) of that file, and flipped off by munmap). This is, obviously, a change in semantics, but surely not even jitters rely on writing to executable file-backed pages (differing cache coherency semantics make this operation intrinsically nonportable in any case).

This prevents the "easy DoS" of open-with-a-flag-to-deny-overwriting -- the only DoS vector now available is the one we already have, "execute something and nobody else can overwrite it while you're executing from it", which seems entirely, y'know, desirable.

File version numbers à la OpenVMS?

Wol — Tue, 31 Aug 2021 13:20:52 +0000

> Many apps would really benefit from these kind of transactional semantics, right now they have to use something like SQLite to get them, but with unit files they could get those semantics directly from the OS.

Again harping on Pr1mos, and I don't know how easy it would be to retrofit to Linux, but files had reader/writer access controls. I don't know whether it was set by the first application to open it, or more likely set in the file system, but you had a choice of "multiple readers (or one writer)", "multiple readers and one writer", and "multiple readers and multiple writers".

So, as it implies, the first person to open the file got it, any other attempts were checked against this lock and the open succeeded or failed depending. When I wrote an accounts package, everything was configured NR&1W, all files were opened read-only unless actually updating the data, and all files were opened in a defined order to prevent deadlocks. (Still didn't prevent one user changing the data underneath another, but the program made sure this didn't matter...)

Cheers,
Wol

The shrinking role of ETXTBSY

Wol — Tue, 31 Aug 2021 12:59:27 +0000

Pr1mos added the ability to rename an open file.

Dunno exactly how it worked, but it was something like the - if the linker hit an open file - it knew the magic sauce to rename it to move it out the way, then write a new one.

Same idea as *nix's ability to delete an open file then put a new one in its place - anything which has the old file open will continue to use that until it closes it.

Cheers,
Wol

The shrinking role of ETXTBSY

Cyberax — Tue, 31 Aug 2021 04:33:44 +0000

You can rewrite the running file on Windows. You can't truncate or delete it, though.

This has more to do with the way the memory mapping is managed on Windows than with anything else.

The shrinking role of ETXTBSY

dancol — Mon, 30 Aug 2021 21:20:59 +0000

No, the Windows way to do things is the right one. You shouldn't be able to mess up a running program by mucking with its binary from underneath. On Windows, you can *move* the old file out of the way and then create a new file, which is what you should be doing on a Unix system too. Windows just enforces good hygiene.

The shrinking role of ETXTBSY

nivedita76 — Mon, 30 Aug 2021 16:03:31 +0000

Does any installer use cp, though? They should already be doing the right thing, no?

What’s the use case for doing cp overwriting an existing executable or shared library?

File version numbers à la OpenVMS?

skissane — Sun, 29 Aug 2021 23:55:10 +0000

File version numbers, as found in OpenVMS (and a bunch of other DEC family operating systems before it) would be a cool solution to this. If an inode can contain multiple data versions, overwriting an in-use executable could create a new data version for the inode. Existing file descriptors would point to the old data version but newly opened file descriptors would point to the new one. The old data version could be automatically deleted when its last open file descriptor closed.

More generally, one could use this to enable what I've heard some people call "unit files". (I don't know if that is standard terminology at all, I heard it from some commenter on HN.) Basically, a unit file acts atomically – when you open it, you get a read snapshot of the file at the time you opened it. When you write to it, your changes aren't visible to other readers until you close the file (or call some kind of "commit" system call). (This would likely imply only one process can have the file open for writing at a time). Many apps would really benefit from these kind of transactional semantics, right now they have to use something like SQLite to get them, but with unit files they could get those semantics directly from the OS.

Of course, I doubt Linux is going to any such features. It is complicated, probably wouldn't be used that much, doesn't fit well with POSIX, could not be supported by existing filesystems without substantial changes (although a few do have some existing COW support, and they might be able to leverage that to implement this feature with only modest effort). But one can daydream about a parallel universe in which UNIX never really took off, and we are all using some open-source clone of VMS. In some ways no doubt an uglier universe, but in other ways a prettier one.

The shrinking role of ETXTBSY

gnu_lorien — Sat, 28 Aug 2021 18:38:11 +0000

"I feel like living in a parallel universe..."

I had the same feeling when reading this article.

I had an experience in a university class around 2006 where the professor claimed that you couldn't overwrite in-use files. I used Gentoo at the time and had never needed to shut down anything to recompile and update my system. The professor insisted this could happen but I couldn't find any scenario from the userspace tools I had available where I couldn't update in-use files.

I've also worked at companies that do cross-platform work for at least 15 years. It's a regular occurrence that things break for the Windows toolchain that don't break for GNU/Linux toolchain because Linux toolchains will happily update in-use binaries. It's one of the things I consider a huge misfeature of Windows that you have to go hunting down whoever is locking a file in order to finish a build. From a user perspective I would be shocked and annoyed if I started seeing ETXTBSY show up. I say good riddance.

Unix atomic actions, or how to replace an executable or library

davecb — Thu, 26 Aug 2021 16:05:03 +0000

Just out of curiosity, does the Linux run-time linker decrement the i_writecount in the library inode? Solaris did...

If so, that suggests that Linux could have the same "you're about to mess up one of your processes" warning if someone inadvertently attempted to use cp or rsync to update an in-use library, just like an in-use program.

Unix atomic actions, or how to replace an executable or library

davecb — Thu, 26 Aug 2021 16:00:14 +0000

Unix was designed with a minimum number of ways to do anything, and a pair of 'atomics'. From as far back as v6,
* open had the O_CREAT flag, which would atomically create a file, or fail if it existed
* mv atomically replaced the inode under a filename, allowing new openers of the file to get the new version, and everyone with an open copy to continue using the old one.

If you think the latter sounds like the read-copy-update Linux uses inside the kernel, you'd be right: it comes from the same seminal papers.

The latter is the canonical way to update a file or library: I used it heavily on Unix and Solaris (I was part of the shared library team), update apparently uses it and I considered it one of the really impressive things about Unix-derived systems. And it fits in nicely as part of the "everything is a file" metaphor.

--dave

The shrinking role of ETXTBSY

flussence — Tue, 24 Aug 2021 19:21:26 +0000

I'm imagining a hypothetical O_REPLACE flag here, which gives the opening process a filehandle that shadows the existing file, and its effects won't become visible to the outside world until the opener explicitly invokes fsync() or close(). Similar atomicity semantics to the tempfile-write-sync-rename pattern, but without the tempfile juggling and the extra problems that causes.

Admittedly it doesn't sound all that easy to implement something like this, even to a kernel layperson.

Interesting discussion on ETXTBSY

immibis — Tue, 24 Aug 2021 16:23:33 +0000

Then the kernel should do it instead. Mapping an executable or library file should be a copy-on-write operation. When the underlying file is replaced the kernel should use the old pages for the existing mappings and the new pages for new mappings.

Of course, that's probably impossible in current Linux.

The shrinking role of ETXTBSY

anton — Tue, 24 Aug 2021 15:48:29 +0000

So someone who uses such a binary, runs it in a 2GB VM, and tries to overwrite this binary gets an out-of-memory condition in a VM rather than ETXTBSY or a crashing binary. So even this unlikely scenario is not really worse off, and in more usual scenarios things just work as intended.

Interesting discussion on ETXTBSY

neilbrown — Mon, 23 Aug 2021 22:19:06 +0000

> I find it baffling that you seem to tacitly assume the existing design is perfect.

I did not mean to imply that, and am sorry that it came across that way.
What I was trying to highlight was that your stated desire about behaviour of tools was already thwarted by reality. The desire came across as a bit naive.

I certainly see the attraction of having simple tools with simple semantics - "Do one thing and do it well". However I know from experience that such tools are usually *too* simple. Different people have different opinions about what the "one thing" should be, and about what it means to do it "well".

There is, and should be, a tension between adding functionality and maintaining simplicity. Both have value and there is no perfect balance. Rather we iteratively find a dynamic balance through robust "conversation" (which involves sharing both opinions and code).

I didn't imagine my proposal about how 'cp' could handle locking would be the final word on the subject, just another contribution to an ongoing conversation. Thanks for adding to the conversation!

The shrinking role of ETXTBSY

anton — Mon, 23 Aug 2021 21:28:09 +0000

What would be the benefit of doing that with mmap(MAP_PRIVATE|MAP_POPULATE) vs. just reading the entire file into the process's anonymous private memory?

Zero-copy unless someone tries to write to the file. The way I imagine it, the write would block until the copying is completed, so this race condition would not exist.

The approach to write new file and rename over the old one would a good one, but despite the inconvenience of ETXTBSY linkers don't use this approach, so maybe the problem with the unwritable directories is more relevant than we think.

The shrinking role of ETXTBSY

nybble41 — Mon, 23 Aug 2021 20:18:20 +0000

What would be the benefit of doing that with mmap(MAP_PRIVATE|MAP_POPULATE) vs. just reading the entire file into the process's anonymous private memory? Of course even if you do that the file could change while you're reading it, so it would reduce the window where the data could be corrupted but not eliminate it altogether.

For that matter, any process that could rewrite or truncate a file while it's in use could also corrupt the data beforehand. ETXTBUSY only protects against *accidentally* corrupting a file by updating it while it's in use, by forcing the update to fail. However, since we don't want the update to fail anyway, the solution which doesn't risk data corruption *or* an ETXTBUSY error is to write the new data to a temporary file and rename it over the original. This does require write access to the parent directory, but that doesn't seem unreasonable to me since logically you are modifying the directory to point to a new file. Any attempt to atomically update the content without replacing the file will run into the issue that mapping follow the file, not the content.

Interesting discussion on ETXTBSY

NYKevin — Mon, 23 Aug 2021 15:40:53 +0000

> And yet .... they already do.

I find it baffling that you seem to tacitly assume the existing design is perfect. It's obviously not. The -T flag, for example, only needs to exist because cp and mv "helpfully" default to interpreting the last argument as -t if it is possible to do so. If you had to explicitly pass -t, then -T would not need to exist, and a whole class of bugs would have been eliminated.

The shrinking role of ETXTBSY

skitt — Mon, 23 Aug 2021 14:11:32 +0000

This isn’t specific to GNU Make, it’s been a feature of Make ever since Stuart Feldman’s first version (see http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.3...) and is part of POSIX (http://pubs.opengroup.org/onlinepubs/9699919799/utilities...).

The shrinking role of ETXTBSY

anselm — Mon, 23 Aug 2021 12:17:48 +0000

You can't kill -9 1; the init process is protected from signals for which it doesn't have an explicitly installed signal handler.

The shrinking role of ETXTBSY

excors — Mon, 23 Aug 2021 11:06:13 +0000

> On my Debian system, all binaries in /usr/bin combined have a total text size of 285MB (as reported by size -t) and all libraries in /usr/lib combined have a text size of 295MB, so keeping the whole text of a few binaries or libraries is unlikely to lead to problems that the system would not soon have otherwise.

Sometimes people run code that isn't shipped with Debian. Back in 2012, Facebook's page requests were handled by a single 1.5GB executable, generated from PHP code transpiled to C++. (https://arstechnica.com/information-technology/2012/04/ex...). I think they switched to a PHP JIT shortly after that, but I imagine other people had (or still have) even larger executables for similar reasons.

The shrinking role of ETXTBSY

anton — Mon, 23 Aug 2021 09:06:38 +0000

Even if the application has used MAP_PRIVATE, that covers how to handle a store from the mmapper, not a write() from somebody else.

That's disturbing. Still, if a MAP_PRIVATE page is written to (e.g., with its original content) in one place, it is copied on that write, and later changes to the original don't affect it.

So that is what could be done on writing to an executed text file: in every affected process, make private copies of the pages of the whole original text (as if on copy-on-write) and populate the mapping with them. There is an opportunity for sharing between several processes that run the same changed binary, but I guess that the benefit is too small and too rare to make that effort. OTOH, the benefit of not having ETXTBSY and not having processes crash when their binary changes is more substantial IMO.

Actually, I would like that also for interpreters (a have had a number of shell scripts crash when I edited them while they were running), maybe by making MAP_PRIVATE|MAP_POPULATE behave that way, or with an additional flag to mmap().

The shrinking role of ETXTBSY

anton — Mon, 23 Aug 2021 08:47:17 +0000

On my Debian system, all binaries in /usr/bin combined have a total text size of 285MB (as reported by size -t) and all libraries in /usr/lib combined have a text size of 295MB, so keeping the whole text of a few binaries or libraries is unlikely to lead to problems that the system would not soon have otherwise. Assuming that you have the ETXTBSY problem on such default VMs at all (what are you using these VMs for where you get ETXTBSY?), I still think that most users prefer the remote change to run out of memory to a certain ETXTBSY or, as seems to be in the works, a crash from having the binary changed during execution.

Interesting discussion on ETXTBSY

mchapman — Mon, 23 Aug 2021 05:01:53 +0000

In other words, if you are copying directories in a script, you need a whole bunch of wrapper code if you want the results to be reproducible.

You can use cp -R source/. dest/, which will do the right thing whether or not dest is an existing directory.

With GNU cp, you can also do cp -R --no-target-directory source/ dest/.

Interesting discussion on ETXTBSY

Wol — Mon, 23 Aug 2021 01:26:37 +0000

And so, very annoyingly, does cp.

If the target is a pre-existing directory, it copies into it. If the target is a pre-existing file it overwrites it. And if the target doesn't exist, it makes a copy called the target.

In other words, if you are copying directories in a script, you need a whole bunch of wrapper code if you want the results to be reproducible.

Cheers,
Wol

Interesting discussion on ETXTBSY

neilbrown — Mon, 23 Aug 2021 01:03:32 +0000

> I do not want my tools doing two completely different things depending on some entirely unrelated state.

And yet .... they already do.

"mv" will prefer the rename() systemcall, but if that returns EXDEV, it will fall back to copy-and-remove-original.

But maybe EXDEV not as "unrelated" as a flock?

The shrinking role of ETXTBSY

NYKevin — Mon, 23 Aug 2021 01:02:21 +0000

OK, I suppose I was a bit vague, so let me be more precise:

Make doesn't have any code which directly calls rename(2), unlink(2), etc. for the purposes of executing a recipe.* Recipes are (effectively) very small shell scripts, and it is rm(1) or cp(1) (or in the case of open(2) for a redirection, the shell) which actually makes those syscalls on make's behalf. Make doesn't "know" anything about how to copy a file, it just "knows" to run cp(1) and check the exit code.

* I have not looked, but I suppose it is theoretically possible that there is some code path where make creates a temporary file or something like that. That's not what I'm talking about here. I'm talking about the files that the user cares about, i.e. the ones that actually appear on the command line or in a recipe, whether explicit or implicit.

Interesting discussion on ETXTBSY

NYKevin — Mon, 23 Aug 2021 00:40:40 +0000

No. A thousand times no.

I do not want my tools doing two completely different things depending on some entirely unrelated state. A given cp invocation should always overwrite, or it should always replace. It must not inspect the phase of the moon to decide which code path to use. That's just an outage waiting to happen. If you want it to support both use cases, add a flag and require the user to be explicit.

The shrinking role of ETXTBSY

willy — Sun, 22 Aug 2021 22:49:06 +0000

Run `cat /proc/self/maps`.

My system shows the 'cat' binary mapped five times. One is executable.

But thanks for explaining to me how the read() system call and the page cache works.

The shrinking role of ETXTBSY

Paf — Sun, 22 Aug 2021 22:02:51 +0000

I think you’re missing a step here.

So, the interpreter doesn’t read in a copy or anything? Unless it’s mmaping it’s going to have a copy of at least part of the file. That’s how read() works.

User space doesn’t work from the page cache unless it’s mmaping.

Interesting discussion on ETXTBSY

neilbrown — Sun, 22 Aug 2021 21:53:36 +0000

> Obviously O_DENYWRITE would require write-permission to the file,

and obviously that would make it useless for shared-library loaders and script interpreters.

So I've completely changed my mind. Only permissions should be effective at preventing a process from writing to a file. A correctly written program should choose not to write to a file that is in-use.
This would require a way to find out if a file is in use. If there are needs in this area, that is the direction that we should innovate.

Maybe shared-library loaders and script interpreters should take a LOCK_SH flock on the file, and 'cp' should replace-instead-of-overwrite if it cannot get a LOCK_EX flock.

Interesting discussion on ETXTBSY

niner — Sun, 22 Aug 2021 08:42:36 +0000

For what it's worth, that "courtesy" has not helped me even once in 25 years. On the contrary, I regularily run into it when debugging and it has me hunting through my open terminal windows looking for that gdb session that's still running. Like many other cases where the computer thinks it's smarter than me, I could do very well without this.

The shrinking role of ETXTBSY

NYKevin — Sun, 22 Aug 2021 08:35:10 +0000

Truncating a file already causes all kinds of interesting race conditions. But they're in userspace, so we all collectively pretend that they're not a problem.

(Because, well, what's the alternative? Any global mutable state will inevitably race, at some level of abstraction, and global mutable state is literally the whole point of a filesystem.)

The shrinking role of ETXTBSY

pabs — Sun, 22 Aug 2021 02:00:35 +0000

Based on some of the discussion early in this thread, it sounds like doing that would be very complicated or impossible.

Interesting discussion on ETXTBSY

neilbrown — Sat, 21 Aug 2021 23:28:21 +0000

> I think this is best left in.

I tend to agree. Further, I think it would be good to make it even more useful.
Currently only executables benefit from ETXTBSY. What if we added an O_DENYWRITE open flag.
Then shared-library loaders could set this, as could script interpreters.

With a bit of work, we could make O_DENYWRITE | O_RDWR work so that the opener can write but nothing else can.
This would be great for swap files.

Obviously O_DENYWRITE would require write-permission to the file, thus avoiding the DOS problems of MAY_DENYWRITE.

The shrinking role of ETXTBSY

developer122 — Sat, 21 Aug 2021 18:14:29 +0000

This seems like a much saner thing to do and I'm surprised nobody figured it out in the 80's.

The shrinking role of ETXTBSY

Jonno — Sat, 21 Aug 2021 11:51:12 +0000

> "eatmydata dpkg …" / "eatmydata apt …" to the rescue.

Or you could use `dpkg --force-unsafe-io ...` / `apt --option DPkg::options::=--force-unsafe-io ...`.

That will only remove the fsync before renaming file X.dpkg-new to X, which gives you about 99% of the speed boost from eatmydata, without risking the apt and dpkg databases and logs.

(Meaning that if something does break you can inspect the logs and simply reinstall any packages that was unpacked since your last system-wide sync to fix your installation. If the broken package is essential it might be a bit tricky to do, but you can always boot from a different boot media, such as the Debian installer CD, and run `dpkg --instdir=/mnt/target ...` to get going again).