Why not just have a one-step spawn? [LWN.net]

Why not just have a one-step spawn?

Posted Dec 21, 2024 15:39 UTC (Sat) by khim (subscriber, #9252)
In reply to: Why not just have a one-step spawn? by josh
Parent article: Process creation in io_uring

A much simpler approach would be to just add some code that would do that setup in the empty process. And we already have memfd_create/execveat combo that can do that.

If you want – add flag to the clone that would call execveat. And then new code in an entirely empty image can do whatever it needs to prepare for the execution of the real binary that you want to execution.

Why shove io_uring into something that already can be done entirely from userspace? Buzzword compliance?

to post comments

Why not just have a one-step spawn?

Posted Dec 21, 2024 16:10 UTC (Sat) by corbet (editor, #1) [Link] (17 responses)

Khim, if you have a better idea, please submit a patch showing it. But please stop insulting the work of others, that does not help anybody.

Why not just have a one-step spawn?

Posted Dec 21, 2024 16:44 UTC (Sat) by khim (subscriber, #9252) [Link] (1 responses)

> But please stop insulting the work of others, that does not help anybody.

Where do you see insults? I've faced the need to mangle simple and easy to understand and implement ideas into pretzels to include all the right buzzwords at my $DAYJOBs often enough that I can easily see buzzword compliance as explicit, or more likely, implicit part of the requirements.

And very often it's even the most important one: if you couldn't cause enough buzz around your idea then it would die (except if there are some concrete tasks for concrete customers that may need it) even if it's pretty good, but with enough buzz around your idea you may push it even if it's totally stupid and would hurt everyone in the long run.

> Khim, if you have a better idea, please submit a patch showing it.

There are no patch because in-kernel parts are already done… years ago, in fact.

And to discuss userspace part we need some idea about who, why and how plans to use that mechanism.

The list of interested parties is not in the article thus it's hard for me to offer anything concrete because it's not clear to me how much flexibility is needed or wanted.

Implementation of posix_spawn is doable but would be significant amount of work without any clear benefits: do we have lots of users of that syscall? If yes, then where are they, if not then why are they so rare?

IOW: I don't see enough of a picture related to that work to judge it fairly and if “buzzword-compliance” is part of reasoning (even if an implicit one) then it could be that io_uring-based solution is the best way forward. Especially if it's a solution-in-a-search-of-a-problem: it's much easier to make someone excited about io_uring solution than about solution that just combines well-known syscalls in a way that makes posix_spawn safer.

Why not just have a one-step spawn?

Posted Dec 21, 2024 16:48 UTC (Sat) by corbet (editor, #1) [Link]

"Buzzword compliance" takes the work of people who are trying to improve the system and casts it as something useless. If it were my work, I would find that insulting. I do not believe that the people working on this are concerned about buzzwords, they are trying to solve real problems. Please try being a bit more respectful toward them.

Why not just have a one-step spawn?

Posted Dec 23, 2024 9:26 UTC (Mon) by gutschke (subscriber, #27910) [Link] (14 responses)

I am still not 100% convinced that khim's solution is necessarily easier nor more robust. But since I was curious whether the proposal to use existing kernel API's in somewhat unconventional ways would be viable at all, I wrote proof-of-concept code and uploaded it to: https://github.com/gutschke/safeexec/blob/main/safeexec.c

Not surprisingly, since we are doing things that weren't quite intended to be done this way, there are warts and pit-falls. If my code was to be turned into a production-quality library, a good amount of additional polishing is necessary. But as is, this is evidence that khim's suggestion can address several of the concerns raised in these comments.

(The best way to play with the code is to run it under the control of "strace". All it does is call "/bin/true" in a very round-about fashion.)

Why not just have a one-step spawn?

Posted Dec 23, 2024 14:39 UTC (Mon) by khim (subscriber, #9252) [Link] (12 responses)

TL;DR version: this approach is better because in case of adoption failure (which is quite likely) one may just throw it away and forget about it, whileas if similar failure would happen with io_uring solution the code and special properties of these chained operations would have to stay in kernel forever.

> I am still not 100% convinced that khim's solution is necessarily easier nor more robust.

It's “easier” in a sense that you can use it in applications for RHEL8+ and Android8+ (and most other distributions also have kernels with memfd_create, too).

That means that you could model your io_uring solution and test it on wide set of real-world tasks (since it can be used in production).

Even if final solution would be to add either a dedicated syscall or set of io_uring operations (plus set of special “chaining” rules needed to make them usable) you would collect lots of data which would tell you what works and what doesn't work.

If you start with addition to the kernel, on the other hand, then all these “large parent processes” deployed in various places wouldn't be able to use it for many years – and by the time when you would have real data collected from real apps… kernel API would be long-established and, most likely, not used (just like posix_spawn is barely used today).

P.S. Of course if you plan to eventually go with io_uring, anyway, then it would be good idea to have API of your safeexec designed in way that would make it easy to switch to io_uring, at some point. Apps wouldn't even need to know that they stopped using “double exec” trick and switched to io_uring on kernels that have io_uring support, it would all be transparent for them.

Why not just have a one-step spawn?

Posted Dec 23, 2024 16:29 UTC (Mon) by bluca (subscriber, #118303) [Link] (2 responses)

> It's “easier” in a sense that you can use it in applications for RHEL8+ and Android8+

Actually I don't think you could use it in either, given it requires writable + executable memory, which is blocked by SELinux by default. Most sandboxing systems restrict that as well, as it's a very commonly used attack vector.

Why not just have a one-step spawn?

Posted Dec 23, 2024 16:36 UTC (Mon) by khim (subscriber, #9252) [Link]

That can be solved if you would crate two mappings: read/write one and read/execute one. Or even just create read/write mapping, then fill it and then change to read/execute before vfork/spawn.

These tricks are already used by JITs and most distributions, even very “enterprise” ones, have knobs to allow JITs, only iOS disabled that completely (and I don't think iOS is in scope for that project).

Changing SELinux settings is needed in any solution, even if you introduce new syscall it's highly unlikely that SELinux wouldn't stop that till you retune it.

Why not just have a one-step spawn?

Posted Dec 23, 2024 17:00 UTC (Mon) by gutschke (subscriber, #27910) [Link]

No writable/executable mapping is used in my proof of concept. Once the ephemeral ELF image has been exec()'d, there is only a single readable/executable mapping.

I use a single mapping for both code and read-only data. That approach slightly simplified the already painfully complicated open-coded serialization of the various data structures that need to be passed into the child. But that could be split into two separate mappings for a production release.

Or instead of passing data as part of the ELF image, all data could be passed into the ephemeral child over a pipe(). Those design details are certainly up for review.

Chained operations in io_uring

Posted Dec 24, 2024 11:57 UTC (Tue) by farnz (subscriber, #17727) [Link] (8 responses)

special properties of these chained operations would have to stay in kernel forever.

The neat thing about the proposed io_uring solution is that the special properties of these chained operations already exist for other reasons - in order to allow you to queue up an I/O operation with an appropriate response on error, chains and hard links already exist^[1], and to allow io_uring to operate asynchronously to process context, it already knows how to handle trying to return to a userspace that isn't running.

The only new things here are IORING_OP_CLONE that creates a new process (not able to run) and IORING_OP_EXEC that replaces the program text and turns it into a ready-to-run process. Everything else already exists in io_uring for I/O purposes.

^[1] The intent is that you can do something like write to a WAL, fsync the WAL if the WAL write succeeds, write to the final location if the WAL write and fsync succeed, and then regardless of success of the WAL and final location writes trigger a futex wake, all in a single submission to the kernel.

Chained operations in io_uring

Posted Dec 24, 2024 14:10 UTC (Tue) by khim (subscriber, #9252) [Link] (7 responses)

> The neat thing about the proposed io_uring solution is that the special properties of these chained operations already exist for other reasons

Have you actually read the article? That one, specifically: Krisman hopes to be able to at least partially lift that constraint in the future.

It's extremely clear to me that interface, as presented, it's not finished and not tested. Or, even worse, tested and is just feed to kernel developers in an insidious way to convince them to adopt huge hairball of API that would be immediately rejected if presented in it's full capacity… that's even worse then “unfinished and untested” API in my book (and I sincerely hope it's not that: Hazan's razor and all that).

> The only new things here are IORING_OP_CLONE that creates a new process (not able to run)

Which is something that Linux haven't supported till today. Currently “load new executable in a process” is one atomic operation that starts from the state where kernel have something mapped and executable in it's address space and ends in the state where kernel have something mapped and executable in it's address space.

An attempt to split that process in two looks innocuous enough, but it's entirely not clear what strange pitfalls it may hit.

> The intent is that you can do something like write to a WAL, fsync the WAL if the WAL write succeeds, write to the final location if the WAL write and fsync succeed, and then regardless of success of the WAL and final location writes trigger a futex wake, all in a single submission to the kernel.

Yes. And that's fine because code before and after comes from the exact same codebase. If some steps are omitted and/or failed then code that started the whole mess could, presumably, handle these failures gracefully.

Compare with io_uring attempt to do clone/exec attempt: you are doing some important cleanup work after clone which is, well, important (or we wouldn't worry so much about doing it in the first place) – and if it fails we execute foreign code, anyway.

This sounds, to me, like “hey, we have added nice security vulnerability to the kernel API, we just have no idea how to exploit it in the wild… contest is starting”!

The most likely consequence would be pile of special cases forbid some “likely exploitable” instructions in the sequence between IORING_OP_CLONE and IORING_OP_EXEC.

With ongoing maintenance when they would be discovered and open questions about what to do about apps that rely on these operations.

Of course safeexec also includes all the same issues (and probably some more), but there's a big difference: because it's not a kernel API and it can be easily embedded into your application by linking it statically there are no need to support all the warts of the first version indefinitely. In can be tuned and fixed [relatively] freely without commitment to support it forever (because each released version is self-contained and would work like it did on the day one).

Chained operations in io_uring

Posted Dec 24, 2024 14:41 UTC (Tue) by daroc (editor, #160859) [Link] (6 responses)

Let's please not get too heated; even though we try to make articles as clear as possible, it's easy to have slightly different understandings of complex technical topics, and the best way to resolve that is usually with examples and more explanations.

In any case — nothing obliges developers to use hard links between io_uring operations. If an important cleanup operation is necessary, and it is not safe to execute the new program if it fails, don't use a hard link. While it is arguably suboptimal to introduce yet another API that must be used correctly or risk security problems, it is hardly the first such API in the kernel. Nothing prevents a poorly-written program from leaking various kinds of state to another program with the current fork()/exec() workflow.

Chained operations in io_uring

Posted Dec 24, 2024 16:04 UTC (Tue) by khim (subscriber, #9252) [Link] (5 responses)

> In any case — nothing obliges developers to use hard links between io_uring operations.

How would anything work without hard links? After IORING_OP_CLONE your process is in “undead” state. It's neither alive nor entirely dead, but the important part: it doesn't have any userspace that may act and do some decisions.

That's the whole point of that patch series: to introduce a way to “clean up” that “undead” process by doing some operations when userspace is entirely gone.

If you wouldn't use hard links… what is supposed to happen? How would non-hardlinked operations work without userspace? What is supposed to happen if some operation would would fail? We don't have any agent that may receive information about failure!

Sure, we would know that combined operation would fail without running the program, but we would just make already stupid situation (where people try to execute programs with non-standard loader, get the message “file not found” and then spend hours trying to understand why program that's clearly there will all the proper permissions couldn't be execute) even worse.

> While it is arguably suboptimal to introduce yet another API that must be used correctly or risk security problems, it is hardly the first such API in the kernel.

It's worse than that: currently it's not “must be used correctly or risk security problems” but “it must be extended more before it would go beyond “proof of concept” phase, because in it's current form it's impossible to use it safely”.

And we have no idea how much more should it be extended to become actually usable. That's precisely why I have said that we need to know who, why and how plans to use that mechanism – because without such information we have no idea what needs to be added to it to make it actually usable.

> Nothing prevents a poorly-written program from leaking various kinds of state to another program with the current fork()/exec() workflow.

Sure, but correctly-written program can do everything correctly. And handle failures safely. Even if glibc fails to do that it's possible, at least in theory. That's currently impossible to do with the new mechanism.

Can it be extended to handle these things? Sure: you can make it possible to receive information about io_uring operations in the parent process. Or introduce some high-level “cleanup” operations. Or do many other extra extensions… but before we would do all that the main question needs to be answered: what we are actually trying to do with that mechanism?

> the best way to resolve that is usually with examples and more explanations.

Sure, but why are you directing that request to me? The main advantage that was touted in the original work way speed that's 6-10% faster than vfork() and 30+% faster than posix_spawn().

I have no idea who would really need that speedup (most of the time time spent in fork/exec is minuscule compared to the time needed to run dynamic loader, verify signatures and so on), but that sounded somewhat sensible.

But if all that complexity (including fight with a kernel corruption after a few spawns) and less reliability than what existing mechanisms provide is justified then it would be really nice to know who executes so many processes that they do care about fork/exec time, why the time needed to actually start process is not impeding their work and so on.

Because if it's some silly specialized unikernel or some kind of cluster management software – then it may very well be handled better with a more focused, more specialized API instead of jenga tower that this patch series starts to build.

Chained operations in io_uring

Posted Dec 24, 2024 19:12 UTC (Tue) by daroc (editor, #160859) [Link] (4 responses)

> How would anything work without hard links? After IORING_OP_CLONE your process is in “undead” state. It's neither alive nor entirely dead, but the important part: it doesn't have any userspace that may act and do some decisions.

It's possible that I've misunderstood how the patch series works, but I thought that if the whole series of operations fails, the program that originally started the operation is notified in the normal way (via an item in the io_uring completion queue). You can see that in this example: https://lwn.net/ml/all/20241209234421.4133054-3-krisman@s...

So a program submits the chain of io_uring operations, and then it either succeeds (and a new process is created) or it fails, and the program that submitted it can choose how and whether to retry. So hard links aren't needed, and it's perfectly possible to write a correct program that closes important files with the current patch series.

Chained operations in io_uring

Posted Dec 24, 2024 19:37 UTC (Tue) by khim (subscriber, #9252) [Link] (3 responses)

> You can see that in this example:

Which test is that? AFAICS the only test function that doesn't use linking, test_unlinked_clone_sequence just issues unlinked IORING_OP_CLONE and then expects this:

if (cqe->res != -EINVAL)
	… Unlinked clone should have failed …

That's it. All other examples use linked operations, as they should.

It's possible that I have misunderstood something, but at least at the first glance it's obvious why it have to be done that: after IORING_OP_CLONE is executed the whole io_uring machinery (I suspect 99% of Linux functionality) becomes, temporarily, “untouchable”… with some operations still permitted – only in linked form. And then it either succeeds (while silently ignoring errors leading to unknown state of the executed process) or fails – as whole.

> So a program submits the chain of io_uring operations, and then it either succeeds (and a new process is created) or it fails, and the program that submitted it can choose how and whether to retry.

Couldn't see that. At least in that patch series and set of examples.

And it's obvious why: if you add that machinery then, suddenly, instead of simple and non-invasive patch that just adds couple of io_uring commands one would need to design completely new machinery which can support inter-process io_uring support! With execution happening in the context of one process and communication channel opened to another process.

Sure, that's not impossible to create, but… do we really want to add so many new subtle features for 6% speedup?

> So hard links aren't needed, and it's perfectly possible to write a correct program that closes important files with the current patch series.

Show me the code. I couldn't find it. And I suspect that it's precisely as I have said: we only see 10% of the iceberg here, the majority of changes, 90% of iceberg is either doesn't exist or is not submitted for review.

Chained operations in io_uring

Posted Dec 24, 2024 19:49 UTC (Tue) by corbet (editor, #1) [Link] (2 responses)

You are really determined to sink this patch series, I'm not sure why.

Normally, you would not use hard-linked operations in the newly cloned child context. If one of the setup operations fails, the entire chain fails, with the status returned to the parent. No silently ignored failures. No unknown state.

Hard links can be used in the execveat() sequence to implement a search path. In that case, continuing after failure is the desired outcome; you want to go until you find something you can actually execute.

I am sorry if the article did not adequately convey that.

Doubtless there is interesting work to do to expand the range of actions available in the just-cloned child context; we will have to see what shape that takes. But I see no reason to suspect some sort of evil plot here.

Chained operations in io_uring

Posted Dec 24, 2024 20:36 UTC (Tue) by khim (subscriber, #9252) [Link]

> You are really determined to sink this patch series, I'm not sure why.

I want to understand what that patch series hopes to achieve, mainly. This part is not reassuring: Krisman hopes to be able to at least partially lift that constraint in the future. And this is even more worrying: The hope is to increase the set of possible operations over time, enabling the implementation of complex logic for the spawning of a new task.

In essence we are supposed to accept some piece of the whole solution without us knowing where the whole thing even leads.

And, worse yet, it's not clear what problem this whole thing is even supposed to solve!

If it's safety of creation of a new process then it's one thing (there are no need for io_uring, we already have all the pieces), if it's 30% speedup for posix_spawn, then it's another thing.

> But I see no reason to suspect some sort of evil plot here.

Evil plot is unlikely. But it really looks like a solution in a search of a problem… and I want to see the problem and, more importantly, explanation why that's the best solution for it.

As was noted in article one alternative solution would be to just create a dedicated system call that would include all the required operations. Or “double exec” if we just want to safely implement posix_spawn.

And if it's “an attempt to see where it goes” then I don't really want to sink but more to “flesh it out”, understand how the full, final, solution would look like and, again, who, why and how would use it.

Because as it stands currently, it's not clear to me what's the goal of all that activity – and that matters much more then minor details of the implementation in the current form.

Even if we would achieve the final goal of being able to execute all io_uring commands in this sequence of these instructions between clone and execute why are we so sure it would be enough.

Where do we plan to arrive with that change and what do we plan to achieve?

> Hard links can be used in the execveat() sequence to implement a search path. In that case, continuing after failure is the desired outcome; you want to go until you find something you can actually execute.

Ah. I see. While this, again, looks like a solution in a search of a problem (why to look up for the executable before executing it? what's the point of moving this pretty much optional functionality into the kernel? do we really want to try to continue after finding “kinda-sorta-suitable” binary that would end up being broken, for some reason?) at least now I understand what I didn't understood about that patch set.

Thanks for explaining it: while I still am not sure how useful would it be to implement what it tries to implement (because, again, I couldn't see the end goal), at least some operations can be implemented in safe manner. That's better than how I understood it working. Thanks for explanation.

Chained operations in io_uring

Posted Dec 24, 2024 23:03 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> Normally, you would not use hard-linked operations in the newly cloned child context. If one of the setup operations fails, the entire chain fails

How exactly is this going to be achieved for processes? As I understand, there's going to be a new visible intermediate state for the process, as the operations are being executed, unless the io_uring sequence locks the entire kernel.

This also can cause a problem for userspace process migration. How do you interrupt the io_uring sequence to suspend it? After reading the patch series, I don't see how it would prevent long-running operations like read() from being introduced into the middle of the sequence.

It really is a poorly-designed API. It is very much in line with the good old UNIX tradition of screwing up process management APIs.

Why not just have a one-step spawn?

Posted Jan 12, 2025 18:17 UTC (Sun) by mrugiero (guest, #153040) [Link]

I believe it would be easier (but would take more execs) to just use execline on your first exec to set up the environment correctly. Nice little scripting language which is already designed for that.