LWN: Comments on "Restartable sequences in glibc"

Restartable sequences in glibc

smurf — Sun, 24 Aug 2025 14:57:05 +0000

Yes, you did miss something: rseq calls cannot be nested. The cover a single range of instructions. This means that you cannot call malloc() (or anything else for that matter) from your restartable code. Thus your use of "struct rseq" and malloc's will not interfere with each other by definition.

Restartable sequences in glibc

vickyBishnoi — Thu, 20 Jul 2023 13:53:55 +0000

Does it possible to disable rseq for selected application?
if yes, how it can be done.

Restartable sequences in glibc

compudj — Wed, 02 Feb 2022 11:41:10 +0000

librseq can be used as an alternative way to register rseq for cases where the libc does not handle rseq. It requires the application to explicitly call a register/unregister API at the beginning/end of each thread's lifetime. It is compatible with the glibc-2.35+ ABI as well and will let glibc handle rseq registration in those cases. See https://git.kernel.org/pub/scm/libs/librseq/librseq.git/

Restartable sequences in glibc

foom — Wed, 02 Feb 2022 04:06:01 +0000

glibc plans to use rseq internally to do useful things like speed up malloc, so it needs to do rseq registration regardless of whether anyone else wants to use it. There doesn't seem to be much point in creating a mechanism to share registration duties, if libc will always "win" anyways.

As far as not exposing any user space abi: if you don't, every user would then need to make a syscall to retrieve the rseq area's location separately per thread? And presumably then cache it in a tls variable for performance? That seems a bit silly and wasteful, when it's easy enough to just make a constant thread offset available in user space to anyone who needs it.

Restartable sequences in glibc

NYKevin — Wed, 02 Feb 2022 02:38:53 +0000

If I'm understanding things correctly, it seems that the only real problem glibc is solving is the following:

1. In order to use restartable sequences, someone needs to alloc a struct rseq and tell the kernel about it. Also, you can't free that struct unless the thread exits or you tell the kernel to stop using it.
2. If multiple different libraries or codepaths try to do that, they will step on each other.
3. Therefore, someone needs to "own" the struct rseq for each thread, and everyone needs to agree on the ownership of this struct.
4. However, once everyone agrees on who owns the struct, there's nothing wrong with foreign code overwriting the struct, so long as it stays within its own thread and doesn't have any reentrancy issues (the struct only needs to be valid over short instruction sequences, and nesting is explicitly unsupported - clobbering somebody else's rseq state is a non-issue as long as you don't move the struct, free it, or clobber it from a signal handler).

glibc fills the role of the "someone" in step 3. However, I don't understand why this role must necessarily be filled by a specific, fixed userspace component at all. If the kernel exposed an API for querying the address of a thread's current struct rseq (which the kernel surely knows), then you could just take a "first to call rseq() wins" approach, and completely sidestep the ownership issue altogether. You would still have the problem that the struct must be freed when the thread exits (and no earlier!), and in practice this might result in glibc trying to be the first to initialize it anyway, but there would be no need for an explicit userspace ABI for this sort of coordination - everybody could just use the kernel to coordinate who owns the struct. OTOH, I suppose there might be some sequencing issues when the thread exits (i.e. during the thread-exiting process, exactly when does it become "safe" to free/reclaim the struct rseq?), but I tend to imagine that there are ways of solving this problem (e.g. it must be allocated on the owning thread's stack, it must be free'd from a different thread after the owning thread is gone, or something similar), and it probably wouldn't be too hard to agree on a convention for how to do that.

Have I misunderstood something?

Musl

corbet — Tue, 01 Feb 2022 21:49:05 +0000

Marginally related: the musl developer isn't entirely happy with the glibc plan and would like to see it delayed.

Restartable sequences in glibc

NYKevin — Tue, 01 Feb 2022 21:37:03 +0000

Is there a compatibility story for those libs getting linked into something that doesn't link to glibc (e.g. because it uses musl or something)? Or are those users just out of luck?

Restartable sequences in glibc

compudj — Tue, 01 Feb 2022 16:24:05 +0000

The code for a restartable sequence critical section sits squarely within a single inline asm. So there is no way one could "call" a data manipulation library from the inline asm without an explicit call, which would move the instruction pointer outside of the critical section, and is not at all a supported use-case.

So I really don't think the scenario you have in mind can realistically happen with the current rseq ABI.

Restartable sequences in glibc

developer122 — Tue, 01 Feb 2022 16:10:39 +0000

I'm thinking in terms of code that may itself make use this restartible sequences feature, within a restartable sequence. It would be a linear sequence of code, but there may be a subtask of one atomic sequence that itself makes the most sense to implement as an atomic sequence. I have a hard time imagining such a senario (why not just handle cleanup for both sequences as part of the code at the outer sequence's cleanup vector, and omit the inner sequence's guards?) but I wouldn't be surprised if such a case existed.

As for how such a situation could ever occur, the article mentions calling code that makes use of restartible sequences, which I suppose could be inlined. So, one bit of restartible code could call a data manipulation library that itself naively tried to create a restartible sequence to protect it's own data structures. Each is trying to protect it's manipulation of it's data structures from access during premption by discarding results that were being worked on if a premption occurred.

The tricky thing here is the matter of cleanup. If something interrupts both sequences by occuring during the nested sequence, then you could restart just the inner sequence but that's wrong because the outer sequence is interrupted and doesn't know it. BUT, if you run just out outer sequence's cleanup code, then the data structures for the inner sequence may be left in an indeterminate state with the changes not being discarded. You can't run both, because only the entry not the exit is defined.

And while we're at it, we're invented the C++ problem of memory cleanup :/

Restartable sequences in glibc

khim — Tue, 01 Feb 2022 14:29:50 +0000

Looks like the notion this is clearly not a feature that is intended for casual or frequent use, but it can evidently produce significant performance gains in systems with high scalability requirements is completely wrong: it would absolutely be the feature which many apps would be “frequently using”… although not directly but because small handful of very low-level libraries would adopt it eagerly.

Restartable sequences in glibc

roc — Tue, 01 Feb 2022 06:54:11 +0000

Oh and rr also pins tracees to a single core (usually) so there are never any CPU migrations.

Restartable sequences in glibc

roc — Tue, 01 Feb 2022 06:53:26 +0000

rr generally doesn't instrument code. Also, we run code on a single core and control scheduling. It is therefore pretty easy to for us to emulate rseq completely. We don't pass the syscall through, instead when rr performs a context switch or signal delivery, we emulate an rsync abort ourselves if necessary.

Restartable sequences in glibc

compudj — Tue, 01 Feb 2022 03:27:31 +0000

In order to facilitate use of rseq for the mere mortals, I've been working on a library called librseq. It is available at https://git.kernel.org/pub/scm/libs/librseq/librseq.git/ . It has two goals:

Its primary goal is to provide a higher-level C API as static inline functions to implement the critical sections for common use-cases on all supported architectures (currently x86 32/64, arm 32/64, powerpc 32/64, s390, s390/x, and mips). It does the heavy lifting: it implements the per-architecture assembly for each per-cpu data access pattern.

Its second goal is to provide a rseq registration API to be used with older glibc (before 2.35) which is also compatible with glibc 2.35.

librseq is available under both LGPLv2.1 and MIT licenses. This library is still under active development, with no official release yet.

Restartable sequences in glibc

compudj — Tue, 01 Feb 2022 01:57:51 +0000

AFAIK glibc expects to do heavy use of rseq. The initial use-case is to speed up sched_getcpu() on various architectures, but they also want to use it other future improvements internal to glibc.

So glibc is not just wiring rseq up for the application. It expects to use it as well even before the application main() is started.

I tried to keep track of all rseq "early adopter" open source projects (e.g. tcmalloc), and they have been made aware that they would have to update their userspace ABI to adapt to the glibc ABI. It was not an issue for them. I maintain librseq, which I have adapted to co-exist with glibc 2.35 and older glibc as well. I have not made any official release of this librseq project yet especially because I was awaiting a final choice on the userspace ABI, which is now happening with glibc 2.35. I have also sent a patch series to Peter Zijlstra to update the Linux kernel rseq selftests so they can co-exist with glibc 2.35. It is queued in the tip tree for the next merge window.

Restartable sequences in glibc

compudj — Tue, 01 Feb 2022 01:51:14 +0000

There is no such thing as nested restartable sequences. A rseq c.s. cannot issue a function call that moves the instruction pointer outside of the critical section range.

Please keep in mind that this is user-space code, so interrupt handlers don't really make sense in this context. What happens when a signal is delivered on top of a rseq critical section is very much relevant though.

What happens in this case is that the rseq c.s. interrupted by the signal handler will be aborted (it's instruction pointer moved to the abort_ip) so when the signal handler returns, the interrupted thread will continue its execution at the abort ip. It's pretty much as simple as that.

This allows using rseq critical sections within signal handlers as well.

Restartable sequences in glibc

compudj — Tue, 01 Feb 2022 01:38:55 +0000

A small correction about the sentence "Immediately prior to entering its critical section, a thread should store the address of its rseq_cs structure into the rseq_cs field of the global rseq structure; it should reset that field to NULL on exit." in the article.

Whereas it is OK to set rseq_cs to NULL when exiting a rseq critical section, it is not actually needed. The only requirement is that the rseq_cs pointer is set to NULL at some point after exiting the rseq critical section, but before reclaim of the memory holding the rseq_cs structure and the code it points to (e.g. dlclose(3) of a shared library).

Removing the requirement for setting the rseq_cs pointer to NULL on exit from a rseq critical section is a significant performance improvement considering that the entire critical section is implemented with very few instructions, which is achieved by letting the kernel detect when it returns to user-space over an instruction pointer which is outside of the range of the rseq critical section. When this is detected, the kernel simply clears the rseq_cs pointer.

Restartable sequences in glibc

developer122 — Tue, 01 Feb 2022 01:26:20 +0000

In the case of nested restartable sequences, what would even be the correct behavior? If sequence A called into code that started sequence B, and an interruption occurred during B, should we jump to A's interruption vector or B's? Technically, they were BOTH interrupted, but the failure of B is part of the failure of A? So we should jump to A's vector?

Restartable sequences in glibc

zx2c4 — Mon, 31 Jan 2022 23:54:25 +0000

https://github.com/rr-debugger/rr/commit/64f0d4ef83bd36ee...

Restartable sequences in glibc

mjw — Mon, 31 Jan 2022 20:04:44 +0000

Did you have to do anything special for rr?
Do you have a pointer to your patch?

For valgrind we opted for now to simply return ENOSYS for rseq after consulting with the glibc hackers to confirm this causes glibc to simply skip the rseq setup when running under valgrind (as if running on a kernel that doesn't implement rseq).

https://bugs.kde.org/show_bug.cgi?id=405377

A real implementation is somewhat tricky if you might have instrumented the restartable sequence. DynamoRIO lists some issues and their current "run twice" approach https://dynamorio.org/page_rseq.html

Restartable sequences in glibc

roc — Mon, 31 Jan 2022 18:42:00 +0000

Coincidentally I just added rseq support to rr.

I'm a bit surprised that glibc is going to default to initializing rseq for every thread. An extra syscall for every thread creation, whether or not rseq will actually be used, seems like unnecessary overhead and this is also going to break every existing rseq user. Wouldn't it have made more sense to require users to call some kind of "ensure rseq initialized" function on each thread before they use rseq?