SA_IMMUTABLE and the hazards of messing with signals

Posted Dec 18, 2021 8:10 UTC (Sat) by roc (subscriber, #30627)
Parent article: SA_IMMUTABLE and the hazards of messing with signals

What is actually needed here is not "a more cautious approach to backporting", but automated tests of basic functionality, which are actually run on every "stable" kernel at least once before it is released, and whose failure blocks the release of that "stable" kernel.

SA_IMMUTABLE and the hazards of messing with signals

Posted Dec 20, 2021 2:29 UTC (Mon) by khim (subscriber, #9252) [Link] (10 responses)

How exactly would tests help? This is clear case of Hyrum's Law: feature which developers haven't designed, planned or even expected to work… yet which users discovered and started using.

Tests rarely (if ever!) help in cases like that. They only verify that aspects of the implementation which developers know about work.

SA_IMMUTABLE and the hazards of messing with signals

Posted Dec 20, 2021 5:00 UTC (Mon) by roc (subscriber, #30627) [Link] (9 responses)

Tests not only *could have* caught this regression, they *actually did*: rr has an extensive automated test suite which caught this regression (and has caught many other kernel regressions). There is no reason why kernel CI tests shouldn't cover at least as much kernel functionality as rr's tests --- they could even just run rr's tests.

As it happens, this isn't Hyrum's Law. Ptracers are supposed to be notified every time a signal is delivered to a tracee (other than SIGKILL). It's clearly documented in the man page:

> While being traced, the tracee will stop each time a signal is delivered, even if the signal is being ignored. (An exception is SIGKILL, which has its usual effect.)

The changes here broke that guarantee.

SA_IMMUTABLE and the hazards of messing with signals

Posted Dec 21, 2021 16:10 UTC (Tue) by walters (subscriber, #7396) [Link] (8 responses)

Robert, I happen to agree with you overall. However, I am doubtful that LWN comments are going to make someone just wake up tomorrow and try to gate kernel commits on rr's test suite.

What is more likely to work (I'd say) is directly getting in touch with one of the CI groups. Maybe https://01.org/lkp/documentation/0-day/lkp-test-suites-de... ?

Bear in mind that this type of "reverse dependency testing" also incurs a maintenance burden; for example, any flakes in the rr test suite suddenly become flakes that kernel maintainers need to deal with. There's questions of versioning (pin to rr commit or track git main), build system, etc.

So if you're serious about this then I think some effort on your side would need to happen both initially to drive it, then there's an ongoing maintenance commitment.

SA_IMMUTABLE and the hazards of messing with signals

Posted Dec 22, 2021 2:04 UTC (Wed) by roc (subscriber, #30627) [Link]

You're right that there's a certain amount of shouting into the wind on my part.

> I think some effort on your side would need to happen both initially to drive it then there's an ongoing maintenance commitment.

I'm sure you're right about this too. Unfortunately I'm not confident the expected value *for us* of doing this would actually be greater than what we're currently doing; we could easily do a bunch of work and then the CI test failures end up being ignored (unless we do the work to diagnose and chase down each regression, which is not all that different from what we have to do today). At a less rational level, it grates to be a tiny, unpaid team doing work to clean up the messes of people being paid megabucks to do kernel development; this is already the case, but upping the stakes in the hope of a net win makes it more so.

SA_IMMUTABLE and the hazards of messing with signals

Posted Dec 22, 2021 2:16 UTC (Wed) by roc (subscriber, #30627) [Link] (6 responses)

> Bear in mind that this type of "reverse dependency testing" also incurs a maintenance burden; for example, any flakes in the rr test suite suddenly become flakes that kernel maintainers need to deal with. There's questions of versioning (pin to rr commit or track git main), build system, etc.

This is all true too.

Ideally there would be a standard test harness and repository for user-space tests of kernel APIs, and we would be able to merge tests from rr into that system. We wouldn't have to merge all of them, and we could run them hundreds of times pre-merge and reject any that failed even once. But as far as I know that harness and repository don't actually exist. Or am I wrong?

Kernel API test harnesses

Posted Dec 22, 2021 3:01 UTC (Wed) by corbet (editor, #1) [Link] (5 responses)

kselftest is probably the droid you are looking for here. I am sure they would welcome contributions!

The Linux Test Project might also be a good home.

Kernel API test harnesses

Posted Dec 22, 2021 9:41 UTC (Wed) by roc (subscriber, #30627) [Link] (4 responses)

Thanks.

Thinking about it some more, I guess I was wrong to suggest that merging rr tests into an existing test framework makes sense. Maybe 0-Day CI is the place for this after all.

Kernel API test harnesses

Posted Dec 22, 2021 14:09 UTC (Wed) by corbet (editor, #1) [Link] (3 responses)

Getting tests merged into kselftest is a good way to get them run in a number of the current testing operations, including those that run for stable kernel candidates.

Kernel API test harnesses

Posted Dec 22, 2021 20:59 UTC (Wed) by roc (subscriber, #30627) [Link] (2 responses)

Yeah, but you wouldn't want to merge a copy of rr itself into kselftest, and that's what would be needed.

Most rr tests are small programs that exercise certain kernel functionality. Running the test requires recording execution of the small program, followed by replaying it, and we verify that the replay worked correctly. Some of the kernel regressions rr caught could have been detected just by running the small programs normally --- because the kernel functionality they use regressed --- and those could be trivially added to kselftest. But many of the kernel regressions rr caught were related to ptrace, signals, etc that were exercised by rr itself.

rr isn't huge (50K-ish lines) but it's written in C++ and I doubt it would be acceptable to paste it into the kernel tree.

I think 0-Day-CI is really the right approach here. But do test failures there block kernel "stable" releases?

Kernel API test harnesses

Posted Dec 22, 2021 21:10 UTC (Wed) by corbet (editor, #1) [Link] (1 responses)

My suggestion would be to submit tests that only run if rr is available. But I'm not the kselftest maintainer, so I can't guarantee they would accept such a thing; some of those folks have been known to read here, maybe they could speak up.

As for blocking stable releases: regression reports will delay them and cause the offending patches to be fixed or removed; see the discussion that follows any of the review postings to see how that works. You can see an example here. The problem is not (usually) responding to reports of regressions, it's knowing that the regressions exist in the first place. As you have often (rightly) pointed out, more and better tests would help a lot in that regard.

Kernel API test harnesses

Posted Dec 23, 2021 20:59 UTC (Thu) by roc (subscriber, #30627) [Link]

Thanks for pointing that out. It looks like 0-Day CI results were not reported there. But results from https://lkft.linaro.org/ were reported there --- and despite the site saying "the mission of LKFT is to improve the quality of the Linux kernel by performing functional testing on Arm hardware" they actually run tests on x86 as well. Maybe that's the right place to add a new test suite.