The 5.3 kernel is out [LWN.net]

The 5.3 kernel is available at last. The announcement includes a long discussion about user-space regressions — an ext4 filesystem performance improvement had caused some systems to fail booting due to a lack of entropy early after startup. "It's more that it's an instructive example of what counts as a regression, and what the whole 'no regressions' kernel rule means. The reverted commit didn't change any API's, and it didn't introduce any new bugs. But it ended up exposing another problem, and as such caused a kernel upgrade to fail for a user. So it got reverted."

Some of the more significant changes in 5.3 include scheduler utilization clamping, the pidfd_open() and clone3() system calls, bounded loop support for BPF programs, support for the 0.0.0.0/8 IPv4 address range, a new configuration option for the soon-to-be-merged realtime preemption code, and more. See the KernelNewbies 5.3 page for lots of details.

to post comments

The 5.3 kernel is out

Posted Sep 16, 2019 9:29 UTC (Mon) by mgedmin (guest, #34497) [Link] (11 responses)

And now I'm curious to know how a performance improvement reduced the amount of entropy.

Fewer I/O operations causing fewer interrupts? Interrupt timings used to increase entropy? Fewer interrupts -- less entropy?

The 5.3 kernel is out

Posted Sep 16, 2019 9:33 UTC (Mon) by yoshi314 (guest, #36190) [Link]

"fail booting due to a lack of entropy early after startup"

i think they key point here is "early after startup". it might be that there is not enough entropy yet to provide.

Entropy loss

Posted Sep 16, 2019 9:34 UTC (Mon) by corbet (editor, #1) [Link] (9 responses)

Exactly that - more efficient I/O led to fewer interrupts, and thus not enough entropy at boot time.

I'm hoping to dig deeper into this, but I'm also sort of buried in conference material at the moment...

Entropy loss

Posted Sep 16, 2019 10:50 UTC (Mon) by darwi (subscriber, #131202) [Link] (8 responses)

A nice summary of the problem can be found below:

- https://lkml.kernel.org/r/20190915081747.GA1058@darwi-hom...

It's a mix of an unfortunate kernel API design (getrandom(2)), missing warnings on usage of such an API (urandom(4), getrandom(2) manpages), bad glibc implementation (a "BSD-like" gentropy(3) since v2.25), and an ext4 disk IO optimization exaggerating the problem considerably.

Entropy loss

Posted Sep 16, 2019 12:58 UTC (Mon) by patrakov (subscriber, #97174) [Link] (7 responses)

What all people who mention "early boot process" fail to understand is that there is no such thing, due to the freedom to assemble an arbitrary initramfs, or because udev can run arbitrary programs from its rules. We already has this lesson: https://freedesktop.org/wiki/Software/systemd/separate-us... . Therefore real attempts to mitigate the issue must be directed not at "not needing the entropy too early", but either at making sure that it is always available, or at admitting the failure and not attempting to fix unfixable.

Entropy loss

Posted Sep 17, 2019 15:38 UTC (Tue) by kazer (subscriber, #134462) [Link] (6 responses)

In a static system there is no entropy. Where do you suggest it is collected from?

Entropy loss

Posted Sep 17, 2019 15:41 UTC (Tue) by patrakov (subscriber, #97174) [Link] (5 responses)

From clock jitter, as measured by timing variations of a memory access pattern. man haveged. Of course subject to validation.

Entropy loss

Posted Sep 17, 2019 17:53 UTC (Tue) by kazer (subscriber, #134462) [Link]

Which means you need to be /running/ the system to get that. It does not just "appear" and be instantaneously available (not in the way required for crng anyway).

Entropy loss

Posted Sep 18, 2019 10:40 UTC (Wed) by nilsmeyer (guest, #122604) [Link] (3 responses)

Perhaps I don't quite understand this but wouldn't it be possible to add these sources of entropy either in the kernel or very early in initramfs?

Entropy loss

Posted Sep 18, 2019 10:58 UTC (Wed) by patrakov (subscriber, #97174) [Link] (2 responses)

In the kernel: explicit ban my Ted T'so (because it is not known how much entropy is in the resulting bitstream), in the initramfs: it's possible but nobody does it because they don't realize the need.

Entropy loss

Posted Sep 18, 2019 14:17 UTC (Wed) by kazer (subscriber, #134462) [Link] (1 responses)

I'm not really sure what you are saying, but it is different to generate randomness sufficient for cryptography ("keep this secret" such as filesystem encryption) and just a random value ("guess my number" such as Xorg MIT cookie). X cookie is particularly bad case to use crng since X security is laughable in other ways already.

In this case reducing file-IO also reduced entropy for crypto-randomness, which was abused by Xorg, which really has not benefit from that level of randomness at all.

Entropy loss

Posted Sep 18, 2019 14:38 UTC (Wed) by patrakov (subscriber, #97174) [Link]

I am saying that a modern CPU (as opposed to an ancient Z80) already has a lot of complex unsynchronized clocks and delays within itself and within the memory controller. Just running a specific, carefully chosen set of instructions (that touch memory and perform arithmetic operations) is guaranteed to take slightly unpredictable time due to "beats" between these independent clocks, even if single threaded, even with interrupts disabled, and even if there is absolutely nothing else running.

So there is, in fact, an unintentionally-created hardware random number generator in every system which does not derive all clocks from a single crystal. There is software ("haveged") that repeatedly benchmarks this kind of instruction sequence using either a high-resolution timer or a cycle counter, and claims to extract entropy from that. Even without RDRAND, and it also works on many non-x86 CPUs. E.g., on mips 24kc in my TP-Link Archer C7, it gets around 200 kilobytes of random data per second. There is also a cryptoapi-based implementation of the same idea in the kernel, and a modern version of "rngd" also has this code. But there are also claims that the output is actually not random (despite passing statistical tests for randomness) and should not be trusted.

The 5.3 kernel is out

Posted Sep 16, 2019 9:51 UTC (Mon) by intgr (subscriber, #39733) [Link] (3 responses)

Before anyone jumps the gun to blame systemd on the ext4 revert (as did Torvalds in his commit message), note that the real cause for boot hanging was not in systemd, but actually in Xorg/GDM/gnome-session using getrandom() for the X11 authentication cookie: https://www.lkml.org/lkml/2019/9/15/314

The 5.3 kernel is out

Posted Sep 16, 2019 11:09 UTC (Mon) by darwi (subscriber, #131202) [Link]

Yeah, I did some quick damage recovery in the early morning to make sure people don't blame systemd :-)

- https://lkml.kernel.org/r/20190916014050.GA7002@darwi-hom...
- https://lkml.kernel.org/r/20190916024904.GA22035@mit.edu
- https://lkml.kernel.org/r/20190916024550.GA1352@darwi-hom...

systemd-random-seed(8) was actually, as Linus thankfully acknowledged later, one of the good users of getrandom(2).

It's __not__ GDM/gnome-session fault either. please check:

- https://lkml.kernel.org/r/20190915081747.GA1058@darwi-hom...
- https://lwn.net/Articles/799271

The 5.3 kernel is out

Posted Sep 16, 2019 23:09 UTC (Mon) by flussence (guest, #85566) [Link] (1 responses)

Presumably this problem would go away if we all just switched to Wayland? ;)

The 5.3 kernel is out

Posted Sep 17, 2019 20:39 UTC (Tue) by adobriyan (subscriber, #30858) [Link]

Or someone could write an en-fragmenter, so that fs could do a little more IO during bootup :-)

The 5.3 kernel is out

Posted Sep 16, 2019 22:21 UTC (Mon) by dkg (subscriber, #55359) [Link]

Has this patch from Ahmed S. Darwish (or this one from Linus) already been merged? If so, this is an unacceptable security regression from the kernel.

Userspace tools like GnuPG and libgcrypt depend on the semantics of the kernel's getentropy system call these days. In particular, they rely on it blocking if the crng is not initialized, which is the right thing to do, and has been documented for years now.

If the kernel is serious about not breaking userspace, it should not change the semantics by introducing this kind of regression.

The 5.3 kernel is out

Posted Sep 16, 2019 22:43 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Can the kernel artificially create some disk activity if it detects blocking?

The 5.3 kernel is out

Posted Sep 17, 2019 7:22 UTC (Tue) by njs (subscriber, #40338) [Link]

Or systemd could do the same in userspace – from the discussion, apparently the reason systemd originally got blamed was because systemd does have a process that calls getrandom(), which it only does to figure out when enough entropy has been generated. So systemd already has the infrastructure to detect when entropy generation is being slow. When that happens, systemd could do things like issue some disk reads, print a message on the console telling the user to wiggle their mouse, and print a link to instructions on how to set up entropy preservation across boots.

The 5.3 kernel is out

Posted Sep 17, 2019 9:07 UTC (Tue) by patrakov (subscriber, #97174) [Link] (2 responses)

We already have jitterentropy_rng - unfortunately and by mistake, as a cryptoalg rng, not as hwrng. It extracts random bits from variations of timing of some CPU and memory operations, i.e. essentially from unsycnhronized clocks in different parts of the system. I will look at its conversion to something (e.g. hwrng) that can actually provide the entropy that Linux accounts for.

The 5.3 kernel is out

Posted Sep 17, 2019 9:41 UTC (Tue) by njs (subscriber, #40338) [Link] (1 responses)

IIRC Ted T'so has a longstanding veto on using jitterentropy to increment the random pool entropy counter in the past, on the grounds that we don't actually know where the apparent randomness comes from and whether it's actually unpredictable or not, esp. against an active attacker who can manipulate microarchitectural state.

The 5.3 kernel is out

Posted Sep 17, 2019 9:44 UTC (Tue) by patrakov (subscriber, #97174) [Link]

Thanks, that's useful to know. But he surely knows that Debian installs haveged by default on systems lacking rdrand, thus circumventing his veto.

The 5.3 kernel is out

Posted Sep 18, 2019 12:53 UTC (Wed) by Vipketsh (guest, #134480) [Link]

I read through most of the LKML discussions regarding this and the impression
I had from some (especially Ted Tso) was that they feel that it's somehow the
kernel's job to make sure that crypto keys are generated securely. Why is
that ?

Thing is that cryptography is hard and any seemingly insignificant minor bug
at any level can be a security disaster -- from a simple uninitalised carry
in an assembly optimised routine all the way to stacking of systems (the WPA
thing). In other words, if you want secure cryptography you need to know
what you are doing and think really, really hard about it. Why would random
numbers be any different ?

The question then is why doesn't the kernel simply tell the user all it knows
and let them deal with it ? Something like: if the user asked for random, hand
out your best randomness *and* your guess to the quality (entropy) of the
randomness. In this way, people that don't care just don't need to care and
the crypto people who would care can look at the quality guess and if it's not
good enough, /user space/ can "do something smart".

Where "do something smart" is definitely not "block until the quality is
good enough", as has been mentioned on the LKML thread multiple times. If
you don't have enough entropy you are not going to get it by just sitting
around doing nothing.