LWN: Comments on "Moving Google toward the mainline"

Moving Google toward the mainline

teksturi — Mon, 18 Oct 2021 10:38:19 +0000

You should consider rolling release stable as your base.

This way every time new rolling stable tag comes your feature branches will be rebased top of that (maybe to new branch). Then your feature maintainer will get info if there is conflict. Some of your maintainers will be little bit more far of from rolling release stable than others, but this will be totally ok. You can very easily see which are bottlenecks.

You can at any time start new kernel internal kernel. Just choose highest version number which all of yours feature branch is compared to rolling stable. Example feature A got big conflict in 5.14 and it takes 3 weeks resolve it. When conflicts are resolved most likely there will be just small conflicts against newest rolling stable (example 5.14.5) and those will be resolved quickly in feature A. Now that bottleneck work in feature A you choose to make new internal kernel. You guys notice that highest as you can go right now is 5.14.3 as there is couple maintainers who are working some issues introduced in 5.14.4. So now 5.14.3 is chosen and testing can start with it.

This way kernel community will be very very happy that you also take stable stuff. Taking 5.14.0 as base make no sense as it gets fixes all the time and then you kinda have to resolve those internal. This is also nice to your maintainers as they can resolve conflicts as they come and some can be aligned all the time with upstream. Usually there will not be many conflict in y.y.x versions. So probably when your conflict resolving work is done you can choose highest stable kernel as internal base.

Moving Google toward the mainline

farnz — Mon, 18 Oct 2021 09:43:27 +0000

Depending on your precise use case for a CPU-only load average, you might want to look at Pressure Stall Information (PSI) as a mainline feature that gets some of what you want.

PSI is a different formulation of the load average concept - instead of looking at total utilization, it looks at how much time is spent with a task blocked completely on a resource. The Facebook PSI microsite has a good explanation of how the different numbers are calculated; basically, though, the `avg` numbers are %age of the time on which either some tasks, or the full set of tasks, are blocked waiting for a given resource to be made available to them. As per the source code comments, for CPU time, full only exists when tasks are restricted from using 100% of available CPU via cgroups, but some is present all the time.

A task is deemed stalled on a resource (CPU, memory, I/O) if the task would run now, but it's waiting for this resource. So, an I/O stall means that the task would be runnable if it wasn't waiting on I/O (whether via blocking I/O calls, or because it's blocked in epoll or the like on an I/O that's not yet completed), while a CPU stall means that the task is runnable, but none of the CPUs it's allowed to run on are idle.

The clever bit is that what's output is stall percentage, not time in which there's a stall; so a value avgXXX=10.0 means that with 10% more of this particular resource (memory, I/O, CPU), there would have been no tasks waiting for resource. This matters when you see a CPU stall of the form "some avg60=10.00"; it means that with 10% more CPU cores, all tasks would have run immediately they were able to; similarly, a CPU stall of the form "full avg300=5.00" means that something limited a cgroup to not use all CPU cores, but if that limit had been raised by 5%, nothing would have waited for a CPU core.

Not the same as a load average, but possibly of use to you in fixing whatever problem you're facing where a CPU-only load average is interesting.

Moving Google toward the mainline

codewiz — Sat, 16 Oct 2021 06:29:22 +0000

Kernel support really depends on the SoC in your device. Each Android release supports a range of kernel versions.

In theory it would be possible to "uprev" the kernel on a particular SoC, but most vendors just don't bother because adding features to discontinued SoCs doesn't help them sell more of their newer SoCs.

Moving Google toward the mainline

SomeOtherGuy — Fri, 15 Oct 2021 21:01:09 +0000

One of the things Google have (I read this in one of their papers) is load averages which don't count processes waiting for IO - I'd love that

I did look at patching it, but load averages are surprisingly complicated - at least in implementation!

(ADVICE WELCOME AND YES I'D LOVE THIS LOAD-AVERAGE - don't question that please)

Moving Embedded toward the mainline

JanC_ — Tue, 12 Oct 2021 08:11:27 +0000

Most hardware is no better than the software when it comes to doing just enough to get something to work (and often the software has to work around the hardware bugs).

Most money is going to marketing, I suppose (and that’s probably necessary to get people to ignore the bad quality & support they get).

Moving Embedded toward the mainline

willy — Sun, 10 Oct 2021 01:21:55 +0000

Exactly. There's plenty of money; they're just choosing not to spend it on software.

Moving Embedded toward the mainline

flussence — Sun, 10 Oct 2021 01:18:28 +0000

Their software divisions are given enough budget to ship a product, not a robust product. You're usually lucky to even get malicious compliance with the GPL2 on one of these things without fighting for it.

Moving Embedded toward the mainline

willy — Fri, 08 Oct 2021 11:58:38 +0000

The embedded world doesn't have a revenue stream? I got my TV for free?

Moving Google toward the mainline

error27 — Fri, 08 Oct 2021 11:23:27 +0000

The link to 2009 was quite interesting to see where the kernel performance challenges were back then. Things like the BadRam patches. I remember that existed but I couldn't say at all what happened to it in the end...

Moving Google toward the mainline

patrick_g — Fri, 08 Oct 2021 06:49:48 +0000

> The Pixel 5, up until a few weeks ago was the newest google headset and it uses 4.19. When it pulls down Android12 it might be slightly newer but I doubt it will even be in the 5 series kernels and that means years old, not months.

I think Android 12 is using a 5.10 kernel.

Moving Google toward the mainline

rahvin — Fri, 08 Oct 2021 05:05:06 +0000

I'm sure you know this but, Depending on version of Android a user could be running kernel 2.xx to 4.xx. Android tends to uses really old kernels and the tech debt porting from 5.xx to 3.xx or even 4.xx is going to be huge. You might be easily able to port from 5.13 to 5.16 (because they are only a few months apart), but tell me it's that easy to go from 5.13 to 4.13 or 3.13.

The Pixel 5, up until a few weeks ago was the newest google headset and it uses 4.19. When it pulls down Android12 it might be slightly newer but I doubt it will even be in the 5 series kernels and that means years old, not months.

Moving Google toward the mainline

zblaxell — Fri, 08 Oct 2021 04:59:27 +0000

One thing we learn very quickly when we try to use software to implement a production service is that there are two kinds of software: software we tested, and software we haven't tested yet. We can say a lot of things about the first kind, like it definitely works for some things and definitely doesn't work for other things, and various behaviors have changed in good and bad ways between versions. We can define boundaries around what we do and don't know about code behavior, and we can assess deployment risks based on empirical data.

All we can say about the second kind of software is that it hasn't been tested on our workload, so we don't know any of those things. Sure, it might have been tested by some group of domain-expert people, and certified by another group of accredited generalist people, and a third group of people with a lot of reddit upvotes swears it's awesome, and some robots didn't notice any of the more common problems--in fact, we'd insist on most or all of that before we bother downloading it for a test build. None of that fancy pedigree matters if we throw our production app on it, and it immediately falls over because we're doing something nobody else does. If we're providing a production service on a commercial scale with the software, it's highly likely we're doing something nobody else does. Even if others start doing what we do, we'd write some new code and be doing something different again. Maybe we're doing something wrong, and our tests (and only our tests) will make the problem visible.

The QA gatekeeper in front of the production server farm has one job: keep the server farm producing at least whatever it is producing now. They can keep running the kernel they already have, so they have no incentive to take risks that might jeopardize that. The gatekeeper will not accept broad assurances of quantity testing--they'll need to be *convinced* to upgrade, with evidence of monotonic improvement in the new versions, or dire and irreparable problems arising in the old versions. "Personally tested by the maintainer and a team of leading experts in the subsystem" is an excellent start, but we'll run our own test suite on it before we call it "tested."

At every node in the integration graph, from developer's laptop to integration tree to merge window to release, LTS, and production deployment, someone is doing testing and deciding whether the code they pulled as input to their node is good enough to push to the output of their node (or in the case of testing robots, snooping on the edges between nodes and advising the node owners). Every node must consider its inputs "effectively untested," or the integration graph doesn't work. That's the whole point of having an integration graph: to combine diverse and isolated pools of domain expertise into a comprehensive testing workflow.

Moving Embedded toward the mainline

mtaht — Thu, 07 Oct 2021 16:27:12 +0000

The embedded world, with the move to extensive offloads, has much more technical debt accumulated, without a revenue stream and developers enabled to pay it down.

Moving Google toward the mainline

bfields — Thu, 07 Oct 2021 15:43:43 +0000

> Upstream is effectively untested.

For what it's worth, as one of the knfsd maintainers, I have a set of NFS-focused test suites (xfstests, connectathon, pynfs, a few smaller tests, run over a variety of NFS protocol versions and security flavors) that I run on anything that I publish. I also run them nightly on the latest trees from Linus, stable, and linux-next, and a few other NFS developers.

I'm by no means the most conscientious. Maintainers do this kind of thing all the time.

I also get pretty regular mail from bots run on mainline and linux-next, and their coverage seems to be improving over time.

"It boots on a dev box and can compile a new kernel" isn't really the current situation.

I mean, I think I'm with you on the basic sentiment, testing is really important and we can and should do better.

Moving Google toward the mainline

jonas.bonn — Thu, 07 Oct 2021 07:01:21 +0000

What value of gc.rerereResolved do people use in practice? The default 60 days doesn't seem sufficient... Or are merges redone often enough that the conflict resolutions get their cached age somehow reset? A bit of extra insight into the practical usage of git rerere for this would be of interest.

Moving Google toward the mainline

tsoni.lwn — Thu, 07 Oct 2021 04:54:27 +0000

Isn't it a nightmare when you have to continuously apply the LTS updates every week on the production kernel which has 9000 patches modifying various core kernel changes? I am sure we will need Engineers just doing the merge conflict resolutions, talking w/ Engineers who wrote these patches, building and testing.

Moving Google toward the mainline

tsoni.lwn — Thu, 07 Oct 2021 04:50:58 +0000

GKI has "tracehooks" approach to make the "core" framework changes working through modules. Lot of vendors using GKI are already doing it including the scheduler and core memory framework changes. There may be very tiny penalty due to tracehooks depending on the path of the code.

Moving Google toward the mainline

pizza — Wed, 06 Oct 2021 22:21:42 +0000

> "It boots on my dev box and can compile a new kernel" is not testing.

Just because the tests they run don't include your particular workloads doesn't mean something isn't tested.

(Over the years, the "can it compile a new kernel" has been a far more useful "test" than most synthetic stress tests..)

Meanwhile, feel free to contribute your own test suite to those working upstream.

Moving Google toward the mainline

Paf — Wed, 06 Oct 2021 22:20:48 +0000

They’re compromising between these two things. They’re not going to hit on the same answer as someone just doing dev.

Moving Google toward the mainline

gps — Wed, 06 Oct 2021 21:56:55 +0000

> """people demanding backports (instead of updating to upstream) realize that they are basically working with a new, untested version of the software anyway"""

This is the Linux Kernel project we're talking about here. Upstream is effectively untested.

Especially when compared to the real world testing needed before mass multi-billion dollar production use.

"It boots on my dev box and can compile a new kernel" is not testing.

Moving Google toward the mainline

pbonzini — Wed, 06 Oct 2021 20:38:37 +0000

It only matters to be close enough. Patches written for 5.13 are easy to forward port to 5.16, and then they will be absorbed in the 5.16 rebase a few months from now.

The team doing the rebases won't even be the same that is doing the upstream contributions.

Moving Google toward the mainline

geert — Wed, 06 Oct 2021 18:08:21 +0000

They also talk about the need to upstream some of their features. To do that efficiently, you have to be on top of the latest development kernel.

Moving Google toward the mainline

Paf — Wed, 06 Oct 2021 17:32:53 +0000

I agree it’s too late to be right for dev, but they’re trying to build a system where they can take this kernel in to production - that’s the one they’re talking about here. And you’d never take this pile you’ve built in to production, even if it’s the right place to do dev.

Moving Google toward the mainline

geert — Wed, 06 Oct 2021 12:25:48 +0000

"At the time of the talk, it was based on the 5.13 kernel, at a time when the 5.15 kernel is in the release-candidate stage. So the project is essentially
one major release behind the mainline." and "The team is working on 5.14 now."

IMHO that's too late: I'm working on v5.16. That is (at the time of the talk) v5.15-rc3 + lots of for-next branches from subsystems I care about.

Moving Google toward the mainline

Sesse — Wed, 06 Oct 2021 10:17:18 +0000

That's not easy for e.g. core memory-management patches.

BTW, I find it amusing that “the prod kernel” now seemingly has the name “Prodkernel”.

Moving Google toward the mainline

taladar — Wed, 06 Oct 2021 09:28:56 +0000

The thing about backports that I always wonder about is if the people demanding backports (instead of updating to upstream) realize that they are basically working with a new, untested version of the software anyway, one created by someone often not terribly familiar with the upstream project.

This might make some sense with central, mature libraries in a stable distro but anything without major reverse dependencies (e.g. almost all binaries or other leaf nodes in the dependency graph) or with a very active development breaking APIs and ABIs all the time should not use the backport model.

Moving Google toward the mainline

tsoni.lwn — Wed, 06 Oct 2021 01:15:50 +0000

How about Google use the GKI model for the Server development as well? Keep all the downstream features as modules and upstream them one by one while keeping the core kernel clean (not so clean w/ hooks but controlled).

Moving Google toward the mainline

ndesaulniers — Tue, 05 Oct 2021 23:05:03 +0000

Forking is a high-interest credit card of tech debt accumulation unless you can 'outrun' upstream.

There are cases where using the credit card make sense, but it's easy to get into a case where you're stuck paying off revolving balance without making a dent in principal.

Getting prodkernel building with LLVM was an exercise in deja-vu; they weren't using branches of stable, so we had to chase backports again that we had already done for Android and CrOS. Makes me wonder how much duplicated effort goes into backports for distros not using stable...