LWN: Comments on "Defragmenting the kernel development process"

Defragmenting the kernel development process

pabs — Fri, 13 Mar 2020 23:42:39 +0000

Fedora retrace is completely unrelated to kerneloops and oops.kernel.org, the latter is vendor agnostic while the former is RedHat/Fedora specific.

Defragmenting the kernel development process

smitty_one_each — Sat, 21 Sep 2019 19:10:23 +0000

It's vaguely comforting that even Really Smart Hackers have as much challenge getting the team synchronized as us mortals.

Defragmenting the kernel development process

dbkm11 — Thu, 19 Sep 2019 23:20:44 +0000

> Torvalds said that lore.kernel.org works so well for him that he is considering unsubscribing from linux-kernel entirely.

Checkout the tool https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/... , it basically fetches mails via lore.kernel.org archives and places them into maildir format for email clients (like mutt et al). This would in fact allow to completely unsubscribe. ;-)

Defragmenting the kernel development process

Cyberax — Thu, 19 Sep 2019 18:20:28 +0000

I remember using an Outlook-To-IMAP proxy to enable mutt, when I was writing a Linux kernel patch.

Defragmenting the kernel development process

jgg — Thu, 19 Sep 2019 12:43:06 +0000

Modern cloud Office365 is perfectly fine as far as deliverability and email content integrity. As long as you don't use Outlook as a client of course.

Where Linux really falls down is that all the usual tools we use for email don't support OAUTH2 - so you often can't actually login to the corp email server anyhow (be it gmail or office365 based). Sigh.

And of course Office365 apparently doesn't support OAUTH for IMAP, only SMTP!

I lost all hope when I saw the Linux team at Microsoft had to setup their own email server.

Defragmenting the kernel development process

neilbrown — Thu, 19 Sep 2019 04:35:40 +0000

> That sounds like my light-weight pull-requests idea

Yes, that's the one.

I think spam is a solvable problem.
If you required the HEAD pushed to always have a signed tag, then you would have a basis for establishing reputation (and you would encourage tag signing, which is a *good* *thing*).

If the signing key doesn't have reputation, the pushed information is limited in some way and it not forwarded to any (public) email lists.
In that case, the author needs to copy some text that was returned by the server into an email message - which can be sent with any old MUA. IT would include a link to find the patch.
I'm not sure how reputation would be gained or revoke, but some combination of automatic flow analysis and crowd-sourcing (if 3 reputable keys vouch for a new key, it gets reputation?)

You wrote that "support in git-daemon is needed" but I think it provides everything you need. You can certainly enable anonymous push, and there are hooks that allow you to check any change to the repo before it happens, so you should be able to prototype something yourself.

Defragmenting the kernel development process

pabs — Thu, 19 Sep 2019 01:53:01 +0000

That sounds like my light-weight pull-requests idea, which was inspired by the fact that some branchable.com sites allow anonymous git pushes.

https://public-inbox.org/git/1486427537.16949.42.camel@bo...

Defragmenting the kernel development process

neilbrown — Wed, 18 Sep 2019 06:12:17 +0000

> At least Exchange (which is probably the most common big corp mail setup) is known to corrupt patches (IIRC, both in- and outcoming).

There is a proposal floating around to have an open git server and allow anyone to submit patches using "git push" instead of "git-send-email".
Exactly how that would work I'm not sure. but one of the things that I do like about gerrit is that submitting patches with "git push" is quite easy.
Maybe that initiative could tip the balance away from gmail.

Defragmenting the kernel development process

meyert — Tue, 17 Sep 2019 20:52:29 +0000

Btw how does retracs correlate to kernel oops? https://retrace.fedoraproject.org/faf/problems/?component...

Defragmenting the kernel development process

broonie — Tue, 17 Sep 2019 11:58:44 +0000

Even with DKIM things are still super unreliable these days - systems will just take a random dislike to mail for reasons that are completely undocumented, and things that look a bit unusual (like technical mail) are especially prone to suffering problems.

Defragmenting the kernel development process

pabs — Tue, 17 Sep 2019 06:35:59 +0000

I had a discussion with the kerneloops developers recently, it sounds like they have been trying to get some interest from the Linux kernel community in the general idea of having access to user oops reports and in kerneloops in particular without any success for years. BTW I've mailed them your question.

Defragmenting the kernel development process

pizza — Mon, 16 Sep 2019 14:47:30 +0000

...Exchange and Outlook have done more to ruin email's utility than everything else combined.

Defragmenting the kernel development process

dezgeg — Mon, 16 Sep 2019 14:08:15 +0000

At least Exchange (which is probably the most common big corp mail setup) is known to corrupt patches (IIRC, both in- and outcoming).

Defragmenting the kernel development process

intgr — Mon, 16 Sep 2019 12:47:09 +0000

> Some years ago, the kerneloops site provided a clear view of where the problem hotspots were

But what happened to kerneloops anyway? I remember setting up the daemon on every one of my machines and servers. Only to discover later that it was defunct. Was it just that nobody cared? Surely if someone cared, the kernel community would be able to find the resources to keep it running?

Defragmenting the kernel development process

idrys — Mon, 16 Sep 2019 11:09:38 +0000

>> A full 90% of linux-kernel subscribers are using Gmail,

> Are they? How sad. You would have thought that we would have learned from the bit-keeper fiasco that depending on non-free tools as a bad idea. It seems not.

I wonder how many of them are using Gmail because of corporate mail setups that suck at dealing with large amounts of mail.

Also, judging from the patches I see sent by Patch Author <author@gmail.com> with the author being Patch Author <author@bigcorp.com>, many corporate mail setups obviously suck at sending stuff as well, and people fall back to Gmail...

Defragmenting the kernel development process

LtWorf — Mon, 16 Sep 2019 09:02:44 +0000

If you don't set dkim up, it's very unlikely for your email to be delivered.

Defragmenting the kernel development process

neilbrown — Sun, 15 Sep 2019 23:53:15 +0000

> None of the tools out there are perfect (far from it), but the majority of them are /far/ better than tracking patches in emails

Are they? I never really had any trouble tracking patches with email(*).
To be a contender to the established method, a new tool must not only be better (which I agree that some are), it must also be no worse. I find them all to be worse (as well as better).

(*) the only real weakness with email is that I would sometimes miss patches. A gentle reminder from the sender after a few weeks of silence always got things moving again and improved the sense of community - some times humans are better than mechanical solutions.

Defragmenting the kernel development process

k3ninho — Sun, 15 Sep 2019 15:37:58 +0000

> Which particular feature(s) unique to gmail are its users actually dependent on?

I can't speak to _unique_ but the "feature" is not rejecting as spam the stuff you've sent from your own mail server. The arc of this development can be read at ex-Symbolics Lisp/ex-Netscape hacker Jamie Zawinski's blog -- https://www.jwz.org/blog/tag/mail/ -- and shows that running your own email servers and communicating widely is hampered by proprietorial relays including Gmail, Microsoft, GoDaddy, Earthlink, Dreamhost and more. I get that spam has broken e-mail and that the paradox of allowing people to post to the SMTP-based social network means that you have to filter later if the posts made were unwanted or malicious.

Kernel development is a social network with some patch transport and discussion in messages and concrete source trees backing these up. Metadata like 'acked-by' and which systems tested the changes or how to replicate found bugs, that stuff needs to annotate the patches and be searchable, then reputations need to be used to retain a high signal-to-noise ratio. And because it's a social network, where Linus (among others) goes, people will follow.

K3n.

Defragmenting the kernel development process

cjwatson — Sun, 15 Sep 2019 12:04:48 +0000

Right, the pseudoheader requirement is a cheap way to prevent most spam in the form of new bugs, but does nothing to prevent spam to existing bugs. For that we have little choice but to muddle along with traditional spam-detection tools and after-the-fact removal. Unfortunately there would likely be an outcry if we started requiring accounts.

(Systems that require accounts aren't immune to spam by any means, but having messages more systematically linked to identities does make some things a lot easier.)

Defragmenting the kernel development process

pabs — Sun, 15 Sep 2019 04:03:19 +0000

> cross-package bugs

Not sure what you mean by that, but with debbugs there are a few things that seem related to what you said:

Bugs can be in package foo but marked as affecting package bar.

You can assign a bug to multiple packages when a change in any single package can fix the issue.

You can mark a bug as being blocked by another bug.

You can clone a bug and then reassign the clone, updating the blocked info at the same time.

The usertags stuff seems relevant too.

Defragmenting the kernel development process

roc — Sun, 15 Sep 2019 02:52:05 +0000

The logic often seems to be "The Linux kernel is a tremendously successful project, so the way Linux kernel development is organized must already be excellent." And of course everyone likes to believe that the way we are used to doing things does not need to change.

Defragmenting the kernel development process

lsl — Sun, 15 Sep 2019 02:24:29 +0000

I guess the possible payoff is just not big enough for spammers to adapt their programs to emit the appropriate pseudo headers that make it a valid bug report.

Defragmenting the kernel development process

marcH — Sun, 15 Sep 2019 02:08:10 +0000

Which particular feature(s) unique to gmail are its users actually dependent on?

Defragmenting the kernel development process

neilbrown — Sun, 15 Sep 2019 01:51:41 +0000

> A full 90% of linux-kernel subscribers are using Gmail,

Are they? How sad. You would have thought that we would have learned from the bit-keeper fiasco that depending on non-free tools as a bad idea. It seems not.

Defragmenting the kernel development process

marcH — Sat, 14 Sep 2019 22:36:39 +0000

> As the discussion wound down, Abbott suggested that what is needed is a kernel DevOps team populated with developers who are good at creating that sort of infrastructure.

That's the core issue. Tools and design choices will keep people bikesh... fascinated and busy discussing for years on end but without some actual and skilled manpower these discussions will forever keep going around in circles.

Speaking of manpower, so-called "DevOps" and validation roles have since forever been less respected/funded/promoted than nobler "developer" roles and the Linux kernel is not much worse than the rest of the industry. Maybe it's changing? Very slowly as any culture change. I think we're starting to see conferences on these topics, did any mention the kernel?

> The first session at the 2019 Linux Kernel Maintainers Summit was a *last-minute addition to the schedule*.

Emphasis mine :-)

Defragmenting the kernel development process

marcH — Sat, 14 Sep 2019 22:23:05 +0000

> no messing about with registering an account just to report a single minor bug.

That's the essence of spam: I wonder how it's prevented in this particular case.

Defragmenting the kernel development process

marcH — Sat, 14 Sep 2019 22:20:57 +0000

> We have many of the necessary pieces to do better, but there is also a lot of fragmentation. Every kernel subsystem does things differently; there is a distinct lack of identity for the kernel as a whole.
> ...
> There are at least seven major testing systems out there (he listed 0day, kernelci.org, CKI, LKFT, ktest, syzbot, and kerneltests) when we should just have one good system.

Seven is too many. One would be too little. Not just to keep some competition alive but simply because the kernel is a really huge project and one size doesn't fit all.

If you think test suites for Wifi, filesystems, graphics and schedulers can re-use validation tools and test code then... it's likely these common things are not even specific to the Linux kernel at all.

Same for bug trackers: re-use is always good but a little diversity and competition doesn't hurt either and again a Linux Wifi bug and a Linux filesystem bug don't really seem much closer to each other than to a bug in openssh or whatever. Granted: they can be tracked on the same git history. Who cares, I'm not even using that filesystem anyway.

Even if all areas of the kernel ever use the exact same tools and processes one day, they should still run different _instances_ to keep things under reasonable and manageable size. Who still reads the lkml? People have been using more focused mailing lists for ages.

Defragmenting the kernel development process

marcH — Sat, 14 Sep 2019 22:04:21 +0000

> None of the tools out there are perfect (far from it), but the majority of them are /far/ better than tracking patches in emails. It amazes me that all this is controversial and would love to understand the reasons behind it better.

Good news, you came to best place: https://www.google.com/search?q=site%3Alwn.net+email

Defragmenting the kernel development process

spwhitton — Sat, 14 Sep 2019 19:20:17 +0000

Well, sourcehut is still in alpha. But something that makes adoption easier than some other platforms is how modular the different sourcehut components are. You can just use what you like.

Defragmenting the kernel development process

wiktor — Sat, 14 Sep 2019 19:08:34 +0000

I'm also interested in whether sourcehut.org is being considered or not (maybe due to sheer size of the LKML).

Another tool that provides data format first and uses git itself as a storage for code review is git-appraise: https://github.com/google/git-appraise

Defragmenting the kernel development process

mpr22 — Sat, 14 Sep 2019 17:27:52 +0000

As a bug submitter, on the other hand, the Debian BTS is my absolute favourite - no messing about with registering an account just to report a single minor bug.

Defragmenting the kernel development process

paravoid — Sat, 14 Sep 2019 17:17:10 +0000

> Hellwig put in a good word for the Debian bug-tracking system, which allows most things to be done using email.

...what? debbugs is a very old piece of software, and it shows. It's not scalable (ironically, bug pages for packages like Linux are among the ones that suffer the most), lacks even relatively simple features (cross-package bugs), lacks any integration with any other tooling, and is confusing to interact with for even the most experienced developers. The fact that changes happen through a custom command language, cannot be previewed before acted upon, are not realtime (you have to wait for your email to be processed) and often take a while to take full effect (due to caching) adds to the confusion. IIRC there were (heroic) attempts in the past to add a... SOAP interface to it but I don't think it's being used much (I may be wrong).

I've been using the Debian BTS for 15 years and I often still struggle. It's probably one of the biggest demotivators I have while working in Debian. That's not a criticism to its maintainers - it was probably great in the 90s or early 00s and has improved a lot in the past few years, but it still pales in comparison to the tools that exist out there today (Phabricator/GitHub/GitLab etc.). Plus it really is a bug tracking system, not a patch management system, so it feels entirely off-topic to the discussion at hand (and my understanding is that Linux has Bugzilla for a BTS?). Patch management-wise, Debian never used its BTS for anything but drive-by small patch submission attached to emails. Most of us actually switched to a Debian-hosted version of GitLab for repository hosting, merge request management & some (limited) CI recently and it has been an exciting journey.

Beyond that, it has always puzzled me how the Linux kernel community -the community that invented and popularized git!- has remained so far behind in its tooling, and stuck in old ways. Bug tracking, code review, patch management/tracking, and CI all seem intertwined with each other and really require unified tooling to manage adequately, for seasoned and new developers alike. None of the tools out there are perfect (far from it), but the majority of them are /far/ better than tracking patches in emails. It amazes me that all this is controversial and would love to understand the reasons behind it better.

Defragmenting the kernel development process

spwhitton — Sat, 14 Sep 2019 16:04:33 +0000

The discussion of using git as a transport layer reminds one of the kind of work Joey Hess has done in the past decade, on tools like git-annex and propellor.

A lot of the challenges discussed in this article are being thought about by people working on <https://sourcehut.org/>. In particular, CI and series tracking for e-mail based workflows.

Defragmenting the kernel development process

pbonzini — Sat, 14 Sep 2019 09:43:08 +0000

> If patchwork could put multiple versions of a patch series into a Git repository, Ts'o said, it would enable a lot of interesting functionality, such as showing the differences between versions.

Let me introduce Patchew! Patchew was started when patchwork seemed to be mostly dead, so it is somewhat similar to Patchwork 2.0. But it has some nice functionality such as version comparison, pushing each submitted series to git (complete with Reviewed-by tags and the like), simple integration with testing and a REST API. It should also be quite easy to write new plugins to automatically parse syzbot or 0day emails and turn them into test failures.

Defragmenting the kernel development process

cyphar — Sat, 14 Sep 2019 09:34:05 +0000

I wonder if this could be taken as an opportunity to start looking at whether [1] is a viable proposal to solve the problems in our workflows.

[1]: https://lwn.net/Articles/793037/