|
|
Log in / Subscribe / Register

Better tools for kernel developers

By Jonathan Corbet
February 6, 2020
By many accounts, the kernel project uses outdated tooling, far behind the state of the art that Kids Today tend to favor. The kernel's workflow has worked well (enough) for years, but there are signs that it may not be sustainable indefinitely. As a result, there has been an ongoing conversation about improving the kernel's workflow, but little has changed so far. The posting of a simple tool called get-lore-mbox is a sign that the rate of change may be about to increase.

The kernel project's reliance on email strikes many as quaint and antiquated. It may indeed partly be a natural outcome of the aging nature of the kernel community; many of the developers there, especially in the important maintainer positions, got started well before tools like web-based Git forges existed. Indeed, some of them got started using punch cards and may still be unconvinced of the virtues of, say, text editors. But the truth of the matter is that there are a number of good reasons for the kernel community's continued reliance on email; there is little else that can handle a community of that size and diversity.

So, while it seems that the future of email (as opposed to, say, proprietary services like Gmail) is uncertain at best, the path toward a replacement in the kernel community is unclear. Developers will have to be convinced that any new tools will make their lives better, not worse; busy maintainers have little patience for "improvements" that slow things down.

The get-lore-mbox tool, posted by Konstantin Ryabitsev is not the product of a vast development effort; it adds up to about 500 lines of Python code. Its core function is simple: given the message ID for any email in a thread of interest, it will download the entire thread from the lore.kernel.org archive into a local mbox file. That alone can be useful for anybody who wants to catch up on an email discussion that may have flown by, but there are some other useful features:

  • With the -a option, only the actual patch series at the heart of the thread will be downloaded; it will be sorted and stored in a form suitable for passing directly to git am. Any useful tags posted as replies (Acked-by, for example) will be automatically added to the relevant patches.
  • The -t option will cause tags posted in replies to the series cover letter to be applied to all patches in the series.
  • If the thread contains multiple versions of a patch series, the latest one will be stored; the -v option can be used to request a specific version if need be.
  • If no message ID is provided, it will read a message from the standard input and extract the ID from there, making single-keystroke operation possible in a number of mail readers.

Kernel maintainers tend to spend a lot of time reading through email threads, organizing the patches found therein, applying tags, etc. The initial response to this tool (including on the private kernel.org users list) has made it clear that they have been wanting something like this for a long time. The developer time that will be saved far outweighs the time it takes to write a gadget like this, but nobody has ever created such a thing.

One might well wonder why this tool didn't appear until now. It would seem that there were two things that needed to happen before this work could be done. One of those was the creation of a reliable archive for kernel mailing lists that could be easily queried by a program. Reliable archives have been scarce in general, and a single archive for all kernel-related lists didn't exist at all; without that, there is nothing for a tool like get-lore-mbox to obtain messages from. The lore.kernel.org archive has fixed that problem; it now seems like an indispensable part of the kernel workflow, but it's worth remembering that it is less than two years old.

Back in 1975, Fredrick Brooks wrote in The Mythical Man-Month that an effective programming team should be organized like a surgical team, with a set of people filling specific roles. The kernel certainly has its surgeon at the top, complete with many of the traits that such professionals are known to display. Brooks said that every team should have two secretaries; the kernel community seems to be able to get by without those. But he also said that each team needs a toolsmith, focused on creating the tools that the team needs.

The kernel project has long lacked a toolsmith. It doesn't even have somebody who can collect the tools created by maintainers for their own needs and make them suitable for more general use. As a result, maintainers often do without, or they use tools they threw together themselves when they really needed to be doing something else; that causes the community to lose a lot of opportunities for better productivity overall.

One of the open secrets in the free-software world is that, while some areas of development are well supported by companies, others tend to languish. Both within the kernel and beyond it, there are many pressing needs that, it seems, are nobody's problem, so nobody is funded to work on them. The costs of this neglect are many: burned-out developers trying to keep projects going in their spare time, security vulnerabilities, missing tests and documentation, and a lack of tools needed by the community as a whole, among others.

If only there were a successful industry consortium that could focus some attention (and resources) on problems of this nature. For the kernel there is one, in the form of the Linux Foundation, which has indeed supported the kernel community in a number of ways, including employing a few key developers and taking over maintenance of the kernel.org system after the 2011 compromise shone a spotlight on the consequences of neglect there. More recently, the Linux Foundation has supported the creation and operation of lore.kernel.org — and the development of get-lore-mbox as well.

The kernel workflow discussions have made it clear, though, that there is a lot more work to be done in this area. The kernel's processes and tools have suffered from a lack of attention for many years; that is partly a result of the fact that, on the surface, things are working smoothly. The kernel community has managed to organize itself well and functions at a level that few other projects can match. When the kernel community has found the time to focus on tools — Git being the primary example — the effects have rippled across the development community as a whole. But reaching that point has required going through a number of crises when the kernel project's processes proved unable to support the community.

It would be a great thing if the next big transition in the kernel's development process could happen without the "crisis" part. If sufficient effort and resources go into the tools that the kernel community will need in the coming years, that might just be possible. That suggests that the work being done at the Linux Foundation should not just continue, but grow into something bigger. Supporting this work (which requires getting its member companies to support it) is one of the best things that the Linux Foundation can do to support the kernel project as a whole.

Index entries for this article
KernelDevelopment tools

to post comments

Better tools for kernel developers

Posted Feb 6, 2020 21:56 UTC (Thu) by milesrout (subscriber, #126894) [Link] (113 responses)

It's getting pretty tiring to hear constantly how terrible the Linux kernel development workflow is. For such a 'terrible' workflow it sure seems to have produced excellent results. Maybe it's not so terrible after all? Maybe, just maybe, it's actually a GOOD process and we should learn from it rather than dismiss it because it doesn't use 'hip' proprietary Microsoft-owned-and-controlled web interfaces. It seems to me that rather than saying 'what can the Linux kernel developers learn from Javascript kids on GitHub' we should be asking 'what can the Javascript kids on GitHub learn from Linux kernel development'. That's a question with a lot more answers that are a lot more interesting:

1. You don't need a 'web interface' to use version control.
2. You don't need a centralised repository for code hosted on a website.
3. You don't need inaccessible proprietary software to discuss things, because email works very well for discussing things already.

get-lore-mbox seems like a wonderful tool and ironically is only possible because the Linux kernel development process is done in the open through a standardised protocol. Projects that organise through GitHub, like the Rust project, have had an infamously terrible time trying to get the GitHub developers to give them any tools for scaling up their organisational efforts to big projects.

> One of the open secrets in the free-software world is that, while some areas of development are well supported by companies, others tend to languish. Both within the kernel and beyond it, there are many pressing needs that, it seems, are nobody's problem, so nobody is funded to work on them. The costs of this neglect are many: burned-out developers trying to keep projects going in their spare time, security vulnerabilities, missing tests and documentation, and a lack of tools needed by the community as a whole, among others.

Free software doesn't come from the corporate world and doesn't need the corporate world. People don't need to be 'funded' to work on free software. Not every piece of software needs corporate backing, a foundation, and corporate interference. Every decent piece of free software came from an single individual person working on something that they found interesting or that was personally useful to them.

Better tools for kernel developers

Posted Feb 6, 2020 22:06 UTC (Thu) by pizza (subscriber, #46) [Link] (2 responses)

> It seems to me that rather than saying 'what can the Linux kernel developers learn from Javascript kids on GitHub' we should be asking 'what can the Javascript kids on GitHub learn from Linux kernel development'

1) Use Good Tools
2) If the Tools Suck, Build Better Tools
3) Goto 1

(I find it interesting that most complaints about email-based workflows basically stem from folks insisting on using crappy email clients...)

Better tools for kernel developers

Posted Feb 7, 2020 0:23 UTC (Fri) by newren (guest, #5160) [Link] (1 responses)

> (I find it interesting that most complaints about email-based workflows basically stem from folks insisting on using crappy email clients...)

That may well be. I happen to prefer the email-based workflow for patch review, mostly because the GUI code review tools I know about were all designed wrong. But I believe a better one could be made that would improve upon email-based workflows (at least for me).

Elsewhere in this thread people are discussing `git send-email` as one reason why, so let me just bring up a different issue that folks tend to be unaware of or overlook: lack of context.

Patches, by their nature, have limited context. This can result in a few problems:
* It is not clear on which commit the patches should be applied on top of. People use various strategies to avoid this problem ("applies on top of next", "meant to build on a merge of en/fill-directory-fixes-more and ds/sparse-checkout-harden", etc.), but it can be messed up and cause problems. It has before.
* The limited context can result in patches applying "cleanly" but incorrectly. I'm not talking about intrinsic problems like semantic conflicts, either, but stuff that is specific to patch-based workflows such as hunks being added to the wrong function in a file, or files placed in the wrong directory -- see [1], [2], and [3] for examples. In fact, these problems are causing git to switch the default backend of git-rebase from the am-backend to the interactive/merge one, with likely deprecation of the am-backend to eventually follow[4].

[1] https://lore.kernel.org/git/CABPp-BGiu2nVMQY_t-rnFR5GQUz_...
[2] https://lore.kernel.org/git/20200108223557.GE32750@szeder...
[3] https://lore.kernel.org/git/xmqqh8jeh1id.fsf@gitster-ct.c...
[4] https://lore.kernel.org/git/pull.679.git.git.1576861788.g...

Better tools for kernel developers

Posted Dec 27, 2020 9:36 UTC (Sun) by Wol (subscriber, #4433) [Link]

Dunno whether it applies elsewhere, but I believe Linus' workflow is that all patches are against the first version in the series, so if 5.12 is the latest kernel and he's prepping 5.13, they all have to apply cleanly against 5.12.

Then I guess once 5.13-rc1 comes out, they all have to apply cleanly against rc1, even if he's up to rc8.

Not proof against cock-ups, but I think interactive-merge would actually be a change for the worse in that case.

Cheers,
Wol

Better tools for kernel developers

Posted Feb 6, 2020 22:23 UTC (Thu) by justincormack (subscriber, #70439) [Link]

It is good because it er, puts email on the web with an API. Everything is better with an API, but email has never had a good universal one.

Better tools for kernel developers

Posted Feb 6, 2020 22:25 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (101 responses)

> It's getting pretty tiring to hear constantly how terrible the Linux kernel development workflow is. For such a 'terrible' workflow it sure seems to have produced excellent results.
It is terrible, especially for newcomers or people who want to contribute just a small patch.

Better tools for kernel developers

Posted Feb 6, 2020 22:29 UTC (Thu) by milesrout (subscriber, #126894) [Link] (100 responses)

Contributing just a small patch is completely trivial! You don't need to sign up for anything, you don't need to register on some proprietary website, you don't have to run proprietary Javascript and give your personal details to Microsoft. You just email a patch using git's *built in* patch-formatting tools.

Better tools for kernel developers

Posted Feb 6, 2020 22:39 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (58 responses)

> You just email a patch using git's *built in* patch-formatting tools.
The last time I had to:
1) Get special permission from IT to use IMAP for mail instead of regular Exchange protocol.
2) Install imapproxy.
3) Install mutt.
4) Set up this whole shebang (which was NOT trivial at all).
5) Finally be able to use git's mail capability (after misusing it once and spamming the LKML with an incomplete patch series).
6) Learn how to use mutt to reply to emails with code reviews (newsflash: mutt sucks).

Sorry, this is not trivial and is extremely convoluted compared to creating PRs on Github (and I've done quite a bit of drive-by contributions using it).

Better tools for kernel developers

Posted Feb 6, 2020 22:47 UTC (Thu) by milesrout (subscriber, #126894) [Link] (50 responses)

It's not the rest of the world's fault that you're stuck using some terrible Micro$oft crap instead of standard email. So let's remove the first four of those, because they're entirely a you problem of you not already having the ability to *send email*, a pretty basic task in today's world. And we'll remove the last of them, because 'learn how to reply to emails' is also a pretty basic task you should also be capable of. There are loads of mail clients out there, nobody is forcing you to use mutt (which doesn't suck).

The process is thus:

Step 1. Learn how to use git's mail capability, for which there are countless articles, wiki pages, tutorials, man pages, etc.
Step 2. There is no step 2, you have completed the process.

It's extremely simple and easy.

Better tools for kernel developers

Posted Feb 6, 2020 23:48 UTC (Thu) by newren (guest, #5160) [Link] (1 responses)

> It's not the rest of the world's fault that you're stuck using some terrible Micro$oft crap instead of standard email. So let's remove the first four
> of those, because they're entirely a you problem of you not already having the ability to *send email*, a pretty basic task in today's world. And
> we'll remove the last of them, because 'learn how to reply to emails' is also a pretty basic task you should also be capable of. There are loads of
> mail clients out there, nobody is forcing you to use mutt (which doesn't suck).

There's a big difference between "sending email" and "sending email in a way that doesn't corrupt patches". Also, Microsoft tools aren't the only one that struggle there, as evidenced by the MUA-specific hints in the git documentation.

> The process is thus:
> Step 1. Learn how to use git's mail capability, for which there are countless articles, wiki pages, tutorials, man pages, etc.

Sometimes people know how to use git's mail capability, use it fine for years, and then something outside of git breaks. For one example, see https://lore.kernel.org/git/CAP8UFD1hTc0dXWmiF6aW6=_7DhB8.... I too struggled with that same problem and remember spending *an entire day* attempting to figure out how to send my patches to git (and it wasn't the only time). I may not be the sharpest tool in the drawer, but I've had hundreds of patches accepted into git.git so if I'm struggling and can point to a thread where other git developers couldn't figure it out themselves and were asking the list, perhaps it's not as trivial as you make it seem.

> Step 2. There is no step 2, you have completed the process.

> It's extremely simple and easy.

I have to disagree. I still prefer the email process (see my three links on why I hate GitHub PRs and most gui code review tools in general[1]) and I think your point earlier in the thread that other projects should learn some good things from kernel development is spot on, but I certainly don't believe it's perfect or even above criticism.

Better tools for kernel developers

Posted Feb 6, 2020 23:51 UTC (Thu) by newren (guest, #5160) [Link]

>I have to disagree. I still prefer the email process (see my three links on why I hate GitHub PRs and most gui code review tools in general[1])

Oops, forgot to include the footnote link; here it is:

[1] https://github.com/newren/git-filter-repo/blob/master/Doc...

Better tools for kernel developers

Posted Feb 7, 2020 0:25 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (34 responses)

> It's not the rest of the world's fault that you're stuck using some terrible Micro$oft crap instead of standard email.
Outlook works perfectly fine for me. It does all I need from it: e-mail, contact management and discovery, shared calendars and meetings and so on. I have no problems with it, it just works.

And my setup is bog-standard, I have nothing whatsoever unusual. If Linux development can't be used with this setup then perhaps it should re-examine itself?

> And we'll remove the last of them, because 'learn how to reply to emails' is also a pretty basic task you should also be capable of.
The style of email communication used in LKML is not easy to replicate in Outlook or other mailers.

> It's extremely simple and easy.
Compared to creating a PR on a git forge? No, it's not.

Better tools for kernel developers

Posted Feb 7, 2020 1:01 UTC (Fri) by pizza (subscriber, #46) [Link] (7 responses)

> Outlook works perfectly fine for me. It does all I need from it: e-mail, contact management and discovery, shared calendars and meetings and so on. I have no problems with it, it just works.

So, why are you bothering with this Linux thing anyway, when Windows is "bog-standard" and works perfectly fine?

(Personally, I consider Outlook and the pathological behaviours it encourages to be not unlike the poorly-thought-out slide decks that led to the Challenger disaster.)

Better tools for kernel developers

Posted Feb 7, 2020 1:09 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> So, why are you bothering with this Linux thing anyway, when Windows is "bog-standard" and works perfectly fine?
Mostly because Linux is standard on the servers. And I'm using Outlook on Mac OS X, btw.

Also, there's very little relation between the Linux kernel and the outdated mail tools used to develop it.

> (Personally, I consider Outlook and the pathological behaviours it encourages to be not unlike the poorly-thought-out slide decks that led to the Challenger disaster.)
Or perhaps the Debian Disaster with removing out the entropy source because there was no code context visible (unlike in PRs in git forges)?

Better tools for kernel developers

Posted Feb 7, 2020 15:25 UTC (Fri) by ballombe (subscriber, #9523) [Link] (4 responses)

s/Debian/OpenSSL/ please.

Better tools for kernel developers

Posted Feb 16, 2020 4:10 UTC (Sun) by Conan_Kudo (subscriber, #103240) [Link] (3 responses)

Well, Debian did that change, it was never present in upstream OpenSSL.

Better tools for kernel developers

Posted Feb 17, 2020 11:04 UTC (Mon) by anselm (subscriber, #2796) [Link] (2 responses)

Yes, but IIRC the guy asked on the OpenSSL developers' list whether it would be an OK thing to do and never got an answer.

Better tools for kernel developers

Posted Feb 18, 2020 2:20 UTC (Tue) by andresfreund (subscriber, #69562) [Link] (1 responses)

> Yes, but IIRC the guy asked on the OpenSSL developers' list whether it would be an OK thing to do and never got an answer.

And it was flagged as depending on uninitialized data by valgrind IIRC. Without comments explaining why there's no danger etc.

Better tools for kernel developers

Posted Dec 24, 2020 4:26 UTC (Thu) by ncm (guest, #165) [Link]

And, also, it was in Debian's packaging of OpenSSH, not SSL.

Better tools for kernel developers

Posted Feb 7, 2020 11:09 UTC (Fri) by excors (subscriber, #95769) [Link]

> So, why are you bothering with this Linux thing anyway, when Windows is "bog-standard" and works perfectly fine?

Linux is often used for embedded devices (via Android, Yocto, etc), where everything is cross-compiled and there's no need for the development environment to be Linux. E.g. you can build Android on macOS, and I suspect you can build it on Windows with WSL, so you can do serious Linux kernel development work without having a Linux desktop.

Better tools for kernel developers

Posted Feb 7, 2020 9:21 UTC (Fri) by dgm (subscriber, #49227) [Link] (20 responses)

> Outlook works perfectly fine for me.
Same here. My Instagram app does everything I want from a communication tool. Why those old Linux farts refuse to work with it just escapes me.
/sarcasm

Better tools for kernel developers

Posted Feb 7, 2020 9:26 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (19 responses)

What exactly is _wrong_ with Outlook? Please note, that "not behaving like a 1980-s mail client" is not a wrong behavior.

Better tools for kernel developers

Posted Feb 7, 2020 9:35 UTC (Fri) by milesrout (subscriber, #126894) [Link] (5 responses)

I would have thought it would be rather obvious to anyone that has ever USED outlook what is wrong with it: absolutely everything.

In particular, though, your argument seems to be 'Outlook is fine because it has every feature you could possibly need' while also simultaneously claiming that the Linux kernel development process is broken because Outlook doesn't support it properly. That seems to suggest that Outlook doesn't have every feature you could possibly need...

Better tools for kernel developers

Posted Feb 7, 2020 9:41 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

"Everything wrong" is not an answer.

> In particular, though, your argument seems to be 'Outlook is fine because it has every feature you could possibly need'
Nope. I'm saying that Outlook has features that I need (and other average corporate email users), not every feature under the sun. And it seems like it's kernel developers that need some strange behaviors from mail.

Better tools for kernel developers

Posted Feb 7, 2020 9:47 UTC (Fri) by milesrout (subscriber, #126894) [Link] (1 responses)

> "Everything wrong" is not an answer.

I have never identified a design decision in Outlook that I like. I hate the default shortcuts, I hate how the buttons are laid out, I hate how the panes are laid out, I hate how the windows are laid out, I hate their horrible excuse for an implementation of standardised email protocols... everything about it is terrible.

> Nope. I'm saying that Outlook has features that I need (and other average corporate email users), not every feature under the sun. And it seems like it's kernel developers that need some strange behaviors from mail.

The oh-so-strange feature of sending plain text email?

Better tools for kernel developers

Posted Feb 8, 2020 12:46 UTC (Sat) by gfernandes (subscriber, #119910) [Link]

>>The oh-so-strange feature of sending plain text email?

... that works perfectly fine in Outlook?

Sorry, but you've still not addressed the very valid question of what exactly is wrong.

Not usable with the Linux kernel workflow could equally be a problem with the workflow and/or the eccentricities of the mail exchange being mandated?

A PR is trivial in comparison to a patch with no context.

Better tools for kernel developers

Posted Feb 11, 2020 22:51 UTC (Tue) by poruid (subscriber, #15924) [Link] (1 responses)

This makes me rant.

Besides the usual nonsensical way that options, account setting or any entity are organised in MS Outlook (e.g. navigate from Advanced to Extra to whatever deeper and then end up again in the dialog the navigation started from, ESC deep down a dialog stack just ending the whole stack), its all-functionality-in-one-application is hell bound: load and startup times are terrible, memory usage is huge, etc etc. Having separate tools that do one thing good and work well together - sounds familiar ? - is much better. The balancing of tradeoffs at Microsoft its development process have very little to do with user demands and much more with squeezing out profits and locking customers in.

Better tools for kernel developers

Posted Feb 11, 2020 23:26 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

FWIW, I have a fairly large Outlook mailbox and the Outlook process private memory size is 180Mb. And this includes all the GUI stuff like rich text editor. I don't consider this even remotely problematic...

Better tools for kernel developers

Posted Feb 7, 2020 12:44 UTC (Fri) by pizza (subscriber, #46) [Link]

Outlook is a Microsoft Exchange (as opposed to "email") client, requiring you to go all-in on Microsoft infrastructure on the backend if you're going to use even a fraction of its feature set. It also requires use of non-Free operating systems.

Both of those make Outlook a complete non-starter before we even get to feature sets.

I'm in this for the Four Freedoms, and I put my money where my mouth is.

Better tools for kernel developers

Posted Feb 7, 2020 13:47 UTC (Fri) by tao (subscriber, #17563) [Link]

How about not being able to send patches without mangling them?

Better tools for kernel developers

Posted Feb 7, 2020 20:44 UTC (Fri) by neilbrown (subscriber, #359) [Link] (10 responses)

> What exactly is _wrong_ with Outlook?

I've never used Outlook, so this is second hand but it appear to not support any straight forward method to reply separately to various points in an email.

When people using Outlook have found the need to do this when replying to me they sometimes mark their comments in blue. Sometimes put [MYNAME] near them. Sometimes just interleave with my original text so I have to remember what I wrote so I can filter it out.
In short, it hard to have a detailed multi-thread conversation with Outlook.

Now imaging trying to do code review using Outlook - code review often requires multiple separate statements, and often needs each to be clearly tied to a specific line of code.

I'm using gerrit in project at present and once when I was grumbling about it at a conference, someone said they really like it because it allows you to comment on individual lines. I thought "so what, I've been doing that with email for decades". I subsequently discovered they used Outlook. Enough said.

Better tools for kernel developers

Posted Feb 7, 2020 20:53 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

> I've never used Outlook, so this is second hand but it appear to not support any straight forward method to reply separately to various points in an email.
Outlook defaults to top-posting, but it supports traditional point-by-point replies as well. It also has tools to quote and unquote blocks of text. Most people are just not exposed to them.

> Now imaging trying to do code review using Outlook - code review often requires multiple separate statements, and often needs each to be clearly tied to a specific line of code.
Indeed, Outlook is not the best tool for it. Additionally, Outlook defaults to a proportional font which is nicer to read than monospace fonts. But it makes it a poor tool to edit or read code.

Better tools for kernel developers

Posted Feb 7, 2020 22:16 UTC (Fri) by neilbrown (subscriber, #359) [Link] (4 responses)

> Outlook defaults to top-posting, but it supports traditional point-by-point replies as well.

I didn't know that - thanks. How easy is it to enable? Are there simple instructions I can point someone to? "Please do *this* whenever you reply to me".. (It would need to work for the web-base outlook as well as the desktop app .... oh, and the phone app and ....)

> Most people are just not exposed to them.

Maybe that simple fact is a big flaw in Outlock. There is little point having functionality that people cannot find and don't even know to look for. I know UI design is hard, but email is so ubiquitous that it really needs the work to be put in.

Thanks.

Better tools for kernel developers

Posted Feb 7, 2020 22:53 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

How easy is it to enable? Are there simple instructions I can point someone to? "Please do *this* whenever you reply to me"

I was able to find these instructions on Microsoft's website, though these are how to set the default rather than how to do it for an individual message.

Maybe that simple fact is a big flaw in Outlock. There is little point having functionality that people cannot find and don't even know to look for.

Older, plain text email clients defaulted to prefixing every line with a quote character because that was the only practical way of quoting a message. When newer clients started to support fancier formatting, that approach didn't work the same, so they used HTML-type quoting. Unfortunately, they don't seem to have come up with an easy way to intersperse quotes and replies, so people defaulted to top posting. That said, even with old-fashioned plain text quoting, there were plenty of people who top posted because it was faster than quote/reply formatting.

Better tools for kernel developers

Posted Feb 8, 2020 0:19 UTC (Sat) by neilbrown (subscriber, #359) [Link] (1 responses)

> though these are how to set the default rather than how to do it for an individual message.

Thanks for the link. I was afraid it might only be settable globally. This related directly to my point below...

> That said, even with old-fashioned plain text quoting, there were plenty of people who top posted because it was faster than quote/reply formatting.

That's true and it doesn't bother me at all - providing they manage to communicate clearly.
I really don't care about the format, but I would like people to think about the message that they want to send, and then to create the message accordingly.
This means they need the tools available to create a useful message. Sometimes top-posting is a perfectly good way to send a message, sometimes interleaving is best. In the cases that I can think of that particularly bothered me, my correspondent clearly *was* trying to communicate usefully, but were impeded by their tool.
Having the tools available needs them available on a message-by-message basis, not as a global setting.

Having quote-with-prefix/quote-without-prefix/don't-quote as a global setting is a bit like having reply-to-sender/reply-to-all/forward be a global setting. (or html/plain-text being a global setting!!!)

Better tools for kernel developers

Posted Mar 3, 2020 17:09 UTC (Tue) by nye (guest, #51576) [Link]

If you set it to "prefix each line", the precise behaviour depends on whether the original message is plain text or HTML, so you don't get the > prefixing for HTML emails (instead you get a coloured border on the side, which is inoffensive and actually quite useful). It's not quite as good as if it let you set this per message, but in practice the way it works seems basically sane and actually a fairly reasonable approach.

Also, when replying to a plain text email, it defaults to plain text so you don't need to change anything.

I don't really want to defend Outlook too much since I kind of hate it, mostly for reasons not covered in this thread, but in this particular way I think its reputation is a bit unfair.

Better tools for kernel developers

Posted Feb 8, 2020 1:20 UTC (Sat) by pizza (subscriber, #46) [Link]

> (It would need to work for the web-base outlook as well as the desktop app .... oh, and the phone app and ....)

The web version of outlook does not support sane quoting. I believe the phone apps are similarly feature-limited.

Better tools for kernel developers

Posted Feb 13, 2020 2:31 UTC (Thu) by chutzpah (subscriber, #39595) [Link] (3 responses)

I have recently had Outlook 365 thrust upon me, configured in such a way that all alternate clients are blocked.

> Outlook defaults to top-posting, but it supports traditional point-by-point replies as well. It also has tools to quote and unquote blocks of text. Most people are just not exposed to them.
I have yet to discover any way to configure it to work this way, at least with the Outlool 365 web client. If you know how to get this functionality, please share.

Better tools for kernel developers

Posted Feb 13, 2020 2:35 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

I don't use the online client, but Googling finds this: https://support.office.com/en-us/article/reply-with-inlin...

Better tools for kernel developers

Posted Feb 13, 2020 2:46 UTC (Thu) by chutzpah (subscriber, #39595) [Link] (1 responses)

Thanks for the pointer, but that is for the desktop client only. Unfortunately the web client does not have any such features that I have managed to discover.

Better tools for kernel developers

Posted Feb 13, 2020 3:10 UTC (Thu) by pizza (subscriber, #46) [Link]

Only Outlook for Windows supports inline quoting. The rest (mac, android, web, etc) don't support it, and Microsoft has never indicated that this will ever change.

Better tools for kernel developers

Posted Feb 7, 2020 16:07 UTC (Fri) by MaximeChambonnet (guest, #135950) [Link] (4 responses)

Please stop... if you are trying to convey linux kernel patches from windows there is a fundamental problem with you to start with...

Better tools for kernel developers

Posted Feb 7, 2020 17:56 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Why is it so?

Better tools for kernel developers

Posted Feb 8, 2020 12:41 UTC (Sat) by jezuch (subscriber, #52988) [Link] (2 responses)

Please stop with this elitist nonsense.

I work at a large investment bank, which is a heavily regulated industry and the Operations are very paranoid that you could accidentally "wall-cross" some sensitive material information (this is within the company, not even leaking it outside!) which means that the working environment is tightly controlled and this means Windows and Outlook. (Which nowadays is not the steaming pile of dung I expected it to be.) I don't love it, but don't you dare tell me that some thing is wrong with *me*.

P.S. We do quite a bit of contributions to Free Software. Some was created within the company and shared with the world.

Better tools for kernel developers

Posted Feb 11, 2020 23:03 UTC (Tue) by poruid (subscriber, #15924) [Link] (1 responses)

Ever noted that people insert screenshot cut-outs in mails sent with MS Outlook ?
If so, on reception, ever saved the image from the mail and found that it shows the whole screenshot? Not good, not in a tightly controlled environment, not good anywhere.

Better tools for kernel developers

Posted Feb 12, 2020 18:28 UTC (Wed) by jezuch (subscriber, #52988) [Link]

Sorry, I can't decide whether this classifies as a straw man or a red herring. Probably both.

(FWIW it never happens here. We have a separate snipping tool which recognizes that corporate users absolutely *love* screenshotting. Not that clueless cropping was ever a problem.)

Better tools for kernel developers

Posted Feb 7, 2020 22:13 UTC (Fri) by DSMan195276 (guest, #88396) [Link] (12 responses)

I've made a few contributions to the Kernel and other Open Source projects, and while I agree it's not terrible, you can't just hand wave away your first step as "this is no big deal":

> There are loads of mail clients out there, nobody is forcing you to use mutt (which doesn't suck).

The number that you can actually use to send patches and reply to mail on the LKML is a lot smaller than you'd expect/like. And with that, besides outlook I don't know *anybody* who uses a desktop mail client anymore - the only time I ever use one is for sending/receiving patches and mail on the LKML and other mailing lists. And the GMail web interface (by far the most popular to use) in particular does not work for the LKML, because it wraps lines at 80 characters and does other silly things, so you can't send patches using that even if you wanted too. It also doesn't support a threaded view of conversations that you obviously need for most (all) mailing lists, so replying to emails via that (which is "supposed to be easy") is also largely impossible.

With that, the list of desktop mail clients is slim and getting slimmer (because nobody uses them...) and `mutt` is the only one I had much success with for the LKML. There `alpine`, which is just as messy to get working as `mutt` in my experience, and there's also `evolution` which somewhat works but requires some settings to get right. I mean, just look at the kernel documentation for setting up your email client:

https://www.kernel.org/doc/html/v5.5/process/email-client...

The fact that it even needs a page that long with a bunch of weird settings should indicate things are a lot messier than you're suggesting. And the only time anybody is ever going to use any of those programs is when they're sending patches, they're otherwise useless tools for almost everybody.

My last point is the big one - the process is no longer "use your existing email client to do this", instead it has become "install and configure one of these obscure programs to interact with our system".

Better tools for kernel developers

Posted Feb 7, 2020 22:29 UTC (Fri) by andyc (subscriber, #1130) [Link]

> And with that, besides outlook I don't know *anybody* who uses a desktop mail client anymore

Nice to meet you... (Cl;aws Mail with own mail server)

Better tools for kernel developers

Posted Feb 7, 2020 23:13 UTC (Fri) by Jandar (subscriber, #85683) [Link] (1 responses)

I know a few people who use any kind of web-mail but those are vastly outnumbered by people using desktop mail-clients like thunderbird.

I'm struggling to think of any web application which isn't inferior to a local running program.

Better tools for kernel developers

Posted Feb 8, 2020 1:18 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link]

I know a few people who use any kind of web-mail but those are vastly outnumbered by people using desktop mail-clients like thunderbird.

I know a lot of people who use web-mail as their interface for their personal mail. It seems to me that it has two big advantages:

  1. It doesn't require they install a new piece of software on their computer.
  2. It requires no configuration.

These are big advantages for people who aren't very computer savvy, not that "aren't very computer savvy" describes many kernel contributors.

Better tools for kernel developers

Posted Feb 10, 2020 8:42 UTC (Mon) by geert (subscriber, #98403) [Link] (6 responses)

I have used the GMail web interface for almost all of my interaction with Linux mailing lists since 2009, except for:
1. Sending patches (git send-email using the SMTP server of my ISP),
2. Replying to patches on mailing lists I'm not subscribed to, and thus downloaded from lore or patchwork (alpine is still in my fingers).

Yes, GMail has lost the ability to not corrupt patches copy-'n-pasted to the web interface many years ago :-(

Better tools for kernel developers

Posted Dec 23, 2020 16:23 UTC (Wed) by tbelaire (subscriber, #141140) [Link] (5 responses)

What exactly does Gmail break? Is it just line wrapping or is it anything else?

I do work on Gmail, if there's a clear test case maybe I could try and fix this. Though I'm a little afraid of touching that part of the code base as there are a lot of tools which parse emails, and I wouldn't know if my change broke something until too late.

I suppose more usefully, how would I test this? I could generate a patch with git, copy it into gmail (after the fix), send it to myself and copy the body from gmail web into a .patch file, which I feed to git?
Or would I need to set up mutt or something in order to get the email in a form git likes? I haven't sent patches by email successfully before.

Can you include the patch file as an attachment instead?

Better tools for kernel developers

Posted Dec 24, 2020 0:34 UTC (Thu) by foom (subscriber, #14868) [Link] (3 responses)

Gmail hard-wraps plaintext mails at 78 columns. It should stop doing so. Fixing this in gmail would also make its plaintext emails more readable on mobile devices.

Other popular mailers have stopped doing hardwrapping years ago, and now simply send long lines as long lines, without inserting any extraneous linebreaks. In order to remain compliant with SMTP standards, which prohibits long lines, you must encode the message as quoted-printable if the lines are too long. But, quoted-printable encoding is reversible and transparent to the content, unlike inserting hard linebreaks.

A previous solution to the mobile problem was to use text/plain;format=flowed, but that format=flowed was a complex spec that had many problems of its own, and never interoperated well. It's fallen out of favor in recent decades (yes, decades!!!), in favor of simply never hard-wrapping text/plain content.

Better tools for kernel developers

Posted Dec 26, 2020 2:25 UTC (Sat) by zlynx (guest, #2285) [Link] (2 responses)

> Gmail hard-wraps plaintext mails at 78 columns. It should stop doing so. Fixing this in gmail would also make its plaintext emails more readable on mobile devices.

It's vague in my memory, but I am almost certain that wrapping around 72 and definitely before 80 is *required* for email user agents.

Yes, yes, it is commonly ignored for pasted or included content, but any email program that you type into is supposed to wrap long line at some point before 80, leaving room for at least three levels of quotes, like "> > > " which is six. So at least 74, more likely 72. 78 is being overly generous really.

This kind of sounds like someone complaining about Google's GMail *following standards*. Weird.

Better tools for kernel developers

Posted Dec 26, 2020 5:02 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

There's no hard requirement that user agents wrap text - it's convention to do so for the purposes of quoting stuff on 80 characters wide terminals, but doing so is inherently lossy when using plain ASCII (there's no way to tell the difference between a 72-character line followed by an 8-character line and an 80-character line that got hard wrapped). RFC5322 does say that agents SHOULD place a 78-character restriction on line length, but the context provided makes it pretty clear that it's talking about written language rather than code.

Better tools for kernel developers

Posted Dec 27, 2020 4:23 UTC (Sun) by foom (subscriber, #14868) [Link]

Sure, that was a common convention in the past, but many other user agents have stopped hard-wrapping years ago, because at this point it does more harm than good.

Better tools for kernel developers

Posted Dec 24, 2020 14:51 UTC (Thu) by geert (subscriber, #98403) [Link]

Using attachments is not an option, as it makes it hard to comment on attached patches.

Test procedure:

1. Generate patch with "git format-patch -1",
2. Revert top change using "git reset --hard HEAD^",
2. Open patch with e.g. gedit,
3. Paste patch into email body using gmail web interface,
4. Send email,
5. Open received email,
6. Select "Download message" from triple-dot menu,
7. Apply patch using "git am" (should apply cleanly),
8. Use "git diff" to check tree contents (should match state with step 1).

The above used to work, until Gmail started replacing TABs in pasted patches by spaces several years ago.
More things may have been broken afterwards (e.g. forced line break).

Thanks a lot!

FastMail

Posted Feb 16, 2020 11:26 UTC (Sun) by CChittleborough (subscriber, #60775) [Link]

(Sorry for posting this so late; I've just got around to reading the comments here.)

FastMail.com is a web mail provider built around the Cyrus IMAP server, a FOSS project to which FastMail make significant contributions (eg., JMAP).

As a FastMail customer*, I currently just use their web interface, but I previously used their IMAP interface with no difficulty.
*Full disclosure: I am a very satisfied FastMail customer.

Unlike gmail, FastMail does not mangle leading whitespace or long lines (at least in the simple test I just performed).

So maybe FastMail would be very useful to kernel developers. Does anyone use it?

Better tools for kernel developers

Posted Feb 21, 2020 21:29 UTC (Fri) by Wol (subscriber, #4433) [Link]

> My last point is the big one - the process is no longer "use your existing email client to do this", instead it has become "install and configure one of these obscure programs to interact with our system".

Thing is, there's a thing called "the principle of least surprise". And MOST GUI systems absolutely *love* to ignore it. If you use an email client that abides by it, you should not have any difficulty communicating with the linux project.

That's why a lot of MS products are so awful - they keep breaking this principle. As soon as someone starts thinking "why the f... is this happening?" there's a problem with the tool. And if the answer isn't blindingly obvious in hindsight (it should be obvious in foresight, too, but that really IS hard) then the tool is broken. Sadly, this is true of a heck of a lot of today's tools.

Cheers,
Wol

Better tools for kernel developers

Posted Feb 6, 2020 22:58 UTC (Thu) by mricon (subscriber, #59252) [Link] (5 responses)

> Sorry, this is not trivial and is extremely convoluted compared to creating PRs on Github (and I've done quite a bit of drive-by contributions using it).

We hope to adapt Git's GitGitGadget [1] to make it possible to submit Github PRs against the kernel tree. This should help with this particular problem without introducing any hard dependencies. This is slightly more difficult than for Git, since there isn't a single mailing list where all patches are sent, but hopefully this obstacle is not insurmountable.

Similarly, we hope to add an alternative frontend to GitGitGadget that wouldn't be tied to Github and can be used with any git hosting service -- such that anyone who is able to host a git tree somewhere on the net can provide an URL, a branch, and a cover letter to have the service automatically submit a patch series to the proper location for review.

[1]: https://github.com/gitgitgadget/gitgitgadget

Better tools for kernel developers

Posted Feb 7, 2020 0:19 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Wow! This would have saved me a lot of time. Is there a way to donate to this project?

Better tools for kernel developers

Posted Feb 8, 2020 23:37 UTC (Sat) by Conan_Kudo (subscriber, #103240) [Link] (3 responses)

Similarly, we hope to add an alternative frontend to GitGitGadget that wouldn't be tied to Github and can be used with any git hosting service -- such that anyone who is able to host a git tree somewhere on the net can provide an URL, a branch, and a cover letter to have the service automatically submit a patch series to the proper location for review.

This is literally one of the things that Pagure tries to offer. It doesn't exactly fit the Linux kernel's requirements right now, since it doesn't know how to process MAINTAINERS and send the message to the correct subsystem mailing lists based on what the diff includes. Nor does it have a mode for sending it as a traditional patch series with a cover letter + diffstat based on the PR description. But Pagure could be easily extended to provide that functionality.

But being most of the way there to supporting that, while being completely Free Software (GPLv2+) and supporting an open data model (all project data in Pagure is stored as git repositories and can be manipulated with Git) makes it a worthy option to consider and contribute to make it work for the kernel, right?

Better tools for kernel developers

Posted Feb 12, 2020 14:25 UTC (Wed) by spwhitton (subscriber, #71678) [Link] (2 responses)

Sourcehut also has this feature.

Better tools for kernel developers

Posted Feb 12, 2020 14:39 UTC (Wed) by Conan_Kudo (subscriber, #103240) [Link] (1 responses)

The problem with Sourcehut is that it provides no alternative workflow for submitting changes. It re-enforces the email-based workflow that myself and many others here have said is not workable.

Sourcehut itself does improve things from the "dumb" email model (such as basically integrating Patchwork's functionality), but itself does not provide any reasonable workflow improvements.

Better tools for kernel developers

Posted Feb 12, 2020 21:29 UTC (Wed) by milesrout (subscriber, #126894) [Link]

What you are saying about the email-based workflow that git was designed from the start to be used with is simply, factually, objectively wrong. Saying that it "isn't workable" isn't just incorrect, it's ignorant.

However if you put even the most basic research into sourcehut you'd know that the major goal of the website is to build online web-based contribution tools that use the "unworkable" email workflow underneath.

Better tools for kernel developers

Posted Feb 8, 2020 8:59 UTC (Sat) by ntj (guest, #130246) [Link]

> 1) Get special permission from IT to use IMAP for mail instead of regular Exchange protocol.

Officially, Exchange is not a protocol. It is not regular either. It uses MAPI, a proprietary protocol, that Microsoft tries to force onto their users as they do with their other garbage including their advertisements. That protocol does not even have an RFC.

To be honest, they officially state they allow you to also use standard protocols like IMAP and POP3, but in many cases you need to wire in your mailbox as an Exchange mailbox, for example on smartphones. (Fortunately you can do this with Gmail, so you are able to use 2 proprietary technologies at once.)

This misinformation and behavior is no surprise from Microsoft however.

Better tools for kernel developers

Posted Feb 6, 2020 22:43 UTC (Thu) by airlied (subscriber, #9104) [Link] (34 responses)

"You just email a patch using git's *built in* patch-formatting tools."

This is a nice throwaway, but try doing it without having a full email server you've run yourself for 20 years, or without access to port 25 due to ISP, or company firewalls.

Setting up git-send-email isn't trivial, compared to using gitlab or github. (gitlab isn't microsoft and used by a lot of things).

Better tools for kernel developers

Posted Feb 6, 2020 22:49 UTC (Thu) by milesrout (subscriber, #126894) [Link] (33 responses)

I can't even *comprehend* not having an email account. Email is the building block on which every account we have on the internet is based. Every account I have anywhere requires me to sign up with an email account anyway. You need to be able to at least *receive* email to be able to sign up to GitHub anyway!

Better tools for kernel developers

Posted Feb 6, 2020 23:15 UTC (Thu) by mpr22 (subscriber, #60784) [Link] (5 responses)

Most of my interactions with the world of email happen via port 443, not port 25.

Better tools for kernel developers

Posted Feb 7, 2020 11:42 UTC (Fri) by meyert (subscriber, #32097) [Link] (4 responses)

Port 25 ist the SMTP server to server port anyway....you probably want SMTP submission protocol port 587

Better tools for kernel developers

Posted Feb 7, 2020 12:01 UTC (Fri) by Karellen (subscriber, #67644) [Link] (3 responses)

Port 25 can also be used, as it was used for years, and can still be used, by all email servers that I am aware of, as the client-to-server email submission port.

Port 587 is completely redundant. It even uses the exact same protocol as that which runs on port 25. You can just set up your firewall rules to translate traffic on port 587 to port 25, and email submission will still work without any other changes. Just use port 25.

Better tools for kernel developers

Posted Feb 7, 2020 18:37 UTC (Fri) by anselm (subscriber, #2796) [Link] (1 responses)

The advantage of using port 587 for client submission is that as a mail server operator you get to apply a different set of policies to connections on that port. For example, submitting e-mail on port 587 often requires the STARTTLS and SMTP-AUTH features (among others), which you can't insist on for server-to-server e-mail delivery on port 25.

In point of fact, various ISPs block clients from accessing TCP port 25 anywhere on the Internet, and force them to submit e-mail on the ISP's own mail servers on port 587 with encryption and authentication (and possibly rate-limited), as a spam-reduction measure.

Better tools for kernel developers

Posted Feb 7, 2020 19:23 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link]

In point of fact, various ISPs block clients from accessing TCP port 25 anywhere on the Internet, and force them to submit e-mail on the ISP's own mail servers on port 587 with encryption and authentication (and possibly rate-limited), as a spam-reduction measure.

I think this gets to the core problem with email as a tool: it no longer works the way people expect it to. Spam and the responses to spam have fundamentally broken email. It is no longer easy to set up a server that reliably communicates with the rest of the world. Demanding that developers use a particular email setup in order to work on the kernel is thus a serious barrier to entry.

Better tools for kernel developers

Posted Feb 11, 2020 10:44 UTC (Tue) by nim-nim (subscriber, #34454) [Link]

587 is not redundant at all. You will typically open 25 without auth (to receive third-party messages), and 587 with auth (to allow your users to create new messages). The security pipeline is different.

Better tools for kernel developers

Posted Feb 6, 2020 23:15 UTC (Thu) by dannyobrien (subscriber, #25583) [Link] (9 responses)

When I find that I cannot comprehend something, I usually discover that I have to work harder on my own comprehension abilities.

For instance, as far as I can see, people here are describing challenges that revolve around sending and receiving email in the specific way that git expects, rather than expressing an inability to send or reply to email in general. Most people have email set-ups that don't work the way that git assumes (including you, I think! You do have a gmail account, yes?), and struggle to find out strategies that can enable them to better work with it.

Better tools for kernel developers

Posted Feb 7, 2020 1:52 UTC (Fri) by lsl (guest, #86508) [Link] (8 responses)

Gmail works just fine with git-send-email (or /usr/sbin/sendmail, ...). You generate an app-specific access token in the web UI and submit to smtp.gmail.com on port 587.

Better tools for kernel developers

Posted Feb 7, 2020 2:04 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (2 responses)

Though in all fairness that is going to change as soon as Google, in all their wisdom, only allows oauth authentication for SMTP and IMAP.

Better tools for kernel developers

Posted Feb 7, 2020 15:56 UTC (Fri) by jgg (subscriber, #55211) [Link] (1 responses)

I've been working on this for a while now, and am getting close to keeping my mutt & exim kernel workflow scheme working post-oauth-apocalypse.

https://github.com/jgunthorpe/cloud_mdir_sync

I have mail inflow working using OAUTH and a REST transport from my Office365 and GMail accounts.

While it seems trivial to say 'oh just use OAUTH' it actually turned out to be quite difficult to wrangle all the peices properly.

I haven't put the last parts in to send email yet, but internally the tool does have the bearer token to authorize SMTP. It needs plumbing into the MTA. Maybe next month. Still waiting for Office365 to actually have a way to send raw email with OAUTH, as MS promised to finally do 4 months ago. Maddening.

But yes, setting this all up is expert level stuff, not something you could expect someone who wants to just use git send-email to sort out. I'm mildly hopeful that someone will build a git plugin that can send email through gmail/o365 fully natively with 0 configuration.. At least for gmail sending email via REST is fairly straightfoward once you get the OAUTH token.

Better tools for kernel developers

Posted Feb 7, 2020 16:23 UTC (Fri) by pizza (subscriber, #46) [Link]

> But yes, setting this all up is expert level stuff, not something you could expect someone who wants to just use git send-email to sort out.

...but it only takes an expert like you to figure it out once, and then it's a lot simpler for other folks to replicate it.

> I'm mildly hopeful that someone will build a git plugin that can send email through gmail/o365 fully natively with 0 configuration.

That might be a natural progression of the work you're doing right now, and I suspect it would be very well received..

Better tools for kernel developers

Posted Feb 7, 2020 4:04 UTC (Fri) by Conan_Kudo (subscriber, #103240) [Link] (4 responses)

And if your Gsuite admin doesn't allow creating app tokens (mainly to force all authentication through 2FA through custom auth), what do you do then?

Better tools for kernel developers

Posted Feb 7, 2020 4:51 UTC (Fri) by pizza (subscriber, #46) [Link] (3 responses)

...Then you quit, in favour of an organization that knows better than to try and force its engineering staff to use the same tools and workflows as its secretaries?

(I'm actually serious here; this sort of thing is the tip of several pathological icebergs...)

Better tools for kernel developers

Posted Feb 8, 2020 23:25 UTC (Sat) by Conan_Kudo (subscriber, #103240) [Link] (2 responses)

This is an unreasonable response. I can count on one hand the number of projects that $DAYJOB relies on that still use email-based workflows for submitting fixes. (Hint: it's only three, and one we don't usually contribute to!) That's out of literally hundreds of projects and the three Linux distributions we contribute to.

There are a number of other concerns that prevent enabling weak app tokens. Most of them are around compliance and preventing account hijacking. Accounts of engineers are often way more privileged than non-engineers. At $DAYJOB, compliance requirements have forced us to mandate it for everyone since we try to maintain a relatively open environment for employees in the company.

As an aside, executive assistants (who you call "secretaries") tend to handle pretty sensitive stuff, and they're usually quite smart. They handle a different problem domain than engineers do, and it's a hard one. I miss being able to ask one to help me with doing travel arrangements and payment work. Doing it myself is a lot of complicated work...

Better tools for kernel developers

Posted Feb 9, 2020 3:59 UTC (Sun) by pizza (subscriber, #46) [Link] (1 responses)

> As an aside, executive assistants (who you call "secretaries") tend to handle pretty sensitive stuff, and they're usually quite smart. They handle a different problem domain than engineers do, and it's a hard one.

Well, of course "secretaries" handle a (very) different problem domain than engineers. (And one of their more challenging responsibilities is insulating executives from the effects of their own policies)

...Meanwhile, that is precisely why each group should have tools catering to their particular needs.

Better tools for kernel developers

Posted Feb 21, 2020 21:36 UTC (Fri) by Wol (subscriber, #4433) [Link]

And Secretaries handle a different domain again ... (sorry, it's one of my pet peeves, my mum was a Secretary and it is a serious legal qualification...)

Cheers,
Wol

Better tools for kernel developers

Posted Feb 6, 2020 23:24 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (13 responses)

This is a straw man. Plenty of people have perfectly valid email accounts, capable of exchanging messages with any other email account, but no straightforward way to use them with git-send-mail. Webmail and Exchange users probably outnumber traditional SMTP/POP3/IMAP clients at this point, and by a large margin.

Better tools for kernel developers

Posted Feb 7, 2020 11:00 UTC (Fri) by marcthe12 (guest, #135461) [Link] (12 responses)

Not to mention some of these issues are either caused by some company using such providers for email mainly gsuite and exchange. Not to mention some peoplemaybe having some email from school days and maybe unwill change email address for example. there need to wrappers for webmail provider to deal with non standard protocols or other auth method (oAuth2 or 2-FA) so they can be used with tradition workflows.

Better tools for kernel developers

Posted Feb 7, 2020 12:58 UTC (Fri) by pizza (subscriber, #46) [Link] (11 responses)

So.. the highly-successful and supremely-productive linux kernel development process needs to rethink its tooling because your employer deliberately chose to use (and configure) its email systems without considering the needs of its engineering staff?

The entire Linux kernel mentality is centered around Fixing Things the Right Way(tm), ie identifying the root cause of problems rather than adding endless layers of workarounds, and that extends to its tooling. If the tools are shitty, then make better ones. Telling them that they need to throw out the tooling they built to be maximally productive because other folks insist on using shitty tooling is not an argument that will gain traction.

Better tools for kernel developers

Posted Feb 7, 2020 17:53 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> So.. the highly-successful and supremely-productive linux kernel development process needs to rethink its tooling because your employer deliberately chose to use (and configure) its email systems without considering the needs of its engineering staff?
Is it super-productive, though?

Better tools for kernel developers

Posted Feb 7, 2020 20:28 UTC (Fri) by roc (subscriber, #30627) [Link] (3 responses)

> your employer deliberately chose to use (and configure) its email systems without considering the needs of its engineering staff?

When you set up an email system you can't know whether, possibly years in the future, your engineers might need to contribute to the kernel.

Better tools for kernel developers

Posted Feb 7, 2020 20:56 UTC (Fri) by pizza (subscriber, #46) [Link] (2 responses)

> When you set up an email system you can't know whether, possibly years in the future, your engineers might need to contribute to the kernel.

You're right, at any given point in time, you can't absolutely know what your users will need at some point in the future, which is why there must be a process for new needs/requirements to be captured, evaluated, and supported. After all, IT is a strategic asset that empowers your colleagues to leverage agile design methodologies to deliver maximum value through effective business analysis to diverse stakeholders while upholding the highest standards of quality and integrity.

Meanwhile, back in the real world, every engineering group I've ever worked for/with has had to set up some sort of shadow IT infrastructure in order to do their jobs.

Better tools for kernel developers

Posted Feb 7, 2020 21:56 UTC (Fri) by geert (subscriber, #98403) [Link] (1 responses)

IT seems to be the only infrastructure where the business people tend to decide/veto decisions for the technical people.
Imagine all transports had to be performed using limousines instead of trucks, as the former "just work fine" for the business people.

Better tools for kernel developers

Posted Feb 7, 2020 22:43 UTC (Fri) by mpr22 (subscriber, #60784) [Link]

> IT seems to be the only infrastructure where the business people tend to decide/veto decisions for the technical people.

Business people are perfectly capable of making terrible decisions about infrastructural matters in all sorts of domains; the Chemical Safety and Hazard Investigation Board would have a lot less to post on their Youtube channel if this wasn't the case.

Better tools for kernel developers

Posted Feb 8, 2020 13:17 UTC (Sat) by gfernandes (subscriber, #119910) [Link] (5 responses)

>>The entire Linux kernel mentality is centered around Fixing Things the Right Way(tm)

...by using the **wrong** tool (aka email)?

Email may have been the **only** way to get things done in the 70's. But that's not too say it was the **right** tool then. Most certainly, it is **not** the right tool **now**.

Better tools for kernel developers

Posted Feb 9, 2020 0:22 UTC (Sun) by andyc (subscriber, #1130) [Link]

> ...by using the **wrong** tool (aka email)?

> Email may have been the **only** way to get things done in the 70's. But that's not too say it was the **right** tool then.
> Most certainly, it is **not** the right tool **now**.

Opinions vary...

Better tools for kernel developers

Posted Feb 9, 2020 4:05 UTC (Sun) by pizza (subscriber, #46) [Link] (3 responses)

Email remains the *only* fully federated communication tool out there.

Everything else has dissolved into balkanized silos whose primary goal is to monetize the user.

Better tools for kernel developers

Posted Feb 12, 2020 17:37 UTC (Wed) by flussence (guest, #85566) [Link] (2 responses)

XMPP works well too, now that the silos have given up trying and failing to weaponise it against each other.

Better tools for kernel developers

Posted Feb 12, 2020 17:56 UTC (Wed) by pizza (subscriber, #46) [Link]

The entire federated XMPP userbase is barely a rounding error when compared to even one of the current IM silos.

...And if you think getting a usable semi-custom email workflow is too difficult... XMPP is an order of magnitude worse, with significantly worse tooling.

(I still have an XMPP server going for my domains. In the past few years, I've received a total of *zero* non-spam messages..)

Better tools for kernel developers

Posted Feb 12, 2020 18:06 UTC (Wed) by rahulsundaram (subscriber, #21946) [Link]

> XMPP works well too, now that the silos have given up trying and failing to weaponise it against each other.

XMPP has failed to get any traction because none of the implementations could be expected to be reasonably compatible with each other. That was the practical consequence of being extensible and having different features bolted on. Matrix has higher momentum now

Better tools for kernel developers

Posted Feb 8, 2020 13:09 UTC (Sat) by gfernandes (subscriber, #119910) [Link] (2 responses)

It's actually quite easy.

Email is so fantastically unfit for purpose, it's not even worth talking about.

WhatsApp and Signal have long replaced email for person-to-person communication for me. It's encrypted by default, and doesn't require me pulling out my hair out gauging out the eyes of whoever I want to communicate with, to setup encrypted communications.

Try doing that with email without getting granny to jump off the cliff.

Better tools for kernel developers

Posted Feb 21, 2020 21:39 UTC (Fri) by Wol (subscriber, #4433) [Link] (1 responses)

> WhatsApp and Signal have long replaced email for person-to-person communication for me. It's encrypted by default, and doesn't require me pulling out my hair out gauging out the eyes of whoever I want to communicate with, to setup encrypted communications.

I avoid WhatsApp, and what on earth is Signal?????

Cheers,
Wol

Better tools for kernel developers

Posted Feb 22, 2020 19:09 UTC (Sat) by jem (subscriber, #24231) [Link]

You know, nowadays you can find answer to questions like this faster by looking them up in Wikipedia...

Better tools for kernel developers

Posted Feb 6, 2020 22:52 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

Eh. I found the *submission* to be trivial (I use mutt as my daily mail driver though, so the stuff Cyberax lamented was handled by my stock setup years ago). Visibility into its *status* is what *really* sucks. Has it been triaged? Ignored? Lost in the firehose? I only get feedback when some step is complete (CI, review, etc.). I'd like to see CI results as they're happening. I'd like to have a nice way of collating the review feedback instead of having to wrangle and manage it myself (e.g., trailer collation, a tiny TODO list from comments). Tracking patch review versions is hard since (apparently) maintainers prefer fresh threads for each patchset version. Is the patchset I'm looking at the latest edition? If not, can I get a link to it? Reverse links would be nice as well.

I understand why things are the way they are, but incremental improvement is possible. I've discussed this before[1] on here[2].

[1]https://lwn.net/Articles/803696/
[2]https://lwn.net/Articles/807136/

Better tools for kernel developers

Posted Feb 7, 2020 0:03 UTC (Fri) by mirabilos (subscriber, #84359) [Link] (4 responses)

Completely agree, except for the git point.

I cannot easily send from git (need to hand-edit the output then feed it into /usr/sbin/sendmail on another machine), and people often don’t accept git format-patch output as attachments.

Back before git, or in the BSDs, where you’d just post inline or attached unidiffs, things were/are easier.

Better tools for kernel developers

Posted Feb 7, 2020 9:34 UTC (Fri) by geert (subscriber, #98403) [Link] (3 responses)

1. What prevents you from having git-send-email on the machine that has /usr/sbin/sendmail? Or configure git to ssh into that machine for running sendmail?
2. Patches created by git-format-patch are very similar to unidiffs, so you can just post them inline like before.

Better tools for kernel developers

Posted Feb 7, 2020 17:25 UTC (Fri) by mirabilos (subscriber, #84359) [Link] (2 responses)

1. The fact it’s a Pentium 233 MMX with 128 MiB RAM and about 40 GB HDD, and I don’t setup any automated ssh into that system as it’s a “bastion” which also has my DD PGP key.

2. Perhaps, yes, but people seem to complain about that as well…

Better tools for kernel developers

Posted Feb 7, 2020 21:51 UTC (Fri) by geert (subscriber, #98403) [Link] (1 responses)

Given you run the Good Old sendmail on that lowly server, I expect there's a way to send email through it?

Better tools for kernel developers

Posted Feb 7, 2020 22:07 UTC (Fri) by mirabilos (subscriber, #84359) [Link]

Incidentally I do (and yes, but only within my LAN), but /usr/sbin/sendmail is a generic MSA interface also implemented by other MTAs, i.e. Postfix.

Better tools for kernel developers

Posted Feb 7, 2020 3:09 UTC (Fri) by re:fi.64 (subscriber, #132628) [Link] (2 responses)

The fact that it works doesn't mean it couldn't be better.

Better tools for kernel developers

Posted Feb 7, 2020 4:30 UTC (Fri) by pizza (subscriber, #46) [Link] (1 responses)

Of course things could be better -- but reasonable folks disagree on what "better" actually means.

Enabling easier contributions from relative newbies is laudable, but if that requires you to kneecap your most productive folks by making their highly refined workflows unworkable, they're going to resist you, and for good reason.

I mean, one could make a decent argument about how if you're not capable of hooking up git-send-email, you probably aren't going to be able to handle the much greater technical challenges of landing actual changes to the kernel itself. Call it a basic weedout task if you will.

Another way of looking at it is that if you want to develop Linux kernel code, you need to use Linux kernel tools and workflows. This is hardly controversial; after all, if you want to develop an iOS app, you are forced to use Xcode on a Mac. Or Android Studio for Android apps, or Visual Studio (on Windows) for Windows applications. Or Vivado for wrangling Xilinx FPGAs. Or.. Or.. Or. Each of those embodies not just a different workflow, but the culture and mentality necessary to be truly productive.

So by all means, let's make things better. But it behoves us to listen to those who have been slogging away in the trenches and see what they come up with, lest we end up with the workflow equivalent of the Chauchat rifle.

Better tools for kernel developers

Posted Feb 7, 2020 6:33 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> Enabling easier contributions from relative newbies is laudable, but if that requires you to kneecap your most productive folks by making their highly refined workflows unworkable, they're going to resist you, and for good reason.
So an email gateway should probably be provided for them. But it's probably a good idea to recognize that the very specialized kernel workflow is a significant barrier.

> Another way of looking at it is that if you want to develop Linux kernel code, you need to use Linux kernel tools and workflows.
I actually developed a non-trivial kernel module in MS VS a while ago, using a third-party tool for GDB integration and remote debugging and editing...

The inverse is also doable, with MinGW+gcc you can compile binaries for Windows on Linux (and run them with WINE).

Better tools for kernel developers

Posted Feb 7, 2020 20:41 UTC (Fri) by roc (subscriber, #30627) [Link] (1 responses)

A classic case of the "success fallacy": "because Linux is winning, every aspect of its development process is justifiably excellent".

The success fallacy holds up right up to the point where a project starts losing, at which point everyone toggles to the failure fallacy: "the project is losing, so its development process must be bad (and we always knew it)".

Other subjects of the success fallacy: microservices, Python decision-making, Windows in the 1990s, Chrome.

Better tools for kernel developers

Posted Feb 7, 2020 21:07 UTC (Fri) by pizza (subscriber, #46) [Link]

> A classic case of the "success fallacy": "because Linux is winning, every aspect of its development process is justifiably excellent".

Which is why any proposed changes need to be incremental and revertable.

Better tools for kernel developers

Posted Feb 10, 2020 14:27 UTC (Mon) by nhippi (subscriber, #34640) [Link] (1 responses)

Of course, the "them javascript kids" attitude is why average age of a kernel maintainers rises every year by about one year[1]. Which is going to create problems a couple decades down the line.

[1] Stetson-Harrison et all

Better tools for kernel developers

Posted Feb 11, 2020 10:16 UTC (Tue) by milesrout (subscriber, #126894) [Link]

I'm 24. I can use email. Being able to set up an email client has nothing to do with age.

Better tools for kernel developers

Posted Feb 7, 2020 5:44 UTC (Fri) by marcH (subscriber, #57642) [Link] (2 responses)

Famous last words: "email just works!"

More seriously: we can discuss tools, habits and preferences until the cows come home but this:

> Reliable archives have been scarce in general, and a single archive for all kernel-related lists didn't exist at all; [...] it's worth remembering that it is less than two years old.

... was utterly, definitely, incredibly, astoundingly, unarguably wrong. I totally understand (!= agree) all the points about closed source software and vendor lock-in[1], slow and bloated web-only interfaces[2], decentralization to resist nuclear wars, but... EPHEMERAL code reviews, seriously? Is it so difficult to understand that some projects preferred early versions and review comments to be "trapped" in the centralized database of some evil corporation/software rather than... lost!?

> without that, there is nothing for a tool like get-lore-mbox to obtain messages from.

Ooohhh... look: a central database to store code reviews. A new, ground breaking concept /s. Long live the very first SEQL client (Standardized Email Query Language). Or was pwclient the first one?

> given the message ID for any email in a thread of interest

"Unique key". Etc.

> there is little else that can handle a community of that size and diversity.

The "size of the kernel" consideration is really overrated.

There's at least one other, highly successful, open-source project of similar size that never used email for code reviews: the user-space operating system named "Chrome" [3]. The workflow is different and I'm certainly not pretending Gerrit would work for the kernel. Still, it's worth nothing the number of commits per month is in the same range (and the CI in a different league) despite Gerrit being one of the grand-daddies. Worst kept secret: a centralized database scales better because it's much easier to filter noise than with a mailing-list. In fact it can lead to staying "too focused" when email (!) notifications are misconfigured.

> But he also said that each team needs a toolsmith, focused on creating the tools that the team needs. [...] Both within the kernel and beyond it, there are many pressing needs that, it seems, are nobody's problem, so nobody is funded to work on them.

Admittedly the complete opposite of Chrome and Android (which gave most of their tools away)

[1] gitlab is an _open-source_ clone of github and a better one in a number of ways https://gitlab.com/gitlab-org/gitlab/issues/20339#note_28... I had to use recently and while not perfect I've been very impressed, well done. Debian adopted it.

[2] ... which actually have decent CLIs more often than you think; how do you think CI works? Wait, you don't have any "plug and play" CI with email, need some elbow grease? Sorry about that. SEQL FTW?

[3] you know it's not a joke: https://www.chromium.org/chromium-os

Better tools for kernel developers

Posted Feb 7, 2020 5:53 UTC (Fri) by marcH (subscriber, #57642) [Link]

> The workflow is different and I'm certainly not pretending Gerrit would work for the kernel. Still, it's worth nothing the number of commits per month is in the same range (and the CI in a different league) despite Gerrit being one of the grand-daddies. Worst kept secret: a centralized^H database scales better because...

After second thought, I don't see why you couldn't have one code review database per subsystem and pull requests between them. The kernel already has multiple mailing-lists after all. While desirable, they wouldn't even need to be all "locked-in" in the same software.

Better tools for kernel developers

Posted Feb 8, 2020 3:17 UTC (Sat) by marcH (subscriber, #57642) [Link]

> Reliable archives have been scarce in general, and a single archive for all kernel-related lists didn't exist at all; [...] it's worth remembering that it is less than two years old.

Just realized it's the exact same problem with IRC: lossy by default (compared to slack etc)

Fascinating...

Better tools for kernel developers

Posted Feb 7, 2020 6:23 UTC (Fri) by kraxel (subscriber, #49444) [Link]

FYI: There is patches (https://github.com/stefanha/patches) doing the same thing.
Ordering patches for 'git am' and collecting acked/reviewed/tested tags from replies.
Uses notmuch as email database, so it works offline, but needs (unlike get-lore-mbox)
some non-trivial setup.

Better tools for kernel developers

Posted Feb 7, 2020 10:57 UTC (Fri) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (8 responses)

And here I thought that LWN was the project's secretary! ;-)

Better tools for kernel developers

Posted Feb 7, 2020 14:18 UTC (Fri) by karim (subscriber, #114) [Link] (7 responses)

This is the bit that got me spilling some coffee: "Indeed, some of them got started using punch cards and may still be unconvinced of the virtues of, say, text editors."

Better tools for kernel developers

Posted Feb 7, 2020 15:57 UTC (Fri) by PaulMcKenney (✭ supporter ✭, #9624) [Link] (1 responses)

Heh! I cannot but plead guilty to charges as read on the first part of that statement!!!

That said, one of the first screen-oriented editors I ever used featured a physical knob that you turned to move the cursor through the text. 512x512 pixels, any color you wanted as long as it was either a pale green or a moderate grey, and frame buffers are for wimps!

And get your touchpad off my lawn!!! :-)

But for me, nothing quite like a few years using punched cards and punched tape to build an appreciation for even line-oriented editors like "ed".

Cue the guys who did their first few years of programming manually switching in binary code along with their jeers of "assemblers are for wimps!!!" :-)

Better tools for kernel developers

Posted Feb 7, 2020 17:34 UTC (Fri) by karim (subscriber, #114) [Link]

All I want to know is why LWN doesn't have a social media function to repost golden nuggets like this ;)

P.S.: Thanks for sharing, this is awesome.

Better tools for kernel developers

Posted Feb 7, 2020 17:14 UTC (Fri) by jem (subscriber, #24231) [Link] (4 responses)

If we still used punch cards we wouldn't need to be bothered with questions like "is the newline character a text line separator or a terminator." I.e. should a text file end with a line feed character or not. Also, no more hard to read overlong code lines.

Better tools for kernel developers

Posted Feb 7, 2020 17:31 UTC (Fri) by PaulMcKenney (✭ supporter ✭, #9624) [Link]

Ah, but the FORTRAN continuation column!

Better tools for kernel developers

Posted Feb 21, 2020 21:48 UTC (Fri) by Wol (subscriber, #4433) [Link] (2 responses)

Well, when the company I worked for first got a daisywheel printer, we discovered we actually needed <cr> to be carriage return, and <lf> to be linefeed. The printer driver read the print file line by line, with a maximum line length of 256. If your line was more than 256 characters long (easily achieved if you wanted bold text on an 80-char line) the driver stuck a line-feed in, splitting what should be one line on to two.

So imho, the *nix convention "line feed means carriage return" is actually broken ...

Cheers,
Wol

Better tools for kernel developers

Posted Feb 22, 2020 2:06 UTC (Sat) by PaulMcKenney (✭ supporter ✭, #9624) [Link]

Bytes were expensive back then! But yes, some of the old drivers would add the <cr> for you. If properly configured, anyway. :-/

Better tools for kernel developers

Posted Feb 22, 2020 19:08 UTC (Sat) by jem (subscriber, #24231) [Link]

The line feed character represents a newline at a higher abstraction level. Unix uses the line feed character to represent a line break in a text file. This has little to do with what character(s) are actually output when the file is sent to a terminal or a printer; the translation is done by the terminal line driver. How the newline should be output is determined by the line settings.

The same goes for input: conventionally, the Return (CR) key is used to terminate a line when typing on a (virtual) terminal, but the line break ends up as a line feed character in a file or in the input to a process.

Better tools for kernel developers

Posted Feb 7, 2020 13:52 UTC (Fri) by dona73110 (guest, #113155) [Link]

>But reaching that point has required going through a number of crises when
>the kernel project's processes proved unable to support the community.

>It would be a great thing if the next big transition in the kernel's development
>process could happen without the "crisis" part.

Why's that?

The long pressure-cooker experience culminating in a crisis is _exactly_ the process that "gives birth" to something beautiful and of real value.

Self-realization is a process which is proved with pain, whether you're talking about source control for the kernel, the declaration of independence, or human life itself.

Better tools for kernel developers

Posted Feb 7, 2020 16:57 UTC (Fri) by smoogen (subscriber, #97) [Link]

So I am going out on a limb and say that the linux kernel environment has had 3 secretaries over the years that would meet Brooks' criteria:

1. The mailing lists. Most of the data is stored and 'collated by grep' for a long time and was more what a 1975 secretary would do with meetings, notes and documents.
2. The git repository. Again keeping track of the source code, who did what and when, why they did it and recaps of decisions.
3. LWN. Jonathan Corbet has done a 20+ year service to the communities for the work he and the other LWN contributors have done to make the high lights of the decisions available to the general audience of 'other groups' needing that information.

The tool maker is definitely something that various people have never had time for full-time and instead has been done ad-hoc.

Better tools for kernel developers

Posted Aug 16, 2021 13:29 UTC (Mon) by muzg666 (guest, #139506) [Link]

hahah 2020 - https://en.wikipedia.org/wiki/Integrated_development_envi...


Copyright © 2020, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds