Reducing patch postings to linux-kernel

By Jake Edge
November 8, 2023

The linux-kernel mailing list famously gets an enormous amount of email on a daily basis; the volume is so high that various email providers try to rate-limit it, which can lead to huge backlogs on the sending side and, of course, delayed mail. Part of the reason there is so much traffic is that nearly every patch gets copied to the mailing list, even when it may be unnecessary to do so. A proposed change would start shunting some of that patch email aside and, as might be guessed, has both supporters and detractors, but the discussion does highlight some of the different ways the mailing list is used by kernel developers.

In a post that was not sent to linux-kernel, but to the Kernel Summit discussion list (ksummit-discuss) and the kernel.org users mailing list, Konstantin Ryabitsev described the problems he sees with the current status of the list. It has nearly 3000 subscribers and is copied on nearly every patch because of a wildcard entry at the end of the MAINTAINERS file; that, in turn, leads to around 3.1 million messages being delivered to inboxes daily based on the roughly 33,000 monthly posts to the list. But that does not work well when delivering to a whole bunch of Gmail addresses:

due to gmail's quota policies, this generally results in anywhere from 50K to 200K messages stuck in the queue all trying to deliver to gmail and being deferred with "this account is receiving too much mail"

His suggested solution is to reroute the wildcard entry so that patches go to the patches@lists.linux.dev mailing list rather than linux-kernel. That will (eventually) reduce the volume on the list, thus "unclog the outgoing queues and speed up mail delivery for everyone". Currently, the get_maintainer.pl tool, which is often used with other tools like "git send-email", will pick up the entry for "THE REST" at the end of MAINTAINERS; the entry says to send everything to linux-kernel. The linux-patches list is available, for those who want it, via the lei tool or by anonymous POP3 for those who want to receive the patches that way—at Gmail, for instance. But direct subscriptions to linux-patches will be vetted so that the mechanism for overwhelming email providers with patches does not recur.

Joe Perches, who maintains get_maintainer.pl and other development tools, called it an "excellent idea"; Borislav Petkov said that CCing patches to linux-kernel was meant for archiving purposes, so a separate list should work just fine for that instead. Others, though, disagreed with one of the downsides of the current situation that Ryabitsev had listed: "due to the sheer volume of messages, LKML is generally seen as useless for holding any actual discussions".

Eric W. Biederman said that he at least skims linux-kernel with some frequency; Christoph Hellwig agreed, adding that he does start discussions on that list as well. Meanwhile, Willy Tarreau suggested that it is a good way to keep abreast of kernel developments:

This way every day I can have a quick glance at all subjects there, that's how I discover new topics, patch series, discussions etc. I think that a non negligible number of LKML subscribers are there for this exact reason.

He said that he would personally miss the patches that got moved to the other list, but he also questioned how much improvement the change would actually bring. Others also wondered about how much traffic would be reduced; Pavel Machek thought that the number of patches picked up by the wildcard was fairly low. Paolo Bonzini said that it might be the configuration of git send-email that was actually causing the patches to be posted to linux-kernel, which means local changes would be needed to alter it; other developers did not think that was a common configuration, however. Laurent Pinchart pointed out that the "submitting patches" document is rather ambiguous about sending patches to linux-kernel:

linux-kernel@vger.kernel.org should be used by default for all patches, but the volume on that list has caused a number of developers to tune it out. Please do not spam unrelated lists and unrelated people, though.
This should be updated, even if for the only reason that the text is quite confusing (in my opinion at least, I'm not sure if it means LKML should be used for all patches, or shouldn't).

He also noted that, unlike some of the other responders, he has completely tuned out linux-kernel since the volume of email "drowns the useful information in noise for me". Greg Kroah-Hartman said that switching the tools to do the right thing will help; for example, the linux-usb mailing list is specified as where patches for the USB subsystem should go, but get_maintainer.pl still also lists linux-kernel for USB patches. The change suggested by Ryabitsev would avoid doing that, "which is a good thing and should cut down on the overall size over time".

If that change is made, though, Julia Lawall wondered how she would be able to review "the discussion that led up to a commit"; currently, looking at linux-kernel is "the obvious place to go for that". Dan Carpenter suggested "lore.kernel.org and b4 and lei", while Pratyush Yadav provided a detailed description of how to use those tools for tracking down a discussion. That procedure will find patches and discussions that occurred before he subscribed, which makes it "more powerful and complete than subscribing to mailing lists". Ryabitsev pointed out that all of the lists that are archived at lore.kernel.org are actually indexed together, so that searches using lore.kernel.org/all will find message IDs or subjects in all of the kernel lists—including linux-patches.

There may well be a discussion about the idea in the Kernel Summit track at the upcoming Linux Plumbers Conference, or perhaps the Maintainer Summit the following day. That is presumably one reason that Ryabitsev posted to ksummit-discuss, though Carpenter noted that it makes a better forum for general topics than linux-kernel these days.

There are probably as many different kernel-development styles as there are kernel developers—a number that grows with each release—so finding common ground between them all is difficult, if not outright impossible. The problems with mail delivery these days are real, sadly, and it is most certainly not only Gmail that causes those kinds of woes. Given that, which only seems to get worse over time, some kind of mailing-list fix is going to be needed; Ryabitsev's plan seems a reasonable approach that may well help. Beyond that, those who are using the large, free email providers may want to consider voluntarily switching their linux-kernel subscription elsewhere in order to improve the service and reliability of the mailing list for everyone else.

Index entries for this article
Kernel	Development model/Patch management

to post comments

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 0:10 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (14 responses)

> due to gmail's quota policies

What policies exactly? This suggests it is based on some policy written down somewhere, but I doubt there is. I've dealt with delivery problems a fair bit and this is far too hand waving. I suspect what is happening is that gmail is returning 4xx try again later codes for some mail based on some unpublished algorithm (aka policy) that google changes on a whim and never shares externally and lkml is a bit of an edge case. That is not necessarily a problem. Having mail in the queue for retry is not a problem. I admin lists.gnu.org and we send millions of emails per day, qemu-devel is the biggest with close to 3k subscribers. The fine article suggests gmail is getting overwhelmed, which sounds highly unlikely to me, lkml is a drop in the bucket for gmail.

It looks to me like this proposal could be decomposed into 3 separate changes from the perspective of a user:

1. Some mail is automated to go to linux-kernel@vger.kernel.org. Start sending it to patches@lists.linux.dev.
2. Stop sending the automated mail to linux-kernel@vger.kernel.org.
3. Don't let people subscribe to patches@lists.linux.dev in general.

If I was considering a change like this on lists.gnu.org & lists.nongnu.org, #3 would need a lot more justification before I ever floated it, because it is removing a feature of the service from users, and I would want to try hard to avoid doing that. As for #2, the right way to do it would be about 6 months after #1 in order to give time for people to adjust, which of course doesn't solve any short term problems with gmail.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 1:59 UTC (Thu) by admalledd (subscriber, #95347) [Link] (11 responses)

The issue is about how gmail (and other larger email providers) think something is fishy because so many of "the same" email(s) are going to so many inboxes. From a glance, qemu-devel as a single list is not even close to needing to deliver millions of emails a day. gmail could easily handle the volume but its reasonable anti-spam/etc policies. Policies which they do have, but are reasonably less listed to reduce rules-lawyer abuse. You can find them/discuss them with gmail support.

Having mail in queue *is* a problem when it starts building up enough that either causes people to worry about missed mail (when instead it is delayed a few hours) or if the backlog is bad enough that it can't catch up at all and builds to infinity.

The steps you listed may be *a* fine solution, but as the article notes that some people have come to rely on/use the "everything is in the big mail-list" for their workflows. Before any such changes are being made, understanding what people are using it for and improving processes is required. Which is the whole point of why the discussion was started, to see what people thought or if there were other solutions to take.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 2:15 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (2 responses)

There are multiple things you said which I think are technically wrong or I disagree with. I don't currently plan to explain those details because I don't find that lwn comment sections with anonymous commenters to be very productive at clearing up misunderstandings, but I'm open to my mind being changed in any specific circumstance.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 4:30 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (1 responses)

Hi! As someone who isn't anonymous I agree with a bunch of the comments above and would appreciate hearing your counterargument.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 5:44 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

Alright. I started on a simple point. I plan to get to more tomorrow evening after work.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 5:42 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (7 responses)

> Having mail in queue *is* a problem when it starts building up enough that either causes people to worry about missed mail (when instead it is delayed a few hours) or if the backlog is bad enough that it can't catch up at all and builds to infinity.

I laughed at this. It brought to mind the quote "That isn't how it works, that isn't how any of this works." Let's start at square one. What is the essential work the LKML outgoing mail server does? Extend that to: has done since it began? For each message posted to the list, it sends that message out to the subscribers. The subscribers changes over time, and we've already been told it is currently around 2,900 email addresses, and we can safely assume averages less than 1% change between each posted message. So, first of all, the LKML mail archive is probably a few terabytes gziped. Then the subscriber list for each message is around 2,900 email addresses, but on average <29 are changed between each posting. That information is definitely < 1 terabyte, maybe < 1 gigabyte. So, the queue could contain all the essential work that LKML has needed to do since it's inception and that could pretty much fit on a single hard drive. There's no infinity going on here. Now, some mail server software or configurations can do things quite inefficiently, I mean, infinity is very easy to program, but that is completely different.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 6:01 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (6 responses)

If mail is received more rapidly than LKML can send it, the issue isn't the amount of storage, it's that the time taken between a mail being sent and it being received by all subscribers trends towards infinity. In the short term, receiving 1000 mails in an hour but only being able to ensure 900 of those are sent to every subscriber each hour isn't a huge deal. But over the course of a day, that means you've received 24,000 mails but only delivered 21,600 of those - 2400 mails are still sitting in the queue. A day later and that number has doubled, and you're probably now receiving more than 1000 mails an hour because some users are re-sending the mails they sent a day earlier and which never got delivered back to them so they think it may have become lost. The problem continues getting worse until people simply stop sending mail, the incoming amount drops below the outgoing amount, and you can recover - but that comes at the cost of the communication simply no longer happening, which means the list is no longer performing the desired function.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 6:55 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (5 responses)

As I said, there are many ways for a program to get to infinity, but you are describing a theoretical problem, not a real practical problem for LKML. To help understand why, it might help to think of an analogy. People are much more familiar with web servers than email servers. In a lot of ways, a mailing list is like a web server with some role reversal, and for most people, someone that runs a mail server for them. So, think of LWN. Corbet or Jake posts an article, we all request it from the server. What would happen if a queue of requests built up on the web server, and the server couldn't process all the requests? Furthermore, imagine the web server is stupid and keeps trying to process all the really old requests first, and so no one can load the page in a reasonable amount of time, and then the site is not serving its function. Then imagine someone said "ya, you know, http request queues can be a problem with infinite request queues, and I think it is reasonable for LWN to restrict that to a small group of people who present a valid reason to use http." If you've administered web servers, you would laugh. I've administered email servers, and so I'm laughing. The basic commonality is that there are are many many ways for a program and a service to stop working, and they will break and stop working, all the time, but most of those things can be fixed with a reasonable effort, as long as someone backs up the data and knows how to get the software installed, and that is why I have a job as a sysadmin and still find time to post here.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 8:32 UTC (Thu) by vegard (subscriber, #52330) [Link] (4 responses)

The gmail mail servers are throttling mails from the vger.kernel.org mail servers, so there is a queue of mails that builds up on the mailing list side. The mail admins have said as much. It's not a theoretical problem.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 9:12 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

> The gmail mail servers are throttling mails from the vger.kernel.org mail servers, so there is a queue of mails that builds up on the mailing list side. The mail admins have said as much. It's not a theoretical problem.

Whatever problem exists, if one exists, I can confidently say that this description is missing essential details which make this description basically wrong or far too vague to be the basis of a discussion of any possible solution. And in any case, I also kind of think what you are saying misses the point of what you are replying to.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 13:04 UTC (Thu) by MarcB (subscriber, #101804) [Link] (2 responses)

Are they really throttling requests from the server (IMHO unlikely) or are they throttling requests to individual Gmail recipients (this is part of Googles' policies)?

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 13:33 UTC (Thu) by jengelh (subscriber, #33263) [Link] (1 responses)

It's the worst of both worlds. And if Gmail blocking incoming mail isn't an insult already, the web MUA munges whitespace or something, reliably breaking patches since 20 years or so.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 14:44 UTC (Thu) by MarcB (subscriber, #101804) [Link]

But they are not blocking mails, they are merely deferring them - and based on a recipient based policy, not a source based one.

I do not see any issue with this. We send around 7 million mails per day to Gmail with an average deferral rate of ~15%. The top cause is over-quota accounts followed by the those per-account rate limits. Queues remain reasonable.

As a sender, that really should not be a problem. Yes, mail to the affected accounts will be delayed, but this is something the users would have to discuss with Gmail. It should also not be super-important for a mailing list.

However, for the over-quota case, it might make sense to unsubscribe accounts if this persists for too long. At the very least, this should be done once Gmail upgrades this to a 5xx error.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 15:57 UTC (Thu) by mricon (subscriber, #59252) [Link] (1 responses)

> What policies exactly? This suggests it is based on some policy written down somewhere, but I doubt there is.

https://support.google.com/a/answer/1366776?sjid=16765502...

Maximum 60 per minute, 3,600 per hour. Someone who subscribes both to netdev and linux-kernel will easily fill up this quota on a single 30-patch series.

> Having mail in the queue for retry is not a problem.

Yes, it's a problem, because the person then cannot receive *any* mail from us. When Greg KH used to send rapid-succession stable patchbomb series of 100-500 messages to linux-kernel, that meant nobody at gmail who is subscribed to LKML was able to receive any mail from us until their quotas cooled off. Thankfully, Greg doesn't do that any more after switching to the dedicated patches list.

Huge queues also degrade performance, but it's less of a concern for us because Postfix is pretty good at managing huge queues and the systems that are sending out these mails are very beefy.

However, I'd like to step back and note that the point of my original post was not about performance -- it was about making linux-kernel more usable for general discussion by moving the patches sent "for archival purposes" to a dedicated list.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 19:09 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

> However, I'd like to step back and note that the point of my original post was not about performance -- it was about making linux-kernel more usable for general discussion by moving the patches sent "for archival purposes" to a dedicated list.

Splitting the list into 2 lists makes a lot of sense to me, making the list have less high volume of posting is good for several reasons. And, my comment that it would be good to give a 6 month notice was too hasty a thought and not a good idea, since as a user, you don't want to receive a message twice, so a much shorter notice would be fine. The main thing I suggested is that more should be done before enacting onerous or strict requirements on people subscribing to the new list, but it is fine to encourage people to use other means of getting those messages. Of course, you could could port over all the subscribers, but not doing it and letting people opt in is fine. Overall, you aren't going to please everyone, and it is fine to decide not to respond to everyone and still implement a change. It is a general problem that the person or people doing the work to administering a service are much less numerous than all the people who use the service, so like engaging in discussion with 1% of the users could wipe out all the time you have for actually administering the service.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 0:57 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (12 responses)

> That will (eventually) reduce the volume on the list, thus ""unclog the outgoing queues and speed up mail delivery for everyone"

"clogged" is not a technical term. A big outgoing queue just means a lot of mail sitting on disk. The reason that could slow down mail delivery for "everyone", is because the mail software is hitting some bottleneck in processing it, either cpu time, disk bandwith, or something, and that can be solved by prioritizing which mail to process or by fixing the bottleneck, and this is not a very hard problem.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 7:10 UTC (Thu) by smurf (subscriber, #17840) [Link] (7 responses)

There is no bottleneck. Any reasonable mail server has a cap on the number of concurrent delivery processes. It's just that all of them are busy trying to deliver mails to Google and get a heap of 4xx replies. Lather, rinse, repeat.

The technical solution would be to forward gmail-directed messages to a separate system that can twiddle its thumbs that way (and eat resources for no good reason) without impacting the rest of the queue.

The real solution is to blacklist gmail and to simply not deliver any LKML email to it until they get their act together and whitelist LKML.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 9:32 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

> There is no bottleneck. Any reasonable mail server has a cap on the number of concurrent delivery processes. It's just that all of them are busy trying to deliver mails to Google and get a heap of 4xx replies. Lather, rinse, repeat.

> The technical solution would be to forward gmail-directed messages to a separate system that can twiddle its thumbs that way (and eat resources for no good reason) without impacting the rest of the queue.

What you are describing is what I call a bottleneck, a lack of some enough of some resource to do work as fast as desired, skimming the 1st sentence on wikipedia, it seems to agree with me https://en.wikipedia.org/wiki/Bottleneck_(software). And the solution you propose is what I would call eliminating the bottleneck: pushing some of the work onto a different machine to free up resources on the first machine.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 10:18 UTC (Thu) by taladar (subscriber, #68407) [Link] (3 responses)

> The real solution is to blacklist gmail and to simply not deliver any LKML email to it until they get their act together and whitelist LKML.

To be perfectly honest there is nothing wrong with the way Gmail handles mail. There is something deeply wrong with the way LKML and other mail-dependent development processes do though. Maybe it is time to consider not catering to the people who refuse to even consider using git and web based pull/merge request and code review tools. The entire problem is just caused by trying to use mail as a high bandwidth broadcast system which is not what it was ever designed for.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 12:38 UTC (Thu) by smurf (subscriber, #17840) [Link] (2 responses)

> To be perfectly honest there is nothing wrong with the way Gmail handles mail.

Sure there is. They're delaying legitimate emails from a legitimate mailing list, and all that accomplishes is to eat up even more resources, on both their end and vger's.

Reducing patch postings to linux-kernel

Posted Nov 14, 2023 18:36 UTC (Tue) by riking (subscriber, #95706) [Link] (1 responses)

They're delaying *all* emails to someone receiving over 60 mail per second. This necessarily means that legitimate mail will be delayed.

Reducing patch postings to linux-kernel

Posted Nov 14, 2023 20:44 UTC (Tue) by smurf (subscriber, #17840) [Link]

… per minute. One per second.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:22 UTC (Thu) by MarcB (subscriber, #101804) [Link] (1 responses)

> The technical solution would be to forward gmail-directed messages to a separate system that can twiddle its thumbs that
> way (and eat resources for no good reason) without impacting the rest of the queue.

The solution is to act like the large scale sender that LKML clearly is: Look at he policies of major recipients and implement suitable queue management that honors those policies. This is what large senders have been doing for decades.

For example, there is no reason to make delivery attempts that almost certainly will be deferred anyway. For example, if you get a Gmail rate limit error for a recipient, simply wait 60 seconds before attempting another delivery to this recipient. If you get it again, wait for one hour. You can reduce the waits if you do more bookkeeping.

If you get a 4xx over-quota also wait some time. If you get a 5xx over-quota, unsubscribe the recipient.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 18:03 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

Yes, MarcB, I believe you are hitting on a likely solution.

> there is no reason to make delivery attempts that almost certainly will be deferred anyway

In fact, you might be punished for doing so, and need to wait longer than you would have otherwise. You have to experiment and ask other mail server operators in order to figure these things out.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 7:57 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (3 responses)

> and that can be solved by prioritizing which mail to process or by fixing the bottleneck, and this is not a very hard problem.

Now you're just adding queues.[1] Give it a few more years, and the problem will simply get worse.

[1]: https://ferd.ca/queues-don-t-fix-overload.html

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 10:18 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (2 responses)

>> and that can be solved by prioritizing which mail to process or by fixing the bottleneck, and this is not a very hard problem.

> Now you're just adding queues.[1] Give it a few more years, and the problem will simply get worse.

> [1]: https://ferd.ca/queues-don-t-fix-overload.html

I assume you are referring to the "prioritizing" in my post. This is a good observation. But, this made me realize the term queue here is not that kind of queue, perhaps it a misnomer to call it a queue at all and it is just weird mail server terminology. I totally forgot this would confuse people. When a new message comes into the mail sever that is meant to go to the mailing list subscribers, it does go into an actual queue, but the situation I've been talking about is where that queue all goes fine and is not clogged, but the delivery attempt fails, for example the gmail 4xx that we are talking about, then the message gets dumped onto disk and scheduled for a retry. In mail server parlance, that is a message in "the queue." But it isn't waiting in line to be handled like the queues in your link, that is why I say it maybe a misnomer. When it next gets retried next, the server checks how long it has been trying, if it has been beyond some period, the server gives up and deletes the message. Then it is gone from "the queue" which wasn't really a queue in the normal sense. In exim, I think by default that happens after 6 days, and the last few days delivery gets retried once every 24 hours. So, the max size of the queue is just 6 days worth of mail. One potential problem, which I was describing, is that mail server software can be kind of stupid and get really busy doing retries and then be slow at sending fresh mail that isn't on a retry. One fix is to just have the server prioritize fresh mail and do less retries, because it is generally not a problem to do a lot less retries than the server is configured to do by default. For example, I've heard that gmail's outgoing servers only retry for 6 hours, and they do a lot less retries than exim does by default.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 10:19 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

> not clogged

Hah. I'm tired should have explained that a little better.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:41 UTC (Thu) by MarcB (subscriber, #101804) [Link]

Maybe I am too deep in mail terminology myself, but IMHO queue is a good fit. However, it is not a static queue.

It is more similar to the run queue of a process scheduler with dynamic priorities and temporary blocks that prevent mails to specific mail addresses or from/to specific IP addresses from being delivered.

LKML shouldn't even be close to Gmail's daily and hourly per-recipient limits. Most likely it is just wasting a lot of time on delivery attempts that cannot succeed and getting back to deliveries that will succeed later than necessary.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 4:40 UTC (Thu) by wtarreau (subscriber, #51152) [Link] (5 responses)

One thing that really annoys me here is that Google provided gmail to collect a lot of data and tried hard to become the #1 centralized e-mail provider on the planet, and now they're dictating how people should communicate just because they reached a critical mass. So because a significant proportion of people use gmail, others have to suffer a degraded quality of service to match what gmail is capable of.

I tend to think that another radical approach could be to just blacklist gmail.com addresses until someone there fixes the problem. It's already particularly irritating that some huge platforms like this impose themselves so vastly, but it's even more annoying when they force anyone else to adapt to their limitations.

Maybe a first step would be to document "be careful, gmail is collapsing and no longer capable of reliably receiving lkml" everywhere a text suggests to subscribe to lkml ?

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:03 UTC (Thu) by MarcB (subscriber, #101804) [Link]

Those limits were essentially in place from day one of Gmail (or very close to it). AFAIK they were even raised over time. They also are not unreasonable.
The only place Gmail is trying to "dictate" anything is mail authentication - but there they have a very strong point, and are not alone in this endeavor.

I also fail to see how this is an actual problem. 50k-200k messages in a queue at any time should not be an issue.

Note that at the volume LKML appears to have, you are in an area where it makes sense to start looking into policies of major recipients and use an adjusted queue management that avoids delivery attempt that will likely fail anyway.

For Gmail in particular, this is very easy, because they have clear policies (it basically is one mail per second and recipient with burst windows of one minute and one hour). So, if you just sent a batch of 60 mails to one recipient in the last 10 seconds there is no point to continue. Instead process the next recipient and get back to the first one after at least 50 seconds.

This is just how mail works at scale, and always has.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:05 UTC (Thu) by marcH (subscriber, #57642) [Link] (1 responses)

> So because a significant proportion of people use gmail, others have to suffer a degraded quality of service to match what gmail is capable of.

gmail offers a "normal" email service. How many mailing lists receive 30,000 messages per month? Email was never intended for that sort of throughput; everyone else uses some sort of database (e.g: Gitlab) instead to manage that much. As mentioned in the main article, even the Linux kernel uses a database now: lore.kernel.org

> I tend to think that another radical approach could be to just blacklist gmail.com addresses until someone there fixes the problem.

I bet no one at gmail is ever going to care about something fixing a non-problem like this.

I wonder why some people with gmail.com addresses subscribe to 30,000 messages per month in the first place... if you really want to use email-as-a-database, then you need some specialized email-as-a-database tools.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:53 UTC (Thu) by MarcB (subscriber, #101804) [Link]

Gmail allows 86,400 mails per recipient address and day, but there are additional limits per minute and hour that will lead to a lot of pointless delivery attempts if not respected.

The volume of LKML really is nothing special, it just seems to be handling its queues in a way that is less then ideal.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 18:36 UTC (Thu) by ssmith32 (subscriber, #72404) [Link]

Yeah, my initial response was similar, but, reading the article, it sounds like gmail is just an example, and there are folks that want a different workflow, regardless.

Reducing patch postings to linux-kernel

Posted Nov 14, 2023 17:03 UTC (Tue) by q_q_p_p (guest, #131113) [Link]

>One thing that really annoys me here is that Google provided gmail to collect a lot of data and tried hard to become the #1 centralized e-mail provider on the planet, and now they're dictating how people should communicate just because they reached a critical mass.

Yeah, and this is what GitHub does too. People never learn.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 6:16 UTC (Thu) by Fowl (subscriber, #65667) [Link] (3 responses)

Presumably Google - which some evidence suggests participates in kernel developement - and also operates Gmail - has been contacted for some immediate technical relief/advice on the "this account is receiving too much mail" problem?

Seems prudent regardless of the outcome of the process discussion.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 9:04 UTC (Thu) by IanKelling (subscriber, #89418) [Link]

Based on experience, I'd give good odds that there has been no contact between anyone who makes policy decisions about how gmail handles incoming mail and Konstantine at linux foundation who made this proposal. I don't know why for sure, but here is a thory: gmail gets requests from all corners of society across the globe, from people trying trying find justice and random desperate people with all kinds of requests who really deserve a response. gmail was created with the idea that google would ignore most worthy and important requests in order to save money. Given that, and other factors, I can think of several reasons for special treatment from gmail policy makers to be intentionally rare and secretive, even among their own employees. And this specific issue, well LKML is an outside group that google has a business interest in, and this is generally a 1st world problem for people who could easily switch to a different email provider. There are good arguments for gmail operators to avoid having any public dealings on this issue.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:26 UTC (Thu) by marcH (subscriber, #57642) [Link] (1 responses)

> Presumably Google - which some evidence suggests participates in kernel developement - and also operates Gmail - has been contacted

You're falling for the common trap of personifying a company with 100,000s of employees.

To be fair the PR departments of such companies work hard to entertain that myth.

Reducing patch postings to linux-kernel

Posted Nov 10, 2023 2:14 UTC (Fri) by Fowl (subscriber, #65667) [Link]

> You're falling for the common trap of personifying a company with 100,000s of employees.

Of course they're not going to be the same people, and for some organizations the PR department can be more effective than any internals coms method, but at least "we created a ticket on the internal issue tracker" has been known to get *something*.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 11:00 UTC (Thu) by bluca (subscriber, #118303) [Link] (10 responses)

If only there was a way to collaboratively develop open source software at scale that didn't rely on a communication medium from the 70s that was never designed for such practices. Oh well, I guess we'll never find out.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 12:54 UTC (Thu) by pwfxq (subscriber, #84695) [Link] (3 responses)

I know I'll be a lone voice, but I find it very funny that the folks behind a key piece of open source software are complaining about how a closed source commercial service they use can't cope with their workloads.

If only someone would step up and write a FOSS alternative to GMail that people could use themselves.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:33 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

At this point even a huge number of Western corporate email addresses are on Gmail, Outlook 365 or (a distant third) Zoho. For personal addresses there's again Gmail and Microsoft (live.com, formerly Hotmail), and even if you add Yahoo and a bunch of country-specific domains that came out victorious in the late 1990s, there's obviously even more concentration.

No one really *wants* to run their own mail server. It's not about having a FOSS alternative, for example there's Kolab. It's just that spam is such a huge issue that running your own mail server is, for most people and even corporations, not just a huge pain in the ass but really a total waste of time.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:37 UTC (Thu) by marcH (subscriber, #57642) [Link]

> ... that the folks behind a key piece of open source software are complaining about how a closed source commercial service they use can't cope with their workloads.

Using email to carry this type of workload is also very unusual. Really unique situation = really unsurprising it meets issues. So yeah, wondering why a closed source, commercial service does not support this is quite surreal. Of course gmail could - but why would they care about that niche?

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:58 UTC (Thu) by MarcB (subscriber, #101804) [Link]

> I know I'll be a lone voice, but I find it very funny that the folks behind a key piece of open source software are complaining
> about how a closed source commercial service they use can't cope with their workloads.

Gmail can easily deal with this load. A Gmail user can receive more mails per day than LKML sends to each address per month.

It seems to me that LKML is running in the per minute and per hour limits because it is managing its queues naively. Maybe over-quota accounts also are an issue.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 16:43 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (5 responses)

I for once would like to try, but Daniel Vetter opened a GitLab feature request for perhaps the biggest blocker (git range-diff functionality) 5 years ago, and no one has followed up. GitHub is even worse with it's obscure garbage collection of commits upon a force-push, because what email provides to Linux developers is, first and foremost, archival.

This doesn't give me much hope for any less-important feature that stands in the way of Linux (at least some subsystems) trying out forges more seriously. (Red Hat uses GitLab for the RHEL kernel, but that's purely backporting and the actual development happens upstream).

https://gitlab.com/gitlab-org/gitlab/-/issues/24096

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 17:20 UTC (Thu) by bluca (subscriber, #118303) [Link] (4 responses)

Github does show diff-of-diff when one updates a PR, it is not obvious though - when an update is pushed, the "force-pushed" in "foobar force-pushed the barbaz branch from abc to def" is a link to the diff-of-diff. Other than that, it is always possible to fetch the changes from a PR locally and run any command one likes.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 18:07 UTC (Thu) by pbonzini (subscriber, #60935) [Link] (2 responses)

It's not a diff of diff. It's a diff of trees that includes anything that was pushed to the source, if it was rebased. (Unless it changed in the last few months).

The unit of review is the patch, and GitLab/Github gets that right, but then it makes it impossible for reviewers to ascertain that prior comments were actually acted upon. The way that many projects work around that is by pushing a lot of fix commits on top, which either destroys bisectability or requires a squash-on-merge workflow. Both are a non-starter for Linux and many other projects.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 18:41 UTC (Thu) by bluca (subscriber, #118303) [Link] (1 responses)

> It's a diff of trees that includes anything that was pushed to the source, if it was rebased.

Yes, it requires a minimum of discipline if one is interested in using that feature - nominally, first rebase and push with no changes, then amend and push again (or other way around). It's doable, but it is also very overrated as a feature, and the vast majority of times it is not really needed, so we bother in only very few cases where it might matter, with extremely large PRs or so - which are frowned upon anyway, so it's rare.

> The unit of review is the patch, and GitLab/Github gets that right, but then it makes it impossible for reviewers to ascertain that prior comments were actually acted upon.

Old comments that refer to lines that have changed are automatically marked as 'outdated', and they can also just be marked as resolved clicking a button, which is what the submitter usually does, and it's much simpler and quicker than spelunking through diffs-of-diffs anyway.

Reducing patch postings to linux-kernel

Posted Nov 9, 2023 18:49 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

Enforcing discipline on the part of submitters never work if there is an easier, less disciplined path. If lack of discipline results in total unusability of the result, rather than just a bit of degradation, it's a non starter.

It is not overrated at all. First, the problem with Linux is that "extremely large PRs" are not that rare once you have thousands of developers. I agree that they're frowned upon, but they happen a lot with a project of the size of Linux.

Second, what's an extremely large PR is also very much up for discussion. Personally, I find something like https://patchew.org/linux/cover.1697532085.git.kai.huang@... invaluable even if the submitter was "not so disciplined" and made some slightly gratuitous changes to the headers. And the UI there sucks, since it was written by a kernel developer (me) rather than someone who knows what they're doing.

Reducing patch postings to linux-kernel

Posted Nov 11, 2023 15:28 UTC (Sat) by marcH (subscriber, #57642) [Link]

> Github does show diff-of-diff when one updates a PR

Only if you don't rebase. In other words: no "git range-diff"

More in last week's discussion: https://lwn.net/Articles/950563/