Next steps for kernel workflow improvement

Posted Nov 1, 2019 17:44 UTC (Fri) by q_q_p_p (guest, #131113)
Parent article: Next steps for kernel workflow improvement

As someone who sends patches to kernel from time to time, I don't have much to complain about current workflow, yet I wish that any enhancement to development process would avoid proprietary solutions like GitHub, still work through SSH (it would be great if it was integrated with git, like git send-email) and preferably wasn't web based (dedicated client that does only what it is supposed to do, would be great).

Next steps for kernel workflow improvement

Posted Nov 1, 2019 19:20 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

My main issue with the email workflow (which has its benefits) is that patches get lost in the firehose. There's no public triage of patches to know anything about them, updates to patchsets is usually done through completely new threads (making it hard to see what was already discussed), joining a mailing list leaves a chasm of "before I was subscribed" and "afterwards" for older information (I read my mail in mutt). If there was better metadata tracking and linking, I'd find it a lot easier.

Next steps for kernel workflow improvement

Posted Nov 6, 2019 5:00 UTC (Wed) by marcH (subscriber, #57642) [Link]

> My main issue with the email workflow (which has its benefits) is that patches get lost in the firehose.

This is one of the main reasons absolutely every code review solution (and any every "firehose" solution for that matter) relies on some sort of *database* featuring all the typical bells and whistles: [live] queries, cross-references, statistics, [CI] triggers, filters, notifications, authentication, etc.

The web vs CLI question is important but secondary, it's "just" the interface and many code review solutions offer both to some degree.

Now what is exciting here is allusions to some _distributed_ database model? Who knows, this could revolutionize code reviews like bitkeeper decentralized and revolutionized version control...?

Next: distributed bug tracking? OK, maybe that wouldn't be useful.

Next steps for kernel workflow improvement

Posted Nov 1, 2019 20:11 UTC (Fri) by logang (subscriber, #127618) [Link] (3 responses)

Yes, I really think any tooling improvements should just start and end with git. Yes, it will take a while for people to upgrade but changes to workflow are going to take a long time anyway and if there's a compelling reason to do so, then people will put the effort in.

My vague ideas for features in git would be:

* Support the entire flow for sending git patches inside git itself. This means branches need first class ways of storing cover letters, versions, recipient lists, etc. Instead of needing to do: format-patch, write cover letter, figure out send lists, notice a mistake, format patch, copy over cover-letter, send. It would nice if git just stored all this with the branch and all you needed to do was 'git send' when it's all ready.

* Support for easily importing patchsets from a mailbox into branches, with the cover letter and recipient lists. (Obviously this will need to solve the base-tree information problem first, possibly by including public repos that already have the base commits with the patches).

* Support for reviewing a patchset inside git itself and having the review comments sent via email to everyone on the recipient list and author, etc.

* Support for branch queues: if people are now importing tons of branches into their repos from their mailboxes, then they need some way of organizing these branches and determining which need attention next

* If the above features start being used by a majoriy, maybe then git could start to allow different transports other than email. So imagine a .git_maintainers file that contains a mapping of email addresses to desired transport. If the recipient's address isn't in this file, it simply falls back to email. A new transport might simply be that instead of emailing the patches they get pushed to a specified branch queue in a world-writable git repo. Sadly, this likely means that git will need to support some spam mitigations too.

* After that, interested parties could probably write a github-like web service that just provides a new front end for git's existing features. Then maintainers that want this could set it up for themselves, or kernel.org could offer this for maintainers that want it.

* And once there's a majority using this flow, adding structured data or tags from CI bots should be a bit easier because it's just a matter of changing the tooling everyone already uses.

Next steps for kernel workflow improvement

Posted Nov 3, 2019 21:14 UTC (Sun) by rodgerd (guest, #58896) [Link] (2 responses)

A lot of what you're describing sounds like the way Fossil works, integrating things like the issue tracker with the source code management: https://fossil-scm.org/home/doc/trunk/www/index.wiki

It's a pity Fossil isn't better-known; they already seem to have solved a lot of these problems.

Next steps for kernel workflow improvement

Posted Nov 4, 2019 14:17 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

From what little interaction I've done with Fossil, there seem to be some intra-version incompatibilities with the database format and the tools (feel free to correct me; I've not interacted with Fossil much beyond cloning sqlite itself). The saga documented for getting a Git repository of sqlite is quite involved[1]. I value stability of source code checkouts very highly.

For those curious, I care because we vendor sqlite which is based on using git repositories for patch tracking before we import the code and then enforce that all changes for the vendoring process are tracked in that repository. Inb4 "vendoring is bad": there's an option to use an existing sqlite, but…Windows.

[1]https://repo.or.cz/sqlite-export.git

Next steps for kernel workflow improvement

Posted Nov 4, 2019 17:26 UTC (Mon) by logang (subscriber, #127618) [Link]

I took a look at fossil's documentation and it really does not have the features I'm suggesting. In fact it uses a completely different model where all repos for a project tend to be synced together instead of being independent and being push/pull (or send-email/am). More over they don't have features like review or features that overlap with what people use their mail clients for (which is what I'm advocating).

Frankly, I think the fossil model is not useful for most open source projects. They don't really have a convincing story for drive-by contribution nor scaling a community. And they pretty much state out right that it would not be suitable for the kernel development model:

>The Linux kernel has a far bigger developer community than that of SQLite: there are thousands and thousands of contributors to Linux, most of whom do not know each others names. These thousands are responsible for producing roughly 89⨉ more code than is in SQLite. (10.7 MLOC vs. 0.12 MLOC according to SLOCCount.) The Linux kernel and its development process were already uncommonly large back in 2005 when Git was designed, specifically to support the consequences of having such a large set of developers working on such a large code base.

>95% of the code in SQLite comes from just four programmers, and 64% of it is from the lead developer alone. The SQLite developers know each other well and interact daily. Fossil was designed for this development model.

Next steps for kernel workflow improvement

Posted Nov 3, 2019 10:23 UTC (Sun) by daniels (subscriber, #16193) [Link]

GitLab at least has a comprehensive API which can be used to pull the feed of recent events, create/modify/etc merge requests and comments on them, and so on, from the client of your choice. There are standalone CLI clients and rich bindings for whichever language you care to use it from. That's true of most web-based services created in the last 5-10 years.