LWN: Comments on "How Red Hat uses GitLab for kernel development"

How Red Hat uses GitLab for kernel development

misc — Sun, 31 Oct 2021 14:12:14 +0000

You might be interested in the Fedeproxy project ( https://fedeproxy.eu/ ) and the Forgefed project ( https://forgefed.peers.community/ )

While federation do not solve the issue of centralization (as we still need some reference point), it reduce the risks of SPOF, and by making forges interoperable, it would also limit problem if one turn bad.

git "volatile" branches

marcH — Mon, 25 Oct 2021 00:41:13 +0000

> There's arguably two separate sets of workflows you want out of your SCM:
> ...
> The ability to share the mutable history is useful in some cases, ...

Yes it would save a lot of confusion if git provided some kind of "volatile" branch marker so publishers can share their intentions and set mutability expectations. Currently this is performed with poor branch naming conventions or worse: pure word of mouth.

Giving the distinction between 1. and 2. a name would elevate it and also clarify the recurring "don't rebase public branches" advice which is exactly what intel-next and others do all the time!? And hey, who knows it could even stop questions like "why can't I commit and push at the same time" from Fossil and other fans.

https://www.google.com/search?q=git+volatile+rebase+branches

Fossil

flussence — Fri, 22 Oct 2021 20:56:23 +0000

I'd liken it to the "preview comment" button here - that one roadbump causes me to put far more care into what I'm emitting than if there'd just been a single post button (even if it doesn't seem like it sometimes). I've used many other UIs for committing what I type to the internet, and it feels like there's a definite correlation between the overall quality of results in some of them and whether they let (or require) you experiment and make mistakes in private.

How Red Hat uses GitLab for kernel development

Klavs — Fri, 22 Oct 2021 13:27:14 +0000

I must say I have the exact same experience with gitlab - their omnibase package is wrapped in ansible and simply "fixes everything" - and upgrades have been very smooth for years.. Major upgrades though - we have found better to do by installing on new server and doing recovery test (which we WANT to test regularly anyways).

Fossil

farnz — Fri, 22 Oct 2021 10:56:18 +0000

There's arguably two separate sets of workflows you want out of your SCM:

Immutable shared history, showing how the product developed. This is curated by the committers to tell a story about the code.
Mutable private history, where you're checkpointing your work in interesting places so that nothing of interest is lost when you try something out. You will later rebase and restack this to become the immutable shared history you push.

The ability to share the mutable history is useful in some cases, but to do it well adds a lot of complexity (see Mercurial's "Changeset Evolution" extension). But the ability to work locally and store up commits of random changes, then to build and test the final history you wish to push, is extremely useful.

Fossil

marcH — Thu, 21 Oct 2021 22:36:18 +0000

Interesting, thanks!

Also realized I forgot to mention Fossil:
https://www.fossil-scm.org/home/doc/trunk/www/fossil-v-gi...
"One way to describe Fossil is that it is "GitHub-in-a-box."

That's probably because I never used it and also because I violently disagree with the lack of history rewriting:

> When every commit is pushed to the parent repo by default, it encourages a working style in which every commit is tested first. It encourages thinking before acting. We believe this is an inherently good thing.

Very ironic considering I use git rebase -i to _test_ a variety of "what if" scenarios every single day.

I find the rest of the page very impressive.

How Red Hat uses GitLab for kernel development

Lennie — Sat, 09 Oct 2021 15:49:38 +0000

I'm really surprised we aren't seeing a more decentralized git usage being adopted.

Their are projects which have build a system which allows issues to be tracked in a special git repository:

https://dist-bugs.branchable.com/software/

How Red Hat uses GitLab for kernel development

fung1 — Thu, 07 Oct 2021 12:22:09 +0000

> SPOF / centralization is the only real problem. All other issues are either fairly easy to solve or solved already (kudos to RedHat for trying to move things forward _and_ sharing their efforts)
>
> So the next (and big!) step is to find some good distributed database model and implement a free software forge based on it. Funny enough that sounds very much like what Bitkeeper and then git achieved.

As of a few years ago, Gerrit moved all comments and discussions into Git objects so they can be replicated and fetched along with the merged commits and proposed changes for a repository. They basically got rid of the RDBMS entirely, except for one UI-related feature for tracking which specific files you've already "seen" when reviewing a particular version of a patch (which has fairly ephemeral utility at best, and almost nobody I know relies on that behavior anyway).

How Red Hat uses GitLab for kernel development

prarit — Wed, 06 Oct 2021 17:33:12 +0000

@johannbg, we looked at the features of many git-forges, and at the time GitLab offered us the best feature set. I'm not saying that going with GitLab is the choice that upstream should make but it has worked for us very well and we're happy with it. As Don said in the video, we continue to work with GitLab on enhancing their software to make it work better both with Red Hat and with upstream.

How Red Hat uses GitLab for kernel development

johannbg — Tue, 05 Oct 2021 18:01:02 +0000

Why did Red Hat chose GitLab ( as opposed to Github,Sourcehut or some other git forge )?

How Red Hat uses GitLab for kernel development

jezuch — Tue, 05 Oct 2021 14:37:29 +0000

That at least was my impression. It was a big company with big projects and a huge build farm, so perhaps it was just overloaded. But from a developer's point of view there were frequent stability issues and the CI team was definitely not sitting bored.

How Red Hat uses GitLab for kernel development

anselm — Tue, 05 Oct 2021 09:49:25 +0000

I'm in charge of our company's Gitlab instance and it doesn't require “constant babysitting”. Gitlab publishes new versions on a regular basis (feature updates every month, and bug fix/security updates in between as needed), but these generally install completely seamlessly and with minimal downtime if any.

How Red Hat uses GitLab for kernel development

zev — Mon, 04 Oct 2021 16:48:20 +0000

Indeed I was not -- that does looks quite useful! Though after a few minutes of poking at it semi-randomly, its patch-discussion linking seems fairly spotty unfortunately (oddly, often failing even in cases where there is a Link: tag right there in the commit message). Nevertheless, I'll probably be making use of that in the future, thanks for the tip.

How Red Hat uses GitLab for kernel development

willy — Mon, 04 Oct 2021 14:19:52 +0000

You must not be aware of https://cregit.linuxsources.org/

How Red Hat uses GitLab for kernel development

taladar — Mon, 04 Oct 2021 10:37:47 +0000

I can't help but feel that Bugzilla is not representative for a tool here because of its highly email-focussed workflow. It is basically the bug tracker for people who already liked the existing workflow instead of one for the people who use bug trackers similar to the ones used in other projects.

How Red Hat uses GitLab for kernel development

jezuch — Mon, 04 Oct 2021 09:52:54 +0000

I was surprised that RedHat don't run their own instance of GitLab. Especially given the concerns that they may be siloing their data in some external company's silos. On the other hand, my previous employer did have their own instance of GL and it required constant babysitting. So yeah I get why they want to outsource the pain :)

How Red Hat uses GitLab for kernel development

bfields — Sun, 03 Oct 2021 19:19:56 +0000

Somehow I missed that!

It's fun to do things like 'git range-diff origin/master mwork@{1} mywork'

Very neat, I've always wanted that--though honestly I feel like I've gotten long OK without.

How Red Hat uses GitLab for kernel development

mkubecek — Sun, 03 Oct 2021 19:15:37 +0000

How about "git range-diff origin/master foo-v1 foo-v2"? Or "git range-diff [prev base]..foo-v1 origin/master..foo-v2" if I still have the branch from v1 review and don't want to rebase it.

How Red Hat uses GitLab for kernel development

marcH — Sun, 03 Oct 2021 18:02:15 +0000

> A bigger issue, though, has to do with robustness. If kernel.org is not reachable, kernel development still goes on; an outage is inconvenient but not really a big problem. Adding a central forge, though, risks creating a situation where, should it go down, no work can get done.

>Perhaps the biggest concern, though, has to do with making GitLab into a single point of failure; what if the company is bought out by somebody who is hostile to Red Hat and its goals?

SPOF / centralization is the only real problem. All other issues are either fairly easy to solve or solved already (kudos to RedHat for trying to move things forward _and_ sharing their efforts)

So the next (and big!) step is to find some good distributed database model and implement a free software forge based on it. Funny enough that sounds very much like what Bitkeeper and then git achieved.

> In that case, it would be relatively easy to pull all of the necessary data out of the system; the Git trees are already mirrored elsewhere. They have a script now that can take all of the comments from GitLab and dump them into a public-inbox instance.

Sure but that would be a huge step back.

How Red Hat uses GitLab for kernel development

marcH — Sun, 03 Oct 2021 17:41:15 +0000

> Git has a native diff tool for ranges of commits: https://git-scm.com/docs/git-range-diff

And neither gitlab not github do range-diff : https://gitlab.com/gitlab-org/gitlab/-/issues/24096

Github is especially bad at it.

Of course using a forge and git range-diff are not mutually exclusive, in fact it's easier than pure email + range-diff.

How Red Hat uses GitLab for kernel development

mathstuf — Sun, 03 Oct 2021 12:24:23 +0000

Alas, the issue has been open for a long time: https://gitlab.com/gitlab-org/gitlab/-/issues/24096

How Red Hat uses GitLab for kernel development

pbonzini — Sun, 03 Oct 2021 10:48:50 +0000

As a Red Hatter and an upstream developer, it's a bit more complicated than this. Despite being on a nominally old version, lots of subsystems in the RHEL kernel are rebased every 4-8 months to the latest version in the upstream kernel. For example, if you compare arch/x86/kvm/ between kernel-4.18.0-333.el8 and upstream 4.18, you get

 72 files changed, 45945 insertions(+), 26903 deletions(-)

If you do the same with 5.13, you get

 36 files changed, 985 insertions(+), 927 deletions(-)

These differences in turn are mostly because the kernel version backports the bugfix parts of what went into the 5.14 merge window. If it weren't for that, it would be even smaller.

As a result, there are two typical kinds of pull request in the RHEL kernel: bugfixes that are verbatim copies of 1-8 upstream commits, and large backports consisting of 200 to 1000 patches. The former are almost entirely trivial to review; often the developer who posts them is the same person who did the work upstream. The latter account for most of the 15000 patches mentioned in the article; of those, about 3-5% patches will have some difference with the corresponding upstream commit, typically due to bugfixes being backported out of order, and even fewer will be exclusive to RHEL. Of the 15000 patches, perhaps 1000 are going to be ever "clicked on" in the Gitlab merge request web pages.

As a result, review used to be 95+% the menial work of finding which patches have differences with the corresponding upstream. Such work is perfect for bots: they check the correspondance between the RHEL commit and the one in the "cherry picked from ..." lines (and the presence of those lines), signal differences, and let humans actually do the interesting parts of the review.

So, on one hand I absolutely love the GitLab-based development process that Prarit and Donald and many others have implemented. On the other hand, I would be wary to suggest that it could be applied to anything but the stable kernel process. In particular, code review on the web at the scale of Linux development is a complete non-starter until GitLab or GitHub implement a UI that maps to "git range-diff".

How Red Hat uses GitLab for kernel development

anmoch — Sun, 03 Oct 2021 06:39:10 +0000

Git has a native diff tool for ranges of commits: https://git-scm.com/docs/git-range-diff

How Red Hat uses GitLab for kernel development

epa — Sun, 03 Oct 2021 06:24:38 +0000

The review comments should be part of the same git branch, as special commits that only have a commit message. Then you can add new commits in response to the review, and it’s easy to see what you changed and verify that the starting point is what was reviewed.

How Red Hat uses GitLab for kernel development

zev — Sat, 02 Oct 2021 22:59:03 +0000

I've yet to actually use it myself, by Konstantin Ryabitsev's b4 tool appears to offer support for comparing revisions of a patch series: https://git.kernel.org/pub/scm/utils/b4/b4.git/tree/b4/co...

Personally, I've recently found myself often wishing that "Link:" tags were more universally used. Commit messages are nice, but it's not at all infrequent that I really want to find the discussion that preceded a patch being accepted. Without a nice lore.kernel.org link right there, finding that is a lot more tedious.

How Red Hat uses GitLab for kernel development

fratti — Sat, 02 Oct 2021 20:35:50 +0000

I've also just found out about the --interdiff switch of git format-patch, so I suppose the ball is in my court to make things easier for maintainers. (Though admittedly, this again places the trust in the submitter to not mess this comparison up, either accidentally or intentionally.)

Perhaps all we need is a --supplants=<message id> for git send-email to get better tracking of patch series.

Thank you for sharing your perspective.

How Red Hat uses GitLab for kernel development

bfields — Sat, 02 Oct 2021 20:01:26 +0000

>I cannot even begin to imagine how painful it must be for reviewers to trust me that I only changed the parts of the driver I said I changed in my cover letter in a subsequent submission of a driver, or to reference back to the comments on the previous submission to see if everything has been addressed.

It's not *that* hard to find the previous version in my saved mail and compare.

You could compare two versions with

$ git checkout -b foo-v1 origin/master
$ git am foo-v1-mbox
$ git checkout -b foo-v2 origin/master
$ git am foo-v2-mbox

and then

$ git diff foo-v1..foo-v2

$ git diff foo-v1^^..foo-v2^^

though I've rarely done that in practice.

Probably this should all be easier, but it's not impossible.

How Red Hat uses GitLab for kernel development

fratti — Sat, 02 Oct 2021 16:47:39 +0000

I appreciate the write-up.

To me, it seems like something that provides some of the benefits of a forge without the full suite of features (and complexity) of a forge would be the best way forward. Say, a better system to manage and track patch series than patchwork currently is, with a way for maintainers to mark a series as merged or rejected, and a way for submitters of patch series to assign a unique identifier to a series so subsequent versions of it are linked to each other and can be compared. Being able to easily diff the diffs of different versions of a series would likely help. While developing, I often wish there was a meta-history for the actual git history wherein I could track the progress on my patches while still keeping a clean patch-centred history. Perhaps something like this already exists, and I'm simply not aware of it.

I'm fairly new to kernel development, mostly having gotten started writing patches and sending them in in the past 2 months, but I cannot even begin to imagine how painful it must be for reviewers to trust me that I only changed the parts of the driver I said I changed in my cover letter in a subsequent submission of a driver, or to reference back to the comments on the previous submission to see if everything has been addressed.