|
|
Log in / Subscribe / Register

Fedora gathering requirements for a Git forge

By Jake Edge
January 29, 2020

Fedora currently uses Pagure to host many of its Git repositories and to handle things like documentation and bug tracking. But Pagure is maintained by the Red Hat Community Platform Engineering (CPE) team, which is currently straining under the load of managing the infrastructure and tools for Fedora and CentOS, while also maintaining the tools used by the Red Hat Enterprise Linux (RHEL) team. That has led to a discussion about identifying the requirements for a "Git forge" and possibly moving away from Pagure.

The conversation started with a post on the Fedora devel mailing list from Leigh Griffin, who is the manager of the CPE team. That message was meant to call attention to a blog post that described the problems that Pagure poses for the CPE team and the path toward finding a solution using the Open Decision Framework (ODF). "We will be seeking input and requirements in an open and transparent manner on the future of a git forge solution which will be run by the CPE team on behalf of the Fedora Community."

The problem statement describes a number of areas where Pagure does not fit within the mission statement for CPE, its current priorities, and the workload for the team. Due to a lack of developer time, Pagure is also falling further and further behind the other Git forges in terms of its feature set. While Pagure was originally developed by CPE member Pierre-Yves Chibon ("pingou") and has been maintained by him and others, largely in their "spare" time, it has languished more recently:

The CPE team has been unable to commit a development team to Pagure for several months now. This situation is unlikely to change based on our current known priorities.

The choices available section lists three possibilities: GitHub, GitLab, and Pagure.

There are no other forges that we could find that had both the product maturity and standing in open source communities, therefore no other solutions are under consideration as the three choices here represent the main git forge solutions on the market.

The idea, then, is to gather requirements for a forge from the various users and stakeholders, then to make an "informed decision" on which of the options to pursue.

To be clear, the outcome here may be a decision to invest heavily in Pagure to meet the requirements or it may be to opt for another git forge to meet the requirements. No option is off the table.

Listing GitHub, and to a lesser extent GitLab, did not sit well with quite a few who commented in the thread. GitHub is closed source, while GitLab is open core—neither of which are seen as wholly compatible with Fedora. Fabio Valentini asked about GitHub and said that he did not "want to see fedora use a closed-source product for such a core service". Beyond that, he is quite happy with Pagure:

Can we please keep pagure? It already has the fedora-specific features we need, and I don't mind a "slow" pace of development. In my experience, it works really well, and I actually *like* to use it (which is not true for GitLab ... which is slow and horrible)

Michael Catanzaro also thought that GitHub should not even be under consideration, which would reduce the choice to either Pagure or the GitLab Community Edition (CE):

Well since we have a request for requirements: I propose requirements #1 and #2 are to be self-hosted and open source. I'm suspect the Fedora community would be outraged if we fail to meet either requirement.

So if we can agree on that much, then we can avoid wasting time by including GitHub in the list of options. That would bring us to a choice between GitLab CE and Pagure. (Are there any other serious options?)

There were suggestions for the lesser-known Sourcehut, Gitea, Gogs, and Phabricator forges as possible other options. Opinions differed on whether those would fit the bill, but, like any non-Pagure solution, they would certainly need to be customized for the Fedora use cases. Michal Konecny did not see that as a way forward:

If we go this way, in a few years we will end up in the same situation as with Pagure today. We will have many custom patches (which we need to take care of) and we will not have manpower to compete with the features of other major git forges.

For GitLab, there is the concern that the company will eventually go in a different direction, which could leave users of the community edition behind. As Bruno Wolff III put it:

Gitlab is open core, which means they have a conflict of interest when accepting improvements from outsiders. They could also stop supporting the open core at any time, leaving users who want infrastructure running on free software in a poor place.

Much of that discussion was not directly addressing what the CPE team is looking for, however. Clement Verna, instead, differentiated the two separate use cases for how Fedora is using Pagure today. There is pagure.io, which is a collection of various repositories for code, documentation, or simply to have a bug tracker, and the package sources repository (src.fedoraproject.org) that is more tightly integrated with Fedora services and practices. The only real requirement for the former is integration with the Fedora Account System (FAS) to allow for single sign-on, he said, while the latter has a whole host of needs:

[...] it needs to be able to integrate with Fedora FAS but also to have the FAS group synced, branch ACLs, a way to integrate with release-monitoring, a way to integrate with bugzilla, a way to integrate with fedora-messaging (RabbitMQ), .... In general I think most of the integration with our infrastructure can be done with any solution either using the solution APIs or plugins system. After we need to compare the cost of developing and maintaining these pieces of glue to integrate everything against the current situation.

Julen Landa Alustiza also split the two use cases; he agreed with Catanzaro that the solution should be open source and self-hosted, though he thought that "self-hosted" could be relaxed if there were a suitable "open source friendly service provider". He added some requirements for both use cases, including privacy and ease of onboarding, along with requirements for each individual use case. Integration with existing Fedora services, such as Bodhi and Koji, is particularly important for the distribution package source repository (which he and others referred to as "distgit").

For pagure.io, he further split the repositories into those used largely for bug tracking (perhaps containing a bit of documentation) and those that host code projects. He had specific suggestions of requirements for each. Standard bug-tracking features and documentation integration are needed for that type of repository, while the code repositories need good searching capabilities as well as a way to see at a glance which repositories are actively changing and which might be good places for newcomers to get started.

Aleksandra Federova suggested splitting off the code-review portion into its own piece. She sees the distribution source repository as a "centrally managed code-review platform". GitHub and GitLab have a workflow that is different from Fedora's; the two styles may clash:

Git Forges play a lot with the idea of users being able to create their own forks of the repository, their own projects, with their own rules. src.fp.org is the integrated platform where Fedora rules are in action. This is different from the use case KDE and Gnome have, as they manage development of projects, while we manage the integration.

Her suggestion of using Gerrit for code review was not met with much in the way of enthusiasm, but her focus was largely on the two big Git forges under consideration. For Pagure, much of the Fedora-specific work has already been done, though there is still plenty more to do. Her point is that a switch away from Pagure will need to address the impedance mismatch between GitHub/GitLab and Fedora's practices.

The discussion was somewhat unfocused; the idea of moving away from Pagure at all seems to have taken many by surprise. In addition, using the Open Decision Framework is new to some, so it was not clear what the next steps might be. Adam Saleh asked Griffin: "[...] how do we actually get a list of requirements?". Griffin said that the discussion was helping to gather that information, but that it would need to be reshaped:

This thread is serving as a source of requirements (although it has meandered dramatically away from that) but I will default to the Fedora Council for how a combined set from the input in this thread and others is collated and presented. When all requirements are gathered from all stakeholders I will share the distilled version out.

It is a bit hard to imagine Fedora moving to GitHub for several reasons: the closed-source nature of the platform and the inability to easily integrate it with the distribution's practices are two obvious ones. At first blush, GitLab CE might make a reasonable fit, though it too would need lots of customization. Given that Pagure is mostly filling the role needed at this point, and avoids the pitfalls of either proprietary or open-core code, it would seem to be the obvious answer.

Since any solution will seemingly require extra effort on the part of the CPE team, making a formal decision in favor of Pagure may make it easier for Red Hat to allocate the people and time needed. Neither of the other two possibilities would appear to provide much in the way of development and maintenance time-saving, but whether that is true will become clearer as the process progresses. It should be interesting to watch.



to post comments

Fedora gathering requirements for a Git forge

Posted Jan 29, 2020 23:49 UTC (Wed) by IanKelling (subscriber, #89418) [Link]

Unfortunately, comfort seems to win

Posted Jan 30, 2020 1:29 UTC (Thu) by mirabilos (subscriber, #84359) [Link] (11 responses)

There weirdly was no widespread argument against Debian adapting GitLab CE for their self-hosted repositories, despite the lack of the #1 communication platform (mailing lists). Shockingly, they didn’t even use the packaged version (which, it turns out, is totally shit anyway) but the pre-made “packages” from the GitLab company.

Sic transit gloria mundi.

Thankfully, I can still self-host (or, currently, use the open-ish FusionForge instance of my employer). It’s worrisome, though.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 2:28 UTC (Thu) by pabs (subscriber, #43278) [Link]

I expect Debian folks who disagreed with the GitLab switch went elsewhere or just reduced their involvement in the community. It certainly felt like a foregone conclusion so people probably felt like there was no chance to change the decision.

At the time SourceHut was not yet an option but if the decision were repeated today I think it would have been a contender.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 9:23 UTC (Thu) by philh (subscriber, #14797) [Link] (5 responses)

At the time I was in favour of Pagure.

This news proves me wrong, which is nice, as it means that Alexander made the right choice for Debian.

Given that nobody else had stepped up to the plate to do the work involved, that choice was absolutely his to make, and I applaud his good judgement.

Note that the only contenders being considered by Fedora are: GitHub, GitLab, and Pagure.

I think it's clear that GitLab is a better choice than GitHub (for Debian at least, and probably for Fedora for similar reasons).

I would guess that Pagure is only in their list now because they already use it, whereas Debian was stuck with an unsupportable fusionforge so continuing with what we had was not an option.

Complaining that the new thing does not have every feature that existed in the old one seems a bit pointless, since continuing with Alioth as it was was not an available option.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 11:15 UTC (Thu) by hmh (subscriber, #3838) [Link] (1 responses)

Well, supply chain attacks are no joke, and in active use by all sort of Actors.

GitLab is Hell to keep tight control on the supply chain, hopefully gitlab-the-company has a team dedicated to this that is forbidden to do anything else (you need to veto every changed line on every dependency, test suits are useless for this).

Hopefully Pagure is less insane on that area?

This was less of a problem when people used forges as a read-only platform as long as the code goes. But the moment you have pull request merging, it becomes trivial to hide undue changes where not many would look.

And one better really look into how the verified commit thing was implemented before trusting that thing, too.

I can use gitlab on Debian and gitlab.com safely from that kind of nastiness because I either never trust a merge or never use it as anything but a read-only depot and communication platform. And I verify all signatures locally after a clone or pull. But I can't really expect others will always do the same, neither at work nor in Debian. Can Fedora?

Debian had little choice but take that risk, but Fedora seems happy enough with Pagure right now, so they do have a second choice.

Obviously, a supply chain attack could always go after the forge-using developer's browser and local machine to try to steal keys and passwords, instead. But I expect that is a risk most of us is exposed to anyway.

And maybe the gitlab CTO will read this and tell us they do have a supply chain security team that cannot be diverted from its true purpose AND have absolute veto power on the supply chain, so you can trust they have enough human power to do their job well at all times, even when it gets in the way of "getting things done" of other teams... It would be really nice, but sounds like a hopeless dream.

Unfortunately, comfort seems to win

Posted Feb 2, 2020 7:39 UTC (Sun) by christian_couder (subscriber, #56350) [Link]

(GitLab employee working on Git itself here.)
GitLab takes security quite seriously.
There is now an application security team with 6 persons (and a vacancy, see https://about.gitlab.com/company/team/org-chart/). Their work is not limited to GitLab itself but all the stack, see for example how Joern Schneeweisz of GitLab was credited in the latest Git security release:
https://lore.kernel.org/git/xmqqr21cqcn9.fsf@gitster-ct.c...
There is also a bounty program, see: https://about.gitlab.com/blog/2020/01/14/celebrating-one-...

Unfortunately, comfort seems to win

Posted Jan 30, 2020 14:03 UTC (Thu) by Conan_Kudo (subscriber, #103240) [Link]

The irony is that this is happening in large part because the Pagure community has been slow to grow. We would likely not be having this discussion at all if other distributions had started adopting Pagure, using it, and contributing to its development.

Fortunately, most of the Fedora community does not want to see Pagure replaced, and would rather see it modestly improved for supporting their workflows, so I foresee a formal assent to allocate resources to Pagure by Red Hat's CPE team.

Part of this is coming from the fact that the CPE team "maintains" over 100 codebases comprising of hundreds of applications and web services. There have been inquiries for *every* application in a similar manner with the exception of three: Koji, Bodhi, and MirrorManager. Koji is maintained by another team, Bodhi is considered critical and irreplaceable, and there are no maintained alternatives to MirrorManager to consider (MirrorBrain stopped being developed several years ago).

The bigger surprise here was this "Open Decision Framework" that nobody knew about before now. This wasn't mentioned in previous inquiries, which has made the conversation very strange.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 18:47 UTC (Thu) by mirabilos (subscriber, #84359) [Link] (1 responses)

Perhaps not Alioth as it was, but FusionForge was still superiour to all the other choices and is changeable.

Ever tried to submit a change to an open core project? Haha… (even if the paid users of it also want it, and it’s trivial, it won’t end up even in the paid version…)

Unfortunately, comfort seems to win

Posted Feb 2, 2020 8:23 UTC (Sun) by christian_couder (subscriber, #56350) [Link]

(GitLab employee working on Git itself here.)
GitLab has on average 185 contributions from the wider community per month growing 128% per year, see: https://about.gitlab.com/community/contribute/

Unfortunately, comfort seems to win

Posted Jan 30, 2020 10:17 UTC (Thu) by lobachevsky (subscriber, #121871) [Link]

I think that change was a good thing. Debian's systems can sometimes seem arcane and the previous cacophony where most everything was hosted somewhere else and in a different VCS was exhausting. With salsa.debian.org I don't have to think about Debian specifics and can find my way around more easily. It greatly increased the times I look into the innards of specific packages and I think this would be a benefit for Fedora as well. As terrible as the Github-monoculture is, the clone that is Gitlab at least lets me look at stuff in a way that doesn't feel foreign, greatly lowering the bar compared to Pagure.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 10:51 UTC (Thu) by nodens (guest, #134696) [Link]

> Shockingly, they didn’t even use the packaged version (which, it turns out, is totally shit anyway) but the pre-made “packages” from the GitLab company.

No, they don't. They use a local fork (no modification AFAIK) and build it from source, as you can see here: https://salsa.debian.org/salsa/salsa-ansible.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 11:00 UTC (Thu) by olasd (subscriber, #64961) [Link]

salsa.debian.org doesn't use the omnibus packages provided by upstream, they use a git checkout of (a light fork of) the upstream repositories.

All deployment manifests are public and available on https://salsa.debian.org/salsa (ansible: https://salsa.debian.org/salsa/salsa-ansible, terraform: https://salsa.debian.org/salsa/salsa-terraform, salsa's GitLab "fork": https://salsa.debian.org/salsa/gitlab) if you want to double-check this assertion.

Unfortunately, comfort seems to win

Posted Jan 30, 2020 23:41 UTC (Thu) by ballombe (subscriber, #9523) [Link]

> There weirdly was no widespread argument against Debian adapting GitLab CE for their self-hosted repositories, despite the lack of the #1 communication platform (mailing lists).

It was also sad to see the 'download page' disappear.
Bye bye tarballs that does not require autoconf/automake to build.
Bye bye signed tarballs.

Fedora gathering requirements for a Git forge

Posted Jan 30, 2020 10:32 UTC (Thu) by highvoltage (subscriber, #57465) [Link] (4 responses)

The open core nature of GitLab hasn't been a problem in Debian since we only use the free bits anyway, and it's a complete solution on its own without needing any of the non-free bits to function. If that was not the case then it wouldn't have been appropriate, but the GitLab instance hosted on salsa.debian.net proved to provide even more functionality that was initially anticipated as useful (the CI runners proved so popular that the team behind it now has to figure out how to make that scale out properly). I don't think that GitLab would be a bad choice for Fedora/CentOS, it's quite possibly the best choice they could go with.

Fedora gathering requirements for a Git forge

Posted Jan 30, 2020 14:25 UTC (Thu) by Felix (subscriber, #36445) [Link]

Could you comment on the resource usage? A Fedora contributor mentioned that pagure uses much less resources than gitlab so we would need to use likely 3-5x the resources (compared to pagure).

The main issue seems to be if Fedora can integrate gitlab with their custom infra tooling (e.g. Fedora Account System, Koji/build infrastructure). Is there some kind of overview which challenges Debian faced and how these were resolved?

I think Debian uses gitlab to version all control files (not sure about the Debian terminology, the packaging "recipes"/spec files). Any issues doing per-branch or per-repo permissions (e.g. we have maintainers per package and each package is a separate repo)?

Fedora gathering requirements for a Git forge

Posted Jan 30, 2020 17:29 UTC (Thu) by m4rtink (guest, #95458) [Link]

The Fedora mailing list discussion mentions[0] that GitLab actually moved some features from the Enterprise Edition to the Community Edition to make it more suitable the Gnome Project, which did migrate to it in the end. While that's certainly nice of GitLab, I wonder how this would go with a smaller project needing GitLab EE features.

Also mentioned in the Fedora ML discussion, full text search is apparently a Gitlab EE exclusive feature[1].

I wonder if the Debian project also had to solve some of the CE vs EE edition issue ? Did GitLab also move some features from EE for Debian & what about the missing full text search ?

[0] https://lists.fedorahosted.org/archives/list/devel@lists....
[1] https://lists.fedorahosted.org/archives/list/devel@lists....

Fedora gathering requirements for a Git forge

Posted Jan 30, 2020 19:21 UTC (Thu) by IanKelling (subscriber, #89418) [Link] (1 responses)

> The open core nature of GitLab hasn't been a problem in Debian since we only use the free bits anyway

There's a lot of problems, and I bet some Debian developers are running nonfree bits. There's a ton of ways admins and users and use optional nonfree dependencies / "integrations", just a normal part of the interface tacitly encouraging users to run them. Want to contribute upstream? Good chance you need to run a nonfree recaptcha program on your computer. A few more problems: https://libreplanet.org/wiki/Fsf_2019_forge_evaluation

> it's a complete solution on its own without needing any of the non-free bits to function

Not in my evaluation. Its a known target of spambots and the recommended solution of upstream is to run nonfree software to stop them. When you say it works "on it's own", its arguably misleading people into giving up their privacy, because salsa.debian.org sent my private web browsing information to gravatar without my consent. As far as I can tell, the free users are treated as beta testers, their "complete solution" is clearly documented and advertised, including in the salsa.debian.org ui: give up your freedom.

Fedora gathering requirements for a Git forge

Posted Feb 8, 2020 1:41 UTC (Sat) by notriddle (subscriber, #130608) [Link]

> Not in my evaluation. Its a known target of spambots and the recommended solution of upstream is to run nonfree software to stop them.

Fedora already uses Wordpress (https://fedoramagazine.org/ source code mentions the Yoast SEO plugin in its source code, which is a Wordpress plugin) and MediaWiki (https://fedoraproject.org/w/index.php?title=Fedora_Projec... is extremely familiar). You are not going to tell me that GitLab has a worse spam problem than those things do.

These apps already provide reCAPTCHA plugins. Fedora, instead, uses this thing <https://admin.fedoraproject.org/accounts/user/new>. They already said that if they adopted GitLab, they'd tie it into their existing SSO service, so it'll be the same thing here.

> As far as I can tell, the free users are treated as beta testers, their "complete solution" is clearly documented and advertised, including in the salsa.debian.org ui: give up your freedom.

If Pagure reaches significant adoption, it'll be targeted by spam bots anyway. Building non-reCAPTCHA spam filters on top of GitLab still sounds a lot easier than building non-reCAPTCHA spam filter *plus the entire rest of the Git Forge functionality*.

Fedora gathering requirements for a Git forge

Posted Jan 30, 2020 15:39 UTC (Thu) by pj (subscriber, #4506) [Link]

There's an Apache-Licensed github clone called 'gitbucket' available at https://github.com/gitbucket/gitbucket

Fedora gathering requirements for a Git forge

Posted Jan 31, 2020 3:25 UTC (Fri) by raof (subscriber, #57409) [Link] (2 responses)

Launchpad!.

Only partially in jest. It is a git forge (along with bzr, should you need that ;)), and it (unsurprisingly!) has a whole bunch of features which are useful for distros' specific use-cases.

Fedora gathering requirements for a Git forge

Posted Jan 31, 2020 12:42 UTC (Fri) by m4rtink (guest, #95458) [Link]

But can you actually deploy it yourself ? IIRC, last time someone tried to self host Launchpad, they found that the graphical theme of the web interface had some weird licensing and there were no public deployment scripts (paraphrasing what I have been told quite a while ago from memory).

I guess things could have been improved since then?

Fedora gathering requirements for a Git forge

Posted Feb 2, 2020 1:32 UTC (Sun) by jzb (editor, #7867) [Link]

If my memory serves, Canonical slow-walked releasing Launchpad in a state that it could be stood up by other communities -- preferring to try to get major projects to host their projects on Launchpad to make it the popular watering hole for FOSS projects and trying to find a revenue stream. It was, ahem, launched in 2004, and source wasn't released until 2009. (See one of Shuttleworth's responses to criticism about lack of source.)

The other problem being, unless something has changed, that Canonical requires use of the so-called "Harmony" CLA to contribute to its projects. It's highly unlikely that many folks in the Fedora community are going to be eager to adopt Launchpad as long as that would require signing the Harmony CLA to contribute upstream.


Copyright © 2020, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds