LWN: Comments on "Fedora's Git forge decision"

Fedora's Git forge decision & CentOS

rahulsundaram — Thu, 16 Apr 2020 17:46:21 +0000

RPM is not part of Fedora infrastructure. What was eliminated is the consideration of Github for hosting Fedora package sources. Whether these independent upstream projects want to move over is to them

Fedora's Git forge decision & CentOS

augustz — Thu, 16 Apr 2020 17:19:16 +0000

Its interesting that they immediately eliminated github from consideration, but aren't things like

https://github.com/rpm-software-management

all the ansible stuff

etc. is this going to be a force migration to gitlab or whatever tool they end up using?

GitHub

robbe — Tue, 07 Apr 2020 20:50:30 +0000

> Also, don't forget being able to scale up and down on demand.

Yes, Public Cloud has advantages there, but …

> (COVID-19 anyone?)

… this is a *very* bad example. You are aware that MS Azure is under severe stress for the last weeks, to the point that starting new instances was not possible for days on end?

Fedora's Git forge decision

flussence — Sat, 04 Apr 2020 23:30:01 +0000

Some variation on "body { display: revert !important; visibility: revert !important }" should work as a general catchall.

("revert" is a recently introduced keyword that tells the browser to discard the style cascade and use its default value - extremely useful in user stylesheets, but unfortunately it doesn't work on form controls.)

Fedora's Git forge decision

simlo — Fri, 03 Apr 2020 17:51:47 +0000

Now GitLab is "open core". If these features are that important for the open source community in large, someone ought to fork GitLab community edition and maintain those features.

Single Sign On for instance is in (LDAP), but no support for using groups, so users have to be granted access inside GitLab, and not by their group membership in LDAP. That seems an easy thing to fix, if just there were an upstream project, which would accept such changes.

Fedora's Git forge decision

jafd — Fri, 03 Apr 2020 17:46:17 +0000

Explain how this is "harmful".

Do you expect news headlines like "IBM stock plummets after a nobody in an internet comment speculates about a possible backstage deal with Gitlab"?

I'm not playing dead. I'm reminding you my opinion is not something you can hope to collect damages from.

I only point at the usual visible earmarks of a duck. It quacks like a duck, walks like a duck. I'm going to bookmark this story to see what happens in a year.

Fedora's Git forge decision & CentOS

cdamian — Fri, 03 Apr 2020 08:38:44 +0000

I find the CentOS blog post about this also more helpful then the one from Fedora:

CentOS: Git Forge decision
https://blog.centos.org/2020/03/git-forge-decision/

Fedora's Git forge decision

AdamW — Fri, 03 Apr 2020 03:31:24 +0000

For the record, as one of the people questioning the decision making process on devel@ quite hard, I absolutely do not believe any kind of kickback or other corrupt practice is involved here, and I absolutely do not believe anyone involved in the process is acting in bad faith. There's nothing to indicate anything of the sort at all.

Fedora's Git forge decision

raven667 — Thu, 02 Apr 2020 17:49:34 +0000

> > > I'm not telling Gitlab gave a kickback to whoever was behind the decision, but it sure looks that way.
> > This quite unfair and harming.
> Man, it's just an opinion from an unimportant nobody. Just how it can harm anyone, I don't see. I'm just telling what it looks like to me, personally.

It's funny how quickly one goes from sharing an opinion accusing someone of corruption, getting the mildest pushback of someone else sharing their opinion that this is likely BS, to rolling over and playing dead where you can't be criticized for what you say because you are an "unimportant nobody".

Obviously this decision has annoyed some people who weren't involved in the process, although the result seems...fine, GitLab is used by Debian and GNOME, although locally and not the hosted version, so it should work, the main consideration is whether to use the hosted version that includes non-free features, or stick with the free codebase that the other major projects use. Maybe IBM/RedHat is thinking of purchasing GitLab and can then resolve the issue by donating the hosting, opening up the code or whatever, the people making the decision don't seem to be corrupt or crazy, so a difference of opinion just reflects some different assumptions in the decision making process. I mean, some group of people of influence in Fedora came to this conclusion after thinking about it, dismissing that as the result of "kickbacks" instead of engaging with the substance of the decision isn't really helpful in understanding the real reasons why this decision were made, how and by who.

Fedora's Git forge decision

jafd — Thu, 02 Apr 2020 16:14:50 +0000

> This quite unfair and harming.

Man, it's just an opinion from an unimportant nobody. Just how it can harm anyone, I don't see. I'm just telling what it looks like to me, personally.

This is also not about *what* they decided, but about *how* they did it. Gitlab, Github, whatever. I understood that transparency in all processes was one of their core values, and it turned out core values can be shoved aside arbitrarily, then a decision announced via a decree, with a "sorry not sorry" responses. Well, okay. Good to know.

For all I know, this might be a practical decision (or it might be not. People got burned with clouds in the past.), I don't question their pragmatism, I question their integrity, which is kind of natural to do when something happens at a company which contradicts some of their declared core values.

per this message: https://lwn.net/ml/fedora-devel/20200330120105.GE4643@sha...

> That "evaluation" and (even the "greater context") seems to have gone
> completely missing here, and no, whatever went on in a disused lavatory
> behind a sign that says "beware of leopard" doesn't count.

Fedora's Git forge decision

pizza — Thu, 02 Apr 2020 15:15:37 +0000

> IBM/Redhat is biggest (only?) sponsor of Fedora infrastructure team,

It's worth noting that historically, RH has strongly resisted outside infrastructure sponsorship.

Meanwhile, if "infrastructure costs" are the actual underlying problem here, the actual costs (including NRE involved in migration!) need to be quantified so possible solutions can be evaluated with that in mind.

Fedora's Git forge decision

zdzichu — Thu, 02 Apr 2020 14:47:12 +0000

This quite unfair and harming. It is far more likely that IBM management is looking where to cut costs and going proprietary looks like a sensible business decision to them.
IBM/Redhat is biggest (only?) sponsor of Fedora infrastructure team, and the shareholders expect money to be spend sensible. Discriminating between free/proprietary may be important for LWN audience, but is not important for general audience.

Fedora's Git forge decision

jafd — Thu, 02 Apr 2020 14:29:55 +0000

That thread was an amusing read. Usually things like this fly around when a corrupt government makes a shady deal with a vendor, bypassing the usual procurement process, journalists start asking inconvenient questions, and officials offer lame excuses and "anyway, the deal is done, so nothing is going to be done about it anyway".

I'm not telling Gitlab gave a kickback to whoever was behind the decision, but it sure looks that way.

GitHub

pizza — Wed, 01 Apr 2020 13:32:39 +0000

> I tend to disagree, working for a multinational company that heavily uses cloud services.
> TCO when you take in account everything [...] tend to be *much* higher on-premise for non-core services.

My own experience with companies of that size is that their internal processes and bureaucracy are responsible for the bulk of that difference in TCO -- and you will end up with the overhead of managing it regardless.

That "offload non-core services to save short-term cash" attitude also is the driving force transition of the IT department from a group that says "How can we help you do your job better?" into the "Computer Says No... <cough>" malaise that actively hinders the few remaining folks that *create* value.

> Also, don't forget being able to scale up and down on demand. If you're on premises and you need e.g. more servers for telepresence if everybody is forced to work from home (COVID-19 anyone?) it might take weeks to compensate.

My organization's "cloud-hosted" services have gone down multiple times since this pandemic started, and that goes triple for our barely-adequate-for-normal-situations VPN service. It turns out all of our cloud provider's other customers are in the same "need to scale externally, fast" boat, so it's taking weeks to compensate anyway.

GitHub

tchernobog — Wed, 01 Apr 2020 12:52:28 +0000

I tend to disagree, working for a multinational company that heavily uses cloud services.

TCO when you take in account everything (running security updates, ensuring high availability, paying for electric bills, replacing faulty hardware, accounting for redundancy, complying with international and local legal requirements...) tend to be *much* higher on-premise for non-core services.

We host code on-premise because of the fear of leaking private data, as with other services that require stringent security (payroll, HR info, etc.)

But we put everything else on the cloud because we've shown benefits in the ballpark of dozens of millions per euros per year. The return of investment took around 3 years to materialize, but after that break-even, we only got better.

Also, don't forget being able to scale up and down on demand. If you're on premises and you need e.g. more servers for telepresence if everybody is forced to work from home (COVID-19 anyone?) it might take weeks to compensate.

Fedora's Git forge decision

brunowolff — Wed, 01 Apr 2020 08:18:49 +0000

I partially figured this out. But it looks like a custom entry will be needed for each web site rather than something general. So I'll need to be really interested in a web site to bother. Right now there is just one.

Fedora's Git forge decision

pizza — Wed, 01 Apr 2020 00:35:42 +0000

Several of the features the CPE team said were deciding factors are only available in the highest-tier "Gitlab Ultimate" edition. Neal Gompa's initial email linked in TFA enumerates many of them, and the thread has grown considerably since.

Fedora's Git forge decision

Cyberax — Wed, 01 Apr 2020 00:33:25 +0000

No integration with single-sign on, as far as I remember.

Fedora's Git forge decision

simlo — Tue, 31 Mar 2020 23:39:45 +0000

Who couldn't they use the Community edition, which Iunderstand as being open sources?

Fedora's Git forge decision

brunowolff — Tue, 31 Mar 2020 19:32:45 +0000

Is there a way to override that with a local style sheet (for say Icecat)?
I

Fedora's Git forge decision

flussence — Tue, 31 Mar 2020 18:38:19 +0000

It was popularised by Google AMP as a way to force people to turn on Javascript so that they may have their eyeballs tracked and harvested — though at least Google had the mercy to remove display:none with a CSS timeout.

GitHub

ibukanov — Tue, 31 Mar 2020 17:34:58 +0000

I thought it was a common knowledge. For example on my previous job we needed to plan for backing up roughly 20 terabytes of data. It turned out that the cheapest option was to get a tape drive and enough tapes. The price of that including all backup media matched roughly paying for 6 months of the cheapest online cloud provider. Plus we would need to upgrade the network line into the office to ensure the data availability withing reasonable amount of time.

GitHub

epa — Tue, 31 Mar 2020 17:23:04 +0000

Indeed, and if the engineering resources are so scarce, why waste them on running an in-house instance of some proprietary software? Better to have it hosted by GitLab themselves (who would surely offer a generous discount) and free up the overworked Fedora sysadmins for something more useful.

GitHub

pizza — Tue, 31 Mar 2020 16:58:46 +0000

You may be correct with respect to purely "host internally" versus "pay an external provider to host it", but that's not the choice being talked about here.

Instead, this is being framed as a choice between "commit significant development/engineering resources to improve an in-house solution so it can meet our needs" versus "pay an external provider to use their off-the-shelf solution to meet our needs"

The migration and customization costs for the latter are invariably (and grossly) underestimated, and that's before one considers the value of keeping critical expertise in-house, especially for an organization that purports to be all about that expertise..

GitHub

tome — Tue, 31 Mar 2020 16:21:52 +0000

Interesting, I didn't know that. Can you cite some sources?

Fedora's Git forge decision

ballombe — Tue, 31 Mar 2020 16:19:31 +0000

> What kind of idiot web developer inlines display:none on the body of a html document?

Apparently this is a new fad:
https://www.parrotsec.org/
does the same

Fedora's Git forge decision

misc — Tue, 31 Mar 2020 15:48:25 +0000

Mhh, isn't hackmd.io free software ?

https://github.com/hackmdio/codimd

GitHub

ibukanov — Tue, 31 Mar 2020 13:14:50 +0000

Typically it is significantly more expensive to use a cloud service than to host the software on self-administered hardware if the organisation has access to competent sysadmins.

GitHub

epa — Tue, 31 Mar 2020 11:28:21 +0000

Indeed, why don't they just host it on GitLab's paid-for service? If you're not running free code you may as well save yourself the bother of hosting it yourself. That also reduces the risk that Fedora admins will be contaminated by having seen non-free source code should they later work on a free replacement.

Fedora's Git forge decision

dottedmag — Tue, 31 Mar 2020 10:11:00 +0000

What a train-wreck.

"It was not our place to question valid use cases or requirements from our stakeholders"

I'm sorry, but this is not a way to do engineering.

GitHub

swilmet — Tue, 31 Mar 2020 08:41:46 +0000

> a not-fully-free version of the GitLab offering

But, is it worse than hosting projects on GitHub?

Many FOSS projects are hosted on GitHub and people find it just fine. Then when it comes to the proprietary version of GitLab, it raises some eyebrows.

For example rpm, dnf etc are hosted on GitHub:
https://github.com/rpm-software-management

Fedora's Git forge decision

bmur — Tue, 31 Mar 2020 01:25:42 +0000

You're exactly right. What kind of idiot web developer inlines display:none on the body of a html document? They are intentionally making it inaccessible.

As someone who has never ventured beyond redhat/centos/fedora style distros, (and even survived the Miguel deIcaza days) it is probably time to at least look at what others are doing.

Fedora's Git forge decision

pizza — Tue, 31 Mar 2020 01:15:39 +0000

Gitlab's Use of "nonfree javascript" is the least of the problems with this decision, because the underlying Gitlab service that CPE will rely on is itself proprietary.

(They're not going to be using "Gitlab Community Edition")

Fedora's Git forge decision

IanKelling — Tue, 31 Mar 2020 00:39:41 +0000

Edit: I shouldn't have said "given up", more like, not valued nearly as highly as it could be.

Fedora's Git forge decision

IanKelling — Tue, 31 Mar 2020 00:37:28 +0000

Even the user stories collected for this got posted to a webpage which renders markdown using nonfree javascript and displays a blank page otherwise https://hackmd.io/@My419-DISUGgo4gcmO1D6A/HJB74lcL8. If you view source, you can view the raw markdown. I'm sure Fedora ships probably 4 different free markdown webpage rendering implementations. Perhaps the foundational value of "choose free alternatives" was effectively given up some time ago.

Fedora's Git forge decision

IanKelling — Tue, 31 Mar 2020 00:23:19 +0000

Very interesting that the Fedora announcement calls it a "foundational service", but there is no mention of the fact that going for a proprietary option is a regression with respect to "Fedora’s Mission and Foundations" https://docs.fedoraproject.org/en-US/project/, for example: "We choose free alternatives to proprietary code"

With this, presumably a normal part of contributing to Fedora will mean downloading nonfree javascript parts of gitlab and running it in your browser.