Rethinking Fedora's compiler policy [LWN.net]

By Jonathan Corbet
April 28, 2021

Now that the Fedora 34 release is out the door, the Fedora project is turning its attention to Fedora 35, which is currently scheduled for release on October 26. One of the changes under consideration for Fedora 35 is this proposal allowing maintainers to choose whether to build their packages with GCC or Clang. This policy change may give maintainers some welcome flexibility, but it has not proved entirely popular in the Fedora community.

It is a longstanding practice for distributors to pick a specific toolchain to be used to build the full set of packages. Mixing compilers and linkers always presents the possibility of incompatibilities leading to subtle bugs, which is the sort of outcome distributors tend to go out of their way to avoid. Adhering to that practice, Fedora has long used GCC as its compiler of choice — a practice dating back to the time when there were no other compilers to use in any case.

Some packagers prefer LLVM

Over the years, though, the LLVM compiler suite and its Clang C compiler have grown in capability and become more popular. There are some programs that will not build with anything else at this point. For such programs, Fedora allows Clang to be used; for everything else, though, Fedora's policy requires building with GCC. This, according to the change proposal, creates unneeded difficulties for Fedora maintainers:

Upstream, the Firefox project builds primarily with Clang/LLVM. Yet we force the Fedora package owner to find and fix issues building with GCC then either carry those custom fixes forward in Fedora or negotiate with upstream to get those changes upstreamed. While this process can be helpful in finding non-portable code, this is ultimately a poor use of the packager's time.

The new proposed policy states that maintainers could use a non-default compiler for their package if they have a "valid technical reason" to do so. Beyond avoiding the above-mentioned Firefox problem, this change, it is argued, will enable the use of the compiler that yields the best result for any given package, thus improving the user experience overall. Beyond that, building with the compiler an upstream project uses for its own development and testing is, perhaps, less likely to tickle compiler-related bugs.

The real motivation behind this proposal, though, might be found in places like this RHEL bug report. The Ruby language has a just-in-time (JIT) compilation feature, and users would like to take advantage of it. But, due to GCC, when built as a position-independent executable, being incompatible with pre-compiled headers, JIT compilation fails with an unhelpful error message. There are various ways that this problem could be addressed; the morbidly curious can find them in the bug report. The conclusion, though, is that none of the options for making this feature work with GCC are deemed acceptable. If, however, LLVM were used, the problem would go away. Thus, unsurprisingly, the Ruby developers would like the freedom to use LLVM when building the Ruby package.

Questioning the change

The first question to arise in this posting of the proposal (it has actually been under discussion for nearly a year) was: who decides what's a "valid technical reason"? Will there be some sort of committee review? The consensus would appear to be that maintainers will be trusted to make the best decisions for their packages. The current version of the proposal differs from the original, which said maintainers should use the compiler that is preferred by the upstream project they are building.

The first posting of the new compiler policy (targeted at Fedora 33) drew a number of complaints, many of which resurfaced this time around. Neal Gompa described it as "an excuse to avoid doing the right thing and leveraging the toolchain that offers the highest quality code generation" and said that, at a minimum, the justifications for this change needed to be spelled out more clearly. Jakub Jelinek (a GCC developer) argued that subtle ABI incompatibilities still exist between the two compilers, and that mixing them increases the chance of introducing difficult bugs.

Mark Wielaard worried about the extra workload for developers of other parts of the toolchain:

For the packages I work on, elfutils, valgrind, debugedit, etc. we have enough trouble keeping up with gcc versions, new flags, optimizations, etc. I don't think we will have time to debug issues because packages are now also built with another compiler. Leaving the system as a whole in a worse state because of it.

Kevin Kofler argued that the best compiler overall should be used for the entire distribution — and that GCC is the best compiler for now.

A few participants, including Wielaard (linked above) and Florian Weimer, pointed out that even if Fedora adopts Clang when building projects that prefer it, the Fedora build will still differ in many ways, including the specific compiler version, the compilation flags used, and the setup of the system as a whole. The end result may well not be a better-tested build environment than was had using the default compiler.

Compiler diversity

One other concern mentioned by Jelinek in the above-linked message merits a mention. He agrees that competition between toolchains is good for everybody involved. While this proposal, on its surface, would appear to facilitate this competition by allowing the best compiler to be used everywhere, he says, the end result is more likely to be an increase in the number of packages that can only be built with LLVM. Upstream projects, especially those dominated by a single large company, often do not have portability across compilers as one of their important objectives. If distributors do not insist on that portability, it may well go away.

This would appear to be a valid concern. The growth of the Chrome-only web shows how this sort of capture can take place. GCC's slow rate of progress in the years before LLVM took off, and the acceleration of development since, highlight the value of this sort of competition. A return to a single-compiler world is not in anybody's interest.

Distributors, being the organizations that wield compilers for the benefit of vast numbers of users, are an obvious place to try to draw a line against a compiler monopoly. Whether distribution policies are the best way to promote compiler diversity is a different question, though. There may come a point where the pain of building with an out-of-favor compiler becomes more than people are willing to deal with, especially in volunteer-driven distributions. The apparent imminent end to Java support in Fedora shows what can happen when the work gets too overwhelming. Burning out package maintainers in the name of portability does not seem like a good way to achieve that goal.

The outcome of this particular change proposal is unclear at this point. Even if it is not adopted this time around, though, it seems likely that the pressure to make that sort of change will only increase in the future. Unless the forces that are driving projects to only support one compiler are somehow reduced, the cost of going against the stream will only grow.

to post comments

Rethinking Fedora's compiler policy

Posted Apr 28, 2021 17:08 UTC (Wed) by Bigos (subscriber, #96807) [Link] (14 responses)

In my opinion, the GCC compiler has shown a lot of progress in user-friendliness (diagnostics, compilation speed, integration with tools, etc.) and overall pace of development in response to Clang. I don't think it is going away anytime soon. Thus justifying the enforcement of use of GCC in Fedora as supporting compiler diversity sounds pretty strange. I would say it is backwards - allowing packages to use any compiler would make it more likely the code bases would be portable across compilers (like the Linux kernel that has gained mainstream support for Clang compilation relatively recently).

If there are issues with ABI incompatibility they should be found, fixed, documented and then continuously tested. This would help not only GCC <-> Clang interoperability but also compiler interoperability between different versions. This probably involves some custom compiler flags (like the -m* flag family) with different settings and defaults across compilers, so I understand it is not an easy job (and probably would involve the distribution packagers).

TL;DR; If GCC wants to keep itself relevant it should keep improving the compiler (and I believe the community is doing a fantastic job at that!) rather than try to enforce its use in a Linux distribution.

Rethinking Fedora's compiler policy

Posted Apr 28, 2021 17:28 UTC (Wed) by zyga (subscriber, #81533) [Link] (11 responses)

I agree with everything except for the extra cost. The motives are clear and noble but is everyone on board? This _will_ put more burden on some people. I think that apart from any technical arguments one needs to consider the cost and decide if we have the capacity and volunteer time to make that happen.

Rethinking Fedora's compiler policy

Posted Apr 29, 2021 1:58 UTC (Thu) by josh (subscriber, #17465) [Link] (10 responses)

It sounds like a tradeoff of burdens. People trying to build software that's designed to build with clang/LLVM will have a lower burden, people who benefit from compiler uniformity will have a higher one.

Rethinking Fedora's compiler policy

Posted Apr 29, 2021 13:51 UTC (Thu) by nim-nim (subscriber, #34454) [Link] (8 responses)

It’s the same trade-off as with Java packages.

Upstreams do not want to handle multiple build scenarii. So they push for “simplification” by arguing everyone should make the same build choices as them (same compiler, same dep versions, ignoring the large overlaps between projects with different preferences, or the need of security patches).

Ultimately, since upstreams do no agree on what the best build choices are, adopting upstream decisions pushes the complexity down to the person that needs to build the parts used by projects with different preferences. And those people burn out, with cascading effects, till there is little left distro side.

But, do not fear, say upstreams. We can not cope with different compilers and deps to build our own project. But we are good for building and maintaining a whole system container to host our project independently!

Yes, right. When you are Google or Amazon, perhaps. Others need to care about the commons.

Rethinking Fedora's compiler policy

Posted Apr 29, 2021 21:27 UTC (Thu) by josh (subscriber, #17465) [Link]

I've seen plenty of projects that only support building with GCC, because they use GCC extensions. And ultimately, that doesn't tend to get fixed by patching the project, it tends to get fixed by adding the requisite feature to clang.

Until recently, the Linux kernel was one of those projects. People have put extensive work into making the Linux kernel work with clang. That work includes people adding features to clang, and people committing to ongoing maintenance of clang support in the Linux kernel. And it does add complexity to the Linux kernel; that complexity is just deemed worthwhile because clang adds useful features as well.

So, sure, if you have the time and bandwidth to work with upstream and commit to helping upstream maintain support for another compiler, and upstream is willing to maintain the complexity added by supporting another compiler, then it might be reasonable downstream to use that compiler. If you don't have that time or bandwidth, you aren't prepared to commit to maintaining it upstream, or it adds enough complexity that upstream isn't willing to maintain it, then it might not be reasonable to expect.

The same arguments apply to supporting less common architectures, or operating systems. Someone has to not only do the work, but maintain the work, and upstream has to be willing to deal with any added complexity.

Rethinking Fedora's compiler policy

Posted May 1, 2021 20:03 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (6 responses)

Realistically, upstream is going to do what is best for upstream, and downstream will then do what is best for downstream. If upstream only supports compiler (or build toolchain, container system, etc.) X, and downstream would prefer to use compiler Y, then downstream has a limited set of options:

1. Suck it up and use X anyway.
2. Convince upstream to support Y. In practice, this probably involves directly helping upstream to maintain support for Y, rather than merely filing bugs whenever Y breaks.
3. Fork or patch the package to support Y.
4. Drop the package.

Although #2 is a possibility, upstream has an absolute right to refuse it, and it is not very productive for downstream to complain about this choice once it has been made. Therefore, it's probably a Good Idea for downstream to have a standardized process for choosing between options 1, 3 and 4. That does not mean that every package should get the same choice, of course, just that there should be a process for making this decision (other than "everyone gets on the mailing list and argues for three days").

Rethinking Fedora's compiler policy

Posted May 1, 2021 22:28 UTC (Sat) by rodgerd (guest, #58896) [Link] (5 responses)

Three imposes huge and unreasonable work on maintainers, and 4 is holding the distro's users hostage.

Really, if people choose not to use a given (compiler, init system, whatever), trying to use distro terrorism to force them to is admission of abject failure.

Rethinking Fedora's compiler policy

Posted May 1, 2021 23:08 UTC (Sat) by pebolle (guest, #35204) [Link]

> distro terrorism

How to top this? Non of distro nazism, distro communism, distro neo-liberalism, distro colonialism really work for me. So it seems you coined a winner here.

Thanks for making free software look even more like a lunatic fringe!

Rethinking Fedora's compiler policy

Posted May 2, 2021 1:27 UTC (Sun) by NYKevin (subscriber, #129325) [Link] (3 responses)

> Three imposes huge and unreasonable work on maintainers, and 4 is holding the distro's users hostage.

So what?

Upstream produces tarballs. Downstream can take them or leave them. If upstream is so inclined, it may voluntarily decide to make changes to accommodate downstream's wishes. But there is no general obligation for upstream to do this, and it would be quite absurd if there was (consider the number of different distros out there). If downstream responds to this reality with hostility and accusations of "holding users hostage," then it is entirely reasonable for upstream to killfile downstream's emails.

Rethinking Fedora's compiler policy

Posted May 2, 2021 10:55 UTC (Sun) by Wol (subscriber, #4433) [Link] (1 responses)

But if, say, downstream relies on llvm (which is not compatible with GCC?), then upstream is basically saying "we don't want you on our distro". Which in the long run is the distro cutting off its nose to spite its face.

If people want to use a different compiler because it supports features GCC can't - or in certain circumstances WON'T - support then you have all the makings of a nasty stand-off down to the distro's intransigence.

I think this is what's getting peoples noses out of joint - the distro forcing technically bad choices onto downstream.

Cheers,
Wol

Rethinking Fedora's compiler policy

Posted May 2, 2021 15:27 UTC (Sun) by pizza (subscriber, #46) [Link]

> I think this is what's getting peoples noses out of joint - the distro forcing technically bad choices onto downstream.

And vice-versa.

Rethinking Fedora's compiler policy

Posted May 2, 2021 14:56 UTC (Sun) by jccleaver (guest, #127418) [Link]

> Upstream produces tarballs.

If only... :/

Rethinking Fedora's compiler policy

Posted May 1, 2021 23:16 UTC (Sat) by amacater (subscriber, #790) [Link]

Seeing the work on reproducible builds in Debian, SuSE and elsewhere - having multiple compilers and running as much code as you can through at lest LLVM and GCC seems eminently sensible. This just one example: building on multiple machine architectures almost requires more than one compiler to be performant.

Rethinking Fedora's compiler policy

Posted Apr 28, 2021 17:32 UTC (Wed) by jwakely (subscriber, #60262) [Link]

> If GCC wants to keep itself relevant it should keep improving the compiler (and I believe the community is doing a fantastic job at that!) rather than try to enforce its use in a Linux distribution.

GCC isn't trying to enforce anything in Fedora. The fact that some GCC developers (who are also Fedora developers) have a particular point of view doesn't mean that "GCC" (as a community or a project or anything else) is trying to do anything here.

Rethinking Fedora's compiler policy

Posted Apr 29, 2021 14:59 UTC (Thu) by Paf (subscriber, #91811) [Link]

“ I would say it is backwards - allowing packages to use any compiler would make it more likely the code bases would be portable across compilers (like the Linux kernel that has gained mainstream support for Clang compilation relatively recently).”

This is addressed in the article. In practice, projects below a certain size will often gravitate towards a single compiler. Upstream will only work with that one, and if downstreams don’t do anything, they’ll move towards that. It’s just easier.

The kernel is more than large enough to generate its own gravity and suck people towards it - Clang is changing so it can build the kernel. And, yes, the kernel is being adjusted to clang support, but that’s because communities of clang users want to build the kernel with it - the kernel is pulling *them* in.

This doesn’t happen with smaller projects. If the project isn’t big enough to drive interest from communities that want specific compilers, it will take the path of least resistance. So if it gets hard to build it with a second compiler, they’ll probably just shrug.

Rethinking Fedora's compiler policy

Posted Apr 28, 2021 18:35 UTC (Wed) by mss (subscriber, #138799) [Link] (2 responses)

Clang used to produce a binary that's slightly slower at run time than when the same code was built by GCC.
Is it sill true in 2021?

On the other hand, I can still see a significant advantage that Clang has over GCC in terms of build speed and compiler memory usage when compiling larger C++ codebases like Firefox/Thunderbird or Chromium.

Rethinking Fedora's compiler policy

Posted Apr 28, 2021 20:24 UTC (Wed) by eru (subscriber, #2753) [Link] (1 responses)

You can probably get either result, depending on what you are compiling, and whether you have picked the best flags for that particular source and compiler. Comparing compilers fairly is hard.

Rethinking Fedora's compiler policy

Posted Apr 30, 2021 18:20 UTC (Fri) by JoeBuck (subscriber, #2330) [Link]

We are fortunate to have two strong compilers; the competition between them has made them both better.

As for what distros should do, I think library packages need to be built with one compiler, but for non-libraries it's less critical. However, it's always helped GCC that, before a new major version is released, someone or someones try building entire distros with the new compiler, to shake out bugs in both and to help produce the porting notes (sometimes additional strictness flags problems with code that built with the previous compiler release but perhaps shouldn't have).

Rethinking Fedora's compiler policy

Posted May 6, 2021 4:28 UTC (Thu) by SomeOtherGuy (guest, #151918) [Link] (14 responses)

Wasn't there a problem recently with a Python cryptography package not being build-able by GCC (because Rust?) so some weird architectures are isolated as LLVM doesn't support them?

I know Fedora is not Debian but do they care?

Rethinking Fedora's compiler policy

Posted May 6, 2021 4:29 UTC (Thu) by SomeOtherGuy (guest, #151918) [Link] (13 responses)

Woo my comment didn't get eaten this time. Don't you love it when you put effort and research into a response only for it to never ever show up. Awesome.

Rethinking Fedora's compiler policy

Posted May 6, 2021 4:44 UTC (Thu) by njs (subscriber, #40338) [Link] (12 responses)

Fedora already allows LLVM to be used for packages that *can't* be built with GCC. Currently that includes all Rust packages.

Rethinking Fedora's compiler policy

Posted May 6, 2021 22:50 UTC (Thu) by SomeOtherGuy (guest, #151918) [Link] (11 responses)

Encouraging both if not upstream would obviously improvement (eg upstream uses X, allowing X and at the maintainers' discretion, Y to be used) could only improve matters.

I don't see how a maintainer exclusively using Y would be different if upstream uses X - provided they cooperate?

I would however not like to see this arch split get worse

Rethinking Fedora's compiler policy

Posted May 7, 2021 0:22 UTC (Fri) by pabs (subscriber, #43278) [Link] (10 responses)

There are efforts towards combining Rust & GCC:

https://github.com/Rust-GCC/gccrs
https://github.com/antoyo/rustc_codegen_gcc
https://github.com/sapir/gcc-rust/

Rethinking Fedora's compiler policy

Posted May 7, 2021 21:50 UTC (Fri) by SomeOtherGuy (guest, #151918) [Link] (9 responses)

Woo github links, I'm sure they're worth depending on

Really?

Posted May 7, 2021 21:56 UTC (Fri) by corbet (editor, #1) [Link] (6 responses)

What, exactly, would be the point of a comment like this?

It's not hard to understand the concerns around proprietary hosting operations like GitHub; I suspect the person you are replying to understands them rather better than you (or I) do. But there is a lot of useful code hosted on GitHub. What is the value in heckling somebody trying to point to some of this code? Please don't do that here.

Really?

Posted May 8, 2021 1:31 UTC (Sat) by SomeOtherGuy (guest, #151918) [Link]

Pabs replied below and pretty much covers it, but there's also the fact that by project number only 99.9% of github is crap

Sure there probably is some stable and reliable software I use every day on there and it may even be the primary mirror, but we should not go "ooh a github link - legitimacy" - as the other guy covers

Really?

Posted May 8, 2021 1:38 UTC (Sat) by SomeOtherGuy (guest, #151918) [Link]

I also offer you this corollary:

(Assuming the projects are relevant and could help Python:)

IF they were dependable and great projects THEN the Python lot would surely use them and give that crypto library to the masses

The contrapositive (logically equiv)

IF the Python lot are not using those THEN they are not dependable and great projects

A little inaccuracy saves a lot of explanation, but you get the idea

Really?

Posted May 8, 2021 1:59 UTC (Sat) by SomeOtherGuy (guest, #151918) [Link] (2 responses)

At the risk of earning the ire of #1 there, I proffer one more thing

Ancient paper by Denis Ritchie I believe (one of them) an essay on trusting trust, in it he builds a toolchain which is able to hide a back door (or some such) into the compiler, that passes itself on to new compilers and is invisible (as they are also adjusted) by nm, and whatever the old version of objdump and friends are

If we were allowing Rust into our computers I would expect no less caution from the Python developers especially on cryptography.

I hope this makes my case of why my sarcastic "oh a github link" comment was justified - we should not use that as a credential

Reference to "Trusting Trust" (WAS: Really?)

Posted May 9, 2021 1:56 UTC (Sun) by josephrjustice (guest, #26250) [Link] (1 responses)

Ken Thompson.

"Reflections on Trusting Trust", his lecture when he was presented with the Turing Award.

Here's a link to an electronic version of the written copy of the lecture: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_... . If one doubts that the electronic copy is an accurate reproduction, I expect that printed copies of the original academic journal it was published in can be found at most reputable academic libraries (at least in English-speaking countries).

This link was the first hit when I just now did a Google search on the phrase "trusting trust" (not including the quotes).

You're welcome.

Reference to "Trusting Trust" (WAS: Really?)

Posted May 10, 2021 22:29 UTC (Mon) by SomeOtherGuy (guest, #151918) [Link]

That was it - TBH I figured we'd all heard about it - so yeah pulling compiler stuff from github seems less wise than "try this wordpress plugin"

Honestly the thing that surprised me was how easy it was for the thing to pass itself off in future compilers, that spooked me a bit, I figured "sure yeah do that" - but to have it insert itself into compilers it builds (with decent accuracy, I imagine it can be defeated) is still REALLY impressive

I'd never question objdump -D

Really?

Posted May 8, 2021 3:07 UTC (Sat) by SomeOtherGuy (guest, #151918) [Link]

Sorry for the quad post, a comment that seems to be empty (808) has showed up with "P" status, I replied to it trying to figure out what it was (asking for help and mentioning I checked the FAQ/RTFM) and I cannot see those on the article, but they have "A" status?

https://lwn.net/Articles/855810/

Like this

I just want to make sure I've not done anything wrong (with the system) but the stuff of substance appears. So nothing is lost anway

Rethinking Fedora's compiler policy

Posted May 8, 2021 0:43 UTC (Sat) by pabs (subscriber, #43278) [Link] (1 responses)

All three projects are in the very early stages and are very unlikely to be usable at this stage. Hopefully they all mature and their changes get merged into the relevant places, the authors of gccrs for example plan to get it merged into GCC.

There are plenty of dependable projects hosted on GitHub, the hosting service doesn't contribute to a project's dependability, as long as the project keeps good backups so they can move to another service.

Of course GitHub being a service owned by a convicted monopolist and running proprietary software isn't great. We need Free Software projects to run Free Software tools on their own infrastructure, which is why my GitHub profile links to mako's "Free Software Needs Free Tools" article.

https://mako.cc/writing/hill-free_tools.html
https://github.com/pabs3

Unfortunately the large tech companies, some of them venture capital funded, have monopolised (and consequently I think shrunk) the market for sysadmins, so various projects (even Python!!) have switched away from self-hosted solutions towards proprietary services, mainly to avoid the sysadmin work necessary with self-hosting both software and hardware, since those with strong sysadmin skills joined large tech companies and others had their remaining skills atrophy after moving to proprietary services.

The network effects of GitHub and the attractiveness of outsourcing your sysadmin work needs mean it is very unlikely we will see much of a switch away from GitHub. The only thing I can see that would disrupt this cycle would be a move away from the Web and towards native apps and development environments based on locally installed software and distributed protocols. I don't expect to see these things within my lifetime though.

Rethinking Fedora's compiler policy

Posted May 8, 2021 1:44 UTC (Sat) by SomeOtherGuy (guest, #151918) [Link]

I do agree with you on the sysadmin thing, I respect them a lot more than I did when I started programming professionally (something I've always tried to be worthy of)

Not sure about the project having backups and being able to migrate, we all know we should backup, not reuse passwords ect, and I will tentatively admit that I'm failing on that front ;)

I expect many of those projects are too - unfortunately.

PHP recently took the change BTW

Rethinking Fedora's compiler policy

Posted May 10, 2021 13:20 UTC (Mon) by scientes (guest, #83068) [Link]

Reading these comments I get the impression that the commenters don't have any experience with developing compilers, such as writing an LLVM (or GCC) frontend.