Rethinking Fedora's compiler policy

It is a longstanding practice for distributors to pick a specific toolchain to be used to build the full set of packages. Mixing compilers and linkers always presents the possibility of incompatibilities leading to subtle bugs, which is the sort of outcome distributors tend to go out of their way to avoid. Adhering to that practice, Fedora has long used GCC as its compiler of choice — a practice dating back to the time when there were no other compilers to use in any case.

Some packagers prefer LLVM

Over the years, though, the LLVM compiler suite and its Clang C compiler have grown in capability and become more popular. There are some programs that will not build with anything else at this point. For such programs, Fedora allows Clang to be used; for everything else, though, Fedora's policy requires building with GCC. This, according to the change proposal, creates unneeded difficulties for Fedora maintainers:

Upstream, the Firefox project builds primarily with Clang/LLVM. Yet we force the Fedora package owner to find and fix issues building with GCC then either carry those custom fixes forward in Fedora or negotiate with upstream to get those changes upstreamed. While this process can be helpful in finding non-portable code, this is ultimately a poor use of the packager's time.

The new proposed policy states that maintainers could use a non-default compiler for their package if they have a "valid technical reason" to do so. Beyond avoiding the above-mentioned Firefox problem, this change, it is argued, will enable the use of the compiler that yields the best result for any given package, thus improving the user experience overall. Beyond that, building with the compiler an upstream project uses for its own development and testing is, perhaps, less likely to tickle compiler-related bugs.

The real motivation behind this proposal, though, might be found in places like this RHEL bug report. The Ruby language has a just-in-time (JIT) compilation feature, and users would like to take advantage of it. But, due to GCC, when built as a position-independent executable, being incompatible with pre-compiled headers, JIT compilation fails with an unhelpful error message. There are various ways that this problem could be addressed; the morbidly curious can find them in the bug report. The conclusion, though, is that none of the options for making this feature work with GCC are deemed acceptable. If, however, LLVM were used, the problem would go away. Thus, unsurprisingly, the Ruby developers would like the freedom to use LLVM when building the Ruby package.

Questioning the change

The first question to arise in this posting of the proposal (it has actually been under discussion for nearly a year) was: who decides what's a "valid technical reason"? Will there be some sort of committee review? The consensus would appear to be that maintainers will be trusted to make the best decisions for their packages. The current version of the proposal differs from the original, which said maintainers should use the compiler that is preferred by the upstream project they are building.

The first posting of the new compiler policy (targeted at Fedora 33) drew a number of complaints, many of which resurfaced this time around. Neal Gompa described it as "an excuse to avoid doing the right thing and leveraging the toolchain that offers the highest quality code generation" and said that, at a minimum, the justifications for this change needed to be spelled out more clearly. Jakub Jelinek (a GCC developer) argued that subtle ABI incompatibilities still exist between the two compilers, and that mixing them increases the chance of introducing difficult bugs.

Mark Wielaard worried about the extra workload for developers of other parts of the toolchain:

For the packages I work on, elfutils, valgrind, debugedit, etc. we have enough trouble keeping up with gcc versions, new flags, optimizations, etc. I don't think we will have time to debug issues because packages are now also built with another compiler. Leaving the system as a whole in a worse state because of it.

Kevin Kofler argued that the best compiler overall should be used for the entire distribution — and that GCC is the best compiler for now.

A few participants, including Wielaard (linked above) and Florian Weimer, pointed out that even if Fedora adopts Clang when building projects that prefer it, the Fedora build will still differ in many ways, including the specific compiler version, the compilation flags used, and the setup of the system as a whole. The end result may well not be a better-tested build environment than was had using the default compiler.

Compiler diversity

One other concern mentioned by Jelinek in the above-linked message merits a mention. He agrees that competition between toolchains is good for everybody involved. While this proposal, on its surface, would appear to facilitate this competition by allowing the best compiler to be used everywhere, he says, the end result is more likely to be an increase in the number of packages that can only be built with LLVM. Upstream projects, especially those dominated by a single large company, often do not have portability across compilers as one of their important objectives. If distributors do not insist on that portability, it may well go away.

This would appear to be a valid concern. The growth of the Chrome-only web shows how this sort of capture can take place. GCC's slow rate of progress in the years before LLVM took off, and the acceleration of development since, highlight the value of this sort of competition. A return to a single-compiler world is not in anybody's interest.

Distributors, being the organizations that wield compilers for the benefit of vast numbers of users, are an obvious place to try to draw a line against a compiler monopoly. Whether distribution policies are the best way to promote compiler diversity is a different question, though. There may come a point where the pain of building with an out-of-favor compiler becomes more than people are willing to deal with, especially in volunteer-driven distributions. The apparent imminent end to Java support in Fedora shows what can happen when the work gets too overwhelming. Burning out package maintainers in the name of portability does not seem like a good way to achieve that goal.

The outcome of this particular change proposal is unclear at this point. Even if it is not adopted this time around, though, it seems likely that the pressure to make that sort of change will only increase in the future. Unless the forces that are driving projects to only support one compiler are somehow reduced, the cost of going against the stream will only grow.