32-Bit x86 support in Fedora

An under-the-radar proposal to stop building i686 kernels for Fedora led to a discussion about dropping support for 32-bit x86 hardware. Any of the hardware that needs these kernels is quite old, but participants in a thread on the Fedora devel mailing list noted that those systems still exist—some run Fedora. As the discussion progressed, though, it became clear that the Fedora i686 kernel has been in rough shape for some time now.

Florian Weimer raised the issue on July 11, likely inadvertently. He pointed to a proposal to stop building i686 kernels for Fedora 27 and noted that if that were to be adopted, it would mean that all x86 hardware still supported by Fedora would have the SSE2 instructions. That would allow the compile flags to be updated to add SSE2 before the "mass rebuild" for that release.

The "drop i686" proposal was marked as "not a system-wide change", but several in the thread objected to that designation. For example, Chris Adams pointed out that "it would remove a supported architecture completely". In addition, several pointed to Atom-based systems that are 32-bit, some of which are still being sold (mostly in the embedded space). Those CPUs do have SSE2, so it is possible that Weimer's request to enable SSE2 support would still make sense. Most other hardware that would be unable to run Fedora without an i686 kernel is at least ten years old.

But the Fedora i686 kernel has been in a state of benign neglect for several years. Josh Boyer said that the kernel team has not been doing much, if anything, for i686 kernels lately:

Boyer concluded by saying that it was "basically luck" if i686 kernels are still working. In fact, Richard W. M. Jones said that he had stopped testing libguestfs with i686 kernels because they were broken most of the time when he tried them using QEMU; nobody was interested in fixing the bugs, he said. Laura Abbott added some perspective on the kinds of bugs that have appeared in i686 kernels; she said that the best way forward was to work with the upstream kernel community:

Ralf Corsepius posited that the lack of interest in i686 for Fedora really stemmed from a lack of interest by Red Hat. Fedora project leader Matthew Miller did not really disagree with that, saying that it is easier to support architectures that a company is paying someone to care about. Since Red Hat is not doing so, and no other company has stepped up, i686 kernels have languished. On the flipside, though:

It is not just the kernel that is suffering, either. Adam Williamson pointed out that there is a ten-month-old bug for an installer feature that does not work correctly on 32-bit x86 kernels. Part of the problem is that there is nowhere for him to refer the bug: "There is no i686 WG/SIG, no more informal but well-known 'group of people who care about and help fix i686 bugs', so I can't do it for that bug", he said in another post.

Hans de Goede seemed to be taken aback by the "zombie" status of the i686 kernel. Boyer pointed to a post he made in February 2015 that laid out the problem from the kernel team's perspective and asked for the community to step up to help fix i686 bugs. Others in the thread listed additional places where the topic had come up as well. No one stepped up, which meant that i686 bugs simply languished.

But De Goede thought that Boyer's message was too much of a "generic, non specific call for help". What would be more helpful, he said, is "a concrete list of things people who care about i686 can work [on]". While Boyer agreed that kind of a list would be useful, he asked: "Are you volunteering to triage all the bugs and create it and maintain that list?" De Goede didn't directly address that question, but did recognize that an i686 SIG needed to be formed in order to move forward. He asked for some guidelines on what needed to happen in order to keep i686 kernels alive for Fedora 28 (the proposal to drop i686 kernels was submitted too late to be considered for Fedora 27).

Miller promptly provided that guidance, at least in draft form; the Fedora Engineering Steering Committee (FESCo) will presumably finalize the guidelines at an upcoming meeting. The idea is that if there is no functional i686 SIG by the Fedora 27 release, the kernel will be dropped for Fedora 28. The criteria for what "functional" means will need to be hammered out.

Corsepius complained that getting rid of i686 kernels showed that Fedora is not really a community-driven distribution. But Boyer strongly disagreed with that:

Moving forward, Stephen J. Smoogen has put out a call to gauge interest in forming an x86_32 SIG. He created a wiki page and listed some preliminary steps that would need to be taken. There has been a bit of activity on the wiki and some interest shown in the thread, but nothing overwhelming at this point. There was interest in a mailing list, so Smoogen created x86@lists.fedoraproject.org on July 18; the archives are rather sparse, so far, but it has only been a day or so.

It is unfortunate when older hardware gets left behind though, as Andy Lutomirski mentioned in the thread, the upstream kernel has good support for i686 (and even i486) these days. But distributions have to apportion their efforts in ways that make the most sense for them. Quite a few Linux distributions are eliminating 32-bit builds, so Fedora is hardly alone if it takes that route. But "community driven" also means that some parts have to be community supported, too, even for distributions like Fedora that have a large company behind them.