CentOS and ARM

The CentOS distribution has long been a boon to those who want an enterprise-level operating system without an enterprise-level support contract—and the costs that go with it. In keeping with its server orientation, CentOS has been largely focused on x86 systems, but that has been changing over the last few years. Jim Perrin has been with the project since 2004 and his talk at Open Source Summit Japan (OSSJ) described the process of making CentOS available for the ARM server market; he also discussed the status of that project and some plans for the future.

Perrin is currently with Red Hat and is the maintainer of the CentOS 64-bit ARM (aarch64) build. CentOS is his full-time job; he works on building the community around CentOS as well as on some of the engineering that goes into it. His background is as a system administrator, including stints consulting for the defense and oil industries; with a bit of a grin, he said that he is "regaining a bit of my humanity" through his work at Red Hat on CentOS.

The initial work on CentOS for ARM started back in the CentOS 6 days targeting 32-bit ARMv6 and ARMv7 CPUs. That distribution is now six or seven years old and it was already old when the developers started working on an ARM version of it. The software in CentOS 6 was simply too old to effectively support ARM, Perrin said. The project ended up with a distribution that mostly worked, but not one it was happy to publish. It improperly mixed Fedora and RHEL components and was not up to the project's standards, so that build was buried.

In January 2015, which was after CentOS 7 was released, the project restarted using that base but targeting aarch64. There was "lots more support for ARM" in that code base, he said. After about six months, there was a working version of the distribution that he and other project members were happy with, so it was time to give the community access to it. Unfortunately, 64-bit ARM chips were not widely available in July 2015, so the project needed to decide where it wanted to go with the distribution.

Community

There are multiple parts of the CentOS community, each of which has its own needs. Hardware vendors are the first and foremost members of the community, because they must create the hardware that all of the others will use. If the hardware does not work well—or CentOS doesn't work well on it—no one will be interested in it.

The second group is the business partners of the hardware vendors. These are early adopters that get the hardware from the vendors and want to "kick things around" to see that the hardware is working for their use cases. CentOS needs to be able to provide help and support for these companies.

There are also early adopters who are not affiliated with the hardware vendors. They buy and break new hardware and are particularly vocal on social media. They will let it be known that they have this new hardware and what software is or isn't being supported on it. They have opinions and a project needs to take care of their needs, he said.

A group that is somewhat similar to early adopters is the maker community. The difference is that early adopters are going to try out business use cases using the system, while the makers will "blast it into space" or run it at the bottom of a lake. Any folks that do "that level of weird things with the hardware" deserve their own group, Perrin said.

Then there are the slower-moving parts of the community. Businesses will typically allow others to work out the bugs in the hardware and software before starting to use it; they have "a more cautious approach", he said. The last group is the end users, who are system administrators and others whose boss bought the hardware; they may not be particularly pleased about using the distribution, but they need to get work done so it is important to try to make their jobs easier.

Some of these communities are "more equal than others", which sounds backwards or odd coming from a community person, Perrin said. But what he is really talking about is timing; you don't need to worry about makers, say, until there is working hardware available. So CentOS needed to take a tiered approach to supporting its various communities.

It all starts with the hardware, naturally. Working with some of the larger vendors on building the distribution for their aarch64 server prototypes was the first step. That was facilitated by the unannounced arrival of hardware at his house. That was "fantastic, but really surprising". From the audience, Jon Masters, who had arranged for some of those shipments, jokingly warned attendees: "don't tell me your address". With a grin, Perrin said: "my electric bill does not thank you".

CentOS started by working with AppliedMicro; that was used as the reference platform starting in March 2015. After that, the project also worked with Cavium, Qualcomm, AMD, and some other vendors that are not public.

Once the hardware is supported, it is time to move on to the early adopters. It was not practical to work with the hardware vendors' business partners as it is not his job to manage those kinds of relationships, he said. But early adopters are different; CentOS wanted to work with folks who are going to be loud about using the distribution. From those efforts, the project learned about some optimizations for aarch64 that were not working well for some users, for example.

More packages

One of the biggest things that helped with that was working with the Fedora and Extra Packages for Enterprise Linux (EPEL) communities to get the EPEL packages working for aarch64. Those packages are valuable for day-to-day work on servers, he said. The CentOS project focused on making the hardware work and making a base set of packages, then getting out of the way. The EPEL group has been "fantastic at packaging up things they think people will need".

Part of the process of creating the distribution is figuring out what software the community wants. The short answer turns out to be that it wants "containers and virtualization". So one of the early projects was to get docker (with a small "d", "not with a large 'D' that is now trademarked", he said) running on aarch64. Docker is written in Go, which meant that the golang package needed to be built.

When the process started, though, the released version of golang was 1.4, which did not support aarch64. The project had to bootstrap a build of 1.5 beta using the 1.4 compiler in a container on an x86_64 system. That "failed immediately" because of a calculation done by docker to determine the page size. It is 4KB on x86_64, but 64KB on CentOS aarch64. That got fixed (and upstreamed) and CentOS was able to build docker by late 2015 or early 2016.

The availability of docker started to accelerate other development on CentOS for ARM. For example, Kubernetes is being ported. The same page-size problem cropped up there, but the Kubernetes developers are quite receptive to patches. Kubernetes 1.4 is "not 100% baked yet" for the distribution but is getting there.

On the virtualization side, users wanted OpenStack. They wanted to be able to do virtualization and virtualization management on ARM. As it turns out, though, the bulk of the need for OpenStack was for network function virtualization (NFV), rather than wanting OpenStack for its own sake. OpenStack is just a stepping stone to NFV, he said. The process of porting OpenStack is under active development right now.

Boring

The overall goal for the CentOS on ARM project is to "get to boring". The idea is that the distribution works just like every other distribution on every other architecture. For some platforms, it has been difficult at times to get to that point. There is a mindset of how software works in the embedded world that doesn't translate well to the server world. If the story is that "this system boots this way, this other one that way", it will not sit well with customers.

So a lot of work was put into community building within the hardware vendor community regarding standards. The idea is that ARM, Intel, AMD, and others all need to work the same way, install the same way, boot the same way, and so on. That means support for PXE, UEFI, ACPI, and so on. There is something of a balance required, though, because at the same time he is beating on the vendors to standardize, he is also asking them to provide hardware for makers and other early adopters.

At this point, there is a functional base distribution that matches what there is on the x86 side. The next step is to get things like Kubernetes and OpenStack working; after that is less clear. He is no longer a system administrator, so he is not attuned to what users may want and need. Part of coming to OSSJ was to hopefully gather some feedback on what users would like to see. He can take that back to Red Hat engineering as input for upcoming plans. Maybe there are tools and technologies that CentOS doesn't even know about that need to be added to the ARM ecosystem; he encouraged attendees to let him know what they are.

In answer to an audience question, Perrin said that installing CentOS for ARM is straightforward, much like the process for x86: download an ISO image, boot that from a USB drive or via PXE. Instructions are available on the wiki. That is for supported 64-bit ARM hardware; for 32-bit hardware, like Raspberry Pi, a specialized image is needed for the platform. The 64-bit Raspberry Pi 3 (RPi3) will ostensibly be supported in three months or so, he said, once the U-Boot bootloader gets UEFI support.

Masters spoke up to note that Perrin is "stuck with" some of the decisions that Masters made. One of those is the 64KB page size, which is good for servers but not as good for embedded use cases like Raspberry Pi. Red Hat (where Masters also works) is focused on the server market where the larger page size makes a lot of sense. Some ARM distributions did not think about that, he said, and will be stuck with 4KB pages that are more suited to embedded use cases.

There are some other hardware choices, which have a similar price point to the RPi3, that could be used for development and testing, Perrin said in answer to another question. The ODROID C2 and C3 boards have 64-bit ARM CPUs, but there is a "giant caution flag" for those kinds of systems. Since the changes for those boards have not been pushed to the upstream kernel, users will be running the CentOS user space with a vendor kernel. That may be just fine, but there have been occurrences in the past where vendor kernels have had problems—a remote root hole in one case.

If you want online hardware, Perrin suggested Packet, where you can get a bare-metal aarch64 system. It is "kind of like AWS" but with ARM hardware.

When asked about 96Boards, Perrin said the company has an "array of good hardware" that doesn't do what is needed for CentOS. The HiKey board is the best, but there are some implementation issues that cause CentOS difficulties and the DragonBoard 410c does not have the right bootloader for CentOS. As Masters put it, the right answer is to spend $1000 to get a real server.

The final question was about whether CentOS is talking with other ARM distributions that do not emanate from Red Hat. Perrin said there is a cross-distribution mailing list; he doesn't see eye to eye with the others on it all the time, but that's true with his colleagues at Red Hat too at times. Driving standards for the hardware helps everyone and the people on the list are trying to do that. That is part of why there has been some effort into supporting CentOS on RPi3; everyone has one, so it is a good way to open up development without having to tell interested people to go buy a $1000 server.

[I would like to thank the Linux Foundation for travel assistance to Tokyo for Open Source Summit.]