A discussion on Linux in space

There was something of a space theme that pervaded the Embedded Linux Conference (ELC) portion of the 2023 Embedded Open Source Summit (EOSS), which is an umbrella event for various sub-conferences related to embedded open-source development. That may partly be because one of the organizers of EOSS (and ELC), Tim Bird, described himself as "a bit of a space junkie"; he made that observation during a panel session that he led on embedded Linux in space. Bird and four panelists discussed various aspects of the use of Linux in space-related systems, including where it has been used, the characteristics and challenges of aerospace deployments, certification of Linux for aerospace use, and more.

The panelists in the room were Lenka Třísková, a lecturer and researcher at the Technical University of Liberec (which is in near Prague, Czechia, as was where the conference was held), who is also the main technical advisor for the Linux4Space project, and David VomLehn, who is the lead flight-software engineer at Astra. VomLehn gave a keynote about Linux in space (YouTube video) earlier in the conference. They were joined by two remote participants, both on the US west coast, thus joining the session at a pretty early hour for them: Steven VanderLeest, chief technologist for Boeing Linux, and Rob Bocchino, from the NASA Jet Propulsion Laboratory (JPL).

Projects

Bird started with what he hoped was "a softball question" about projects that the panelists had been involved with that used Linux for aerospace applications; was that work done for "production" deployments or simply research, he asked, and "were they successful?" He started with VomLehn, who quickly listed two rocket projects, two satellite projects (plus one where it sounds like only the military knows if it was successful), and a lunar lander. They all used Linux, and the first series of satellites was successful, one of the rockets was successful (he is awaiting launch of the other); the second series of satellites was not successful and the company behind the lunar lander ran out of money before launch, unfortunately. Bird asked if any of the failures were Linux-related; "absolutely not", VomLehn said, the satellites all failed due to hardware problems.

VanderLeest said that his experience is "more on the air than the space side"; ten years ago he worked for a small company doing research on certifying Linux for safety-critical systems and for flight certification in particular. Now he is at Boeing, which has used Linux for both air and space applications. As far as he is aware, Linux has only been certified to level "D", which is one of the lower software assurance levels; he is currently working on a project to develop a Yocto-based Linux system for flight certification at the higher levels, up to and eventually including "A".

Bocchino said that he had worked on the ASTERIA low-earth-orbit satellite based on a 6U CubeSat, which was deployed in 2017 from the International Space Station (ISS). It ran Linux and was "a very successful mission". Its original mission was for exoplanet detection and it successfully demonstrated high-precision pointing of its space telescope; it is the smallest satellite to ever perform those maneuvers, he said. After the main mission was completed, the team was able to upload new software to the satellite to test out other capabilities.

While he was not directly involved with it, the Mars Ingenuity helicopter is another big Linux success story at JPL. He did help develop one of the components that is used by the helicopter; the F' (or F Prime) flight-software framework is used by ASTERIA and Ingenuity. Bird said that he had been watching Ingenuity with great interest; "it's like the little helicopter that could". Bird also pointed out that F' is open-source software (it is available under the Apache 2.0 license).

Třísková said that in early 2020 she did some consulting on the CubeSat instruments for the Hera Mission, which is an asteroid-investigation mission that will be launched by the European Space Agency (ESA) in 2024. The two CubeSats run Linux, but she noticed that there is no common Linux platform for them, which is why she became involved in Linux4Space, which is an effort to build that platform.

In his keynote, VomLehn had said that "space is hard", Bird said, which is something of a "standard mantra" in the industry; he asked the panelists what kinds of requirements there are for Linux in aerospace that make it particularly difficult. VomLehn said that generally there is both a mass budget and a power budget for a project; both of those have to be met. Radiation is an enormous problem in space; "I would love it if there were standard solutions for that, both hardware and software." Bandwidth is another problem area; the lunar-lander project he worked on had only 500bps of bandwidth available to it for parts of its mission. Even in low earth orbit, bandwidth is going to be a problem until satellites can plug into things like Starlink, he said.

Beyond the physical characteristics that these systems need to withstand (such as radiation, vibration, and temperature), there is a need to provide evidence that the system itself is reliable, VanderLeest said. Once you start adding human crew and passengers into the mix, the requirements for those assurances get quite rigorous. The system must be shown to be both reliable, meaning that it does do what it is meant to do, and safe, which means that it does not do what it is not meant to. Doing that is a costly process and every added feature means that the cost rises, so there is a focus on a small footprint of only the critical features for the system and mission.

Bird suggested that part of the reason that realtime operating systems (RTOS) have inertia for aerospace applications is due to the "provability of assurance". He had attended a flight-software workshop recently where people were talking about 100% test coverage for the lines of code; "that's somewhat shocking for a project that has 25-million lines of code". VanderLeest noted that, while Linux itself has that many lines of code, any given instance of it will only use a subset. The two largest pieces are drivers and architectures; only a limited number of drivers and a single architecture will be present in a running system. That may still add up to a million lines of code, though, "and that's a lot of lines to prove".

VomLehn noted that simply hitting the lines of code in testing is not necessarily sufficient; there is also "feature coverage" where you actually execute things in the sequences they would follow when "real commands" are being executed. VanderLeest bobbed his head in agreement. Bird said that "it's a hard problem", which both of the panelists acknowledged.

Why Linux?

That led Bird to ask: "Why are we here? Why should we be using Linux?" Třísková said that for the CubeSats, they had decided to use Linux for the non-mission-critical software, in part because of the amount of other software that can be used on Linux. For example, there is lots of code for doing data and image processing; in addition, the drivers for the different protocols used in space are all available for Linux. Beyond that, there are a lot of people who know and understand Linux, which makes it easier to hire people to work on the project. And "it's a lot of fun", she said.

Bird asked Bocchino why they chose Linux for ASTERIA. Like Třísková, Bocchino said that Linux is an easier environment to work in, with a familiar shell and set of tools. It turns out that if you are willing to tolerate some risk, the shell can be used during the mission. Both the ASTERIA and Ingenuity teams have used shell access in that way.

Bird raised the idea of doing AI processing on these systems, which was another topic from the workshop; he knows that it is fairly straightforward to get that kind of code running on Linux but wondered whether an RTOS has the services needed to do so. VanderLeest said that "Linux provides a collaborative environment for innovations, whether it's AI or anything else"; it gives access to "cutting-edge technology more quickly". In addition, when vulnerabilities are identified, they "tend to be fixed much quicker" in Linux, he said.

Another important thing to consider, VomLehn said, is that "there is just so much Linux out there"; in some sense, the functional coverage comes from the amount of use that Linux gets. In terms of assurance, "process is paramount when it comes to reliability"; the kernel-development process and, especially, the amount of varied testing it gets on lots of different platforms "really drives the reliability up". There are ways to mathematically prove the correctness of programs, which is not the approach the Linux community has taken, but it has come up with a different process that produces "demonstrably excellent results".

Realtime

Bird asked about the realtime requirements for aerospace and whether the realtime patch set for Linux would satisfy them; are there still concerns about whether Linux can hit realtime deadlines? Bocchino said that ASTERIA was using an earlier version of the kernel with the realtime patches, but he suspects that what he found then is still true today: "Linux is just never going to be a hard realtime OS." So if there are hard realtime deadlines in the system, which cannot be missed, "then don't use Linux for that".

On the other hand, soft realtime is often good enough, he said. If not, the system can be partitioned into a part that is running an RTOS and a part that is running Linux, as was done on the Mars helicopter. VomLehn said that he would "quibble a little bit about how hard 'hard' is in Linux realtime"; if you look at graphs from a well-tuned system, it "has pretty well-bounded latencies". A microcontroller with an RTOS has lower latencies, which is "nice if you are trying to prevent something like an explosion". But a lot of the other parts of the system, such as the guidance software, can tolerate an occasional missed deadline.

VomLehn stressed the importance of tuning the system properly; he said that using realtime Linux for things like controlling rockets is possible. VanderLeest said that a system is only realtime to him if he can prove deterministically that the worst-case latency is still within the requirements of the system. For many of the use cases he is looking at, those requirements are measured in milliseconds "and Linux can perform at that level".

Bocchino said that his experience had been that the worst-case latencies for realtime Linux "can often be pretty bad", though maybe that could be fixed with tuning. If you have an application that can "tolerate a missed deadline every now and then, then it works fine"; if you cannot miss any deadlines, though, his experience is that it would be hard to use Linux. Třísková agreed with that assessment; mission-critical pieces that are, say, preventing explosions are better done in an RTOS. There are plenty of other pieces that can benefit from the power and availability of software that Linux brings.

Cost

Bird wondered whether Linux provided an advantage in terms of development cost; his theory was that if there was less effort needed to work on the OS, that would allow for more time on the science and other aspects of the mission. VomLehn definitely agreed that Linux was an advantage due to "the sheer massive number of features" that come with Linux and its overall ecosystem. He is convinced that it is less costly to develop these systems using Linux.

As an example of the range of the ecosystem, VomLehn noted that there is a satellite out there running Python, which surprises a lot of people; its developers wanted to do as much processing on the satellite as possible because of severe bandwidth limitations for transmitting results. He has hired "a lot of sharp people" who are not from the aerospace industry, but who know Linux. In combination with other aerospace veterans, it works out quite well because of the capability of the Linux platform. Bocchino added that having developers who already know Linux makes it more cost-effective than trying to retrain them for another environment.

Bird then asked about certification, noting that he had seen requirements for a design-driven process that is quite different from what Linux uses; "does that mean it's impossible to certify Linux for certain space applications?" VanderLeest said that he definitely did not believe it was impossible; there are already examples of Linux being certified at fairly low assurance levels. Going further is "a very hard problem", but he and his team do not think it is impossible.

The certifications are based around a process where requirements are described and a design is created before code is written, which is decidedly different than the approach Linux takes. But there are guidelines for reverse-engineering the requirements and design; "there is a design to Linux, it's just emergent". There are also implied requirements for Linux. VanderLeest and his team think that can be successful with that approach "and we're driving towards that".

VomLehn pointed out that there is a large company that is launching people into space using a Linux-based system. Since that company, SpaceX, is a competitor of his, he did not want to say the name, though Bird filled it in with a chuckle. The SpaceX system is two-tiered, with a low-level microcontroller and RTOS tier and a Linux executive above that. VomLehn's company and others are using Linux as well, and SpaceX has gotten a crew rating for its systems, which is the highest level of assurance. It took years of work with NASA to get there, but it does show that it can be done.

The use of commercial off-the-shelf (COTS) products in space-based systems was next up. Bird noted that an earlier CubeSat talk (YouTube video) mentioned the use of COTS parts that were subjected to a higher level of testing; he was perhaps a bit surprised at that approach given that space missions cost so much already. But VomLehn pointed out that an FPGA that costs a few dollars may cost $30,000 for space-rated version, so the cost difference can be considerable.

VanderLeest said that, 25 years ago or so, there was a shift in the thinking about processors; instead of designing special-purpose space processors that had redundancy and fault-tolerance built-in, the industry switched to standard processor designs and added the fault-tolerance at the system level. The purpose-built processors were hard to keep up-to-date with the latest innovations elsewhere, which he likened to the shift to Linux being seen today. By switching to a more commodity OS, the aerospace applications get the updates and innovations that continuously come from the community.

Bird asked for questions from the audience; the first (and only, as it turned out) was not that, exactly, but more of a lengthy comment from realtime Linux developer John Ogness. He wanted to clarify a misconception that he had already heard a few times at the conference and in this panel session: "The realtime Linux team is committed to hard realtime". They are not at all aiming for soft realtime, "so a single missed deadline is a completely failed system". If there are systems where the realtime patch set is not providing that, it is likely either a misconfiguration or a bug in the user-space code, Ogness said. In any case, the realtime team is looking for feedback on any problems that people are encountering, so he encouraged the space-Linux community to get in touch if the realtime patches were not meeting its goals. That was met with approval (thumbs up) from several panel members and a round of audience applause.

As time ran out on the session, Bird asked his final question; he wondered what impact the open-source licenses had on the use of open source in aerospace. In particular, were there things that the industry did in order to work with the GPL as the license of Linux? VomLehn said that the industry generally does not sell rockets, but sells "launch services" instead; that means that there is no real distribution of the code. He said that Astra employees are encouraged to work with upstream communities and there are parts of his kernel work that he "would be delighted to be able to give back"; he hopes that can happen eventually. There are various legal restrictions on what the industry can say and do, but he thinks there is still room for those in the industry to be good open-source community members.

A YouTube video of the panel session is available.

[I would like to thank LWN's travel sponsor, the Linux Foundation, for assisting with my travel to Prague.]