LinuxCon: Dragons and penguins in space

One of the best things about large conferences like LinuxCon is that the extensive program draws in speakers from outside the core Linux and free software ecosystem. Such was the case at the North American edition of LinuxCon 2012 in August, which featured a presentation from an aerospace engineer at Space Exploration Technologies (SpaceX). The company is evidently using Linux extensively on the ground and in its launches, although it provided frustratingly little detail.

Beyond the cloud (literally)

The speaker, Jim Gruen, is a relatively new hire at SpaceX, working on flight software. He started off by explaining exactly what the company does and how it operates. Its long-term mission is to make space flight as routine as air travel; for the near-term that means competing for contracts from NASA for the space agency's private space flight programs, such as the Commercial Orbital Transportation Services (COTS) program and the Commercial Crew Development (CCDev) program. Each program incorporates several rounds of proposals and milestones overseen by NASA, ultimately ending in a flight mission with specific objectives. SpaceX has flown two successful NASA contract missions. COTS Demo Flight 1 (C1), in 2010, sent the company's unmanned "Dragon" capsule through two Earth orbits then splashed it down into the ocean. COTS Demo Flight 2/3 (C2+) followed in May 2012, which combined the COTS 2 objective of rendezvousing with the International Space Station (ISS) and the COTS 3 objective of docking with ISS.

Although that slate of projects would certainly qualify as interesting stuff in just about anyone's book, Gruen's explanation of SpaceX's operations was intriguing as well. The company believes strongly in vertical integration as a cost-cutting measure, to the point where it manufactures in-house 80% of what it puts into space. It buys raw metal and manufactures the parts for its rockets and capsules, and it designs, prototypes, and produces its own computers, circuit boards, and even chips. The goal of this approach, he said, is to have everyone working in the same building, and enable them to try new experiments very quickly.

With that background material out of the way, he explained how the company uses Linux. For starters, space flight generates an enormous amount of data, including flight telemetry, traffic between ground stations, media feeds from the spacecraft, and so on. Streaming, storage, and analysis of this data is done on Linux — though it is not a task set unique to SpaceX or to space flight, he admitted.

Gruen's team works on avionics, the navigation and control systems on the company's spacecraft. The team is responsible for the complete life cycle and operation of the equipment, he said: board bring-up, bootloading, hardware initialization, straight on up through the user-space software. The company's C1 flight was a proof-of-concept run for its Dragon capsule design, and on that mission it did not run Linux. However, the C2+ model and subsequent revisions do run Linux. This does not mean that Linux is merely running on an auxiliary computer, he emphasized: Dragon's core systems are Linux, a custom in-house distribution using the uboot bootloader with C++ code running on top. Linux handles the triply-redundant avionics system, the thrusters, and even the pyrotechnics (which in space-faring lingo refers to the explosive charges used to do things like deploy parachutes for re-entry). He also showed images from the C2+ mission's ISS docking procedure, which used computer vision software running on Linux to locate the docking port and align the spacecraft with the station.

Event horizon

Gruen's overview of the Dragon vehicle and its use of Linux was interesting, to be sure. Unfortunately, the overview was more or less the full extent of the detail available. He was accompanied by representatives of SpaceX who sat in the front row and who would not allow him to go into any specifics about the hardware or software of the system, nor to take questions from the audience. The room was packed to overflowing, and the session let out with plenty of time still on the clock.

Gruen attributed the restrictions on his subject matter to the US State Department, whom he said classified SpaceX's business as "building dangerous weapons". Consequently, he expressed his excitement to be giving the talk, but added that he was "going to present as much as I can without breaking any laws and going to jail". That is certainly an unenviable bind to be in, but the upshot was that the audience learned little about SpaceX's Linux systems — and about the challenges or discoveries its developers have encountered along the way.

What makes that limitation puzzling is that so many Linux developers were in the audience for the session — Gruen commented more than once that there was code running on Dragon that had been written by people there in the room. In fact, Linux is so widespread in the scientific community that it would have been a surprise to hear that Linux was not the platform of choice. After all, Linux has been trustworthy enough to run nuclear weapons simulations for the US Department of Energy for years, and reliable enough to run medical devices; it is not a big stretch to hear that it runs on an orbital capsule as well.

It was unclear how much of SpaceX's taciturnity was due to government regulation and how much was by choice. SpaceX is in a highly competitive business, to be sure, and has the right to work in private, but it seems a bit implausible to argue that how the company uses upstream code like Linux constitutes a trade secret. Is there any credible chance that a competitor such as Orbital Sciences is running Windows on its spacecraft and has something substantial to gain from hearing that SpaceX sees better performance from Linux's scheduler, or which GRUB limitations made uboot the bootloader of choice?

SpaceX's reluctance to discuss details stands out, because attendees heard several other talks about Linux in high-security and scientific environments just days earlier. For example, Kipp Cannon of the Laser Interferometer Gravitational-Wave Observatory (LIGO) collaboration spoke at GStreamer Conference about his group's use of Linux to capture and analyze live laser interferometry data from gravity wave detectors. Cannon's group uses the largest GStreamer pipelines ever made, on massive machine clusters, to process LIGO signals fast enough recognize events in time for astronomers to aim telescopes at them before the events end. Certainly getting to and docking with ISS is a tremendous technical challenge, but it is not a drastically bigger challenge than the real-time detection of gravitational waves from black hole collisions a galaxy or more away. LIGO is a collaborative effort, but it too has fierce competition from other experiments, both for funding and for results.

As for the security factor, the implication was that SpaceX's work is regulated by the US Government, although it is not clear why that is the State Department's purview. But the GStreamer Conference also had a presentation from researchers at the US Department of Defense's Night Vision and Electronic Sensors Directorate (NVESD), which uses Linux and open source software to calibrate and test night-vision equipment and create new algorithms for combining multiple sensors' media streams into usable displays. They made it quite clear that the algorithms they develop are classified, while still explaining how they used GStreamer and other open source software, and even contributed code back upstream. Like NVESD, SpaceX's core projects might be confidential, but the software engineering problems that constitute the daily grind are likely familiar to developers everywhere.

That is probably the main point. I am not particularly interested in spacecraft avionics or infrared sensor algorithms, but it would have made for a more interesting LinuxCon session if SpaceX had talked about some of the challenges or design decisions it has faced in its software program, and how it overcame them. For example, Gruen mentioned that the company uses the kernel's soft real-time support. It would be interesting to hear why Dragon does not use hard real-time — which seems at first glance like a plausible requirement. It would even be worthwhile to hear the story if the solution was to ditch a standard Linux component and write an in-house replacement. Consider the space capsule's storage system, which surely has high reliability and fail-over requirements. There are plenty of computing environments with demanding specifications; hearing how various Linux filesystems fared — even if those that do well in other high-performance applications were not up-to-snuff on Dragon — would have been informative.

Upstreaming

But in the long run there are more important factors than a single interesting talk. Any company can choose to isolate its internal code from the upstream projects on which it relies; the downside is that doing so will increase its own development costs over time. It will either have to expend resources maintaining internal forks of the software that it branches (and back-port important features and bug fixes from newer releases), or periodically perform a re-base then re-apply its own patch sets. Both options increase — in the time required and the complexity involved — the longer that a company commits to them.

Google has walked this path in years past. As we covered in 2009, historically the company maintained its own internal kernel code, rebasing every 17 months. The resulting maintenance effort included merging and debugging its own feature patches, plus backporting hundreds of features from newer upstream releases. Google had its own reasons for not upstreaming its kernel work, including reluctance to share what it regarded as patches of no use to others, but eventually it found the maintenance headaches too painful and modified its kernel development process.

Interestingly enough, the NVESD speakers commented that the DOD greatly prefers its developers to send their patches back to upstream projects — including, in this case, GStreamer — rather than to start their own forks and repositories (and subsequently maintain them). The SpaceX talk mentioned that the Dragon missions generate an enormous amount of video data, but did not go into detail about the software the company uses to stream or analyze it. If it uses GStreamer for the task (which is certainly a possibility), consider how much it stands to gain by interacting in the open with other industrial-sized GStreamer users like NVESD and LIGO — and vice versa.

Perhaps the State Department is simply more secretive than the DOD, but my suspicion is that SpaceX plays close to the vest largely due to the natural tendency for companies to keep their work private (particularly in a company that places a high value on vertical integration). Almost everyone experiences some reluctance when first dipping its toes in open source waters. Indeed, coming to LinuxCon was a good first step for SpaceX. Perhaps it will take a page from its clients at NASA and open up more, particularly where upstream projects like Linux are involved. After all, Gruen's talk was informative and entertaining, and it was nice to hear that Linux has proven itself to be a valuable component in the nascent space flight industry. One merely hopes that next year the company will come back to LinuxCon and engage a little more with the rest of the free software community.