Truly portable C applications

Programming language polyglots are files that are valid programs in multiple languages, and do different things in each. While polyglots are normally nothing more than a curiosity, the Cosmopolitan Libc project has been trying to put them to a novel use: producing native, multi-platform binaries that run directly on several operating systems and architectures. There are still some rough edges with the project's approach, but it is generally possible to build C programs into a polyglot format with with minimal tweaking.

Actually portable executables

Justine Tunney, the creator of the project, calls the polyglot format she put together "αcτµαlly pδrταblε εxεcµταblεs" (APEs). Every program compiled to this format starts with a header that can simultaneously be interpreted as a shell script, a BIOS boot sector, and macOS or Windows executables. This lets APEs run across Linux, macOS, FreeBSD, OpenBSD, NetBSD, Windows, and bare metal on both x86_64 and Arm64 chips. When interpreted as a shell script, the program detects which architecture and operating system it is running on, and (by default) overwrites the first few bytes of the program on disk with an ELF header pointing to the code for that architecture and system, before re-executing itself. This does mean that the binary is no longer portable, but it can be restored by overwriting the ELF header again.

By building separate versions of the program for different architectures and then combining them with this polyglot trick, the project promises portability between different architectures without the overhead of an emulator or bytecode virtual machine. This approach does have downsides — APEs are larger than normal C binaries, although still smaller than many produced by other languages — but for projects where the portability is a benefit, the tradeoff may be worth it.

Still, at the cost of increased compile time and binary size, APEs nearly hold up their promise of running across all major operating systems. There are some systems where they don't run correctly. The biggest source of problems is, ironically, one of Linux's tools for handling foreign binaries: binfmt_misc. In typical configurations, the Linux kernel knows how to run two kinds of binaries: ELF files and shell scripts. The binfmt_misc mechanism lets the user register additional types of program to be run with special helpers. When correctly configured, binfmt_misc can let users transparently run programs for other architectures under QEMU, or run Windows programs under Wine — which is precisely the problem. On Linux systems configured like that, APE programs get run under Wine, instead of running natively.

Luckily, the workaround is quite simple: just add a more specific binfmt_misc rule for APE binaries. This also has the advantage of saving the extra startup latency of first invoking the program as a shell script and then again as a binary. Another problem for APE programs is that some older shells (such as Zsh before version 5.9, released in 2022) don't handle binary data being part of shell scripts well. Because of these problems, the documentation actually recommends installing a binfmt_misc handler, even though it is not strictly necessary on most systems.

The tooling

Producing programs in APE format is relatively straightforward. Stock GCC and Clang can both be configured to do so, with the appropriate headers and linker scripts. But there are a number of configuration flags to get right, so the Cosmopolitan Libc project has also created cosmocc, a wrapper around both GCC and Clang that takes care of building for multiple architectures and linking them together into an APE.

Cosmocc is, itself, an APE. So interested readers can download it and experiment with compiling their own software. Like all APEs, it is statically linked, and so can just be unpacked and run in place. Most C programs should build relatively painlessly under cosmocc. One potential stumbling block is that any dependencies also need to be built for multiple architectures, so relying on distribution packages is unlikely to work.

The Cosmopolitan Libc project does include several common dependencies, such as ncurses, zstd, zlib, SQLite, and others in its repository. So if a project only uses common third-party dependencies, it is usually fairly straightforward to build the necessary libraries. For projects that use GNU Autotools, the superconfigure project provides tools for integrating Cosmopolitan Libc into the build process.

One obvious question when building programs that should run across multiple operating systems is how to translate system calls between platforms. Unfortunately, a complete solution is simply not possible — some operating systems support operations that just aren't available on others — but Cosmopolitan Libc does a good job of covering the common operations. The documentation lists exactly which functions are available on which platforms. Generally, most common POSIX operations are usable without issues. Any more recent, specialized, or obscure APIs may require the use of the C preprocessor to select the right implementation. That is, of course, the normal state of affairs for portable software. But at least with Cosmopolitan's polyfills, programs can focus on only doing that for the small portion of functions where it is truly necessary. These functions are where Cosmopolitan Libc gets the "Libc" portion of its name from; while the tooling does support linking programs with the GNU C library, musl libc, or any other local C library, the usual case is to link to Cosmopolitan's Libc.

I attempted to build several pieces of software (GNU Bash, TCC, and some software of my own) with the Cosmopolitan toolchain in the course of writing this article. Overall, it was remarkably straightforward. I ran into a few confusing compiler errors until I found the correct set of configuration flags, but after configuring the projects to expect static linking and to use cosmocc and its associated linker, everything did work as promised.

Additional facilities

The fact that Cosmopolitan has to be involved at every step of the building and linking process in order to produce APE files allows it to pull a few other tricks, as well. In addition to being an executable and a shell script, every APE file is also a Zip archive. This is due to the fact that Zip archives store their directory information at the end of the file, so the information can be appended to the APE binary without interfering with the already-overloaded first few bytes of the format.

A few of Tunney's other projects take advantage of this, including redbean, llamafile, and others. Redbean is a web server that serves dynamic web sites made using Lua and SQLite from its own Zip archive; the project lets programmers distribute these web sites as self-contained portable executables, suitable to be run by end users. Llamafile, which LWN covered earlier this year, is a project that packages LLM weights alongside the code required to run them, for ease of distribution and archiving.

Other projects have used the ability to treat APE executables as Zip archives to bundle data files like a copy of the timezone database as a sort of super-static-linking that ensures the program always has the resources that it needs to run. Data files aren't the only thing required to be distributed alongside programs, however — many licenses also require distributing copies of the license, or copyright attributions.

Cosmopolitan Libc's tooling automates compliance with those licenses by making sure that licenses and attributions for all of the Libc code or (supported) third-party libraries that are linked into the final APE file are embedded in a special linker section. Cosmopolitan Libc itself is licensed under the ISC license (a simplification of the MIT/BSD licenses), but some of the third-party components that Cosmopolitan either uses directly or makes available to compiled programs are licensed under the BSD, MIT, or Apache 2.0 licenses. The Cosmopolitan compiler itself includes GPL-licensed code from GCC, but all of the code linked into finished binaries is licensed under a permissive license (unless one chooses to statically link the GNU C library), so the final result is not covered under the GPL.

Cosmopolitan Libc is a fairly active project; it has regular, small releases. The most recent is v3.9.6. As might be expected for any project with such ambitious compatibility goals, there are usually a number of small bugs affecting different platforms open at any given time. The project is fairly stable, however, with the basic ability to compile programs across multiple systems unchanged since the initial release.

So, while the project does still have its rough edges, it's a promising tool for people who want to write highly portable software. The automated license compliance, static compilation, and easy bundling of non-software dependencies are nice additions to the already compelling idea of running the same binaries on multiple platforms.