Testing compat_linux on NetBSD with the Linux Test Project

NetBSD has long had support for running Linux binaries via its kernel-level compat_linux feature, but test coverage for it was less complete than some might hope. In order to provide better testing for compat_linux, Google Summer of Code (GSoC) participant Henrique Brito opted to work on enabling the Linux Test Project (LTP) test suite to compile and run on NetBSD. At EuroBSDCon 2026, Brito's mentor, Stephen Borrill, provided a report on the project, and the status of LTP on NetBSD. The work has already resulted in some minor fixes, and a good list of additional problems to solve.

I did not attend EuroBSDCon 2026 in person, but had the opportunity to watch Borrill's presentation shortly after it was live-streamed. The video is available along with the other EuroBSDCon 2026 videos on a PeerTube instance, and the slides are linked from his web site.

Borrill, who has been a NetBSD developer since 2007, said that he runs a company that develops thin clients and servers based on NetBSD. Many thin clients need to access virtual desktops using a proprietary Independent Computing Architecture (ICA) application from Citrix. There is no NetBSD-native version of this client, but there is a Linux version that will run under compat_linux, and Borrill maintains its package for the NetBSD Packages Collection (pkgsrc), which is a repository of third-party software.

He ran into a new version of the ICA client that would not run under compat_linux. Instead, it would fail with an ENOSYS error, indicating a system call that was not implemented. That led him to take an interest in compat_linux, and he added an entry to NetBSD's list of suggested projects for GSoC to emulate missing Linux system calls.

compat_linux

The first GSoC project did not solve Borrill's problem; that project led to a native implementation of the splice() system call that can be called by compat_linux. The ENOSYS error he had been running into was fixed separately when the timer_create() call was added; after that, Borrill said, he didn't care much about emulating missing system calls. He was, however, interested in testing linux_compat.

The original title of Borrill's talk included the phrase "Linux emulation" to describe compat_linux, which he said is not quite right; NetBSD runs Linux code natively, with "virtually no performance impact", when compat_linux is enabled. "All we're doing is having a small shim that's making it look like it's Linux in terms of things like system calls and how we access files." The mapping of Linux system calls to NetBSD calls is defined for each architecture in /sys/compat/linux/arch/<ARCH>/syscalls.master, where ARCH would be amd64, i386, etc. The NetBSD Internals documentation, though slightly dated, explains how the operating system examines an executable binary and determines which kernel ABI to use.

The compat_linux feature is not enabled by default in NetBSD's GENERIC kernel, but it is available as a module. Borrill said that "everything's been moved to modules" for security reasons. compat_linux translates Linux system calls into NetBSD system calls and looks for any shared libraries that the application depends on in a "shadow root" directory under /emul/linux (or /emul/linux32 for 32-bit applications). For example, if a Linux binary tries to open /usr/bin/ldd, then NetBSD will first look to /emul/linux/usr/bin/ldd before falling back to /usr/bin/ldd.

As one might expect from NetBSD, compat_linux has support for quite a few architectures, including 32-bit and 64-bit applications for both x86 and Arm. Borrill noted that NetBSD also includes support for running FreeBSD, SunOS, and Ultrix binaries and "used to support a lot more, you know, a really crazy amount". However, the need to run many of the other operating system's binaries has slowly died away—as well the hardware needed to run those binaries in numerous instances.

Testing compat_linux

NetBSD has a native testing framework called Kyua, but it does not have extensive coverage for compat_linux. Rather than trying to add that support into NetBSD's regular test suite, Borrill thought the project might be able to build LTP for NetBSD and use it instead. He noted that LTP has tests for "all sort of things; but for the moment we're only looking at the syscall side of it". He said that consists of about 376 collections of tests with about 1,300 tests in total.

The first decision was how to build LTP; rather than trying to cross-compile it, Brito and Borrill opted to build it with a Linux version of GCC under compat_linux instead. They chose to use the SUSE 15.5 compatibility package from pkgsrc, created a Linux compatibility package for GCC 12 and uploaded that to pkgsrc. Once the test suite was running, he said, they noticed that most of the tests that were run passed, but many were being skipped.

Brito thought that it would turn out to be a boring project if everything passed; Borrill looked into it and noticed that Brito had started with NetBSD 10.1. That release has a compat_linux that presents itself as Linux 3.11, which was released in 2013. Obviously, quite a bit had changed since that kernel was released; the SUSE 15.5 packages for NetBSD, for instance, expect Linux 5.14.21. Testing system calls that were added since 3.11.6 would certainly increase the odds of finding problems as well as help to ensure that more Linux applications would work.

Borrill said that they could just change the configuration to declare a different version of the kernel for compat_linux, but it was decided that a better fix would be to use NetBSD 11, which has a compat_linux that declares 6.3.10 as the Linux kernel version, even though NetBSD 11 had not been released yet. However, the fourth release candidate of NetBSD 11 was available, and the final release was announced on July 30, about two months after the start of coding for this year's GSoC.

Not surprisingly, they also had to skip some of the tests because of features that were not available on NetBSD. For example, Borrill said that LTP attempts to test filesystems that NetBSD does not support, such as Btrfs and XFS, so they restricted the tests to ext2, ext3, ext4, and tmpfs. Ultimately, Brito created a manager script (compat_linux_test_project) to handle running LTP against compat_linux. It accounts for the differences between NetBSD and Linux to compile LTP properly, run tests, and skip tests known to (for example) hang or panic the kernel. They also decided to package the work for others to use with an eye to publishing it via pkgsrc. It is not yet merged into the official pkgsrc tree, but it is currently available in the emulators/compat_linux_test_project directory of Brito's pkgsrc fork on GitHub.

Borrill suggested that they try compiling everything statically, rather than dynamically, so that there would be no additional requirements "like all the Linux libraries and things like that". The package came out to about 80MB in size when compiled dynamically, but when built statically it turned out to be about 5.1GB. "I built this while I was down at the pub and I had time to order and drink a whole pint while it was just trying to build the package into a tarball at the end." That is probably a no-go, he said, but the option to compile everything statically is still available for anyone who wants it.

It takes about 25 minutes to run all of the tests, but the script allows users to specify which system calls they would like the test to avoid having to re-run the entire batch. For example, this command would test the readv(), writev(), open(), and close() system calls.

    $ compat_linux_test_project -s readv, writev, open, close

The script also has options for users to re-run tests and compare them against previous. "We've also included a sort of default baseline of what we believe the current state of affairs is, so you can just compare the run on your machine with the defined baseline we already have."

Failure types

linux_compat has implemented most of the Linux system calls "in that you can generally have a first stab at taking a Linux binary and expect it to work", Borrill said. The ways that it doesn't work, though, are interesting and varied; there are many types of failure that might present themselves.

The easiest ones to deal with are features where it's known that "it's not supported and we don't care about that". Likewise, some system calls will be missing entirely; for example, NetBSD does not support any of the system calls related to Linux namespaces. Later in the talk, he mused that the missing namespace system calls would be nice to have for NetBSD but, "definitely outside the scope of a GSoC project".

In some cases, a test may exercise a system call supported by compat_linux that "might look right on the face of it" but fail because of a Linux-specific flag that is not supported on NetBSD. In other cases failures may turn up something that NetBSD just hasn't implemented yet, or it could be "something that only makes sense within the Linux world".

There are also environmental variables and configurations to consider when running LTP on NetBSD; Borrill noted that LTP looks for the .config file for the Linux kernel to see what to run, but since compat_linux does not supply a kernel, it also lacks a configuration file. Brito's script takes care of supplying the appropriate .config so that LTP skips unsupported subsystems and operations. The last type of failures he mentioned were the exciting ones "where everything crashes, machines die, all those sorts of things".

Examples

He then cited some specific examples, such as lseek() which has two flags that are not implemented: SEEK_DATA and SEEK_HOLE. That caused the test to stop early, so it did not flag later tests as failed.

Many of the file-open tests failed because compat_linux did not support the open() system call's O_TMPFILE and O_PATH flags. A fix for that, he said, might be to actually implement those in the NetBSD kernel so that compat_linux inherits them natively. Another option, for things that would not be useful for NetBSD, would be to implement it only within the compat_linux layer. An example of that is the rt_sigqueueinfo() system call used to send a signal and data to a process; that is only available via compat_linux.

One of the interesting failures, he said, was that the behavior of Linux diverges from BSDs when iovcnt = 0 is passed to writev(). The BSDs, he said, return EINVAL as specified by POSIX—Linux returns "0" instead, indicating that, as requested, no data was transferred. "The tests were failing, but by all the specs, we were doing the right thing. And that's because zero is an invalid value, and so we were returning EINVAL." That would not be implemented natively, but could be implemented in compat_linux in two or three lines to "sort of be bug-compatible" with Linux.

But does compat_linux always need to be bug-compatible? He pointed out that LTP's tests for setsockopt() and shmat() look for known Linux bugs that are not present in NetBSD. "So what do we do with those? Do we just ignore them? Or do we need to be bug-compatible like we were with the writev() one?"

One test failure that was discovered had "a certain sense of irony", he said, because it was related to a system call added in a previous GSoC. The call copy_file_range() causes a kernel panic on NetBSD when off_in or off_out, which give the target ranges to copy from a file and to a file (respectively), are not NULL; the call should support non-null values for the file offset. It was simple, and the fix has already been committed to NetBSD.

What's next?

The GSoC project focused on testing x86 and x86-64. Borrill said that one of the obvious next steps would be to work toward testing on multiple architectures, in particular Arm. "Some of the more esoteric architectures we're probably not going to bother with". Supporting Arm, however, will require updating the SUSE compatibility packages available on pkgsrc to support additional architectures. He said that he had not yet had time to do that, but he thought it would be really useful and planned to "get a Raspberry Pi up and actually give it a go" once he was home from the conference.

As part of the project, Brito has put together a spreadsheet of the test results of system calls considered fundamental; that is, those that are used more often by Linux applications. Borrill said that, because Brito had done the legwork to identify not only which system calls were failing but why, there is now "a shopping list of tasks we can pick up", which may be useful not only for compat_linux, but NetBSD itself.

Borrill wrapped up his talk by thanking Google for supporting NetBSD through GSoC, Brito for his work on the project, and Leonardo Taccari who became involved in NetBSD as a GSoC participant, later became a mentor, and now helps to organize the project's involvement in the program. "He's really gone up the management tree there. His pay grade hasn't gone up, but he's been fantastic at organizing everyone".

Questions

The first person to be called on said: "In the past, I remember that futex() support was a big issue, and that it wasn't well supported in the Linux emulation. Is it better now?" Borrill replied that it was not; it was just one of the things he had not specifically covered during the talk. He noted that it was documented in Brito's spreadsheet.

Another audience member wanted to know whether compat_linux had support for versioning when there were "different versions of the same thing, like earlier version and later version". For example, what if the behavior of a system call was different between Linux 3.11 and 5.15? Did the compatibility layer support both versions, or just emulate the most recent version. Borrill said that the short answer was that he didn't know; NetBSD does have system-call versioning, but "we don't do any of that in the Linux compatibility". It does, he added, support two versions of SUSE binaries, with the latest being 15.5. He referred the questioner to the spreadsheet for more information.