LWN.net Weekly Edition for December 13, 2018

Welcome to the LWN.net Weekly Edition for December 13, 2018

This edition contains the following feature content:

• Measuring container security: are containers more or less secure than virtual machines? Answering that question requires finding a way to measure the relative security of the two options.
• A filesystem corruption bug breaks loose: how a filesystem-corrupting bug came about and some questions it raises.
• Kernel quality control, or the lack thereof: a disagreement over a stable-kernel backport sparks a debate on the kernel development process.
• DMA and get_user_pages(): fixing a longstanding race condition in the kernel.
• Toward race-free process signaling: an attempt to fix another longstanding race condition, this one in user space.
• Large files with Git: LFS and git-annex: two options for managing large files in Git without the associated pain.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Measuring container security

There are a lot of claims regarding the relative security of containers versus virtual machines (VMs), but there has been little in the way of actually trying to measure those differences. James Bottomley gave a talk in the refereed track of the 2018 Linux Plumbers Conference (LPC) that described work that targets filling in that gap. He and his colleagues have come up with a measure that, while not perfect, gives a starting point for further efforts.

Bottomley introduced himself as a "container evangelist" for IBM. He used to help convert businesses to becoming part of the open-source community. Working at Parallels on that is how he got involved with containers. He is also a kernel developer and maintainer.

Containers and hypervisors

He began with some "container basics". The difference between containers and hypervisors, which run VMs, is where the interface is located. Hypervisors are based on emulating hardware, he said; they can bring anything up that expects to talk directly to the hardware (e.g. a Linux kernel). Containers are about virtualizing the subsystems of the operating system (OS). The interface is the system-call interface provided by the OS.

If you look at the "container revolution" today, "everything is Linux". There is really no other kind of container out there, he said. That is because of the "hardness guarantees" of the Linux system-call interface and its stability over time. There has been no container system for Windows because its system-call interface changes with every release and the line between user space and the kernel is blurry—and changeable.

Containers all run on a single kernel, while hypervisors run a host kernel and a guest kernel per VM. There is a benefit to that for containers because one kernel means that there is only one resource manager for the whole system. Resource sharing between containers provides agility; there is instant scaling up or down without needing to "inflate balloons" or do any of the other tricks that are needed for hypervisors. Resource decisions can be made more efficiently for containers since they aren't done across a virtual hardware interface.

Docker is rather famous in the container world, but containers are not really what led to that fame. Docker provided a way to box up an application and all of its dependencies so that what got tested on a developer's laptop was exactly the same as what runs on the host. Docker is really nothing more than "an application packaging and transport system" that was enabled by containers, Bottomley said. One of the "dirty secrets of hypervisors" is that you can't build and run a VM on your laptop and then deploy it to, say, the Amazon cloud; there is a mismatch in drivers that will prevent that from working.

Security risks

The crux of his talk is that there is a great benefit to the sharing that containers allow, but sharing increases the security risk. Hypervisor advocates seek to exploit this by noting that hypervisors use a small interface, while "containers are shit" because they can use any of the 300 Linux system calls. Those system calls are the most easily exploited part of Linux, so container security is clearly worse, they would argue. In truth, Bottomley said, hypervisor advocates have a bit of container envy because of all the excitement around containers. They want to "Make Hypervisors Great Again", he said, showing an image of a red baseball cap with that slogan. Luckily, most of what the hypervisor advocates are saying is "fake news", he said with a grin. Bottomley's web-based slides are available; use the arrow keys to navigate.

The real problem with all of this advocacy around security is a lack of facts. He didn't provide any real facts about containers either, just "waved my hands around". There is no "intellectual rigor" in any of these debates. Part of what he and his colleagues are trying to do is to put together some way to measure the relative security to provide some rigor.

Containers on Linux all use control groups (cgroups) and namespaces. LXC, Docker, Mesos, and others all build on the same interfaces. Hypervisors all use different interfaces. The container API was agreed upon at the Kernel Summit in 2011. The various players converged on a unified upstream API to avoid a repeat of the split between Xen and KVM.

The cgroups are available for resource control; there are many of them, each controlling a different resource. Those cgroups can be turned on or off for any given container; this makes the definition of a container less hard and fast than that of a hypervisor, which has a fixed interface based on the virtual hardware. There are also lots of namespaces for virtualizing various kernel resources, which can be used or not by different container solutions. User namespaces, in particular, are a tool that can help secure containers.

Docker advocates will say that the Linux container API is "almost impossible to use", but that's not true. It is a myth that orchestration people push, Bottomley said. Docker is just the start of what containers can do and it is the source of a lot of the security problems because most Docker containers do not use a user namespace. That means that "real root" is used inside the container, which is a real security problem, as hypervisor advocates are eager to bring up. One of the discussions in the containers microconference the previous day was about how to get Docker to incorporate user namespaces into its containers, he said.

Attack profiles

In order to counter the hype, there is a need to find a measure to define "security" or "containment". Even when security experts talk about "security" they often cannot agree because there are no numerical measurements that can be compared dispassionately. He asked his research group to help him figure out some kind of numerical measure that could be used. What they came up with is something called the "attack profile"; it is a good first approximation of a measure for security.

An internet-exposed application takes input from the net, makes calls down the stack until it reaches the hardware then traverses back up to give a response. As it does that traversing, it could encounter bugs that would allow attackers to exploit the system. So the vertical attack profile is the number of lines of code encountered as the application performs its normal function multiplied by the bug density. But the bug density is a constant for the kernel as a whole, which is the shared piece, so it can be thrown away, leaving just the lines of code as the rough measure of the attack profile.

Where the discussion gets interesting is that some of the attack profile belongs to the tenant and some to the hosting provider. In a hypervisor system, the provider is only responsible for a relatively thin layer that provides the hardware emulation, so the tenant is responsible for everything else. In a container system, the tenant's responsibility stops at the system-call interface; the tenant is responsible for much less of the vertical attack profile.

Hosting providers would generally prefer their tenants to use VMs, rather than containers, because it makes the provider responsible for less code, Bottomley said. While providers might claim that hypervisors make their tenants more secure, they really are simply trying to shift responsibility for the kernel to the tenant.

The horizontal attack profile looks at the whole system and all of its containers or VMs; it gives a measure of the overall chance of being attacked by the exposed shared code. The kernel has a large horizontal attack profile as it exposes a huge number of interfaces.

This large horizontal attack profile is also why there are no providers of bare-metal containers; they all run containers on top of a hypervisor. IBM's Blue Mix cloud did provide bare-metal containers because IBM wanted to show that even though the hypervisor fans claim that containers are hundreds of times less secure than hypervisors, it simply isn't true, Bottomley said. If it were, the Blue Mix cloud should have seen many more exploits than the hypervisor-based container providers—and didn't. "There must be a reason for this", which is what led to the research and the idea of using attack profiles as a measure of security.

An exploit in the horizontal attack profile is the worst kind for a hosting provider. It means that a container can attack other containers via the system-call interface. That profile is roughly the number of lines of code in the kernel multiplied by its bug density.

He and his team decided to measure the horizontal profile of Docker versus that of a Kata container running on KVM. They used Ftrace to determine which kernel functions were being called from the Kata container through KVM to the host versus the functions being called by a Docker container on the host. He put up a graph showing the number of unique kernel functions accessed by each for three separate tests (for Node.js, Redis, and Python Tornado). While Docker is worse than Kata for all three, it is not many times worse. It is not a factor of 100, more like 10-30% he said.

The tests that were run are well-behaved applications, whereas a more comprehensive test would perhaps use a fuzzer to better simulate misbehaving attackers. But if the system calls invoked by an application can be identified, a reasonable seccomp filter could be applied to the Docker container so that it could only use those calls. He is not promising that Docker can be made completely secure, but he is promising that there is a way to secure Docker such that its risk matches that of a hypervisor within a few tens of percent. Putting a good seccomp filter in for Docker is definitely difficult ("horrible") but it can be done, he said.

Now that there is a measure, it can be minimized. Experiments can be run with different container descriptions; those descriptions can be crafted to reduce the horizontal attack profile. This is what projects like Google's gVisor and IBM's Nabla containers are trying to do, he said. With Nabla containers, he set out to be able to claim that they are more secure than hypervisors based on the horizontal attack profile measure because he knew that would get press attention.

But he also knows that measuring the horizontal attack profile is not the end point. "I think this is the beginning of the conversation of how we measure security." He has a "funny feeling" there are much better ways to measure the security of these systems; "now we need to find out what they are".

There are some problems with using horizontal attack profile as a measure. The kernel contains many bugs, but not all are exploitable; there is a need to incorporate some measure of "exploitability" into the equation. He suspects that the interface description plays a larger role than the number of lines of code. System calls that have more sweeping effects are more likely to harbor exploitable bugs. But that is all speculation at this point.

Bottomley's best guess is that the next generation of security measurements will focus more on the interfaces. Some interfaces are "inherently more exploitable" than others; there are kernel system calls that are often exploited and some that no one has ever found a way to exploit. Somehow that needs to be taken into account. Perhaps having a way to monitor calls to the dangerous system calls "with eBPF or something", and disallowing any of the insecure uses, might be a way to help secure the kernel interfaces. That would be useful "independently of whether we use it for containers or not".

Other kinds of containers

Sandboxing is a way to run a container that does not expose it to the full Linux interface. Instead, some of the system calls are emulated inside the sandbox. That starts to sound "a bit like a mini-hypervisor running and calling itself a container", he said. But a sandbox is a way to get security containment without doing full hypervisor set up and image building.

So, for example, Nabla containers do not use namespaces, just cgroups; namespaces are not needed because of the system-call emulation. In reality, though, Nabla containers do use a network namespace because that is required by Kubernetes. But emulation in the sandbox means that the code is not shared between containers, thus reducing the horizontal attack profile, at least in theory. It does push the responsibility for that emulation layer on the tenant, however, because it is not shared. With Nabla containers, IBM has found a way to push the emulation layer back into the remit of the provider, though Bottomley said he wouldn't go into details as there is a whole other 45-minute talk needed to describe it.

The difficulty is to get the sandboxing right without pushing the whole thing out to the tenant as hypervisors do. A benefit of the cloud should be that it shifts the burden of infrastructure security away from the tenant. Providers have response teams to handle various infrastructure problems, which should relieve the tenant from having to handle them. Most hosting providers are trying to make as much as possible be the responsibility of the tenant, but that runs counter to the advantages of the cloud. In the end, though, "it doesn't mean that your crap application is any less crap", he said; tenants do need to focus on securing their applications, but they shouldn't need to do more than that.

Two of the better known sandboxing systems are gVisor and Nabla, he said. GVisor has rewritten many system calls in Go, while Nabla uses unikernel techniques to create a library operating system with a container profile. Nabla is meant to create containers that are more secure than hypervisors, at least based on the horizontal attack profile measure that his team developed.

He put up a graph (using the same data as in a graph in this blog post) showing the number of unique kernel functions accessed for the same three tests. One thing to note about the results, he said, is that gVisor came in worse than most of the other solutions; that is because it uses Go for the system-call emulation and the Go runtime is "incredibly profligate in terms of system calls". For the horizontal attack profile measure, that makes it appear to be far less secure, though that may not actually tell the full story. As might be guessed, the graph shows that Nabla fares better than any of the others: Docker, Kata, gVisor, and gVisor on KVM. Nabla "is not significantly better than hypervisors", but it is better on all of the tests; on the Node.js test Nabla uses less than half of the kernel functions than Kata does. He is hoping this "puts to rest" the argument about whether hypervisors or containers are more secure; "the answer is neither, unless you build them correctly".

He also noted that the analysis of these tests found an anomaly in the way that Kata containers accessed the Linux kernel; he asked an intern to see if it could be exploited. He wanted to see if there was a way to run code in a Kata container that could oops the host kernel. The intern was successful in doing that and sent him a video of the exploit just that morning; it was written up on the Nabla blog after the conference. The exploit is not particularly serious; it exploits the Plan 9 filesystem interface that Kata uses to transport filesystem operations from the guest to the host. But it does show that "you have to be incredibly careful when you set up interface descriptions", even for VMs; without the right interfaces, "you may not be getting all of the benefits of a virtual machine", Bottomley said.

Next up

He then turned to some ideas for the future. Separation or segmentation of system calls within the kernel might be a useful thing to look at. If, for example, system calls could be each run in their own tenant-owned address space, they could no longer interfere with each other, which might be helpful. Perhaps running parts of the kernel in user context would be another way to segment system calls, though "Linus [Torvalds] is going to scream" when people start experimenting with that. Another possibility is to use supervisors, using something like Linux Security Modules (LSMs), to correct problems in the system-call interface. We are now starting to measure where those problem areas are, he said, so we may be able to craft an LSM that protects the system calls better than the existing LSMs, which are all based on "security, handwavy stuff of 'we think these interfaces are insecure'".

There is also a need to look at the vertical attack profile. Hosting providers should be trying to reduce that for tenants, not by reducing the stack, but by taking responsibility for more of it. There is a need to look at ways of moving up the stack and taking that responsibility, he said.

In conclusion, there is "lots of exciting progress to be made". The field of measuring container and hypervisor security is in its infancy; the current measure is crude. There is a need for second and third generation measures that go much further. There is, Bottomley said, a need for much more research into all of this.

A YouTube video of the talk is available.

[I would like to thank LWN's travel sponsor, The Linux Foundation, for assistance in traveling to Vancouver for LPC.]

Comments (15 posted)

A filesystem corruption bug breaks loose

Kernel bugs can have all kinds of unfortunate consequences, from inconvenient crashes to nasty security vulnerabilities. Some of the most feared bugs, though, are those that corrupt data in filesystems. The losses imposed on users can be severe, and the resulting problems may not be noticed for a long time, making recovery difficult. Filesystem developers, knowing that they will have to face their users in the real world, go to considerable effort to prevent this kind of bug from finding its way into a released kernel. A recent failure in that regard raises a number of interesting questions about how kernel development is done.

On November 13, Claude Heiland-Allan created a bug report about a filesystem corruption problem with the 4.19.1 kernel; other users joined in with reports of their own. Initially, the problem was thought to be in the ext4 filesystem, since that is what the affected users were using. Tracking the problem down took a few weeks, though, because few developers were able to reproduce the problem. There were some attempts at using bisection to find the commit that caused the problem, but they proved to be worse than useless, as they identified the wrong commits and caused developers to waste time on false leads.

It took until December 4 for Lukáš Krejčí to correctly bisect the problem down to a block-layer change. Commit 6ce3dd6eec, added during the 4.19 merge window, optimized the handling of requests in the multiqueue block layer. If there is no I/O scheduler in use, and if the hardware queue is not full, this patch causes new I/O requests to be placed directly into the hardware queue, shorting out a bunch of unnecessary processing. It's a harmless-seeming change that should make I/O go a little faster.

Things can go bad, though, if the low-level driver for the block device is unable to actually execute that request. This is most likely to happen as the result of a resource shortage — memory, perhaps, or something related to the hardware itself. In that case, the driver will return a soft failure, causing the I/O request to be requeued for another attempt later. While that request sits in the queue, the block layer may merge it with other requests for adjacent blocks, which should be fine. If, however, the low-level driver has already done some of the setup for the request, such as creating scatter/gather DMA mappings, those mappings may not be updated to match the larger, merged request. That results in only part of the request being executed by the hardware, with bad effects on the data involved.

The problem was partially fixed with this commit, but one more fix was required to fix a new problem caused by the first. Both fixes were included in the 4.20-rc6 release; they also found their way into 4.19.8. The original patch was never selected for backporting to older stable kernels, so those were not affected.

How this happened

Naturally, some developers are wondering how a problem like this could have made it into a final kernel release without having been noticed. Surely automated testing should have been able to find such a bug? Not all parts of the kernel have exhaustive automated testing regimes (to put it charitably), but the block layer is better than most. That is a function of the severity of bugs at that layer (which can often cause data loss), but also of the relative ease of testing that code. There are few hardware-specific issues to deal with, so testing can usually be done in virtual machines.

This particular case, though, turned up a hole or two in the testing regime. It required a filesystem on a device configured to use no I/O scheduler at all (a relatively rare configuration on the affected filesystems), and that device needed to run into resource limitations that would cause it to temporarily fail requests. The driver for the device also needed to store state in the request structure and use that state in subsequent retries. Finally, the multiqueue block layer must be in use, which only happens by default for SCSI devices as of 4.19. That last change is unlikely to have been picked up by many developers or testing setups, since kernel configuration files tend to be carried forward from one release to the next.

As a result of all those factors, nobody doing automated testing of the block layer reported this particular problem, and developers found themselves unable to reproduce it once it came to light. Ubuntu users who install from the kernel-ppa repository, instead, got a shiny new configuration file with their bleeding-edge kernel and were thus exposed to the problem. This sort of occurrence is one of the reasons why developers tend to like to remove configuration options; more options create more configurations that must be tested. In this case, block maintainer Jens Axboe has said that he will be writing a new test for this particular issue. He also noted that "this is the first corruption issue we've had (blk-mq or otherwise) in any kernel in the storage stack in decades as far as I can remember".

Doing better next time

There have been some suggestions that the kernel community should have done more to protect users once the problem was discovered. It is not clear that there is a whole lot that could have been done, though. Arguably there should be a mechanism to inform users that a given kernel might have a serious issue and should not be used, but the community lacks such a mechanism. This particular issue was, in fact, less visible than many since it was not discussed on the mailing lists. The only people who were aware of it were those who were watching the kernel bugzilla — or who were busy restoring their filesystems from backups.

Laura Abbott noted that the problem left Fedora users in an awkward position: they could either run a 4.19 kernel that might mangle their data or run the 4.18 kernel, which was no longer being supported. Perhaps, she said, there should be a way to respond to problems like this?

Willy Tarreau responded that he ensures that the previous long-term stable release works on the systems he supports for just this reason. Dropping back to 4.14 is unlikely to be a pleasing alternative for many users, but it's not clear that something better will come along. Some people would surely like it if the previous release (4.18 in this case) were maintained for longer but, as stable maintainer Greg Kroah-Hartman put it, that "isn't going to happen"; the resources to do that simply are not available.

Kroah-Hartman did have one suggestion for situations like this: tell him and the other stable maintainers about it. In this case, he was only informed, by accident, shortly before the bug was tracked down and fixed. There is not much that could have been done even if he had known sooner, since nobody knew what the origin of the problem was. But keeping the stable maintainers in the loop regarding serious problems that have appeared in stable kernels can only help to get the fixes out more quickly.

One other aspect of this bug is that, depending on how one looks at it, it could be seen as resulting from either of two different underlying issues. One is that the multiqueue block layer is arguably still not sufficiently mature, so it is turning up with severe bugs. The other is that maintaining two independent block subsystems is putting a strain on the system and letting bugs get through. One's point of view may well affect how one views the prospect of the legacy block API being removed in the next merge window, which is the current plan. Ted Ts'o let it be known that this idea "is not filling me with a lot of joy and gladness". But for many others, the time to make this transition is long past and, in any case, the number of devices that can only use the multiqueue API is growing quickly.

The good news is that problems of this severity are rare and, when they do happen, they get the full attention of the developers involved. Some early adopters were burned, which is never a good thing, but the vast majority of users will never be affected by this issue. Some testing holes have been identified that will hopefully be closed in the near future. But no amount of testing will ever reveal all of the bugs in the system; occasionally a serious one will escape and bite users. With luck and effort the number and severity such events can be minimized, but they are not going to be entirely eliminated anytime in the near future.

Comments (31 posted)

Kernel quality control, or the lack thereof

Filesystem developers tend toward a high level of conservatism when it comes to making changes; given the consequences of mistakes, this seems like a healthy survival trait. One might rightly be tempted to regard a recent disagreement over the backporting of filesystem-related fixes to the stable kernels as an example of this conservatism, but there is more to it. The kernel development process has matured in many ways over the years; perhaps this discussion hints at some of the changes that will be needed to continue that maturation in the future.

While tracking down some problems with the XFS file cloning and deduplication features (the FICLONERANGE and FIDEDUPERANGE ioctl() calls in particular), the developers noticed that, in fact, many aspects of those interfaces did not work correctly. Resource limits were not respected, users could overwrite a setuid file without resetting the setuid bits, time stamps would not be updated, maximum file sizes would be ignored, and more. Many of these problems were fixed in XFS itself, but others affected all filesystems offering those features and needed to be fixed at the virtual filesystem (VFS) level. The result was a series of pull requests including this one for 4.19-rc7, this one for the 4.20 merge window, and this one for 4.20-rc4.

More recently, similar problems have been discovered with the copy_file_range() system call, resulting in this patch set full of fixes. Once again, issues include the ability to overwrite setuid files, overwrite swap files, change immutable files, and overshoot resource limits. Time stamps are not updated, overlapping copies are not caught, and behavior between filesystems is inconsistent. Chinner's patch set contains another set of changes, almost all at the VFS level, to straighten these issues out.

Mainline quality assurance

The discovery of these bugs has brought a fair amount of disappointment with it. It seems clear that these new features were not extensively tested before being added to the kernel; certainly no automated tests had been added to the xfstests suite to verify them. Dave Chinner put it this way:

In time, these bugs will be fixed and users of all filesystems should benefit. Tests are being added to help ensure that these features continue to work in the future. This is clearly a necessary effort; Chinner and Darrick Wong are performing a service for the kernel community as a whole by taking it on. This work does raise a couple of interesting issues, though.

The first of those is the prospect of regressions in programs that have come to depend on the behavior of these system calls as it is supported in current kernels on specific filesystems. Hyrum's law suggests that such users are likely to exist. Or, as Chinner put it: "the API implementation is so broken right now that fixing it is almost guaranteed to break something somewhere". That could lead to some interesting discussions if users start to complain that kernel updates have broken their programs.

Stable updates

The other question that arises concerns the backporting of all these fixes to the stable kernel updates; indeed, it was the selection of one of the VFS fixes for backporting that set off the current conversation. The XFS developers have long been hostile to the automatic inclusion of their patches in stable updates, feeling that the work to validate those patches in older kernels has not been done and that the risk of creating new regressions is too high. As a result, XFS patches are not normally considered eligible for backporting, but that exclusion does not extend to fixes at the VFS layer.

In this case, Chinner stated that the current set of fixes has been validated for the mainline with a testing regime that runs billions of operations over a period of days; anything less risks not exposing some of the harder-to-hit bugs. Backporting those fixes to a different kernel would require the same level of testing to create the needed confidence that they don't create new problems, he said, and the XFS developers are too busy still fixing bugs to do that testing now.

Chinner followed up with a lengthy indictment of the kernel development process as a whole, saying that it is focused on speed and patch quantity rather than the quality of the final result. The stable kernel process, in particular, is "optimised to shovel as much change as possible with /as little effort as possible/ back into older code bases". He pointed out that changes often appear in stable releases before they show up in a real mainline release (as opposed to an -rc release), which doesn't leave a whole lot of time for real stabilization. It is not, he feels, a small problem:

Sasha Levin responded that the current process is the best that we can do at the moment:

For the time being, the VFS and XFS patches will not be included in the stable kernel updates. Once the fixes are complete and the filesystem test suites have been filled out, Wong said, it should be possible to safely backport the whole set. At that point, this particular issue will be solved, but that is not likely to happen until after the 4.21/5.0 kernel release.

For the longer term, there is still the problem that, as Wong put it: "New features show up in the vfs without a lot of design documentation, incomplete userspace interface manuals, and not much beyond trivial testing". One might well argue that this problem extends beyond VFS features. The kernel community has never had much of a process around the addition of APIs visible to user space; there are no real requirements to ensure adequate documentation, testing, or consistency between interfaces. The results can be seen in our released kernels, and in the API mistakes that just barely escape release because the right developer happened to notice them in time.

Over the years, the kernel community has matured considerably in a number of ways. One need only look back to the days when we had no source-code management system, no rules on regressions, and no release-management discipline to see how much things have improved. The last few years have seen some big improvements around automated testing in particular. For all of our problems, the quality of our releases is quite a bit higher than it once was, even if it is not what it should be. Given time, it is reasonable to expect that we can build on that base to further focus our processes on the quality of the kernels we release, if that is something that the community decides it wants to do.

Comments (39 posted)

DMA and get_user_pages()

In the RDMA microconference of the 2018 Linux Plumbers Conference (LPC), John Hubbard, Dan Williams, and Matthew Wilcox led a discussion on the problems surrounding get_user_pages() (and friends) and the interaction with DMA. It is not the first time the topic has come up, there was also a discussion about it at the Linux Storage, Filesystem, and Memory-Management Summit back in April. In a nutshell, the problem is that multiple parts of the kernel think they have responsibility for the same chunk of memory, but they do not coordinate their activities; as might be guessed, mayhem can sometimes ensue.

Hubbard began by laying out the goals of the session. The idea is to make sure everyone knows about the problem; even though it has been discussed in various mailing-list threads and such, everyone may not be up to speed on it. He was hoping to get a consensus on a long-term fix; he has put out a few RFCs, but the solutions have been a bit contentious.

It is not that difficult to crash the kernel using a common code pattern, he said. An application pins pages in memory, which causes the kernel to use get_user_pages() or similar to map the user-space memory into the kernel's address space. It then sets up DMA for the pages. Though it was the RDMA microconference, the problem exists for all DMA operations (for, say, a GPU or FPGA), not just remotely over the network, Hubbard said. The drivers will then mark the destination pages as dirty and release them with put_page() (or release_pages()). But if the pages are backed by a non-RAM-based filesystem, things can go awry. This pattern has been "known" to be reasonable since 2005 or so, but it is actually broken.

The underlying problem is that the page buffers can be stripped off while the page is pinned and undergoing I/O; the writeback code believes it has written the dirty data, thus the buffers are not needed, but then the driver marks the pages dirty again when the I/O completes. Once that happens, writeback may come along (again) expecting that the page buffers are still there. Essentially, the filesystem is unaware that a page can be marked dirty outside of its purview. There are two subsystems that do not agree on which is the maintainer of the dirty state. All of that means that DMA to file-backed memory is not truly supported by Linux today, he said.

Hubbard went into some more detail on the crash, which can be seen in his slides [PDF] that also reference a lengthy email from Jan Kara about the problem. Kara was listed as one of the session leads, but he could not make it to LPC.

In answer to a question from an attendee, Hubbard said that the buffers are being freed because reclaim thinks it is going to get further than it does. It cannot progress because get_user_pages() has taken a reference to the pages. Reclaim has already released the buffers, but then has to back out. Williams added that if reclaim stays out of the way, everything works just fine; the problems occur when reclaim gets to the pages before the driver does.

Mel Gorman said that filesystems fundamentally assume that page_mkwrite() will be called before a page can be marked dirty. So when reclaim comes along, the page may be dirty, but the filesystem thinks it is safe to get rid of the buffers. He noted that dirty is not really a binary state.

Someone asked how big of a problem this is, does it occur every day, week, or month? Hubbard said it is reliably reproducible for him; he has advised customers to avoid doing DMA to file-backed memory. But it is rare enough that people write this kind of code, test it successfully, and then it fails when it is deployed in a more stressful scenario, an attendee said. Another suggested that the less reproducible a bug is, the more important it is to fix it. Hubbard agreed, saying he remembered a bug that took a month to reproduce, which meant it ended up taking him a year to fix it. Wilcox added that it definitely needs fixing as more and more systems are running into it.

Proposal

Hubbard then proposed that get_user_pages() "make a note" in the page structure that effectively says "get_user_pages() was here". One problem is that struct page is full—a perennial problem. An idea that Wilcox had come up with is to use the LRU next and previous pointers in struct page for flags and a reference count to track this; the pages would be removed from the LRU list while they are pinned, Hubbard said.

The way Hubbard is proposing to do this is to replace all of the affected put_page() and release_pages() calls with new put_user_page*() calls that will call put_page() after dealing with the tracking information and restoring the pages to the LRU list. His RFC patch set converts the InfiniBand driver, but there are around 100 other places that need to be converted. Some of those will need a fair amount of analysis to determine how to convert them, he said.

Williams cautioned that he is using the LRU pointers in DAX and that Jérôme Glisse is also using them in his heterogeneous memory management (HMM) patches. Wilcox admonished that he had freed up "a lot of space" in struct page for DAX and HMM; "there's going to be space for both of you". It is important to fix this, Hubbard said, since nothing stops users from doing it, as they have since 2005; "it just doesn't work", Williams added.

After the first RFC was posted, Andrew Morton asked how Hubbard was going to ensure that all of the conversions have been done. There is a need to not only solve the problem, but to show that it has been solved. To that end, he wants to add an assertion into the code that will trigger if put_user_pages() has not been called when it should have been. He needs a bit in struct page to track that, however. Wilcox said that he had another bit Hubbard could use, especially if it was only a temporary use while the conversion was in progress; "I've been saving it for you John", Wilcox said with a grin. Williams cautioned that "the first bit is free"—to widespread laughter.

Part of what he is trying to accomplish in the session is to raise awareness that this patch set is coming, Hubbard said. The RFC is just six patches that includes the InfiniBand conversion, which has been reviewed by Infiniband developers. But he wants to take some time to go into each subsystem and make sure that he understands how it is using the memory. For now, there is a dummy put_user_page() that simply does a put_page().

All of the work he had described so far just tracks the pages that have been pinned with get_user_page(). Once you have that information, you need to do something with it. Stopping the buffer removal is needed and can be done by holding off try_to_unmap() for those pages. But filesystem developers say that holding pages in that state is not a good idea, so Kara and others have said that the kernel could still allow writeback, but do it with bounce buffers that will be unaffected by whatever else is done with those pages.

revoke()

Williams said that he wanted to talk about APIs for revoke(), which would help with these problems where an mmap() region is shared and being used for DMA. If another process wants to truncate or punch a hole in the file in the region where DMA is being done, "you are screwed", at least for DAX.

But, as he has in the past, Glisse pointed out that there are some things that cannot be revoked. It will only work on devices that support it. Some drivers can stop the DMA, but others just pin the memory and have no way to stop the DMA and unpin the memory. Wilcox said that it is hardware dependent and, even for NVMe, which has a command to cancel outstanding I/O, most devices just do not implement it. That memory cannot be reused until the hardware says it won't DMA there anymore, he said. Williams suggested that the revoke() call should just wait for the device, but for some devices that wait might be forever, an attendee said.

Everything that calls get_user_pages() needs to be audited, but Williams was concerned that some drivers don't really know whether their pages have come from get_user_pages() or not. Hubbard said that he had to pass some extra tracking information down in some of his under-development conversions. If he could permanently free up a bit in the page structure to track whether these were pages pinned with get_user_pages(), Wilcox wondered, would that remove the need to change all of the call sites, since put_page() could simply do the right thing?

There was some thought that might work, but it didn't last long. There would be no way to distinguish other uses of put_page() from one that should do that "right thing". There would be a reference count, but there is no way to know that any given put_page() is the one from the driver that should clear the bit. So the call sites in the drivers still would need to change. It would help tracking these kinds of pages so that the extra information would not have to be passed down from upper layers, however.

The performance numbers that accompanied the RFC patches were questioned by an attendee. While they showed little performance impact for the changes, the numbers were orders of magnitude lower than what these kinds of devices should be able to do. The concern is that the numbers are so low that they shouldn't even be compared to determine what impact, if any, the changes actually made. As can be seen in a thread from after LPC, there was a measurement problem; once it was resolved, though, the impact still is minimal.

Returning to the subject of revoke(), Williams said that his goal was to get rid of the distinction between short-term and long-term DMA. That distinction was added so that DAX could simply fail any attempt to use its memory for long-term DMA. It is not just RDMA that does it; there are other devices, such as video-offload devices, that also pin memory forever. He suggested that maybe file leases could provide a model for handling the problem. if another process does a file truncate operation, the lease mechanism could provide a way to pull that memory away from the device.

Jason Gunthorpe suggested simply failing the truncation operation when the memory is in use for DMA. Wilcox said that he had been advocating that for years, but that other kernel developers will not allow it.

The pages belong to the filesystem, Boaz Harrosh said, so the mistake is in letting other parts of the kernel handle them in ways the filesystem cannot see. He suggested that DMA users take out a range lock for the part of the file that is being used for DMA, but Williams said that they cannot "rewrite the universe" to say that everyone must get a range lock. That will also have performance impacts that are unacceptable, Wilcox said.

It is clear that there are no easy solutions, but the planned path seemed agreeable to most. Wilcox pointed out that by removing the pages from the LRU list in get_user_pages(), writeback would never find them, so the crash cannot occur. For DAX, things are not so rosy, but the RDMA developers seemed willing to try to handle being told the memory they had pinned for DMA was going away, at least in exceptional circumstances. It certainly seems like a topic that will come up again—possibly multiple times over the next few years.

A YouTube video of the session is available.

[I would like to thank LWN's travel sponsor, The Linux Foundation, for assistance in traveling to Vancouver for LPC.]

Comments (9 posted)

Toward race-free process signaling

Signals have existed in Unix systems for years, despite the general consensus that they are an example of a bad design. Extensions and new ways of using signals pop up from time to time, fixing the issues that have been found. A notable addition was the introduction of signalfd() nearly 10 years ago. Recently, the kernel developers have discussed how to avoid race conditions related to process-ID (PID) recycling, which occurs when a process terminates and another one is assigned the same PID. A process that fails to notice that its target has exited may try to send a signal to the wrong recipient, with potentially grave consequences. A patch set from Christian Brauner is trying to solve the issue by adding signaling via file descriptors.

PIDs increase for each new process up to the maximum value, and then go back to the beginning. For the maximum value, most distributions use the conservative value of 32768 to avoid breaking legacy systems. However, users can consult and change the maximum value in /proc/sys/kernel/pid_max. Signal-related APIs identify processes by PID. The disadvantage of this method is that, in the lifetime of a system, the same PID is reused as processes are created and terminated. What happens if a process has finished and another one has taken its PID? The PID value stays valid. Other processes, unaware of the situation, may try to send signals to the wrong process. This may have consequences as serious as terminating the wrong service. This race condition requires the PID space to wrap between the creation of the two processes, which is not uncommon.

/proc/pid/kill proposal

The recent discussion started when Daniel Colascione proposed adding a file called kill to each process's /proc directory. Writing the numerical value of the desired signal to that file would send that signal to the selected process. The race was solved — or attempted to be solved — by holding the /proc directory open, thus preventing the PID from being reused at the wrong time. The discussion showed that other developers were considering the same problem in parallel.

While the problem was well understood, a debate started about the implementation. One part of the discussion concerned whether opening the /proc directory is enough to prevent PID reuse, in which case the patch would not have not been necessary in the first place. A small test case was developed and showed that the answer is no. A modification in the kernel, like the proposed patch, is needed to solve the issue.

A heated debate followed on a proper API to deliver the signal: should it be a write to the file or another system call? Jann Horn suggested an ioctl(), followed by Tycho Andersen, who agreed that this would simplify the permission checks. Colascione replied by supporting the choice of using write() and stated that it is unsafe to call ioctl() on an unknown descriptor. In case of a mistake, we do not know what the effect of a random operation will be (this is, however, similar when writing to an unknown file). His other option was adding a new system call.

Another part of the debate was started by Aleksa Sarai who added namespaces to the mix: there is a risk of processes sending signals between PID namespaces in situations when they normally do not have that ability. He suggested that only processes from the same PID namespace should be able to send such signals.

In the same thread, Brauner mentioned that he is working on a similar solution and proposed postponing the patch review to after the discussion at Linux Plumbers Conference that was then two weeks away. Colascione and other developers were interested to know more. Colascione also added some context to the discussion by noting that there had been previous, but unsuccessful, attempts to solve this problem. There are two options to fix the issue and keep the interface race-free: either keep a PID reserved when the handle is open, or keep a reference to the struct pid instead of the PID value.

Signaling by /proc/pid

After LPC, Brauner submitted a new patch set. It proposes to solve the signal delivery issue by using file descriptors to identify processes; these descriptors would be obtained by opening a process's /proc directory. The solution Brauner proposed is to store a handle to the process's struct pid in the inode associated with that descriptor; this gives a stable handle that does not have the disadvantages of the simple PID number. It turned out that the patch could be simplified; Eric W. Biederman explained that the handle is already present in the inode reference and proc_pid() is enough to get the handle from a file descriptor.

While the first part of the patch deals with getting the handle, the second part of the patch set implements sending the signal itself; it is done using a new system call named procfd_signal(). This system call operates on a file descriptor of a process; the previous discussions convinced Brauner that this is a solution preferred over an ioctl(). The new system call has the following prototype:

    long procfd_signal(int fd, int sig, siginfo_t *info, int flags);

It sends the signal sig to the process identified by the file descriptor fd. The optional info argument is a pointer to siginfo_t provided by the caller (used when sending realtime signals), and flags is reserved for future use and should be zero. On success, the system call returns zero; in case of an error it returns -1 and errno is set to the detailed error code: EBADF if the given file descriptor is not valid, EINVAL if the signal value is invalid or the file descriptor does not refer to a process, EPERM if the caller does not have sufficient permissions to send a signal to the target, and ESRCH if the target process does not exist.

The submission caused discussions of both the implementation and the use of signaling via file descriptors. While there has been no direct opposition, the developers noted a number of issues that should be taken into account. Sarai started a discussion about sending signals to other namespaces. As a result, a check has been added so that sending signals is possible only to processes in child PID namespaces. This avoids problems when file descriptors leak between namespaces, for example when the root file system is bind-mounted into a container. Adding the possibility to send signals to ancestors can be always added in the future.

The debate on which system call to use restarted with Andersen again preferring an ioctl() interface. Colascione and Brauner argued instead for a new system call. This kind of debate happens quite often in the kernel community. Some developers prefer adding a new ioctl(), because they think that adding a system call is too complex. On the other hand, ioctl() is considered a worse API. Andy Lutomirski added a twist to the debate and proposed a better version of the ioctl() system call. The discussion finished without a clear conclusion.

An example of how to use the mechanism has been posted in the cover letter. It is simple: the programs opens the right /proc/pid directory and then sends the signal with all parameters.

The second version of the submission included both 32 and 64-bit versions of the system call with two different entry points. Lutomirski objected, explaining that this design should be avoided for new system calls. The two versions are necessary due to the differing definition of struct siginfo_t. An easy way to avoid creating multiple entry points was eventually found, and the patch set was reposted as taskfd_send_signal() with the same argument types.

The submission tries to fix a problem that has been experienced by multiple people and the developers seem motivated to have the work done. The solution goes in the direction of following the long-established convention of using file descriptors. There is no conclusion yet if this approach will be accepted — probably more iterations will still be needed. However, it seems likely that we will get an improvement in the robustness of signal usage in the not-that-far future.

Comments (85 posted)

Large files with Git: LFS and git-annex

Git does not handle large files very well. While there is work underway to handle large repositories through the commit graph work, Git's internal design has remained surprisingly constant throughout its history, which means that storing large files into Git comes with a significant and, ultimately, prohibitive performance cost. Thankfully, other projects are helping Git address this challenge. This article compares how Git LFS and git-annex address this problem and should help readers pick the right solution for their needs.

The problem with large files

As readers probably know, Linus Torvalds wrote Git to manage the history of the kernel source code, which is a large collection of small files. Every file is a "blob" in Git's object store, addressed by its cryptographic hash. A new version of that file will store a new blob in Git's history, with no deduplication between the two versions. The pack file format can store binary deltas between similar objects, but if many objects of similar size change in a repository, that algorithm might fail to properly deduplicate. In practice, large binary files (say JPEG images) have an irritating tendency of changing completely when even the smallest change is made, which makes delta compression useless.

There have been different attempts at fixing this in the past. In 2006, Torvalds worked on improving the pack-file format to reduce object duplication between the index and the pack files. Those changes were eventually reverted because, as Nicolas Pitre put it: "that extra loose object format doesn't appear to be worth it anymore".

Then in 2009, Caca Labs worked on improving the fast-import and pack-objects Git commands to do special handling for big files, in an effort called git-bigfiles. Some of those changes eventually made it into Git: for example, since 1.7.6, Git will stream large files directly to a pack file instead of holding them all in memory. But files are still kept forever in the history.

An example of trouble I had to deal with is for the Debian security tracker, which follows all security issues in the entire Debian history in a single file. That file is around 360,000 lines for a whopping 18MB. The resulting repository takes 1.6GB of disk space and a local clone takes 21 minutes to perform, mostly taken up by Git resolving deltas. Commit, push, and pull are noticeably slower than a regular repository, taking anywhere from a few seconds to a minute depending one how old the local copy is. And running annotate on that large file can take up to ten minutes. So even though that is a simple text file, it's grown large enough to cause significant problems for Git, which is otherwise known for stellar performance.

Intuitively, the problem is that Git needs to copy files into its object store to track them. Third-party projects therefore typically solve the large-files problem by taking files out of Git. In 2009, Git evangelist Scott Chacon released GitMedia, which is a Git filter that simply takes large files out of Git. Unfortunately, there hasn't been an official release since then and it's unclear if the project is still maintained. The next effort to come up was git-fat, first released in 2012 and still maintained. But neither tool has seen massive adoption yet. If I would have to venture a guess, it might be because both require manual configuration. Both also require a custom server (rsync for git-fat; S3, SCP, Atmos, or WebDAV for GitMedia) which limits collaboration since users need access to another service.

Git LFS

That was before GitHub released Git Large File Storage (LFS) in August 2015. Like all software taking files out of Git, LFS tracks file hashes instead of file contents. So instead of adding large files into Git directly, LFS adds a pointer file to the Git repository, which looks like this:

    version https://git-lfs.github.com/spec/v1
    oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
    size 12345

LFS then uses Git's smudge and clean filters to show the real file on checkout. Git only stores that small text file and does so efficiently. The downside, of course, is that large files are not version controlled: only the latest version of a file is kept in the repository.

Git LFS can be used in any repository by installing the right hooks with git lfs install then asking LFS to track any given file with git lfs track. This will add the file to the .gitattributes file which will make Git run the proper LFS filters. It's also possible to add patterns to the .gitattributes file, of course. For example, this will make sure Git LFS will track MP3 and ZIP files:

    $ cat .gitattributes
    *.mp3 filter=lfs -text
    *.zip filter=lfs -text

After this configuration, we use Git normally: git add, git commit, and so on will talk to Git LFS transparently.

The actual files tracked by LFS are copied to a path like .git/lfs/objects/{OID-PATH}, where {OID-PATH} is a sharded file path of the form OID[0:2]/OID[2:4]/OID and where OID is the content's hash (currently SHA-256) of the file. This brings the extra feature that multiple copies of the same file in the same repository are automatically deduplicated, although in practice this rarely occurs.

Git LFS will copy large files to that internal storage on git add. When a file is modified in the repository, Git notices, the new version is copied to the internal storage, and the pointer file is updated. The old version is left dangling until the repository is pruned.

This process only works for new files you are importing into Git, however. If a Git repository already has large files in its history, LFS can fortunately "fix" repositories by retroactively rewriting history with git lfs migrate. This has all the normal downsides of rewriting history, however — existing clones will have to be reset to benefit from the cleanup.

LFS also supports file locking, which allows users to claim a lock on a file, making it read-only everywhere except in the locking repository. This allows users to signal others that they are working on an LFS file. Those locks are purely advisory, however, as users can remove other user's locks by using the --force flag. LFS can also prune old or unreferenced files.

The main limitation of LFS is that it's bound to a single upstream: large files are usually stored in the same location as the central Git repository. If it is hosted on GitHub, this means a default quota of 1GB storage and bandwidth, but you can purchase additional "packs" to expand both of those quotas. GitHub also limits the size of individual files to 2GB. This upset some users surprised by the bandwidth fees, which were previously hidden in GitHub's cost structure.

While the actual server-side implementation used by GitHub is closed source, there is a test server provided as an example implementation. Other Git hosting platforms have also implemented support for the LFS API, including GitLab, Gitea, and BitBucket; that level of adoption is something that git-fat and GitMedia never achieved. LFS does support hosting large files on a server other than the central one — a project could run its own LFS server, for example — but this will involve a different set of credentials, bringing back the difficult user onboarding that affected git-fat and GitMedia.

Another limitation is that LFS only supports pushing and pulling files over HTTP(S) — no SSH transfers. LFS uses some tricks to bypass HTTP basic authentication, fortunately. This also might change in the future as there are proposals to add SSH support, resumable uploads through the tus.io protocol, and other custom transfer protocols.

Finally, LFS can be slow. Every file added to LFS takes up double the space on the local filesystem as it is copied to the .git/lfs/objects storage. The smudge/clean interface is also slow: it works as a pipe, but buffers the file contents in memory each time, which can be prohibitive with files larger than available memory.

git-annex

The other main player in large file support for Git is git-annex. We covered the project back in 2010, shortly after its first release, but it's certainly worth discussing what has changed in the eight years since Joey Hess launched the project.

Like Git LFS, git-annex takes large files out of Git's history. The way it handles this is by storing a symbolic link to the file in .git/annex. We should probably credit Hess for this innovation, since the Git LFS storage layout is obviously inspired by git-annex. The original design of git-annex introduced all sorts of problems however, especially on filesystems lacking symbolic-link support. So Hess has implemented different solutions to this problem. Originally, when git-annex detected such a "crippled" filesystem, it switched to direct mode, which kept files directly in the work tree, while internally committing the symbolic links into the Git repository. This design turned out to be a little confusing to users, including myself; I have managed to shoot myself in the foot more than once using this system.

Since then, git-annex has adopted a different v7 mode that is also based on smudge/clean filters, which it called "unlocked files". Like Git LFS, unlocked files will double disk space usage by default. However it is possible to reduce disk space usage by using "thin mode" which uses hard links between the internal git-annex disk storage and the work tree. The downside is, of course, that changes are immediately performed on files, which means previous file versions are automatically discarded. This can lead to data loss if users are not careful.

Furthermore, git-annex in v7 mode suffers from some of the performance problems affecting Git LFS, because both use the smudge/clean filters. Hess actually has ideas on how the smudge/clean interface could be improved. He proposes changing Git so that it stops buffering entire files into memory, allows filters to access the work tree directly, and adds the hooks he found missing (for stash, reset, and cherry-pick). Git-annex already implements some tricks to work around those problems itself but it would be better for those to be implemented in Git natively.

Being more distributed by design, git-annex does not have the same "locking" semantics as LFS. Locking a file in git-annex means protecting it from changes, so files need to actually be in the "unlocked" state to be editable, which might be counter-intuitive to new users. In general, git-annex has some of those unusual quirks and interfaces that often come with more powerful software.

And git-annex is much more powerful: it not only addresses the "large-files problem" but goes much further. For example, it supports "partial checkouts" — downloading only some of the large files. I find that especially useful to manage my video, music, and photo collections, as those are too large to fit on my mobile devices. Git-annex also has support for location tracking, where it knows how many copies of a file exist and where, which is useful for archival purposes. And while Git LFS is only starting to look at transfer protocols other than HTTP, git-annex already supports a large number through a special remote protocol that is fairly easy to implement.

"Large files" is therefore only scratching the surface of what git-annex can do: I have used it to build an archival system for remote native communities in northern Québec, while others have built a similar system in Brazil. It's also used by the scientific community in projects like GIN and DataLad, which manage terabytes of data. Another example is the Japanese American Legacy Project which manages "upwards of 100 terabytes of collections, transporting them from small cultural heritage sites on USB drives".

Unfortunately, git-annex is not well supported by hosting providers. GitLab used to support it, but since it implemented Git LFS, it dropped support for git-annex, saying it was a "burden to support". Fortunately, thanks to git-annex's flexibility, it may eventually be possible to treat LFS servers as just another remote which would make git-annex capable of storing files on those servers again.

Conclusion

Git LFS and git-annex are both mature and well maintained programs that deal efficiently with large files in Git. LFS is easier to use and is well supported by major Git hosting providers, but it's less flexible than git-annex.

Git-annex, in comparison, allows you to store your content anywhere and espouses Git's distributed nature more faithfully. It also uses all sorts of tricks to save disk space and improve performance, so it should generally be faster than Git LFS. Learning git-annex, however, feels like learning Git: you always feel you are not quite there and you can always learn more. It's a double-edged sword and can feel empowering for some users and terrifyingly hard for others. Where you stand on the "power-user" scale, along with project-specific requirements will ultimately determine which solution is the right one for you.

Ironically, after thorough evaluation of large-file solutions for the Debian security tracker, I ended up proposing to rewrite history and split the file by year which improved all performance markers by at least an order of magnitude. As it turns out, keeping history is critical for the security team so any solution that moves large files outside of the Git repository is not acceptable to them. Therefore, before adding large files into Git, you might want to think about organizing your content correctly first. But if large files are unavoidable, the Git LFS and git-annex projects allow users to keep using most of their current workflow.

Comments (39 posted)