Debian Technical Committee overrides systemd change

Debian packagers have a great deal of latitude when it comes to the configuration of the software they package; they may opt, for example, to disable default features in software that they feel are a security hazard. However, packagers are expected to ensure that their packages comply with Debian Policy, regardless of the upstream's preferences. If a packager fails to comply with the policy, the Debian Technical Committee (TC) can step in to override them, which it has done in the case of a recent systemd change that broke several programs that depend on a world-writable /run/lock directory.

The Filesystem Hierarchy Standard (FHS) specifies that the /var/lock directory should be used to store lock files for devices and other resources shared by multiple applications. On Debian, /var/lock is a symbolic link to /run/lock. The /run directory is created as a tmpfs filesystem specifically for run-time files by systemd-tmpfiles during system startup.

Debian Policy still cites the FHS, even though the FHS has gone unmaintained for more than a decade. The specification was not so much finished as abandoned after FHS 3.0 was released—though there is a slow-moving effort to revive and revise the standard as FHS 4.0, it has not yet produced any results. Meanwhile, in the absence of a current standard, systemd has spun off its file-hierarchy documentation to the Linux Userspace API (UAPI) Group as a specification. LWN covered that development in August, related to Fedora's search for an FHS successor.

Locking up /run/lock

The /run/lock directory was deprecated in Systemd v258; rather than dropping the directory entirely, though, the project has changed the default to making /run/lock writable only by root, which is stricter than the permissions Debian had shipped with previously rather than making it world-writable as in the past. The plan is to get rid of /run/lock entirely in the v259 release, though users (or distributions) can still retain the legacy behavior by adding a configuration file in /etc/tmpfiles.d to override systemd's defaults and create the directory with the desired permissions.

The Debian project just released a new stable version, Debian 13 ("trixie"), in August, and work has begun on Debian 14 ("forky"). The current stable version of Debian shipped with systemd v257, so users on stable will not be affected by these changes. But v258 has entered Debian unstable where the change to /run/lock broke other software, such as the Unix-to-Unix Copy program (UUCP) and the cu utility. Use of the directory is not limited to vintage utilities; Zbigniew Jędrzejewski-Szmek objected to removing /var/lock in v259 as it would break alsa-utils and create additional work for distributions:

Doing this would this way just creat a foottrap for distributions: if they notice the change, they'll just create a local override, so we get a more complicated system in total with zero benefit to anyone. If they miss it, things will be broken for a while until users report it. And then they'll add the override.

On August 13, Marco d'Itri—who is listed as a maintainer of the systemd package—filed a bug against the uucp package reporting that systemd v258-rc1-1 had broken uucico, along with filing a bug against the systemd package, which cited the FHS entry for /var/lock. He said that a compromise might be to make the directory writable by the dialout group rather than world-writable. He also mentioned that there was a previous effort in 2014 to modernize software that uses UUCP-style locks to use flock() instead, but it stalled out.

"Dead and severely outdated"

Rather than temporarily reverting the behavior, systemd maintainer Luca Boccassi argued that a world-writable directory in /run is a security risk. Any process could write as much as it wanted to /run, which could effectively DoS the system by exhausting space or inodes; filling up /run would then cause critical services, such as udev, to stop working. The FHS, he said, is "dead and severely outdated".

The issue had already been discussed by the systemd project; Lennart Poettering had responded that he did not see the point of /var/lock "in the modern world", but distributions were free to do as they see fit:

Consider this more a passing of the baton from upstream systemd to downstreams: if your distro wants this kind of legacy interface, then just add this via a distro-specific tmpfiles drop-in. But there's no point really in forcing anyone who has a more forward-looking view of the world to still carry that dir.

Poettering argued that distributions could make their own choices, though it made him shudder to think of allowing unprivileged programs to fill up a directory. Boccassi echoed that sentiment in his response to the Debian systemd bug. Any package could ship a configuration for tmpfiles.d to create the directory and assume responsibility for it as well:

I certainly won't try to stop anyone wishing to do it, but also I do not wish for these old workarounds to ship in this package either.

There's ~2 years time until Forky ships, and that should be plenty of time to either add this workaround elsewhere, or fix remaining programs to use BSD locks, or both, so I'm not going to hold back the new version from testing for this niche case, sorry.

Boccassi closed the bug with the "WONTFIX" tag.

Debian Policy

On September 1, d'Itri responded that upstream systemd's opinion was not relevant in this case. Debian policy requires the directory for lock files of serial devices, though he had also opened a bug to revisit that, since the practice of using /var/lock for serial-device locks dates back to the 1980s. However, /var/lock is provided by systemd, so he reasoned that it is a systemd bug unless another package was identified to take ownership of creating the directory. "But you cannot just decide that the policy violation does not exist."

Thorsten Alteholz opened a bug with Debian's Technical Committee on September 15. He asked for advice on how to proceed since the systemd bug had been marked WONTFIX.

So what do you recommend how to go on from here? Change Debian policy (as asked in #1111839), revert the change in systemd, find a Debian wide solution or let every package maintainer implement their own solution?

Bdale Garbee weighed in on the bug as well. He said that he uses cu "almost constantly for interacting with embedded serial consoles on devices a USB connection away from my laptop". He was frustrated with the change imposed by systemd "with no warning", and looked forward to the TC's response.

On September 24, Matthew Vernon responded to the bug, with a CC to the systemd maintainer alias. He said that it seemed that Debian Policy required FHS compliance, "at least until we come up with a transition plan", and asked if systemd would please revert the change. There was no response from any of the systemd maintainers.

Override

Vernon updated the bug on September 29, and said that the TC had discussed the situation at its last meeting. The conclusion was that systemd should comply with policy, and thus FHS. He included a draft ballot text that the TC would vote on, which noted that the committee was "sympathetic" to the argument that flock() would be a better solution, but that "an important part of the role of a Debian Developer is ensuring that software in Debian complies with Debian Policy". A final ballot for the committee to consider was posted on October 2.

The ballot contained three options; all three required that the systemd package provide /var/lock "with relaxed enough permissions that existing Debian software that uses /var/lock for system-wide locks of serial devices (and similar purposes) works again". The committee would exercise its power under the Debian Constitution to override the systemd maintainers. The options were:

1) This change to systemd must persist until a satisfactory migration of impacted software has occurred and Policy updated accordingly.

2) This change to systemd must persist until Policy has been updated to allow otherwise.

3) This change to systemd must persist until the TC allows otherwise, which the TC expects to do once a suitable transition plan has been agreed.

Vernon replied on October 6 to say that a decision had been reached and that option one had won.

Unlocking

Debian's continued use of UUCP-style locking does seem to be more than a little bit dated. The FHS 3.0 is clearly reaching the end of its useful life, if not actually expired.

As a comparison, Fedora's uucp package has a patch to use lockdev instead. The upcoming Fedora 43 release includes systemd v258, and /run/lock is not world-writable. It seems like Debian could borrow Fedora's approach for the uucp package, though that would not solve the problem for any other Debian packages affected by the /run/lock change. There is also third-party software to consider, of course.

To date, Boccassi has not responded to the conversation; I emailed d'Itri to ask why he did not make the change himself, since he was aware of the bug. He replied on October 13 and said that "a maintainer cannot force a decision on another maintainer." Since he and Boccassi disagreed about the change it was left to the TC to decide. He said that he planned to prepare a merge request to implement the TC's decision, "as I have already agreed with Luca", within the week.