|
|
Subscribe / Log in / New account

User=0day considered harmful in systemd

By Jake Edge
July 12, 2017

Validating user input is a long-established security best practice, but there can be differences of opinion about what should be done when that validation fails. A recently reported bug in systemd has fostered a discussion on that topic; along the way there has also been discussion about how much validation systemd should actually be doing and how much should be left up to the underlying distribution. The controversy all revolves around usernames that systemd does not accept, but that some distributions (and POSIX) find to be perfectly acceptable.

The bug was opened in late June by GitHub user "mapleray". It describes setting up a systemd service file with a "User=0day" entry, which means that the service should run as the 0day user. However, mapleray found that it ran as root instead, which is, at the least, rather surprising. It turns out that usernames starting with a digit are disallowed by systemd—so it ignores the line and puts a warning in the log. Since there is no user specified, systemd falls back running it as the default user: root.

Lennart Poettering replied that systemd was functioning as intended. In his mind, 0day is not a valid username, thus systemd should not accept it. The comment didn't address whether a warning was sufficient, however. He duly marked it as "notabug" and closed it. Others were not so sure, however.

Many of the GitHub comments focused on whether 0day should be treated as a valid username. Some distributions do allow it; even those that disallow it via adduser will allow it when used with the lower-level useradd command (or with adduser --force-badname). POSIX is much more liberal than most distributions; it allows many different kinds of usernames, including names starting with "-" or "." and names containing ".", any of which might confuse various command-line programs (e.g. chown), users, and administrators.

Falling back to root

But others thought that focusing on username validity was misplaced. Whether or not 0day is accepted by systemd as a valid username, many found it hard to see why the service should be started running as root when an invalid username is encountered. As "rain-1" put it: "The real problem here is that the unit is run as root, despite '0day' not describing the root user, isn't it?" To some, at least, systemd is clearly doing the wrong thing; "RealDolos" summed it up this way:

Same as systemd should not pick and run some random binary when the specified ExecStart is invalid, it shouldn't run a service under a random uid if it cannot find the specified user (uid 0 being the random uid fairly picked by the Debian PRNG, of course)

Shortly thereafter, trolls showed up in the bug report, which led to the issue being locked so that only project members could comment. Poettering posted a summary of his thoughts on the bug, but it did not address the issue of whether systemd should ignore invalid usernames in User directives (and run the unit as root). For backward compatibility reasons (i.e. newer service files used by older versions of systemd), complaints are issued about syntax errors, like unknown directives or directives with syntactically invalid settings, but the unit file is still run, he explained. On the other hand, semantically invalid settings (e.g. non-existent, syntactically valid usernames) will cause a fatal error for the unit.

Meanwhile, Daniel Skowroński posted about the bug to the oss-security mailing list. He focused on the root fallback aspect, but that didn't stop others from reiterating some of the discussion about username validity. However, Ben Tasker suggested an all-too-plausible scenario where the root fallback behavior could lead to real problems:

It'd be all too easy for a reviewer to look at the unit file, note it runs as '0day', double check that the package creates a user called '0day' and be happy that it's going to work. Hopefully someone *would* notice but that might not happen until after a vulnerability in that particular package has been remotely exploited (giving root access, oh dear), which is a situation that's in no-one's interest.

Like several others in the thread, Tasker thought that a CVE should be assigned. But Simon McVittie was concerned about the scope of CVE assignment growing to problems that are not truly security related: "[...] if the working definition of a vulnerability gets stretched too far towards things that are 'just a bug', it will reduce the perceived importance of fixing CVEs promptly, harming the overall level of security in software". While Tasker agreed with McVittie's overall concern, he was adamant that this bug qualified for a CVE:

So, as a result, you've potentially got a daemon listening for connections from the outside world, whilst running as root (despite the fact you're expecting it to run as an unprivileged user). If someone can convince that daemon to run arbitrary code, it does so with superuser privileges.

IMHO the discussion on how that unit file makes it onto the system in the first place is an irrelevant distraction.

Mistakes happen, whether that's a package reviewer missing the connotations of the *valid* username in the unit file, or some eejit leaving a unit file world writeable (I've seen kernel modules with 0777 on public facing production systems in the past).

Kurt Seifried assigned a CVE for the issue, CVE-2017-1000082, on July 6.

Another sub-thread on oss-security concerned the locked issue on GitHub. Pali Rohár noted that the lock blocked further discussion there, which is "really *bad* for security related problems". But others pointed out that the only recourse maintainers have when bug discussions start going off the rails is to lock them. The alternative is far worse, Alan Coopersmith said:

Honestly, given the level of flaming and trolling that happens on issues like this, locking the report is the only sane option I can see once everyone started piling on. Forcing FOSS maintainers to accept infinite amounts of shitposting is a horrible way to reduce security by burning out all FOSS maintainers quickly and leaving software abandoned.

Changes coming?

Overlapping some of that discussion was a thread on systemd-devel asking some questions about the bug and whether the fallback behavior would change. Systemd developer Zbigniew Jędrzejewski-Szmek replied that it might:

I do agree that we might want to completely reject unit files when some crucial lines fail to parse, for example ExecStart or WorkingDirectory or User, but it's not as obvious as one would think at first.

When new configuration options are added, the same unit file can almost always be used with older systemd, and it'll just warn & ignore the parts it doesn't understand. Similarly, various configuration options might be unavailable on some architectures and with some compilation options. The current behaviour of warn&ignore provides for "soft degradation" in those cases.

To do this properly, we would need to figure out which options are a) important enough, b) supported on all compilation variants and architectures, and then add a generic mechanism to make errors in them fatal.

Much of the systemd-devel discussion, once again, turned to the question of why valid usernames (by some definitions) were being rejected by systemd at all. Poettering posted a lengthy defense of the practice and pointedly disagreed with the POSIX rules (which are quite lax). He noted that the systemd username rules came from an interest in supporting the most portable subset of usernames across distributions. He also pointed out that the users in question here are "system users", rather than regular users that might want a username like "j.random.hacker" (which systemd would treat like 0day):

Also, do note that system users are different concepts than regular users: system users are concepts required for system services which are usually put together by developers, packagers and administrators who hopefully understand these issues to some point and pick good names instead.

Jędrzejewski-Szmek ("keszybz" on GitHub) created a pull request (PR) to fix the fallback problem that has been merged into the master branch. Meanwhile, version 234 of systemd, which would include that fix and other changes since the release of 233 back in March, is planned for July 12 (though it has not been tagged as of this writing). So it would seem that units with invalid User settings will not be run (as root or any other user) in systemd moving forward.

Perhaps the best description of why this bug caused such an uproar came in a comment on the issue that will be closed by Jędrzejewski-Szmek's PR; Justin Azoff said:

If I typo User=app as User=app0 the unit will fail to start. If I typo it as User=0app the unit will start as root. This behavior is absolutely insane.

If a unit file has User=something it can be said for certain that the one thing that the administrator absolutely does not want is the unit to start as root.

One can argue that systemd should not be checking usernames for itself, but should instead consult the system by using getpwnam()—many did. But systemd is rather opinionated about how things should work, naming, and so forth, so it should not come as an enormous surprise that it enforces its rules on system usernames. As Poettering pointed out—several times—systemd is free software; users, distributions, and others are all welcome to modify it to suit their needs. So, whether it makes sense or no, that part of the behavior will likely stand.



to post comments

User=0day considered harmful in systemd

Posted Jul 12, 2017 17:26 UTC (Wed) by tao (subscriber, #17563) [Link] (8 responses)

It should be noted that the proper syntax for chown is <user>:<group>; using '.' is a BSD:ism (even SuSv2 from 1997 specifies ':'). This of course doesn't help much since many legacy scripts still use the BSD syntax, but it might at least be good to keep in mind for future scripts.

User=0day considered harmful in systemd

Posted Jul 12, 2017 17:38 UTC (Wed) by dskoll (subscriber, #1630) [Link] (7 responses)

As Poettering pointed out—several times—systemd is free software; users, distributions, and others are all welcome to modify it to suit their needs.

In my opinion, that is not an appropriate to response to a bug that's this dangerous (potentially). It's a copout.

User=0day considered harmful in systemd

Posted Jul 12, 2017 17:41 UTC (Wed) by dskoll (subscriber, #1630) [Link] (6 responses)

OK, I read some more context and it seems to me he was talking about the strict username rules rather than treating 0day as root. So it makes more sense and I retract most of my previous comment, though I still think playing the "it's free so you can change it" card is a bit of a copout.

User=0day considered harmful in systemd

Posted Jul 12, 2017 18:00 UTC (Wed) by drag (guest, #31333) [Link] (2 responses)

Systemd is intended to be consumed by people designing operating systems. So it's expected that they should understand the software they are working with as well as be willing and able to make modifications when necessary It's a upstream project intended to be consumed by experts. If end users get burned by a systemd default it's at least partially the distribution's fault for not taking the time to configure it correctly for the users.

What we have here, at least in some ways, is a balancing act between the need for secure defaults and the need to have service files not only be backward compatible, but compatible between different distributions.

I think the solution to this problem may actually be a sort of 'systemd-lint' program. Something you can run that will check your service files for bogus lines, bad users, and other common pitfalls. There are multiple places that you can find configurations for any single service definition and requiring visual inspection of the code for correctness is really just setting up users and system administrators to fail.

User=0day considered harmful in systemd

Posted Jul 12, 2017 18:39 UTC (Wed) by zdzichu (subscriber, #17118) [Link] (1 responses)

Regarding last paragraph of your reply, systemd-verify exists. Maybe it needs to be extended to check if rvalues are correct – https://www.freedesktop.org/software/systemd/man/systemd-...

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:23 UTC (Wed) by zuki (subscriber, #41808) [Link]

Of course it checks that rvalues are correct ;)

User=0day considered harmful in systemd

Posted Jul 13, 2017 0:21 UTC (Thu) by firstyear (subscriber, #89081) [Link] (2 responses)

I think the response is still an issue. Just because it's open, and I can modify it, doesn't mean my changes will be accepted. Either I need to push the patch uphill to get it accepted (and in this case, with an upstream that "disagrees" with the concept), or I now need to carry the patch and maintain it potentially indefinitely. Both are bad situations.

This response is a massive copout, it gives the appearance of open source without having to actually be open in your work flows or attitude.

User=0day considered harmful in systemd

Posted Jul 14, 2017 10:59 UTC (Fri) by niner (subscriber, #26151) [Link] (1 responses)

As a distributor, you can ship a systemd modified according to your tastes and opinions. As a user you can run a systemd modified to your tastes and opinions.

What makes you think that software is only free if you can impse your opinions and tastes on the original authors?

User=0day considered harmful in systemd

Posted Jul 14, 2017 14:17 UTC (Fri) by phred14 (guest, #60633) [Link]

Seems to me that there is a difference between configuring upstream to fit your wishes and rules, and patching upstream for that same purpose. Having no idea of how stable systemd code is between releases, I have no idea how often such a "policy patch" might tend to break, requiring further manual action rather than something simply scripted. Certainly if such a policy issue were configurable it would be better for everyone.

User=0day considered harmful in systemd

Posted Jul 12, 2017 17:55 UTC (Wed) by davidstrauss (guest, #85867) [Link] (43 responses)

> One can argue that systemd should not be checking usernames for itself, but should instead consult the system by using getpwnam()—many did. But systemd is rather opinionated about how things should work, naming, and so forth, so it should not come as an enormous surprise that it enforces its rules on system usernames.

It's not possible with the defined meaning of User= to simply use getpwnam(). The User= directive takes either a user name or a UID [1], making getpwnam() insufficient on its own. For numeric IDs, getpwuid() gets used instead. This means that the parser has to decide which to use, and that decision has to be made based on the string following User=. Attempting getpwuid() or getpwnam() first isn't safe, as I'll explain.

It's unsafe to just attempt getpwnam() first, given the potential to match on a user with an all-numeric name. Matching this way would allow an attacker to hijack a service's operation (causing it run under a user they control) if the attacker could cause a user to get created with a *name* that matches the UID configured in the User= directive.

Attempting getpwuid() first on all-numeric User= directive could also open an attack vector if (1) a named user exists with a matching, all-numeric name and (2) the UID matching the number in User= doesn't yet exist. An attacker would then just have to spam user creation until a user exists with a UID that matches the number after User=. This seems like an even higher risk (assuming the number is in the valid UID range) than the former attack because there's no need to control the names of the created users.

With all that considered, it's safest for the parser to make a decision whether to treat User= as a UID versus a named user based on the value of User= rather than relying on trial-and-error syscalls. I could see an argument for splitting into UserName= or UserID= instead of an overloaded User=, but that wouldn't fix the original concern here, which is quiet fallback to root on failure. UserName=0day would still need a defined fallback if that user doesn't exist (or, more likely, scripted creation with something like adduser failed). I won't defend the fallback that was in place, but the logic here may be more complex than users, administrators, and even some systems programmers assume.

Just for context, I am a systemd contributor.

[1] https://www.freedesktop.org/software/systemd/man/systemd....

User=0day considered harmful in systemd

Posted Jul 12, 2017 19:13 UTC (Wed) by anselm (subscriber, #2796) [Link] (8 responses)

If we do accept the POSIX definition for user names, the only problem is really what to do with all-numeric ones (not a great idea, but who knows). According to the GNU project the suggested method is to treat every user name (even an all-digit one) as a string first, and only interpret it as a UID if no user with that name exists on the system. That does indeed have the issue that you mention.

If it was up to me I'd introduce optional prefixes, e.g., #123 for the UID, 123, and !123 for the user name, “123”. That would let a systemd administrator make clear that an all-digit string in the configuration is either meant to be a UID (in which case it is simply taken verbatim) or else a user name (in which case it is checked for existence and a hard error is reported if it does not exist; no fallback to a numeric UID happens). If there is no prefix, the default matching algorithm as per GNU can still be used for compatibility (possibly with a warning).

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:12 UTC (Wed) by mitr (subscriber, #31599) [Link] (4 responses)

> According to the GNU project the suggested method is to treat every user name (even an all-digit one) as a string first, and only interpret it as a UID if no user with that name exists on the system

That’s actually a POSIX requirement, see e.g. (man 1p chown). AFAIK POSIX is consistently using the “username first, UID if not found” semantics.

(I do agree that numbers-only usernames are fairly crazy; though perhaps more crazy now that uid_t is 32-bit than in the past, when it used to be 16-bit; back then it might have seemed neat to use phone numbers as usernames, because the ranges were so distinct, and user names based on real-world surnames are problematic due to marriages and divorces.)

Anyway, there _are_ actually users who use number-only usernames, inadvisable as it is, so every application should be interpreting such strings consistently with the basic POSIX commands. (If systemd wanted to refuse to work with such usernames, _and reject such input_ instead of reinterpreting it inconsistently with the rest of the system, *shrug*, works for me.)

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:15 UTC (Wed) by mitr (subscriber, #31599) [Link]

> AFAIK POSIX is consistently using the “username first, UID if not found” semantics.

… which also demonstrates that the possibility to use all-numeric usernames is there intentionally, not an oversight. (That still does not make it a good idea, of course.)

User=0day considered harmful in systemd

Posted Jul 13, 2017 13:27 UTC (Thu) by TRS-80 (guest, #1804) [Link] (2 responses)

I run an Ubuntu server where the usernames are numeric; they are in fact the student's ID number. I've never actually encountered a problem with usernames being interpreted as UIDs.

User=0day considered harmful in systemd

Posted Jul 15, 2017 0:22 UTC (Sat) by rahvin (guest, #16953) [Link] (1 responses)

Is the system configured so that the numeric username is also the UID?

User=0day considered harmful in systemd

Posted Jul 15, 2017 5:48 UTC (Sat) by TRS-80 (guest, #1804) [Link]

No, the UID is calculated using the macOS method, which is to take the result of the first 32 bits of the Active Directory GUID AND 0x7fffffff.

User=0day considered harmful in systemd

Posted Jul 12, 2017 21:01 UTC (Wed) by davidstrauss (guest, #85867) [Link] (2 responses)

> If it was up to me I'd introduce optional prefixes, e.g., #123 for the UID, 123, and !123 for the user name, “123”. That would let a systemd administrator make clear that an all-digit string in the configuration is either meant to be a UID (in which case it is simply taken verbatim) or else a user name (in which case it is checked for existence and a hard error is reported if it does not exist; no fallback to a numeric UID happens). If there is no prefix, the default matching algorithm as per GNU can still be used for compatibility (possibly with a warning).

That is, essentially, the same as the alternative I mentioned of splitting the directives into UserName= and UserID=. Your proposal just uses a special prefix on values instead of new directives. Requiring a prefix for UIDs would have the additional disadvantage of breaking existing configurations. Leaving User= as a deprecated directive (with the existing heuristics for username versus UID treatment) would keep existing units working while supporting explicit username/UID for new units.

That said, I don't think the current heuristic and User= directive are all that bad. I agree with Lennart that system users ought to use conservative, distro-portable names. Falling back to an error instead of User=root should be enough to prevent issues cropping up from typoed unit files and missing system user configurations.

User=0day considered harmful in systemd

Posted Jul 13, 2017 4:02 UTC (Thu) by pj (subscriber, #4506) [Link] (1 responses)

He said 'optional' not 'required' prefixes.

User=0day considered harmful in systemd

Posted Jul 13, 2017 19:33 UTC (Thu) by davidstrauss (guest, #85867) [Link]

> He said 'optional' not 'required' prefixes.

It's not really "optional" if it falls back to behavior that would be possibly triggering a warning and use a different matching algorithm (the GNU one) than the one currently in place.

User=0day considered harmful in systemd

Posted Jul 12, 2017 19:39 UTC (Wed) by dskoll (subscriber, #1630) [Link] (29 responses)

What is the use case for having a systemd unit file specify a user that doesn't exist in the system passwd file? I can't think of one and can think of good reasons why it's a giant security hole... what happens if that UID is latter added to the system as some user? If we agree that there isn't one, then User= should always treat its argument as a Username and should fail if the name doesn't exist in the passwd database, even if it's all-numeric.

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:40 UTC (Wed) by drag (guest, #31333) [Link] (8 responses)

When you have a unit file that is intended to be used in multiple distributions and take advantage of a standard system user when present and fall back to root when it's not.

It's fairly common approach for init stuff, as sad as it is. So it's not just a systemd thing. It mostly just has just adopted a already existing standard practice. (at least it seems to me). I have some legacy systems (historically Solaris environment from the bad old days dragged into the 21st century) were uid conflicts exists and many times when installing rpms in centos6/7 users that are normally present are absent and the rpm postinstall stuff complains bitterly, but the services tend to still work.

It's probably a bit of a flaw in modern *nix systems that 'system users' versus 'user users' are treated as pretty much the same things and the only division really is one of convention of usage and some basic assumptions about numerical values being below 500 or whatever.

Incompatible distributions

Posted Jul 13, 2017 10:13 UTC (Thu) by NAR (subscriber, #1313) [Link] (3 responses)

"When you have a unit file that is intended to be used in multiple distributions"

It might have been a goal of systemd to use distribution-agnostic unit file instead of the distribution-specific shell scripts, it seems that this goal is not achievable. What if some distribution runs e.g. web services as www-user, and other as httpd-user? I think the best the software author can do is to provide a unit file template and let the distribution packager update it to the specific distribution. Or forget the whole distribution-thing as it is now and converge on some standard (I guess it would happen only over the dead bodies of the distribution creators...).

Incompatible distributions

Posted Jul 13, 2017 10:40 UTC (Thu) by bof (subscriber, #110741) [Link]

Thanks to the very nice systemd override features a distribution with "nonstandard" User= requirements, can easily install a /usr/lib/systemd/system/someunit.d/distrouser.conf with the distribution specific correct setting.

Incompatible distributions

Posted Jul 14, 2017 11:08 UTC (Fri) by niner (subscriber, #26151) [Link] (1 responses)

Why would one distro run web services as www-user and another as httpd-user? Only because back then, there was no standard or guide. But if software contains a service file with a user name suggestion right from the start, it's hard to see why distros would ignore that recommendation and forever ship with a patch. Choosing a name is work for distros that they will happily leave up to upstream. So yes, systemd's goal of standardizing mundane parts across distros is achievable and is quite successful in practice.

Incompatible distributions

Posted Sep 28, 2017 12:41 UTC (Thu) by mstone_ (subscriber, #66309) [Link]

It's pretty hard to change existing installs, and the benefits just aren't worth the effort. It might sound nice to use the same username everywhere, but in practice it isn't that hard to figure out.

User=0day considered harmful in systemd

Posted Jul 14, 2017 10:47 UTC (Fri) by dskoll (subscriber, #1630) [Link] (3 responses)

I cannot imagine a scenario in which "fall back to root" would be desirable. It seems to me intentionally abetting that sort of behaviour is verging on the criminal.

User=0day considered harmful in systemd

Posted Jul 14, 2017 15:17 UTC (Fri) by drag (guest, #31333) [Link] (2 responses)

It's the way things have been done. Systemd isn't responsible for introducing this behavior.

Maybe it should be something that should be fixed, but it'll end up breaking things if it is.

User=0day considered harmful in systemd

Posted Jul 14, 2017 15:31 UTC (Fri) by mchapman (subscriber, #66589) [Link] (1 responses)

> It's the way things have been done.

If you're talking about initscripts, I think you'll be hard-pressed to find an initscript that tries to run a program as an unprivileged user, but if that fails continues to run that program instead as root. I really don't think this has ever been "the way things have been done".

> Systemd isn't responsible for introducing this behavior.

systemd doesn't even have that behaviour. There is no "fallback to root" logic in systemd.

What systemd did have was a situation where the User= directive was ignored if it could not be parsed. I can understand why people keep mistaking this as some kind of "fallback" -- it certainly looks and acts like one, especially if you don't look at your logs. But it has nothing to do with whether a particular user exists on the system or not. If the User= directive is successfully parsed but the user it identifies does not exist on the system, systemd will fail the unit when it attempts to run a command for it.

Anyway, "0day" was (and still is) one of the unparsable values for this directive, however since a recent commit [1] this parse error now results in the unit failing to load rather than the directive being ignored.

[1] https://github.com/systemd/systemd/commit/bb28e68477a3a39...

User=0day considered harmful in systemd

Posted Jul 14, 2017 16:57 UTC (Fri) by jspaleta (subscriber, #50639) [Link]

Hmm so the logic of the commit message makes a subclass of configuration options that will always fail if an error occurs if parsed.
That seems reasonable. As a short hand I'll refer to this subclass as "sensitive directives"

Question the first:
Would it make sense to give local admins the ability to extend the set of sensitive directives with more directives of their choosing?
The commit comment refers to other supplemental directives that can't be guaranteed to parse correctly across all system configurations in the general case from a unit writer perspective.

I would think however that a local admin may want to use their discretion and local system knowledge to treat specific supplemental directives as sensitive and have the units fail to start if the directives fail to parse to avoid unexpected behavior.

I would also naively think that distributors who are supplying unit files to end users might find it useful to use some automation and audit for parse errors on a much larger set of directives. Since these distributors generally set the compile options, they are in the best position to test to make sure supplemental directives outside the default set of sensitive directives fire correctly.

Question the second:
What's the best practice for local admins/unit writers/distribution integrator to be notified of directive parse errors in the unit files on system? Is there a way to automatically audit the disabled/inactive units for these sorts of errors?

Question the third:
Should this be taken further and create a subclass of directives for local admins to populate which much be present and parse correctly in all units?

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:54 UTC (Wed) by davidstrauss (guest, #85867) [Link] (14 responses)

> What is the use case for having a systemd unit file specify a user that doesn't exist in the system passwd file?

There isn't one, nor did I argue there was one. The remainder of your questions seem to assume that was my position, so I don't know how to address them.

My comment was about the security implications of using syscalls to validate input as a username or UID, specifically responding to the argument I quoted in my comment. I only mentioned fallback on lack of username/UID match as an aside -- and I agreed that fallback to root is bad.

User=0day considered harmful in systemd

Posted Jul 14, 2017 10:49 UTC (Fri) by dskoll (subscriber, #1630) [Link] (13 responses)

I asked the question to to criticize you, but to suggest that the desirable behaviour should be to always treat the User value as a user name and to always fail if that value is not found in the password database. It seems to me that's the safest approach.

User=0day considered harmful in systemd

Posted Jul 14, 2017 12:05 UTC (Fri) by mchapman (subscriber, #66589) [Link] (6 responses)

> I asked the question to to criticize you, but to suggest that the desirable behaviour should be to always treat the User value as a user name and to always fail if that value is not found in the password database. It seems to me that's the safest approach.

That is actually what happens... sort of.

systemd does call getpwnam() between the fork() and execve() when launching a command. That is the precise point at which it maps a username to a UID, a GID, supplementary GIDs, etc.

But getpwnam() can *not* be called at the point at which the unit is loaded into systemd. It is perfectly fine for the user not to exist at that particular time. Consider what happens during a package installation, for instance, where the unit file is dropped into the filesystem and then, during some post-installation script, adds the user to the system. There's a window of time where the unit can be loaded without the user actually existing. Or consider the case of the user being removed from the system after the unit has been loaded into systemd... systemd has no way to know that the user no longer exists, and even if it did it can't drop the unit as the service may still be running!

And finally, there are cases where it is intended that the system user database not contain the username specified in the unit. The DynamicUser= directive and the nss-systemd module interact together to provide dynamic users that exist for the duration of a service's runtime only.

In short, systemd already does getpwnam(), and it does it precisely where it should do it.

The problem that started this whole shemozzle off is that there were certain User= values that cause the entire directive to be ignored altogether. This meant that the default value, User=root, was used instead. getpwnam() called in the child process would look that up, map it to UID 0, and run the command with that persona.

With Jędrzejewski-Szmek recently merged pull requests, the User= values that would have caused the directive to be ignored now cause the unit to fail to load completely.

User=0day considered harmful in systemd

Posted Jul 14, 2017 12:19 UTC (Fri) by mchapman (subscriber, #66589) [Link]

> The problem that started this whole shemozzle off is that there were certain User= values that cause the entire directive to be ignored altogether. This meant that the default value, User=root, was used instead. getpwnam() called in the child process would look that up, map it to UID 0, and run the command with that persona.

Actually, it'd probably be more correct to say that the default behaviour, in the absence of a User= directive, is that systemd doesn't bother changing persona at all. systemd does not assume that a "root" user exists.

User=0day considered harmful in systemd

Posted Jul 14, 2017 14:30 UTC (Fri) by dskoll (subscriber, #1630) [Link] (4 responses)

My understanding was that systemd would not call getpwnam() for a username that's numeric. Or maybe another interpretation is that it would interpret it as a UID iff getpwnam() failed. Was this understanding wrong?

User=0day considered harmful in systemd

Posted Jul 14, 2017 14:53 UTC (Fri) by dskoll (subscriber, #1630) [Link] (2 responses)

Indeed, the systemd.exec man page (Debian Stretch) says "Takes a single user or group name, or numeric ID as argument.

Well, it shouldn't. It should insist on a name and not an ID.

The use case someone mentioned for an ID (that the user name is in a remote database) is not persuasive. How does it make more sense to manage a bunch of machines with "User=4242" in the unit files rather than create a local user on each machine with that ID and use "User=foouser"? Changing the ID is just as onerous in each case.

User=0day considered harmful in systemd

Posted Jul 14, 2017 15:09 UTC (Fri) by mchapman (subscriber, #66589) [Link] (1 responses)

> The use case someone mentioned for an ID (that the user name is in a remote database) is not persuasive.

Well, the use case isn't supported anyway. The user must be in the system's user database -- i.e. must be able to be returned by a getpw*() call -- whether a username or a UID is used to look it up. systemd will not execute a command for a user that isn't in the user database.

> How does it make more sense to manage a bunch of machines with "User=4242" in the unit files

If I had my way I would have User= only take a username, never a UID. I still wouldn't have it allow all-digit usernames though, because that's just plain weird.

But that kind of change would be hard to introduce now. I don't know why anybody would do it, but there's probably somebody relying on User=4242 doing something useful on their systems.

User=0day considered harmful in systemd

Posted Jul 14, 2017 15:12 UTC (Fri) by mchapman (subscriber, #66589) [Link]

> Well, the use case isn't supported anyway.

Ah sorry, I misread your comment.

systemd should work just fine with a remote database for users. It's just calling the standard libc functions, after all, and the user database can be in LDAP or whatever.

User=0day considered harmful in systemd

Posted Jul 14, 2017 15:00 UTC (Fri) by mchapman (subscriber, #66589) [Link]

> My understanding was that systemd would not call getpwnam() for a username that's numeric. Or maybe another interpretation is that it would interpret it as a UID iff getpwnam() failed. Was this understanding wrong?

Oh, you were talking about all-digit User= values specifically? systemd will assume that if the value in User= only consists of digits, then it should be interpreted as a UID. While spawning the child process it will attempt to get the credentials for that UID using getpwuid(). It never calls getpwnam() in this case, so if you have an all-digit username on your system the only way to reference this in User= would be to use that user's UID.

So only getpw*() call is chosen, and if it returns an error the command is not executed. It doesn't try one then the other, nor is there any "fallback" to root.

User=0day considered harmful in systemd

Posted Jul 14, 2017 21:17 UTC (Fri) by davidstrauss (guest, #85867) [Link] (5 responses)

Two issues are still getting mixed together here:

  • What heuristic should determine the intention of a packager or administrator when evaluating "User=" values?
  • How should systemd handle a failure to implement the configuration (as determined by the heuristic)?

I think people are over-focusing on the former because it's related to the User=0day example (and would have handled that case), but the latter is more relevant to ensuring "fail safe" behavior. Even if systemd could read the mind of the person who wrote the unit file (i.e. always knowing whether to treat it as a username or numeric UID), the latter risk exists if the necessary user is missing. A broken heuristic just happens to make the latter risk more likely.

It's also not possible to just "always treat the User value as a user name" (as you suggest) because of the existing semantics of User=, which thousands of packages (and countless sysadmin-configured units) assume includes support for numeric UIDs. Treating every User= configuration as a username would make handling of the latter case even more important because legacy User=$UID configurations would generally fail to match when treated as usernames. That's why systemd has addressed the more fundamental issue: failing safely when user lookup fails.

User=0day considered harmful in systemd

Posted Jul 15, 2017 0:08 UTC (Sat) by anselm (subscriber, #2796) [Link] (2 responses)

Whatever the case may be, systemd shouldn't execute stuff as root that is not meant to be executed as root. That seems to be sorted now, which is good.

One could argue that systemd also shouldn't try to be smarter than is good for it, e.g., by enforcing its own undocumented assumptions about what usernames ought to look like that may or may not be grounded in reality. There's no doubt that systemd is a nifty piece of software and a huge improvement on anything that came before it, but sometimes a little less would actually be more.

User=0day considered harmful in systemd

Posted Jul 15, 2017 0:43 UTC (Sat) by rahvin (guest, #16953) [Link] (1 responses)

Has anyone seriously argued systemd should treat a username starting with a number as root? Because I honestly haven't seen anyone say it is, in fact I'd argue that's the one thing pretty much everyone agrees with and the patch to fix this already appears to have been merged. As someone else said, you don't ever want a program to assume that if the user doesn't exist it should run as root instead because the very act of naming a user (even if improperly done) means they didn't want it to run as root.

Anyway, in my mind that part of the discussion is already settled, patch is already merged, just need the release to come out. The publicity on the bug helps, people can now look at their service files and see if they have this happening and they can evaluate it with a simple scripted check using standard tools.

User=0day considered harmful in systemd

Posted Jul 15, 2017 1:07 UTC (Sat) by anselm (subscriber, #2796) [Link]

Has anyone seriously argued systemd should treat a username starting with a number as root?

I don't think so. Where systemd goes out on a limb, as far as I'm concerned, is by assuming that a username that starts with a digit is invalid (and that therefore the directive could be ignored altogether, such that in the absence of a User= directive the unit is by default executed as root). We've heard Lennart claim that this is the “least common denominator” among various Linux distributions, but we've also heard from people who aver that digits at the beginning of a username (or even all-digit usernames) are perfectly acceptable. The paragraph on User= in systemd.exec(5) certainly doesn't mention that restriction but it probably should.

User=0day considered harmful in systemd

Posted Jul 17, 2017 11:30 UTC (Mon) by dskoll (subscriber, #1630) [Link] (1 responses)

Anyone have any idea how many unit files rely on specifying a numerical UID for User= ? I bet few to none.

Systemd introduced a huge disruption to the Linux init system. Saying we have to keep supporting numerical UIDs in the User= directive seems pretty silly to me.

User=0day considered harmful in systemd

Posted Jul 18, 2017 0:35 UTC (Tue) by neilbrown (subscriber, #359) [Link]

> Saying we have to keep supporting numerical UIDs in the User= directive seems pretty silly to me.

certainly having an /etc/systemd/system.conf config option to disable numerical UIDS in User= would be welcome.
Shouldn't take too many lines of code, and creating a pull request on github is fairly easy.

User=0day considered harmful in systemd

Posted Jul 13, 2017 1:51 UTC (Thu) by peterhoeg (guest, #4944) [Link] (2 responses)

> What is the use case for having a systemd unit file specify a user that doesn't exist in the system passwd file?

Users can come from other places than the passwd file such as LDAP and NIS which may be unavailable at the time of systemd parsing the unit files.

User=0day considered harmful in systemd

Posted Jul 13, 2017 6:00 UTC (Thu) by anselm (subscriber, #2796) [Link] (1 responses)

Where in the systemd documentation does it say that user names must be resolvable to UIDs when unit files are loaded? Surely it should be sufficient for systemd to be able to figure out a UID for a user name when the program in question is about to be started.

User=0day considered harmful in systemd

Posted Jul 13, 2017 15:25 UTC (Thu) by mezcalero (subscriber, #45103) [Link]

See the man page about that:

https://www.freedesktop.org/software/systemd/man/systemd....

Specifically:

> … If DynamicUser= is not used the specified user and group must have been created statically in the user database no later than the moment the service is started, for example using the sysusers.d(5) facility, which is applied at boot or package install time…

User=0day considered harmful in systemd

Posted Jul 13, 2017 14:30 UTC (Thu) by giggls (subscriber, #48434) [Link] (1 responses)

Using a system where the user account data comes from a database (ldap, ...) instead of a plain file might be a valid use case even for system accounts.

Especially when they need to have the same userid across different machines.

I used to do this for the sudo group in some setups.

User=0day considered harmful in systemd

Posted Jul 14, 2017 10:52 UTC (Fri) by dskoll (subscriber, #1630) [Link]

In that case, those unit files should depend on the databases being started first and should be re-parsed at that time. It gets a bit ugly... you need to parse the unit files once to get dependencies and not care about unknown users at that point, but care about unknown users when you actually go to start the service. I think it's manageable, though.

User=0day considered harmful in systemd

Posted Jul 13, 2017 10:14 UTC (Thu) by edeloget (subscriber, #88392) [Link] (2 responses)

> Attempting getpwuid() first on all-numeric User= directive could also open an attack vector
> if (1) a named user exists with a matching, all-numeric name and (2) the UID matching the
> number in User= doesn't yet exist. An attacker would then just have to spam user creation
> until a user exists with a UID that matches the number after User=.

You fully understand that to create a user, you need to be an administrator, right? I mean, you already have the right to write in /etc/passwd which pertains to the root:root user. If you can do this, you're able to do any attack you want, unless I'm mistaken.

If the security scenario you envision to justify that it's not safe to do either getpwnam(3) or getpwuid(3) starts by not trusting the root user then, of course, you're somewhat at risk :)

User=0day considered harmful in systemd

Posted Jul 13, 2017 12:12 UTC (Thu) by matthias (subscriber, #94967) [Link]

I guess this only applies to systems, where an unprivileged user has the ability to create other unprivileged users. This could be possible by a suid executable that only allows to add new users, but not to execute arbitrary code. Of course it has to be carefully checked that this executable works as intended (as with all suid executables).

I am sure there are such systems out there, even if it is not the default configuration.

User=0day considered harmful in systemd

Posted Jul 13, 2017 19:23 UTC (Thu) by davidstrauss (guest, #85867) [Link]

> You fully understand that to create a user, you need to be an administrator, right?

No, you do not if an attacker can trigger a systems management tool that provisions a user. This applies to web hosts, SaaS vendors, universities with self-enrollment tools, and anyone else who allows low-privilege users to trigger user provisioning. Moreover, a user often has some control of the user they've caused to be provisioned, making a service running under that user all the more vulnerable.

Additionally, if you do rely on something like LDAP for users (which I think is not a good design for system users), users can get created by anyone with control of LDAP -- directly or indirectly in the sense of self-registration.

User=0day considered harmful in systemd

Posted Jul 14, 2017 12:15 UTC (Fri) by ballombe (subscriber, #9523) [Link]

The solution is to add a UID= directive for numeric UID instead of trying to guess.

User=0day considered harmful in systemd

Posted Jul 12, 2017 18:08 UTC (Wed) by smckay (guest, #103253) [Link] (4 responses)

It's surprising and somewhat concerning to me that it's possible for systemd to run a service as root when a non-root user has been explicitly requested. Okay, systemd has this nice generic handling of syntax errors that makes it a little difficult from an engineering perspective to obtain the correct behavior. That doesn't make this issue notabug.

User=0day considered harmful in systemd

Posted Jul 12, 2017 18:54 UTC (Wed) by anselm (subscriber, #2796) [Link] (3 responses)

I can see how it might be distasteful to a systemd developer to have to special-case the User= directive. On the other hand, it is usually better engineering to fail towards safety. Simply disregarding the directive by way of the usual mechanism in systemd is easy and consistent, but potentially dangerous and certainly not what a user would expect – based on outcomes, not starting the unit at all is the much better option compared to silently starting it as root.

User=0day considered harmful in systemd

Posted Jul 12, 2017 21:30 UTC (Wed) by raven667 (subscriber, #5198) [Link] (2 responses)

Its not just special casing the User directive though, if you go down that road you need to special case every directive that might have security relevance, anything specifying SELinux or AppArmor, capabilities, namespaces, etc. and deal with the fact that different distros are set up slightly differently and that applications should include unit files in their source that may include directives that don't exist on old systems, including security features, but should still work across a wide range of old and future systems.

User=0day considered harmful in systemd

Posted Jul 12, 2017 23:06 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

It's reasonable to expect that if you explicitly go to the trouble of specifying a user for a unit that isn't root, that unit won't inadvertently – and silently – be executed as root. This is not a backwards-compatibility issue since Unix has supported different users from the very start. I'm not as convinced that directives controlling new(ish) security features should automatically lead to hard errors on older systems that don't offer these features at all; perhaps emitting very obvious warnings that SELinux or AppArmor is not supported on this system and that therefore these features cannot be enabled is the reasonable tradeoff here, just so the administrator knows what to expect. It depends.

What certainly should not happen at all, ever, is that important and ubiquitous security settings are silently ignored (or ignored with a generic warning) because systemd doesn't like a given configuration value. Arguably an error message such as “invalid username ‘0day’, using ‘root’ instead” would have called attention to the situation being discussed here, where silently falling back to root is an obviously bad idea no matter what distribution you're using and whether you're in the past, present, or future.

User=0day considered harmful in systemd

Posted Jul 13, 2017 2:50 UTC (Thu) by zuki (subscriber, #41808) [Link]

The warning (printed in red, multiple times, every time the unit is loaded) was:
> /etc/systemd/system/test.service:28: Invalid user/group name or numeric ID, ignoring: 0day

Anyway, that's how it was. systemd was patched to say:
> /etc/systemd/system/test.service:28: Invalid user/group name or numeric ID: 0day
> test.service: Failed to create test.service/start: Unit test.service is not loaded properly: Exec format error.

User=0day considered harmful in systemd

Posted Jul 12, 2017 19:53 UTC (Wed) by robert_p (guest, #110578) [Link]

Sorry but I don't really see any argument here

> One can argue that systemd should not be checking usernames for itself

Not sure if that's the best criticism, I don't think it makes sense. But one could argue that how things are handled are suboptimal either way. First introducing somewhat arbitrary restrictions, then silently ignoring them and default back to root.
But what's the case for it?

> But systemd is rather opinionated about how things should work, naming, and so forth,

Ok... ? The sentence starts with "but", yet there's no argument made. They are 'opinionated about something they shouldn't be? That doesn't really say anyhting.

> so it should not come as an enormous surprise that it enforces its rules on system usernames.

Well, I guess it did come as a surprise, hence the discussion.

> As Poettering pointed out—several times—systemd is free software; users, distributions, and others are all welcome to modify it to suit their needs.

"If you don't like it do it better!" the universal last line of defense. Of course that's true. People can adjust it. They could patch systemd. It isn't an argument for handling it the way it's done in the first place, though.

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:12 UTC (Wed) by judas_iscariote (guest, #47386) [Link] (1 responses)

"but should instead consult the system by using getpwnam()—many did."

getpwnam, additionally to what davidstrauss said...will call the the NSS module stack.. one should not do NSS lookups from PID 1.

User=0day considered harmful in systemd

Posted Jul 12, 2017 20:29 UTC (Wed) by zuki (subscriber, #41808) [Link]

Yes! To clarify: the nss module stack can do anything it wants, e.g. query the LDAP database over the network, and it does that synchronously. So systemd does not call getpwnam() when parsing unit files, but instead after forking in the child that will become the service process.

User=0day considered harmful in systemd

Posted Jul 13, 2017 4:49 UTC (Thu) by OrbatuThyanD (guest, #114326) [Link]

hoping that this is the straw that breaks the systemd camel's back.

User=0day considered harmful in systemd

Posted Jul 13, 2017 8:32 UTC (Thu) by cyperpunks (subscriber, #39406) [Link]

Might be wise for systemd maintainers to take some input from community before they end up here: https://lwn.net/Articles/196523/

User=0day considered harmful in systemd

Posted Jul 13, 2017 15:19 UTC (Thu) by mezcalero (subscriber, #45103) [Link] (5 responses)

A couple of more notes that I think are worth mentioning:

1. systemd is not just the consumer of user names via User=/Group=, it's also the creator of users specified that way. Specifically, If you combine User=/Group= with DynamicUser=1, then systemd will automatically allocate a user for the time the unit is running, and release it afterwards. When creating users we need to be particularly careful with usernames, as under no circumstances we should be a vehicle for creating users with names that wouldn't be permissible under local policy otherwise.

2. I am pretty sure the POSIX username rules are very much underspecified, and there's a good reason nobody implements them. They permit fully numeric usernames, and do not require any size limit (not even a minimum length, suggesting that an empty string might be a valid username, if you read the spec by the word). Real-life implementations depart from the POSIX rules in both directions: upstream shadow-util is both more liberal than POSIX (as it permits one trailing $ in the name, for compat with Windows, which POSIX wouldn't allow) and more strict (as it doesn't permit leading numerals — and hence no fully numeric usernames either). Fedora patches shadow-utils and explicitly replaces these rules by different ones, that do permit leading numerals, but still no fully numeric usernames. RHEL then patches shadow-utils again, and is even more liberal: it permits umlauts in usernames. Yuck. Debian of course is Debian, makes no decision, and instead added a configurable setting, where you may specify a regex instead, of your choice. I mean, what could rules be good for, unless you can actually bend them the way you want? ;-) systemd's rules describe the set of names that are OK by all of these implementations, and hence closest to what shadow-utils implements, except for the Windows thing and for the fact that we enforce a minimum and maximum username length.

3. systemd-logind doesn't validate user names in contrast to this, for the reason that in that case we just consume stuff, and do not create stuff, and we also do not define the ecosystem (unit files are after all defined by systemd, originally, and User= only exists and only consumes stuff written as native unit, there is no further legacy to care about and keep compatibility to), but just integrate with an existing one. In that case we assume that PAM passes to us pre-validated data (even though it's not particularly careful with that).

Lennart

User=0day considered harmful in systemd

Posted Jul 13, 2017 15:47 UTC (Thu) by smitty_one_each (subscriber, #28989) [Link] (2 responses)

It seems that an option to load a suitably secure CSV file of valid User names that are whitelisted and checked against the User= entry, before launching a unit, would be one way to increase security, while still supporting traditional behavior.

User=0day considered harmful in systemd

Posted Jul 20, 2017 1:26 UTC (Thu) by welinder (guest, #4699) [Link] (1 responses)

There is no problem for which a CSV file is the solution.

There are more different ways of interpreting CSV files than there are programs that read them. There is even an RFC adding one to the number of competing standards.

User=0day considered harmful in systemd

Posted Jul 20, 2017 19:54 UTC (Thu) by flussence (guest, #85566) [Link]

Soon we'll be rid of the /etc/passwd menace and replace it with systemd's much better documented and modern DBus+XML API!

Just kidding. You meant TCB, didn't you?

User=0day considered harmful in systemd

Posted Jul 13, 2017 16:28 UTC (Thu) by smoogen (subscriber, #97) [Link] (1 responses)

Maybe it is time to have a Portable Operating Systemd Interface (POSDI?) written up so that people can agree on what 21st century conventions should be versus finding them out via surprise. Is that something that could be started at the next systemd conference? [This is not meant to be a troll of any sort. It is meant that maybe it is time to get these things proposed, written and agreed on so that operating systems, programmers etc aren't feeling surprised or that if they are forking from POSDI that they know why they are doing it.]

User=0day considered harmful in systemd

Posted Jul 19, 2017 14:38 UTC (Wed) by cesarb (subscriber, #6266) [Link]

A first step could be to update the FSSTND for the systemd world. For instance, the latest release I found for it (FHS 3.0) mentions only PID files and sockets in /run, while recent systems also create things like /run/user/<pid> there.

Fully expected this thread to be full of Trolls

Posted Jul 15, 2017 0:51 UTC (Sat) by rahvin (guest, #16953) [Link]

I fully expected this thread to be full of the Lennart "fan club" trolls. I'm pleasantly surprised. It's got to be progress where we can have a thread with Lennart and systemd named in the article and it's not full of trolling. Even a year ago half the posts would be the "fan club" out in force criticizing anything he's involved with. Yay for progress!


Copyright © 2017, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds