The case of the supersized shebang

Posted Feb 18, 2019 21:16 UTC (Mon) by flussence (guest, #85566)
Parent article: The case of the supersized shebang

What a mess. Several failures had to align perfectly for this to happen:

1. NixOS doing weird things that only work because the Perl 5 runtime contains a hack to compensate for them (what do they do for other languages without a similar hack, and why is this one different?)
2. The kernel using a fixed-size buffer with a weird legacy size (zero relation to PATH_MAX or ARG_MAX)
3. There were no regression tests for this part of the kernel yet
4. The kernel Bugzilla is functionally a bitbucket in a separate universe from normal kernel development
5. As others have pointed out, the backporting process is only as smart as Youtube's recommendation algorithm

Probably a few more I've overlooked too. After all that, we're left with only a revert and a vague promise to fix #3. That seems like a weak response at best.

The case of the supersized shebang

Posted Feb 18, 2019 21:54 UTC (Mon) by grahamc (guest, #111068) [Link] (2 responses)

> 1. NixOS doing weird things that only work because the Perl 5 runtime contains a hack to compensate for them (what do they do for other languages without a similar hack, and why is this one different?)

Our Perl tooling is an older part of the NixOS ecosystem, and has received a bit less attention lately when compared to other languages.

Perl's hack is not to compensate for us exactly, but presumably someone long ago who was bitten by the same problem. This hack existing is the reason we've allowed such a weird shebang to exist: it allowed us to and didn't cause us problems. Frankly, nobody noticed the shebang until the kernel broke it.

We handle other languages much more sanely.

> 4. The kernel Bugzilla is functionally a bitbucket in a separate universe from normal kernel development

Yes, this was a bit of a frustrating and troubling learning curve for us.

The case of the supersized shebang

Posted Feb 18, 2019 21:56 UTC (Mon) by grahamc (guest, #111068) [Link] (1 responses)

Oops, I meant to expand my reply to include: We're now fixing how Perl dependencies are handled, to not depend on gigantic shebangs.

The case of the supersized shebang

Posted Feb 18, 2019 22:25 UTC (Mon) by rweikusat2 (subscriber, #117920) [Link]

There isn't really anything wrong with "depending on gigantic shebangs" with Perl as the perl program is supposed to handle this case to enable consistent behaviour regardless of #!-implementation limits like processing only one argument directly attached to the #! or truncating the line prior to parsing it (the perlrun manpage mentions a historic 32 character limit).

A 127 character limit is documented behaviour for #! and Linux execve (the final byte being occupied by a 0).

The patch (and everyone involved with that) is entirely to blame here as it changes documented, observable behaviour based on speculations about possible errors based on the wrong assumption that nothing but the kernel ever interprets the content of a #!-line. But perl has been intepreting this line since at least 5.004, probably longer (that's the earliest example I know of).

The case of the supersized shebang

Posted Feb 18, 2019 22:08 UTC (Mon) by joncb (guest, #128491) [Link]

That was pretty much my takeaway too.

Fixing #4 seems like the way to go here. If the bug report managed to reach the right ear then the stable thing wouldn't have happened. Bonus points if this didn't require human input.

There's a possible angle around having an explicit negative tag (i.e. "do not backport this") so that the system can distinguish between "human has decided this patch should not be backported" and "human has not evaluated this patch for backporting".

The case of the supersized shebang

Posted Feb 18, 2019 22:32 UTC (Mon) by NYKevin (subscriber, #129325) [Link] (9 responses)

> 2. The kernel using a fixed-size buffer

That's not even slightly unusual. There are tons of places where syscalls and other kernel interfaces impose arbitrary limits on the sizes of things. Just about the only major exceptions are read()/write()-style interfaces such as file-likes and sockets.

> with a weird legacy size (zero relation to PATH_MAX or ARG_MAX)

Frankly, I agree with the kernel here. Shebang lines are a convenience, and more importantly they're intended for use by the (userspace components of the) distro, and are accidentally useful to end users, but random other developers have (IMHO) no business writing shebang lines. The distro knows where Python is installed, so that it doesn't need to write #!/usr/bin/env python to force a PATH search. The distro has the prerogative (and responsibility) to stick interpreters in sensible directories like /usr/bin instead of some crazy deep path under /opt, so that short truncation is not a problem either. In short, shebangs are remarkably well-suited to the specific problems that distros need to solve when implementing standard binaries using a custom interpreter.

The fact that shebangs also happen to be useful to end users sticking handy scripts in ~/bin is purely an accident. The idea that shebangs can or ought to be "portable" boggles the mind; just write a wrapper shell script. It's cleaner, more portable, and involves substantially less abuse of /usr/bin/env and other such "clever tricks."

The case of the supersized shebang

Posted Feb 18, 2019 22:48 UTC (Mon) by rweikusat2 (subscriber, #117920) [Link] (7 responses)

> Frankly, I agree with the kernel here. Shebang lines are a convenience, and more importantly they're intended for use
> by the (userspace components of the) distro, and are accidentally useful to end users,

https://www.in-ulm.de/~mascheck/various/shebang/4.0BSD_ne...

The assumption that people who aren't "involved with a distro" must be "end users" (who have no business programming anything) is ... interesting ...

The case of the supersized shebang

Posted Feb 19, 2019 1:24 UTC (Tue) by KaiRo (subscriber, #1987) [Link]

IMHO, even assuming that "end users" have no business programming is, erm, interesting, as I thought one point of FLOSS was that *anyone* can start programming and contributing.

The case of the supersized shebang

Posted Feb 19, 2019 9:36 UTC (Tue) by brooksmoses (guest, #88422) [Link] (5 responses)

I think you're misunderstanding NYKevin's point -- he specifically said "writing shebang lines", not "programming".

If you're not writing code that's part a distro's userspace, you should (generally) be writing code to be portable across different distros. (Even if you're an end-user with only one computer, you may well change your distro in the future and want to run your old code.) However, a shebang line is not reliably portable -- to pick an example that I've recently been having pains with, "#!/usr/bin/python2" assumes that your distro has Python version 2 installed in /usr/bin/python2. If you instead have only /usr/bin/python2.7, or /usr/local/bin/python2, or something else, then it doesn't work. And shebangs don't follow the PATH, so you have to specify an absolute path.

In the particular case I was having pains with, the relevant code was in a testsuite, and it would have been much nicer to have Autoconf find the right interpreter and then call it explicitly via something like "$PYTHON testfile.py".

The case of the supersized shebang

Posted Feb 19, 2019 13:48 UTC (Tue) by NAR (subscriber, #1313) [Link] (2 responses)

However, a shebang line is not reliably portable

I guess that's why /usr/bin/env was invented for...

The case of the supersized shebang

Posted Feb 19, 2019 16:36 UTC (Tue) by rweikusat2 (subscriber, #117920) [Link]

Sort-of. It does a PATH seach internally. This may find some perl executable and it may even be the one supposed to execute a particular script. However, it might as well not. perl is a fairly stable execution environment, however, it has its share of "Let's break working stuff because it's just WRONG!" (that it works, presumably :->) people who add essentially random code changes[*] whose purpose doesn't seem to be known to anyone.

[*] Eg, starting with perl 5.16, xs functions can't be loaded via Dynaloader anymore unless a new keyword is added to the existing code. The documentation makes it very clear that loading xs-functions via Dynaloader Is Just Wrong[tm], but there's no positive justification for the change.

The case of the supersized shebang

Posted Feb 19, 2019 16:46 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

No, that is not what /usr/bin/env was invented for. /usr/bin/env was invented for setting up different environment variables - that's why it's called "env."

The case of the supersized shebang

Posted Feb 19, 2019 15:11 UTC (Tue) by rweikusat2 (subscriber, #117920) [Link]

He wrote about "distro developers" vs "end user sticking scripts in ~/bin". This is a false dichotomy which is so false that it's laughable and all features of the system are ultimatively supposed to be useful to users.

*If* you want to run other people's code, don't stick interpeters in /opt/var/local/ultima-thule/lib/bin/asterisk/moff just because this makes perfect sense to you.

The case of the supersized shebang

Posted Feb 19, 2019 17:01 UTC (Tue) by zblaxell (subscriber, #26385) [Link]

> However, a shebang line is not reliably portable

I, for one, would prefer to solve that problem.

i.e. have a namespace that I can reference from portable scripts, like "#!/prefix/org/python/python/2.7.5", where the python installation puts in as many symlinks as required to accurately state the level of backward compatibility achieved.

The case of the supersized shebang

Posted Feb 19, 2019 19:38 UTC (Tue) by jccleaver (guest, #127418) [Link]

> Frankly, I agree with the kernel here. Shebang lines are a convenience, and more importantly they're intended for use by the (userspace components of the) distro, and are accidentally useful to end users, but random other developers have (IMHO) no business writing shebang lines.

That's... kind of insane. Is this point of view common? That might explain all the scripts I see written by devs (often java devs, but not always) that are kept chmod 644 and have no shebang.

The first thing I do is make them executable and put the proper shebang in, unless there's some specific external $PATH call I need to wrap into it, which is just dumb... or weird.

This actually feels like a decent shibboleth among *nix admins and... non-*nix developers.

The case of the supersized shebang

Posted Feb 18, 2019 22:39 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link]

Several failures had to align perfectly for this to happen:

This is very often the case. A functioning quality control system has multiple safeguards to protect defective products from being released, so any time one is released it must be the result of multiple failures. That's why it's important to look at all the points of failure and try to fix them all rather than laying the blame on a single scapegoat.