LWN: Comments on "The case of the supersized shebang"

The case of the supersized shebang

cleverca22 — Wed, 16 Oct 2019 00:24:40 +0000

https://github.com/NixOS/nixpkgs/pull/55786

and i have already written a PR to do exactly that

Patch backports

nix — Fri, 01 Mar 2019 16:30:52 +0000

That's more or less what I do locally, with most of my systems containing no real persistent state (either all contents are rsynced nightly from the big network-critical server, or they're just outright NFS-mounted from it) and running latest stable and sometimes random rcs with local hacks... and said server running a "stable stable" which survived at least a couple of weeks on the other boxes, with obviously-crucial security fixes cherry-picked back into that if need be. (I tried the distributed systems approach, having N of every distributable crucial service on different machines, and found that unless you spent *ages* analyzing you ended up introducing N single points of failure rather than just one, so I've gone back to the "one great big single point of failure which is at least easy to identify" approach. It is also too heavy to lift and won't fit through the door so is probably fairly theft-proof.)

This approach seems to work. I haven't lost any filesystems on the big server for, oh, almost a year now! :P

Patch backports

flussence — Fri, 01 Mar 2019 01:48:17 +0000

I'm thinking the whole concept of having a -stable branch might be wrong if this is how it works in practice. The label “stable” is always going to be wishful thinking at best, since the halting problem applies.

One could argue it'd make sense to bless individual kernel versions as “stable” after a grace period passes with no complaints raised, but the existing process needs to be fixed so those complaints are heard before that can happen.

Patch backports

nix — Sun, 24 Feb 2019 22:52:54 +0000

I use stable kernels because I want serious bugfixes, stability fixes, and security fixes but don't want to run -rc kernels on the systems my job depends on that house all my data thankyouverymuch. This... does not seem like a terribly unusual requirement, to me. I specifically do not avoid updating because I "don't want the latest version": if I could reliably update without rebooting I'd do it within minutes of every stable kernel coming out (but ksplice is fiddly for a self-compiled kernel and requires patch-by-patch analysis to determine which changes can be applied and kgraft is just as bad, AIUI: not a thing you can just throw a new stable kernel at and say "magically update me to this without rebooting").

Like most people operating small numbers of machines rather than huge failovered farms, I upgrade at irregular intervals, when a stable kernel with a bugfix seemingly serious enough to make it worth the annoyance-cum-terror of rebooting and flushing all my caches comes along -- though I suspect most people don't routinely read the git log and patch series of everything that hits -stable the way I do. (Rebooting is much less bad for performance than it used to be, thanks to bcache caching all the seeky metadata, but rebooting my core server is *always* terrifying: what if it never comes back up? It always has so far but this is a PC which means it's shit by definition, and I am not confident that all-Intel-mobo-plus-Intel-UEFI-only-one-corp-to-blame means it's reliable before the OS has started, not when I've *seen* the thing lock up once or twice when trying to enumerate its USB ports, exhaust some sort of watchdog timer, and autoreboot again before completing POST. I'm tempted to switch to kexec just to avoid most of that terror, but unfortunately kexec is even *less* routinely tested so the terror quotient would be greater. Yes of course I have backups, and backups of backups, and backups of backups of backups, but terror does not yield to common sense. I keep the ludicrous levels of backups anyway.)

The case of the supersized shebang

flussence — Sun, 24 Feb 2019 09:17:49 +0000

For a long time perl *also* had a bug where if the line looked like /path/to/perl$anyversion, it'd try to run it directly. That meant perl5 wouldn't run perl6 scripts at all, and probably would also break in weird situations with two perl5 environments on the system. It was fixed very recently, IIRC.

Patch backports

NAR — Sat, 23 Feb 2019 21:32:25 +0000

I guess people use -stable kernels because they explicitly do not want the latest version (due to fears about regressions, being locked to a version number, etc.). I don't know if they want a new -stable kernel every 3-4 days or just take a look at the top of their preferred -stable tree at their convenience (maybe once every quarter) and use that version.

Patch backports

nix — Fri, 22 Feb 2019 15:44:21 +0000

If that was the way it worked, then even serious bugs wouldn't get fixed more often than once every three months. That would render the stable kernels almost pointless, since you could always just upgrade to the also-stable Linus kernel the fixes came out of instead.

The case of the supersized shebang

epa — Fri, 22 Feb 2019 08:31:51 +0000

Sounds like a handy trick to speed up your shell scripts!

The case of the supersized shebang

rweikusat2 — Thu, 21 Feb 2019 15:02:15 +0000

......... :-))

binfmt_misc

mm7323 — Thu, 21 Feb 2019 08:24:09 +0000

Couldn't you just use binfmt_misc to match on #! and then run a process that can open up the script and figure out the required interpreter and arguments in userspace?

That should surely be possible for a distro that wants to do unusual things and only costs an extra exec() for starting up scripts which aren't known to be that fast anyway.

The case of the supersized shebang

neilbrown — Wed, 20 Feb 2019 22:59:08 +0000

Time for a new patch tag I guess:

Fixes: NOTHING - don't ever back-port this to stable

The case of the supersized shebang

pebolle — Wed, 20 Feb 2019 22:32:47 +0000

I see. So it seems - or, put another way, strace showed me - that this behaviour of /bin/sh is what allows execlp() to do its, well, magic.

The case of the supersized shebang

foom — Wed, 20 Feb 2019 22:08:28 +0000

Some shells *also* handle this themselves.

E.g., with bash, if execve fails with -ENOEXEC, it will reset all the shell state and evaluate the #!-less script file directly in the post-fork bash process, rather than exec'ing anything at all (neither /bin/sh nor /bin/bash!)

Patch backports

Spack — Wed, 20 Feb 2019 20:32:41 +0000

How come a test kernel can feed a stable kernel with backported patches? I would have expected that only patches within the Linux stable kernel 5.0 would be allowed to be backported to previous versions?

The case of the supersized shebang

smitty_one_each — Wed, 20 Feb 2019 13:43:05 +0000

It's a shame that something bad happened, but, considering the man-years of effort put into the kernel and the princely sums paid to so many of the contributors...it almost seems a left-handed compliment that such boo-boos are so rare.

The case of the supersized shebang

pebolle — Wed, 20 Feb 2019 11:55:01 +0000

Michael Kerrisk's TLPI (p. 575) points to execlp() and execvp() for that behaviour. A quick test showed that execlp() indeed treats such an executable file "as though [it] started with a line containing the string #!/bin/sh".

The case of the supersized shebang

epa — Wed, 20 Feb 2019 11:19:07 +0000

OK, according to the historical lore linked later in this discussion, it's the shell that is responsible for treating an executable as a shell script if it looks like one. Also Perl's exec() call does it. But the raw system call and C library do not: trying to execl() an executable file which just contains 'echo hello' will fail. It needs the shebang line.

The case of the supersized shebang

meuh — Wed, 20 Feb 2019 09:35:37 +0000

Artificial intelligence gives artificial answers

The case of the supersized shebang

epa — Wed, 20 Feb 2019 08:13:50 +0000

Isn’t there an ancient convention, predating shebang lines, that if it doesn’t look like a binary executable it gets run through /bin/sh by default?

Wrong interpreter

marcH — Wed, 20 Feb 2019 06:49:08 +0000

Falling back on the default shell is not the same than falling back on the current shell.

The case of the supersized shebang

jeremyhetzler — Wed, 20 Feb 2019 04:10:03 +0000

More on how shebang works, and gory details of how it differs in various systems:

https://www.in-ulm.de/~mascheck/various/shebang/

The case of the supersized shebang

scientes — Wed, 20 Feb 2019 01:55:02 +0000

I came here to post the same thing. The article really should be fixed. Linux has nothing shell specific, and one of systemd's goals was to remove that dependancy, so reading this kind of irked me (really?????).

The case of the supersized shebang

perennialmind — Tue, 19 Feb 2019 23:47:10 +0000

It looks like musl libc doesn't have the rickety fallback to crazytown. I for one hope they leave this POSIX compliance "bug" as is.

The case of the supersized shebang

ewen — Tue, 19 Feb 2019 23:19:01 +0000

Leaving aside the initial regression (the #! line has always been prone to truncation; as other comments say it used to be truncated earlier in older Unixes, and older software like perl has workarounds for this for decades), the discussion about the regression induced in *stable* kernels is more worrying. The discussion seems to be saying "whitelisting patches for the kernel stable tree is too hard and does not scale, so let's select them automatically and then let people blacklist the ones that shouldn't be backported". The "spotting ones to blacklist" seems even less likely to reliably scale, with more false negatives ("spot the problem patch amongst the hundreds automatically added this week" missing some that should have been flagged "don't backport").

If the "stable" kernels are going to have not-manually-selected/verified changes in them, and a shorter release cycle, it seems to me they'd increasingly become "alternative bleeding edge" kernels with older features, but "assorted newer patches added in for flavour." At which point I wonder who would run them? Those wanting actual stability probably end up relying on their distro kernel teams manual review, and those wanting the bleeding edge probably want the new features too. In other words the explicit manual selection of "stable" changes, and the QA, is what makes them "stable" kernels. Which seems increasingly not to be happening with the upstream kernel stable trees, because it's a lot of work.

As a sysadmin my desire for "stable" anything is (almost) no regressions, and some fixes for critical issues. Almost by definition with a preference for stability (ie, availability/reliability) over changes.

Ewen

The case of the supersized shebang

jccleaver — Tue, 19 Feb 2019 19:38:03 +0000

> Frankly, I agree with the kernel here. Shebang lines are a convenience, and more importantly they're intended for use by the (userspace components of the) distro, and are accidentally useful to end users, but random other developers have (IMHO) no business writing shebang lines.

That's... kind of insane. Is this point of view common? That might explain all the scripts I see written by devs (often java devs, but not always) that are kept chmod 644 and have no shebang.

The first thing I do is make them executable and put the proper shebang in, unless there's some specific external $PATH call I need to wrap into it, which is just dumb... or weird.

This actually feels like a decent shibboleth among *nix admins and... non-*nix developers.

Wrong interpreter

rfunk — Tue, 19 Feb 2019 19:11:34 +0000

Wherever the shell fallback is implemented, the result is the same, so that seems irrelevant unless you're also going to patch all the shells that implement that fallback.

Wrong interpreter

dona73110 — Tue, 19 Feb 2019 19:00:55 +0000

> So if I understand the situation correctly, this patch was intended to avoid *maybe* getting the wrong interpreter (a situation that Unix users have known about for some 40 or 50 years) by falling back to the default shell -- which is almost certainly the wrong interpreter!

That's what corbet said but it's not exactly what's happening - if you look at the patch there's no default to a shell, it just returns an error that the file was not executable.

The shell is the one that takes this "can't execute" error as a cue to interpret the file shell instead :-x

Typo

tome — Tue, 19 Feb 2019 18:49:57 +0000

Uh oh, I cluttered the clutter... and now this. I'd best be quiet.

The case of the supersized shebang

dona73110 — Tue, 19 Feb 2019 18:49:38 +0000

Yeah, that line

>By default, the interpreter is a shell, which will interpret the file as a shell script

is not really correct; it's a shell convention but the kernel will definitely _not_ use a default interpreter if the file does not start with #!

Wrong interpreter

rweikusat2 — Tue, 19 Feb 2019 18:30:07 +0000

-*- language -*- close to the start of some text file causes Emacs to switch to the major mode for language.

The case of the supersized shebang

Anssi — Tue, 19 Feb 2019 17:21:48 +0000

> If, however, the file starts with the characters "#!", the remainder of the first line will be treated as the name of the interpreter to use (and possibly arguments to be passed to that interpreter).

Well, only a single argument, not "arguments".

The case of the supersized shebang

DrHyde — Tue, 19 Feb 2019 17:21:21 +0000

I believe that http://catb.org/jargon/html/V/vaxocentrism.html applies, except with modern Linux instead of VAX.

The case of the supersized shebang

zblaxell — Tue, 19 Feb 2019 17:02:19 +0000

Not really? Assuming it's a one-time adjustment during install, you just prepend, and in many cases you don't even need BEGIN:

#! /nix/store/mbwav8kz8b3y471wjsybgzw84mrh4js9-perl-5.28.1/bin/perl
use lib qw(
/nix/store/x6yyav38jgr924nkna62q3pkp0dgmzlx-perl5.28.1-File-Slurp-9999.25/lib/perl5/site_perl
/nix/store/ha8v67sl8dac92r9z07vzr4gv1y9nwqz-perl5.28.1-Net-DBus-1.1.0/lib/perl5/site_perl
/nix/store/dcrkvnjmwh69ljsvpbdjjdnqgwx90a9d-perl5.28.1-XML-Parser-2.44/lib/perl5/site_perl
/nix/store/rmji88k2zz7h4zg97385bygcydrf2q8h-perl5.28.1-XML-Twig-3.52/lib/perl5/site_perl
);

possibly adjusted to deal with the subtle differences between BEGIN { unshift(@INC, 'foo' }, use lib qw(foo), and -Ifoo--I remember there are some, but I don't remember what they are.

IIRC Perl 4 doesn't have 'use' or 'BEGIN', so perl4 definitely needs the shebang hack...but perl4 also implements a lot of bizarre stuff like "execute perl embedded in non-MIME-encoded email message", and...not the shebang hack (or at least it's not documented). Perl4 and NixOS are separated by a decade, so I doubt perl4's limitations had much to do with NixOS's design; however, early-2000's bugs in the Perl5 implementation could have been a problem for a project starting in 2003 (I know it was a problem with several of mine!).

Anyway, the answer I was looking for is apparently "only Perl and Guile do that, and NixOS does it for historical reasons, and they're going to stop now."

The case of the supersized shebang

zblaxell — Tue, 19 Feb 2019 17:01:52 +0000

> However, a shebang line is not reliably portable

I, for one, would prefer to solve that problem.

i.e. have a namespace that I can reference from portable scripts, like "#!/prefix/org/python/python/2.7.5", where the python installation puts in as many symlinks as required to accurately state the level of backward compatibility achieved.

The case of the supersized shebang

NYKevin — Tue, 19 Feb 2019 16:46:52 +0000

No, that is not what /usr/bin/env was invented for. /usr/bin/env was invented for setting up different environment variables - that's why it's called "env."

The case of the supersized shebang

rweikusat2 — Tue, 19 Feb 2019 16:36:45 +0000

Sort-of. It does a PATH seach internally. This may find some perl executable and it may even be the one supposed to execute a particular script. However, it might as well not. perl is a fairly stable execution environment, however, it has its share of "Let's break working stuff because it's just WRONG!" (that it works, presumably :->) people who add essentially random code changes[*] whose purpose doesn't seem to be known to anyone.

[*] Eg, starting with perl 5.16, xs functions can't be loaded via Dynaloader anymore unless a new keyword is added to the existing code. The documentation makes it very clear that loading xs-functions via Dynaloader Is Just Wrong[tm], but there's no positive justification for the change.

The case of the supersized shebang

tux3 — Tue, 19 Feb 2019 16:17:06 +0000

Oh I just realized what message you were replying to, my mistake.

The case of the supersized shebang

tux3 — Tue, 19 Feb 2019 16:16:00 +0000

Experimentaly "from __future import ..." seems to work fine with a #!, or even random comments before.
It seems python is happy as long as it's the first line actually executed, though the error message is confusing.

The case of the supersized shebang

roblucid — Tue, 19 Feb 2019 16:15:13 +0000

As a sysadmin who used Perl 4 there were key advantages over shell, otherwise some utilities would have to be C. Perl was much faster than shell by reducing fork and calls to utilities, it gave access to the C library, so could read system DBM files for example directly.

Indeed it re-read the #! line, IIRC to allow perl flags to be set and magic with env as perl at that time wasn't established in /usr/bin/perl.
Perl had extreme portability, it would use some tricks to work on 'broken' systems too.

The later Perl 5 gave more flexibility, but using dynamic libraries made the installation more fragile.

The case of the supersized shebang

dbaker — Tue, 19 Feb 2019 16:01:44 +0000

One of my projects has some interesting hacks because Haiku has /bin/env (no /usr at all, IIRC)