Weekly Edition
Return to the Distributions page
| |||||||||||||
De-bashing DebianA report that a thousand or more Debian packages will no longer build from source would be a cause for concern at any time, but when the project is trying to button up for a release, it is especially unwelcome. It turns out that the initial estimates were overblown, but that a change in the dash shell to make it POSIX-compliant, exposed a number of "bash-isms" that had crept into the Debian build scripts. In the process of tracking things down, various problems with checkbashisms were found as well. As reported to debian-devel by Raphael Geissert, the dash shell recently changed:
dash recently added support for the magic variable $LINENO, which was the last
piece to make it POSIX compliant. However, this change made the autoconf-generated configure scripts use dash to execute the script's code. Without
support for LINENO, configure scripts exec to bash automatically.
That caused multiple Debian packages to "fail to build from source" (FTBFS). Geissert's estimate for the number of packages affected was a rather eye-opening 1504. That led Petter Reinholdtsen to ask:
What about reverting this change in dash until after Squeeze is
released? Now [seems] like a bad time to make thousand of packages in
Debian fail to build from source. :)
Several folks pointed him at the bug report, where that plan was being put into action. The newer dash was put into the experimental release tree, while the unstable—eventual Squeeze—tree got a dash without the POSIX upgrade. The original move to dash was done for performance and POSIX-compliance reasons. At the time, it broke various things because scripts that specified /bin/sh were really written to run with bash and did not conform to the POSIX shell standard. The checkbashisms script was developed to help find these problems so that they could be fixed. Unfortunately, as the Debian folks just found, checkbashisms produces a lot of false positives as well as some false negatives. As it turns out, building the archive with both the old and the new dash yielded a list of "only" 124 packages that failed to build. Still a concern, but not one that merited the outcry that the first message elicited. Neil Williams was rather annoyed both at the dash change and the original report: "(I've heard of off-by-one errors but off-by-one-order-of-magnitude is a stretch.) [...] Please, let's get some accurate figures before raising things like this." But as David Weinhall points out, the underlying problem is not with dash at all:
You're getting things the wrong way around. The version of dash that
will be put in experimental will be the correct one, the one in unstable
will be the crippled one. The reason things [fail] isn't because of
dash, but because of sloppy programming on behalf of people that still
believe that bash is the say all and end all when it comes to shell
scripts.
But Lucas Nussbaum, who ran the comparative archive builds, wondered about the whole exercise of removing bash-isms from the Debian build scripts. He is concerned that a lot of work is being done for POSIX-compliance which "only a minority of users care" about and to shave a few seconds from boot time.
Was is really the right path to follow? Wouldn't it have been easier to
work on bash to make sure that it supports everything POSIX requires,
and to improve its performance a bit?
Now we are going to patch configure scripts to make sure that they can run correctly with dash. What's next? Rewrite C programs that use GCC-specific extensions? There is quite a list of commonly used bash-isms on the page about the dash move linked above. For good or ill, bash is the shell that many folks use and test under. The fact that it will happily run /bin/sh scripts that have used bash extensions is perhaps unfortunate in some ways, but it's also something that isn't going to change. In order for Debian to stay dash-pure, it will require an ongoing effort to patch the build scripts that come from upstream. It may be that there are better uses of Debian developers' time. (Log in to post comments)
De-bashing Debian Posted May 27, 2010 6:13 UTC (Thu) by mhelsley (subscriber, #11324) [Link]
LTP faces similar issues. They have been removing bash-isms so LTP can run on embedded machines with dash.
De-bashing Debian Posted May 27, 2010 19:56 UTC (Thu) by vapier (subscriber, #15768) [Link]
the focus of LTP moving to POSIX shell was not dash, but any POSIX compliant embedded shell. a better example would be the shells used in busybox since those are the ones that appear much more often in the embedded world.
Dash Posted May 27, 2010 6:21 UTC (Thu) by sladen (subscriber, #27402) [Link] The change is perhaps less painful than the article implies, most of the work to the Debian packages has been on-going since/bin/sh was switched to Dash in Ubuntu 6.10 (October 2006). LWN also covered the process one year ago in the article "A tale of two shells: bash or dash" (July 2009).
Dash Posted May 28, 2010 9:58 UTC (Fri) by joey (subscriber, #328) [Link]
Actually, Debian has been working on removing bashisms since before Ubuntu existed.
De-bashing Debian Posted May 27, 2010 13:54 UTC (Thu) by Tobu (subscriber, #24111) [Link] Many bashisms tend to improve maintainability. For example, [[ ]] helps prevent quoting errors, and >() is more foolproof than creating temporary files or fifos. I don't see a big performance impact in supporting some of these extensions in an extended mode for dash, to get a bit more speed than bash without doing a lot of pointless porting.
De-bashing Debian Posted May 27, 2010 16:57 UTC (Thu) by jengelh (subscriber, #33263) [Link]
And with systemd [see recent LWN article], eventually there is less need to use shell for booting.
De-bashing Debian Posted May 28, 2010 11:45 UTC (Fri) by nescafe (subscriber, #45063) [Link]
Hopefully not -- bash syntax makes for perfectly readable conf files. One of my biggest annoyances with upstart is that Scott invented yet another poorly documented config file syntax instead of just embedding bash and using it for all his config needs.
De-bashing Debian Posted May 27, 2010 20:01 UTC (Thu) by vapier (subscriber, #15768) [Link]
the usage of globs on the RHS of tests makes life a lot nicer too
POSIX: case ${foo} in *"simple string"*) bar=true;; *) bar=false;; esac
i'll take bash every time
De-bashing Debian Posted May 28, 2010 13:23 UTC (Fri) by mpr22 (subscriber, #60784) [Link]
By all means, use bash in your scripts... just don't lie to the computer by telling it the script is written in sh.
De-bashing Debian Posted Jun 3, 2010 18:03 UTC (Thu) by vapier (subscriber, #15768) [Link]
this article is about moving from bash to POSIX shell, not about bashisms showing up in scripts using /bin/sh as their interpreter
De-bashing Debian Posted Jun 7, 2010 12:20 UTC (Mon) by nye (guest, #51576) [Link]
>this article is about moving from bash to POSIX shell, not about bashisms showing up in scripts using /bin/sh as their interpreter
In that case I have completely misunderstood this article, as I read it as being entirely about bashisms showing up in scripts using /bin/sh as their interpreter.
De-bashing Debian Posted May 28, 2010 10:00 UTC (Fri) by joey (subscriber, #328) [Link]
The correct thing to do if you have a good reason to use a bash-specific feature in your shell script is to put #!/bin/bash at the top of it.
De-bashing Debian Posted May 27, 2010 17:10 UTC (Thu) by iabervon (subscriber, #722) [Link]
It seems to me that the common flaw with these scripts is not that they're using bash features, but that they're running /bin/sh. If they ran the executable that's supposed to support the features they need, and didn't make the assumption that any shell that supports all of the POSIX features also supports bash features, they would still work.
Of course, it may be worth giving up bash features for implementation efficiency, but changing the used dialect to a better one is a separate issue from making sure to have support for the dialect you're using.
De-bashing Debian Posted May 29, 2010 18:42 UTC (Sat) by giraffedata (subscriber, #1954) [Link] If they ran the executable that's supposed to support the features they need, and didn't make the assumption that any shell that supports all of the POSIX features also supports bash features, I don't think the programs are making that assumption. What they're assuming is that those Bash features are POSIX. And it's hard to do otherwise. A great deal of software engineering is done by trial and error, not the more laborious reading of specs. If your trials happen to use Bash, there's no visible error. Gcc has the -ansi option to support trial and error development of ANSI-only C programs. Why doesn't Bash have a similar thing? I write #!/bin/sh at the top of my shell programs to express my desire that it work with any POSIX shell, but in fact I always test with Bash and the program often doesn't work with anything else.
De-bashing Debian Posted Jun 1, 2010 21:39 UTC (Tue) by lab (subscriber, #51153) [Link] Gcc has the -ansi option to support trial and error development of ANSI-only C programs. Why doesn't Bash have a similar thing?man bash: --posix Change the behavior of bash where the default operation differs from the POSIX standard to match the standard (posix mode).
De-bashing Debian Posted Jun 2, 2010 1:42 UTC (Wed) by giraffedata (subscriber, #1954) [Link] Gcc has the -ansi option to support trial and error development of ANSI-only C programs. Why doesn't Bash have a similar thing?man bash: --posix Change the behavior of bash where the default operation differs from the POSIX standard to match the standard (posix mode). That's not what that is. It affects behaviors where something is valid in both POSIX and Bash but means something different, but it does not affect behavior where something is valid in Bash and not valid at all in POSIX. I.e. it doesn't force a program to fail just because it exploits a bashism. That's what gcc -ansi is.
De-bashing Debian Posted May 27, 2010 17:16 UTC (Thu) by jengelh (subscriber, #33263) [Link]
Ironic. Replacing the use of bash by a POSIX shell, wasting the just-won time for forking extra processes like printf(1), because echo -e is not guaranteed to exist.
De-bashing Debian Posted May 28, 2010 0:35 UTC (Fri) by MegabytePhreak (subscriber, #60945) [Link]
According to http://linux.die.net/man/1/dash, printf is a builtin in dash, so it shouldn't need to fork any more that using echo -e, assuming it is also a builtin.
De-bashing Debian Posted May 27, 2010 19:00 UTC (Thu) by EV (subscriber, #5528) [Link]
If bash performance is an issue, where's the bash LLVM project ;-)
Just run configure under bash explicitly Posted May 27, 2010 22:38 UTC (Thu) by proski (subscriber, #104) [Link] I think a reasonably simple short term solution would be to run configure scripts in bash:
That would avoid crippling dash while making the build systems use the same shell they were using before dash started supporting $LINENO.
Upstream developers using Debian would run
Just run configure under bash explicitly Posted May 27, 2010 23:48 UTC (Thu) by ABCD (subscriber, #53650) [Link]
The proper way to do that would be to run `env CONFIG_SHELL=/bin/bash /bin/bash ./configure`, so that configure knows to use /bin/bash to start any subshells (for instance, in the shebang of config.status).
De-bashing Debian Posted Jun 1, 2010 21:43 UTC (Tue) by lab (subscriber, #51153) [Link] I never got it. Tell me again why Bash is a bad thing?
De-bashing Debian Posted Jun 2, 2010 11:17 UTC (Wed) by nye (guest, #51576) [Link]
Well, switching to a minimal POSIX shell can measurably improve performance when you're running short scripts - as happens frequently when booting.
The real problem though isn't so much using bash or bash-specific features; it's using bash-specific features while declaring your script to be POSIX compliant by specifying /bin/sh as the interpreter. This is a straightforward bug which could simply be fixed by specifying /bin/bash, but too many people don't even realise that they've made the mistake.
De-bashing Debian Posted Jun 3, 2010 21:08 UTC (Thu) by oak (subscriber, #2786) [Link]
And this is an issue because people want systems not having Bash to boot up and otherwise work. It might be possible to install Bash to these systems afterwards for interactive use, but system itself running scripts isn't interactive work.
(Some of the reasons why these systems might not have Bash pre-installed is that it's large or that they disagree with its licence and therefore have something like Busybox.)
|
|||||||||||||