Bash gets shellshocked

It's been a crazy week for the Bash shell, its maintainer, and many Linux distributions that use the shell. A remote code-execution vulnerability that was reported on September 24 has now morphed into multiple related vulnerabilities, which have now mostly been fixed and updates released by distributions. The vulnerabilities have been dubbed "Shellshock" and the technical (and mainstream) press has had a field day reporting on the incident. It all revolves around a somewhat dubious Bash feature, but the widespread use of Bash in places where it may not really make sense contributed to the severity of the bug.

First bug

The bug, which was evidently introduced into Bash in around 1992, (ab)uses the environment variable parsing code by adding extra commands after defining a function. It was not really common knowledge that you could even define a function by way of an environment variable. It is a little-used feature that was also, evidently, not strenuously tested. An attacker who could set an environment variable could do:

    VAR=() { ignored; }; /bin/id

Normally, just on general principles, one avoids giving attackers ways to set environment variables in shells. But, as it turns out, there are a number of ways for an attacker to do so. The easiest (and perhaps best known) way is to use the Common Gateway Interface (CGI) protocol. All CGI programs get invoked with certain environment variables set whose values are controlled by the client (e.g. REMOTE_HOST, SERVER_PROTOCOL, etc.). If Bash is invoked, either because the CGI program is a Bash script or through some other means, it will parse the environment variables and execute any code the attacker tacked on. Game over.

There may not be all that many Bash-specific CGI programs in the wild, but many Linux distributions (notably not Debian or Ubuntu) make exploiting the bug easier still: they link /bin/sh to Bash. So, any /bin/sh CGI scripts (of which there still probably aren't all that many) are vulnerable. More worryingly, CGI programs in any language that use the shell (e.g. via system(), popen(), or similar) may well be vulnerable.

Beyond CGI programs, there are a number of other possible vectors for attack. DHCP clients often invoke shell scripts for configuring the network and use environment variables to communicate to those scripts. Mail transfer agents (MTAs) may be affected; Exim and qmail are both vulnerable. Restricted OpenSSH logins (using the ForceCommand directive) can bypass the restrictions using Shellshock. And so on. Red Hat has compiled a list of some of the affected (and unaffected) programs.

Fixes

Michał Zalewski has a blog post from September 25 that describes the original bug along with the first patch. That patch worried Zalewski and others since it simply stopped the parsing once the function definition had been parsed; it would no longer execute code placed after the definition. But, as he pointed out, it still allowed an attacker to send a HTTP header like:

    Cookie: () { echo "Hello world"; }

As Zalewski described, that first patch was fragile because it made two assumptions about the Bash parser—both of which were later shown to be incorrect, as updates sprinkled throughout the post attest. He advocated Florian Weimer's approach, which puts the functions defined in environment variables into a different namespace by adding a prefix and suffix to the names. That should avoid allowing environment variables to be unknowingly set to functions by web servers and other programs.

Weimer's patch was eventually merged with a few tweaks (e.g. changing the suffix from "()" to "%%") by Bash maintainer Chet Ramey. So there is now an easy test to determine if a system is susceptible to the bugs:

    $ foo='() { echo not patched; }' bash -c foo
    bash: foo: command not found

Meanwhile, distributions have been a bit whipsawed: updating Bash, seeing more bug reports, and updating again. At this point, things have mostly settled down on that front. All that remains is for users to update their systems. Since both Debian and Ubuntu use the Debian Almquist shell (Debian ash or dash) for /bin/sh, there is likely far less risk of an exploit, though Bash should still be updated.

More changes may be coming. Christos Zoulas suggested that a flag be added to govern importing functions through environment variables with a default to "off". That is a change he has made for NetBSD's Bash: "It is not wise to expose bash's parser to the internet and then debug it live while being attacked." Others have agreed that it would provide a stronger defense against other, unknown flaws in the parser. Scripts that use the feature (which seem to be few in number) could be changed to turn the feature on.

It should be noted that attacks are ongoing in the wild. For example, the LWN web logs are full of attempts to exploit the vulnerabilities.

While there is plenty to worry about with regard to Shellshock, some in the press have gone a little overboard. It is unlikely, for instance, that vast numbers of embedded Linux devices are vulnerable, medical-related or otherwise. The problem of embedded Linux devices that can't be updated is certainly real (and likely to bite us at some point), but Bash is not typically installed in the embedded world. Most are likely to be using BusyBox, which uses ash, so it is not vulnerable. Another large chunk of Linux devices, Android systems, use mksh (MirBSD Korn shell) rather than Bash.

Since Bash is a heavyweight shell, with a long startup time and high memory requirements, one might wonder why many Linux distributions make it the default shell for shell scripts. Debian and Ubuntu moved away from Bash for shell scripts for just those reasons. Slackware uses ash for its install scripts as well. These bugs may lead to a push to switch to a more minimal shell as the default for scripts in more distributions. This is likely not to be the last Bash vulnerability we see—especially now that security researchers (and attackers) are focused on it.

The "function definition via environment variable" feature seems to be of limited utility. Also, since it isn't all that well-known, it has largely escaped scrutiny until recently. Weimer mentioned that the feature appears to be used by test harnesses. The search he did in Debian's code repositories bears that out. While it may be tempting to disable the feature, as Ian Jackson tried, the namespace fix is backward compatible so existing users can continue to use it. Movement toward reducing or eliminating Bash for non-interactive uses throughout Debian (i.e. eliminating #!/bin/bash), though, seems to be picking up some steam.

Like with OpenSSL and Heartbleed, Shellshock has exposed a project that is both critical to many Linux systems and is not completely healthy. Diego Pettenò described the problem in a blog post. Bash has a single maintainer, with a somewhat cathedral-like development process, which led Pettenò and others to be concerned about the shell long before Shellshock. It would seem prudent for the Linux community to be on the lookout for these kinds of problems now that we have been bitten twice recently.

There are a number of programs that underlie a basic, functioning Linux system, but it is not entirely clear what that number is, nor what projects belong in that set. OpenSSL was an obvious member (though largely ignored until recently); Bash is less so, even though it is now clear that it is used in ways that can easily lead to system compromise. It is probably long past time that some kind of inventory of this "critical infrastructure" is done. Once the projects are identified, some kind of health assessment and/or security audit can be done. We can be sure that those kinds of assessments are being done, at least informally, by attackers and black hats—we just don't get the benefit of their analysis.

$ ls -sh /bin/bash
1000K /bin/bash

$ ls -sh /usr/bin/perl /usr/lib/libperl.so.5.18.2 
 12K /usr/bin/perl  1.6M /usr/lib/libperl.so.5.18.2

$ ls -sh /bin/dash 
120K /bin/dash

 _____________________________________________ 
/ It looks like your script is over 100 lines \
\ long; did you mean to write this in Perl?   /
 --------------------------------------------- 
    \
     \
                                   .::!!!!!!!:.
  .!!!!!:.                        .:!!!!!!!!!!!!
  ~~~~!!!!!!.                 .:!!!!!!!!!UWWW$$$ 
      :$$NWX!!:           .:!!!!!!XUWW$$$$$$$$$P 
      $$$$$##WX!:      .<!!!!UW$$$$"  $$$$$$$$# 
      $$$$$  $$$UX   :!!UW$$$$$$$$$   4$$$$$* 
      ^$$$B  $$$$\     $$$$$$$$$$$$   d$$R" 
        "*$bd$$$$      '*$$$$$$$$$$$o+#" 
             """"          """"""" 

  # hello() { echo "Hello, CLI user"; }
  # export -f hello
  # bash
  ## hello
  Hello, CLI user
  ## env | grep -A1 Hello,
  hello=() {  echo Hello, CLI user
  }