LWN: Comments on "Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)"

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

flussence — Sun, 10 May 2015 16:55:03 +0000

I don't have flash installed.

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

dezgeg — Sun, 10 May 2015 11:18:23 +0000

Perhaps that was an issue with Flash and not Firefox?

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

flussence — Sat, 02 May 2015 01:29:58 +0000

/tmp here is currently 100kB with a youtube video open in Firefox 37.0.2 and switching between the resolution settings, so hopefully they've fixed the root cause bug since then.

It started happening a few months back and I started avoiding video in the browser as a side effect, so there's a gap in my memory, but here's all the useful detail I can recall: The first symptom was long YT videos would stop after a few minutes and refuse to restart, that went away after I rebooted the browser. Second was that bash's tab-completion completely stopped working with a "no space left on device". That led me to run df which showed /tmp 100% full, running du on it showed only a handful of tiny files, then I tried lsof and it pointed to a bunch of deleted filehandles eating up all the space. After some trial and error I figured out those were video data.

And yes, this stuff should've all been written to ~/.cache/mozilla/ (which is there, creation timestamp on it is from 2013), but it wasn't for some reason. I'm not using any other software that interacts with Firefox data, so that can't have been it. Maybe my messing with about:config settings had caused it. Or fixed it. Maybe it'll break again.

And looking at lsof /tmp right now, it *does* have an invisible /tmp/mozilla-temp-1642885559 file open, but it's 0 bytes large. Not sure what that's for.

perl < /tmp/foo.pl

mathstuf — Fri, 01 May 2015 20:19:54 +0000

> It's turtles (er, pre-processors) all the way down - where does it stop being a "program" that "noexec" should be applied to and instead be considered as data?

When prefixed with a single quote.

perl < /tmp/foo.pl

perlwolf — Fri, 01 May 2015 15:32:05 +0000

Even worse, trying to get interpreters to manage "noexec" via tricks moves from a well-defined meaning to a fuzzy one.

For an interpreter, "noexec" prevents the interpreter from being executed - but not its data (the script). (Using a shebang script allows the script to also be controlled, but it is only visible to the interpreter that the script has already been vetted in this way with some effort on the part of the interpreter.) If you try to treat scripts as "executable" (to apply a kernel-internal flag), there is no clear boundary for the limit. Almost any program can be viewed as an interpreter for a language, and the data that it processes as a script in that language. Is a crontab file a program (to be executed by the cron interpreter) or data (to be processed by the crontab program to create another file)? A makefile? A data file that will be transformed into a crontab or a makefile with m4 or sed or a perl script? It's turtles (er, pre-processors) all the way down - where does it stop being a "program" that "noexec" should be applied to and instead be considered as data?

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

dirtyepic — Fri, 01 May 2015 05:08:16 +0000

That sounds like a configuration issue. Firefox doesn't store anything in /tmp by default. It should all be in ~/.mozilla/firefox or ~/.cache/mozilla/firefox. With 87 tabs open right now my /tmp is 169M. Are you using something like profile-sync-daemon?

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

k8to — Fri, 01 May 2015 03:56:27 +0000

I have used this to good effect copying files to do my job over bastion hosts I was supposed to use that were misconfigured. I was able to copy the files to the "locked down" system into /proc/shm and then out again, since the admins had set every other location to be unwriteable for the transiting user.

perl < /tmp/foo.pl

gmatht — Fri, 01 May 2015 00:43:13 +0000

There probably isn't much point getting Perl to honour the noexec flag if you can just pipe the script you want run into Perl's stdin, or convert it all to a "perl -e ..." one liner. This is true of most scripting languages; e.g. bash can also take input from stdin and "-c". Removing the ability of bash to take commands from stdin might be unpopular ;).

The only real advantage I've heard of noexec on /tmp is that it makes it harder to craft evil syscalls, and breaks scripts (not all of which are friendly): http://serverfault.com/a/72357/183440.

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

gutschke — Thu, 30 Apr 2015 23:42:45 +0000

"perl" unfortunately does not appear to honor "noexec", but the dynamic linker seems to do so these days.

This is a result of "ld-linux" calling "mmap(..., PROT_READ|PROT_EXEC, ...)", which is not allowed for files on "noexec" file systems.

I wish, all other command interpreters would be taught similar tricks. But that's likely to break a ton of existing software. And it'll be a multi-year effort anyway, as there are just so many different command interpreters that could be used for this type of exploit.

So, yes, mounting all world-writable directories as not executable is a good start. But nowhere close to comprehensive.

Besides, depending on the exact nature of the malware, it might not be restricted to putting files into a world-writable directory. It could very well decide to put them into a directory that legitimately needs to allow executables.

Furthermore, given that this particular malware appears to be able to create a crontab entry for "root", I suspect it actually runs as "root". If that is true, all bets are off. A sufficiently determined privileged piece of malware, can do all sort so things that make it very hard to discover. Some of these attacks are technically disturbingly easy to implement.

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

marduk — Thu, 30 Apr 2015 23:19:33 +0000

It was legible, thereby obfuscating the fact that it was Perl :P

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

flussence — Thu, 30 Apr 2015 23:18:43 +0000

I seem to remember having some .deb-based distro break after manually putting a noexec tmpfs on /tmp, but that was a few years back so hopefully it's been fixed.

A slightly bigger problem in those modern distros you mention is that Firefox thinks /tmp/ is infinitely large and puts webpage media in there. With a generously-sized tmpfs (half a GB - the default size on my netbook) it takes about 2 visits to Youtube to fill up and the entire desktop begins breaking in weird ways. Only a browser restart clears it out, because Firefox helpfully unlink()s the FDs on open... in the name of security of course.

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

Otus — Thu, 30 Apr 2015 23:12:30 +0000

Wait... Perl requires obfuscation?

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

sjj — Thu, 30 Apr 2015 23:10:55 +0000

On my Ubuntu 15.04 /tmp is mounted rw,nosuid,nodev,relatime, ie not-noexec.

Perl actually used to die back in the day if /tmp was mounted noexec. Dpkg likewise. Don't know if that is still the case.

Surprising that a "sophisticated" attack would fail instead of using "perl /tmp/foo.pl" or "ld-linux-x86-64.so.2 /tmp/foo.bin" though...

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

gutschke — Thu, 30 Apr 2015 19:41:26 +0000

I just checked on an image of a reasonably recent Ubuntu system that I happened to have lying around.

"/tmp" is mounted with "noexec", but "/run/shm" is world-writable and allows executable files. So, a simple scan of "/proc/mounts" for suitable paths is likely to find a path that is vulnerable.

Unboxing Linux/Mumblehard: Muttering spam from your servers (WeLiveSecurity)

ledow — Thu, 30 Apr 2015 19:32:23 +0000

"Mounting the tmp directory with the noexec option prevents the backdoor from starting in the first place"

So... what exactly kind of system is this infecting to spread? Because I don't know of a modern distro that DOESN'T do this.