A fuzzy issue of responsible disclosure [LWN.net]

A fuzzy issue of responsible disclosure

Posted Aug 30, 2022 15:24 UTC (Tue) by tytso (✭ supporter ✭, #9993)
In reply to: A fuzzy issue of responsible disclosure by anarcat
Parent article: A fuzzy issue of responsible disclosure

In general, I don't use USB sticks to transfer files these days. People can send me e-mail, or send me a pointer to a Dropbox or Google Drive link. I do have an Apricorn Aegis Encrypted USB thumb drive[1] (another brand is IronKey) which has a keypad on the device, and for which you have to enter a pin code in order to unlock the key --- and if you enter the pin code wrong three times in a row, it will zero the AES key used to encrypt the contents of the drive.

[1] https://apricorn.com/flash-keys/

This USB thumb drive only gets plugged into trusted machines, and it's where I store things like backups of my SSH and GPG private keys, etc.

In general, plugging in a USB thumb drive for which you don't have complete confidence in the provenance of the image is dangerous. As you say, even if the primary file system types are hardened, there are plenty of file system images which are not regularly getting tested, not just for security bugs, but also for functionality bugs. For example, ext4 and xfs will pass most xfstests tests in the auto group for the default configuration. But for other file systems, there are rather more failures (and again, these are just functional tests; not security/fuzzing testing):

udf/default: 436 tests, 14 failures, 272 skipped, 3292 seconds
  Failures: generic/075 generic/091 generic/095 generic/112 generic/127
    generic/249 generic/263 generic/360 generic/455 generic/482
    generic/563 generic/614 generic/634 generic/643
vfat/default: 438 tests, 23 failures, 282 skipped, 4135 seconds
  Failures: generic/003 generic/130 generic/192 generic/213 generic/221
    generic/258 generic/299 generic/309 generic/313 generic/426
    generic/455 generic/467 generic/477 generic/482 generic/495
    generic/563 generic/569 generic/633 generic/645 generic/676
    generic/688 generic/689
  Flaky: generic/310: 20% (1/5)
f2fs/default: 666 tests, 5 failures, 217 skipped, 3904 seconds
  Failures: generic/050 generic/064 generic/252 generic/506 generic/563
btrfs/default: 935 tests, 9 failures, 232 skipped, 12835 seconds
  Failures: btrfs/012 btrfs/219 btrfs/235 btrfs/277 btrfs/291
  Flaky: btrfs/172: 20% (1/5)   generic/297: 80% (4/5)
    generic/298: 60% (3/5)   shared/298: 20% (1/5)
exfat/default: 665 tests, 22 failures, 546 skipped, 1794 seconds
  Failures: generic/309 generic/394 generic/409 generic/410 generic/411
    generic/430 generic/431 generic/432 generic/433 generic/438
    generic/443 generic/455 generic/465 generic/490 generic/519
    generic/563 generic/565 generic/591 generic/633 generic/639
    generic/676
  Flaky: generic/310: 20% (1/5)
ext2/default: 711 tests, 6 failures, 467 skipped, 3108 seconds
  Failures: generic/347 generic/455 generic/482 generic/614 generic/631
  Flaky: generic/225: 60% (3/5)
reiserfs/default: 658 tests, 27 failures, 408 skipped, 4525 seconds
  Failures: generic/102 generic/232 generic/235 generic/258 generic/321
    generic/355 generic/381 generic/382 generic/383 generic/385
    generic/386 generic/394 generic/418 generic/455 generic/520
    generic/533 generic/535 generic/563 generic/566 generic/594
    generic/603 generic/614 generic/620 generic/634 generic/643
    generic/691
  Flaky: generic/547: 40% (2/5)
Totals: 4933 tests, 2428 skipped, 506 failures, 0 errors, 33349s

So yeah, you might think it's an xfs or ext4 file system, but there are no guarantees that this is the case. In fact, it's much more likely to be a vfat file system. It may be that plenty of people plug random USB sticks into their computer *all* the time. But lots of people also install software programs by using "curl <url> | /bin/sh", as well. Or download a random software package over the network and install it. People do lots of security-inadvisable thing *all* the time.

You're right, viruses and other malware have been spread via floppy disks long before the internet. Fortunately, with the internet, it means we don't need to use USB thumb drives to transfer files any more. And if you have a high security, air-gapped system, then you need to very much pay attent to how you transfer data using removeable storage devices. It can be done securely, but you have to be super careful, and it doesn't start by giving your USB thumb drive to a NSA or KGB or Mossad agent's laptop, and then immediately plugging it into your air-gapped computer, and mounting the sucker. Instead, you might start by disabling the automounter on your air-gapped computer, and then using fsck to examine the file system *before* you mount the image. Or you might use a userspace FUSE program (for example, fuse2fs for ext2/ext3/ext4 file systems) to access the removeable storage device.

to post comments

A fuzzy issue of responsible disclosure

Posted Aug 30, 2022 16:32 UTC (Tue) by anarcat (subscriber, #66354) [Link] (17 responses)

This USB thumb drive only gets plugged into trusted machines, and it's where I store things like backups of my SSH and GPG private keys, etc.

I understand where you're coming from with this: I have a similar device (a yubikey), and I consider machines where I put it to be trusted. Furthermore, it's not supposed to be modifiable by the host, so even if I would plug it into other machines, it shouldn't (in theory again) be possible to compromise it. I still consider it a security breach if I lose custody of it, however, because it could be replaced by a fake or something.

In general, plugging in a USB thumb drive for which you don't have complete confidence in the provenance of the image is dangerous. [...] It may be that plenty of people plug random USB sticks into their computer *all* the time. But lots of people also install software programs by using "curl url | /bin/sh", as well. Or download a random software package over the network and install it. People do lots of security-inadvisable thing *all* the time.

That's kind of a straw man argument, isn't it? It's not because some people advise you to install their software through "curl | sh" that we shouldn't harden the kernel from compromise due to a bug in a filesystem driver. In fact, just now there's been discussions about hardening kernel drivers against crashing, why shouldn't we do similar work with filesystem implementations?

Fortunately, with the internet, it means we don't need to use USB thumb drives to transfer files any more.

I think you are overstating people's capacity of solving this problem. I know that *I* have had this problem numerous times: sometimes it's interoperability between platform (e.g. AirDrop works on Macs, not on linux or windows, "i don't have a dropbox account", "what is syncthing|wormhole|google drive anyways?"), or just straight out lack of bandwidth (e.g. "there's no way I can transfer you this 4GB video through my dropbox over this crap satellite link).

Maybe *you* don't need USB thumb drives to solve this problem, but I keep finding people who constantly have this problem, from film makers to secretaries. It's a real problem.

[NSA attack scenario] Instead, you might start by disabling the automounter on your air-gapped computer, and then using fsck to examine the file system *before* you mount the image. Or you might use a userspace FUSE program (for example, fuse2fs for ext2/ext3/ext4 file systems) to access the removeable storage device.

Okay, now we're talking. :) That's interesting: are you saying that fsck should be able to detect (and fix?) a compromised filesystem... some filesystems don't even have `fsck`, if my memory is correct...

I guess maybe we should start teaching our users to:

not open an untrusted filesystem on their computer (that includes USB thumb drives, but also (microsd/CF) flash cards, external backup drives, etc)
if they really need to, then first run fsck on the device
then specify the filesystem type when mounting, to prevent the kernel from being mistakenly lead towards a bad filesystem driver
if necessary, use fuse (in a VM?) to access the drive

Am I missing anything?

Thanks for the response!

A fuzzy issue of responsible disclosure

Posted Aug 30, 2022 16:56 UTC (Tue) by fenncruz (subscriber, #81417) [Link] (13 responses)

If your security depends on your users always doing everything right and never skipping any part of a 10 part checklist, they you've already lost and your system is compromised.

A fuzzy issue of responsible disclosure

Posted Aug 30, 2022 21:53 UTC (Tue) by tytso (✭ supporter ✭, #9993) [Link] (12 responses)

If your security depends on your users always doing everything right and never skipping any part of a 10 part checklist, they you've already lost and your system is compromised.

Some 10-15 years ago I visited the Banco du Brasil, where they had the good taste to migrate the majority of their desktop. They also had the good sense to do a security upgrade of all of the USB ports on their desktops using epoxy. If you want to run a high security facility, such as in a major financial institution or a government secure facility, the only smart answer is "Just Say No". No complicated checklists are required; you just make it physically impossible for any of your users from using the USB ports.

A fuzzy issue of responsible disclosure

Posted Aug 31, 2022 10:38 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

The keyboards are typically USB these days. Ban keyboards as well?

A fuzzy issue of responsible disclosure

Posted Aug 31, 2022 13:39 UTC (Wed) by anton (subscriber, #25547) [Link] (10 responses)

If you take security seriously, buy hardware that accepts PS/2 keyboards, and use PS/2 keyboards and mice. No USB needed.

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 0:46 UTC (Thu) by pabs (subscriber, #43278) [Link] (4 responses)

Hmm, I wonder if anyone has fuzzed the PS/2 code in Linux.

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 13:24 UTC (Thu) by anton (subscriber, #25547) [Link]

PS/2 does not support mass storage devices, so an attack through a corrupt file system image is not possible through PS/2. Looking beyond the topic at hand, the attacker will have a harder time seducing a naive user to plug something into the PS/2 ports (which are used up by keyboard and mouse).

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 13:36 UTC (Thu) by anarcat (subscriber, #66354) [Link] (2 responses)

Blast from the past... Are there *any* computers still shipping with PS/2 ports nowadays? I haven't seen those in a while on new computers. Certainly not in any laptop for years, at least.

Also, PS/2, is that the thing that would fry your motherboard if you plugged (or unplugged? I forgot) it after boot? Seems like they fixed that at some point, but I guess it's pointless to fuzz a stack where "plugging it in" crashes the *hardware* in the first place...

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 19:47 UTC (Thu) by james (guest, #1325) [Link] (1 responses)

Laptops, no, but if you buy a separate motherboard for a desktop PC, it's likely to have PS/2. Apparently some gamers like it because USB involves polling, which implies the dreaded latency -- and it seems that mass-market motherboards are all aimed at gamers, judging by the prevalence of RGB headers and snazzy colour schemes.

I actually got a bit emotional this week, retiring a Proper Green board (Intel brand) for a black and white ASUS one. But both of them had PS/2 and VGA (which was also introduced with the PS/2).

A fuzzy issue of responsible disclosure

Posted Sep 2, 2022 13:19 UTC (Fri) by geert (subscriber, #98403) [Link]

Yeah, mass-market PCs seem to target only two user bases: light office work, or heavy gaming.
That's how I ended up with a large case with a window, and RGB LEDs lighting up the void around the PCIe slots. The DIMM slots are maxed out, though, which was the key factor dictating motherboard size.

Oh yes, it has a PS/2 keyboard/mouse combo port, but no VGA connector.

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 12:56 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

And a PS/2 mouse?

Epoxying USB ports looks like they're doing "something", but it only deters the really simplest attacks and shows the incompetence of the sysadmins.

Windows (which I assume they're using) supports USB port lockdown that can simply disables ports, and enables them for trusted devices only. It can also work in tandem with secure boot to prevent bootloader attacks.

A fuzzy issue of responsible disclosure

Posted Sep 2, 2022 7:46 UTC (Fri) by daenzer (subscriber, #7050) [Link]

> Windows (which I assume they're using) supports USB port lockdown that can simply disables ports, and enables them for trusted devices only.

FWIW, this is possible with Linux as well, e.g. using https://github.com/USBGuard/usbguard .

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 13:34 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

You could epoxy the keyboard and mouse ports after they're plugged in too.

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 15:31 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Why not turn the whole computer into a solid block of epoxy? Let's stop with half-measures.

A fuzzy issue of responsible disclosure

Posted Sep 1, 2022 16:23 UTC (Thu) by anton (subscriber, #25547) [Link]

What is the attack scenario this measure should help against?

One reason not to do that is that (cheap) keyboards fail relatively often. So if your attack scenario makes it necessary, use durable keyboards.

A fuzzy issue of responsible disclosure

Posted Aug 30, 2022 21:47 UTC (Tue) by tytso (✭ supporter ✭, #9993) [Link] (2 responses)

That's kind of a straw man argument, isn't it? It's not because some people advise you to install their software through "curl | sh" that we shouldn't harden the kernel from compromise due to a bug in a filesystem driver. In fact, just now there's been discussions about hardening kernel drivers against crashing, why shouldn't we do similar work with filesystem implementations?

As the saying goes, "You don't have to run faster than the bear to get away; you just have to run faster than the guy next to you." If there are easier ways to get security-naive users to run malicious code, then there's not a huge amount of effort to install a vault door if the walls are made of paper-mache.

In addition, who is "we"? If you would like to volunteer to do that work, or if your company is willing to hire software engineers to do that work --- and remember, it's not enough to do this for ext4 and xfs --- but for every single file system in the kernel --- that's great! I'm certainly willing to work with someone who is willing to volunteer to do that kind of work for ext4. The problem is that it's a huge amount of work, and there aren't enough volunteers or funded head count to do this work. Given we don't have infinite amounts of headcount, we need to prioritize how we deploy our resources.

That's interesting: are you saying that fsck should be able to detect (and fix?) a compromised filesystem... some filesystems don't even have `fsck`, if my memory is correct...

It's going to depend on the file system, but in general, there are many maliciously compromised file systems which will be detected (and fix) by an fsck program. At the very least, it makes the job harder for the attacker because they now need to figure out how to corrupt the file system such that it can evade the checks by both the kernel and the fsck program. Often the fsck program will do more checks because they aren't as concerned about performance than the kernel implemenation of the file system.

And of course, you can run the fsck or the fuse driver in a VM. For that matter, mounting the file system image in a guest kernel in a VM can also provide a lot of protection.

One other thing you can do if you want to be really paranoid is to copy the file system image from the "USB storage device" to a file on your local media. File system code assumes that the storage device is in the Trusted Computing Base, which means that if you read block N at time T, and without modifying it, you read it again at time T+X you'll get the same data. Or if you write a block at time T, and read it later on, you get the same data back. But if the "USB storage device" is a malicious device that doesn't always behave like a storage device, this can cause Hilarity to Ensue. (Note that if you have a malicious USB device, it might also have a keyboard and mouse interface, and it might be able to inject interesting commands like "sudo ...." into a window when you're not looking.) So you don't trust the USB thumb drive to actually be a valid USB storage device --- well, you've got other problems, but this is another example of why you should never take a random USB thumb drive you find lying on a parking lot and slam it into your desktop on your company's intranet. :-)

A fuzzy issue of responsible disclosure

Posted Aug 30, 2022 23:22 UTC (Tue) by mjg59 (subscriber, #23239) [Link]

> If there are easier ways to get security-naive users to run malicious code, then there's not a huge amount of effort to install a vault door if the walls are made of paper-mache.

And people are doing the work there. Projects like Flatpak are making it easier to distribute third-party software in a way that enforces stronger boundaries between the distributed code and anything security sensitive. Scaling this to cover the curl | sh scenarios is more work, but I'd bet that the number of people who plug in USB keys is larger than the number of people frequently running curl | sh. This is an argument that works for you only as long as you're not the slowest person in front of the bear - if everyone else speeds up, you're suddenly going to be the target.

(USB keys aren't the only thing I'm worried about here - user namespaces mean that unprivileged code can also exercise the filesystem code, which means malicious code that's nominally sandboxed still has a large attack surface for privilege escalation. The fact that mount passes the filesystem type as a string also makes this tedious to fix with seccomp…)

A fuzzy issue of responsible disclosure

Posted Aug 31, 2022 10:48 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> As the saying goes, "You don't have to run faster than the bear to get away; you just have to run faster than the guy next to you." If there are easier ways to get security-naive users to run malicious code, then there's not a huge amount of effort to install a vault door if the walls are made of paper-mache.

Note that the analogy falls apart a bit in computer security. While you do have to run faster than N people when there are N bears, in computer security, the bears can clone themselves such that you now need to run faster than N+1 people (and so on). Additionally, the bears can be upgraded to be faster and some have a zombie trait that makes anyone caught into a bear themselves. Don't forget that Bear 2.0 models can be spawned in "anywhere" for all anyone knows and can even have temporary invisibility.

While I don't think malicious filesystems is quite on the list, I don't think it will take long to make…interesting cases happen if/when it rises near the top of any "viable attacks" list. And yes, the real world does require prioritizing things because there are severe bottlenecks in the accomplishing of such tasks. However, that just tells me that at least *new* code should better consider "what if the disk lies?" kind of situations so that we're at least not exacerbating some future "please update your kernel every day for new fs fixes" state.

A fuzzy issue of responsible disclosure

Posted Sep 5, 2022 18:42 UTC (Mon) by nix (subscriber, #2304) [Link]

> In general, plugging in a USB thumb drive for which you don't have complete confidence in the provenance of the image is dangerous

It's worse than that. My fairly new Fairphone 4 started randomly rebooting recently. I was all worried, as is usual when a £650 piece of hardware starts malfunctioning, and then I discovered that the cause was the SD card plugged into the phone, which had aged out and gone read-only (and possibly messed up its contents in other ways?) at a bad instant and had produced an ext4 fs that reliably caused a panic (well, it panicked my desktop box so I suspect the same sort of thing was happening on the phone). The provenance of this fs was perfectly normal, only ever written on two devices both of which I control: all it took to corrupt this FS image was aging hardware.

(Unfortunately I threw the card away before I remembered the existence of e2image, or I'd have sent you a nice metadata dump. Whoops...)