|
|
Subscribe / Log in / New account

Bricking systems using rm

By Jake Edge
February 10, 2016

Deleting all files starting at the root (i.e. rm -rf /) is generally ill-advised; it is almost always a mistake of some sort. But, even if it is done intentionally, a permanently unbootable system—a brick—is not expected to be the result. The rm command can cause all of the Extensible Firmware Interface (EFI) variables to be cleared; due to some poorly implemented firmware in some systems, that can render the device permanently unable to even run the start-up firmware.

The problem was first reported to the Arch Linux forums in a thread that has been moved so that it is only available to registered users. Essentially, the user "9233" reported running:

    rm -rf --no-preserve-root /
The reason given was to "get rid of the pretty bloated Arch installation" on an MSI notebook. As noted in the post, there are plenty of other ways to accomplish that (e.g. reformat the partition), but that was the one chosen. After doing so, the system would no longer even start up the firmware after a power cycle, thus it couldn't boot or even allow the firmware settings to be changed.

Participants in the thread rapidly narrowed it down to the EFI variables that are accessible via the /sys/firmware/efi/efivars/ directory. These variables are stored in non-volatile RAM (NVRAM) on the motherboard of the system. Even though the Universal EFI (UEFI) specification clearly states that wiping all of the boot variables should still result in a bootable system, the UEFI implementation on the laptop did not conform. When those variables were deleted by the rm command, the laptop would no longer even reach the power-on self-test (POST) that is the first step in booting.

Some in the thread (and in discussions at Hacker News and reddit) tried to blame systemd for the problem, but the systemd bug filed was quickly closed by Lennart Poettering. Systemd needs the ability to write to EFI variables to reboot the system in some scenarios, so it mounts the efivarfs as a read-write filesystem on /sys/firmware/efi/efivars as he noted in a later message. He also pointed out that the filesystem can be remounted read-only with a change in /etc/fstab.

None of that completely ended the systemd squawking, but it was clear to most that the real problem lies in the UEFI implementation. That broken firmware may not be fixed soon—if ever—so the conversation turned to how to work around the problem. There were some who advocated mounting efivarfs as read-only until something needed to write to it. Poettering seemed adamant that systemd would not do so, but distributions could certainly remount efivarfs as read-only, though it might require changes to systemd for the reboot scenario.

But the kernel is traditionally the place where broken hardware is "fixed", as others in the thread argued. To that end, Matthew Garrett offered up a quick fix that simply disallowed deleting most UEFI variables. "For a short-term fix, let's just block deletion of most variables while we figure out a better approach." A longer-term solution has also been proposed by Peter Jones (with an Acked-by from Garrett).

Jones's approach is to make most of the UEFI variable "files" in efivarfs immutable, except for a whitelist of those variables that are known to be safe (at least so far). Making the files immutable means they cannot be changed, but root (or processes with the CAP_LINUX_IMMUTABLE capability) can use the chattr -i command to make them mutable again. The idea is to protect these variable files from programs like rm, but to allow programs that do need to change them the ability to do that. That way, accidental (or even intentional) deletions will be prevented unless an administrator makes a conscious effort to circumvent the restrictions.

While the real problem lies in the firmware implementation, it is not uncommon for the kernel developers to try to find ways to avoid giving users unpleasant surprises. Bricked laptops certainly qualify. One can wonder about whether it is sensible to delete all files from the root, but there is no question that it shouldn't lead to unusable hardware—or even warranty service.

It should be noted that any attackers who get root access to a system that has this flaw can exploit it. They can, of course, do lots of other destructive things, but permanently disabling the hardware is not exactly expected. Those who have affected systems should be clamoring for a firmware update from their manufacturers. Unfortunately, the only way to figure out if a system is impacted may be to try to brick it, which is likely to make folks just a little shy about doing so.

Index entries for this article
SecurityFirmware


to post comments

Bricking systems using rm

Posted Feb 11, 2016 2:34 UTC (Thu) by kjp (guest, #39639) [Link] (3 responses)

OT I think using the "--one-file-system" rm option would have helped here. Doesn't seem like anything good can come out of trying to remove /proc and /sys. Or just selecting which subdirs in / you want to remove...

Bricking systems using rm

Posted Feb 11, 2016 23:43 UTC (Thu) by ms_43 (subscriber, #99293) [Link] (2 responses)

Actually it would be nice if --one-file-system were the default: it would prevent the popular "oops, didn't expect that rm -r on that thing deletes the NFS mounts too, recursively" accidents...

Bricking systems using rm

Posted Feb 12, 2016 5:15 UTC (Fri) by josh (subscriber, #17465) [Link] (1 responses)

Or "didn't realize that would nuke my bind-mounted home directory", for that matter. (Chrome OS build chroots used to have this problem, but then they switched to using mount namespaces to fix that.)

Bricking systems using rm

Posted Feb 12, 2016 11:31 UTC (Fri) by fishface60 (subscriber, #88700) [Link]

rm --one-file-system probably also needs to be beefed up to do more than just check whether st_dev differs, since if your chroot is on the same filesystem as your home directory, it would fail to notice that check.
You'd probably need something like what systemd has: https://github.com/systemd/systemd/blob/b26fa1a2fbcfee7d0...

Bricking systems using rm

Posted Feb 11, 2016 7:04 UTC (Thu) by rahvin (guest, #16953) [Link] (2 responses)

I don't disagree with a modification to the kernel to try to prevent such things from happening but this is clearly a bad UEFI implementation that should be fixed by the manufacturer including replacement of damaged boards and issuing of a new bios that doesn't suffer from this problem. With this in mind any changes in the kernel need to be lightweight and focused on not screwing things up for everyone with compliant UEFI implementations.

People should avoid MSI's products until they fix their UEFI implementation IMO. If people don't punish MSI for this we'll keep ending up with these broken UEFI implementations.

Bricking systems using rm

Posted Feb 11, 2016 12:15 UTC (Thu) by fratti (guest, #105722) [Link] (1 responses)

The few consumers aware of this issue are not enough to make a dent in MSI's revenue. Also, if you've ever tried to report a technical issue to a hardware manufacturer, you'll know that they simply have no way to contact a person that knows what you're talking about.

While yes, this is clearly the fault of the UEFI firmware, you're going to have to ask what the most realistic approach is to preventing as many problems caused by bad UEFI firmware as possible. To that end, setting UEFI variables to be immutable is probably the overall best solution, since it doesn't cause issues for existing applications, and can easily be changed in the future.

Bricking systems using rm

Posted Feb 12, 2016 15:47 UTC (Fri) by ksandstr (guest, #60862) [Link]

>The few consumers aware of this issue are not enough to make a dent in MSI's revenue.

More worryingly, there's reports that Thinkpads are also rendered unbootable. In that case however, it seems that correct function can be restored with a combination of selecting a boot device in the BIOS menu (or the per-boot menu), and a BIOS reflash which the administrator can perform. Of course MSI hardware's failure to even POST is worse, but given that even the trusted brand of tens of thousands of neckbeards doesn't implement UEFI to the efivarfs & systemd authors' reading, avoiding a guaranteed general fix (in favour of an interim kernel patch!) seems like it's not even in the same ballpark with the right thing.

Setting permissions

Posted Feb 11, 2016 10:22 UTC (Thu) by epa (subscriber, #39769) [Link] (2 responses)

Could efivarfs grow a mount option to allow writing to files but not unlinking them?

Setting permissions

Posted Feb 11, 2016 11:24 UTC (Thu) by lgeorget (guest, #99972) [Link]

> Could efivarfs grow a mount option to allow writing to files but not unlinking them?

We could finally make use of the 'u' extended attribute ('undeletable'). :-) (although with a different meaning than described in the man page of chattr).

Setting permissions

Posted Feb 11, 2016 19:20 UTC (Thu) by pjones (subscriber, #31722) [Link]

Yes, but it's horrible for compatibility. For instance, to rearrange the boot order requires removing the variable and writing a new one.

Bricking systems using rm

Posted Feb 11, 2016 14:20 UTC (Thu) by sytoka (guest, #38525) [Link] (9 responses)

I thinks that rm could be modified to forbid removing folder ".."

rm -rf .*

Why "rm -rf *" doesn't remove ".file" and "rm -rf .*" remove parent folder ".." ? It's really to dangerous and has nothing to do in a CLI ;-)

Bricking systems using rm

Posted Feb 11, 2016 22:24 UTC (Thu) by sjj (guest, #2020) [Link] (7 responses)

man 7 glob. rm doesn't expand the file globs (*, .*), the shell does.

Bricking systems using rm

Posted Feb 12, 2016 7:24 UTC (Fri) by sytoka (guest, #38525) [Link] (6 responses)

OK. So why glob '*' doesn't return .file (so not '.' folder) and '.*' return '..' ?

Maybe 'rm' could had a protection for '..' when use without a special flag. How many people have destroy trees of files like this?

Bricking systems using rm

Posted Feb 12, 2016 13:13 UTC (Fri) by jwilk (subscriber, #63328) [Link]

> OK. So why glob '*' doesn't return .file (so not '.' folder) and '.*' return '..' ?

The asterisk generally matches any character, with one exception:
If a filename starts with a '.', this character must be matched explicitly.

This makes sense. I certainly want "foo*" to match "foo.txt".

> Maybe 'rm' could had a protection for '..' when use without a special flag.

I once did

chmod 700 * .*

when cwd was /root.
Don't try this at home^W^Won production servers.

Bricking systems using rm

Posted Feb 12, 2016 13:20 UTC (Fri) by k8to (guest, #15413) [Link] (2 responses)

Special treatment of leading-dot files is apparently an error in ancient UNIX that became traditional for "hiding" things in the home dir. Basically someone tried to hide .. for issues similar to the ones you are raising, and sloppily hid .everything. The rest is our shared folly thereafter.

Bricking systems using rm

Posted Feb 12, 2016 15:13 UTC (Fri) by roblucid (guest, #48964) [Link]

Sloppy or efficient?
When the whole address space was 32KiB and 1 MIPs was fast, just checking the first character seems like a good feature.

Bricking systems using rm

Posted Feb 13, 2016 20:15 UTC (Sat) by sjj (guest, #2020) [Link]

It was 40 years ago today
V6 Unix taught itself to glob,
it's been going in and out of style,
still guaranteed to raise a "huh?"
We're Sgt. Unix's Bastard Offspring OS.

Bricking systems using rm

Posted Feb 12, 2016 15:20 UTC (Fri) by hummassa (subscriber, #307) [Link]

PLEASE START USING zsh.

;)

Bricking systems using rm

Posted Feb 12, 2016 23:03 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

There is the "dotglob" option you can set in bash to have the "*" glob character match a leading dot.

Bricking systems using rm

Posted Apr 12, 2019 9:44 UTC (Fri) by metux2 (guest, #131412) [Link]

> I thinks that rm could be modified to forbid removing folder ".."

Doesn't need to be modified, it's implemented that way for aeons. Posix mandates that '.' and '..' are explicitly excluded from recursion. Otherwise rm -Rf .* would *always* wipe the whole disk.

Yes, Lennart refuses to read manpages and specs, so he made that horrible mistake in systemd (even declared the corresponding bug report as invalid and insisted in his wrong assumption was the way rm always worked ...) - but he's certainly isn't an example for serious software engineering.

--mtx

Bricking systems using rm

Posted Feb 11, 2016 14:42 UTC (Thu) by mm7323 (subscriber, #87386) [Link] (2 responses)

Could selinux protect this? Granted it's perhaps not always enabled/enforcing, but where it is, it would seem like a good thing to prevent malicious code trying to brick boxes. It would probably be only need the addition of a simple policy which could be quickly distributed while a more permanent solution is considered.

Bricking systems using rm

Posted Feb 11, 2016 19:13 UTC (Thu) by pjones (subscriber, #31722) [Link]

I did actually consider this - making everything there have a label that programs other than efibootmgr or others run in a role without the delete permission. I don't think it's as good - first, we can't solve it generically and just say "anything that uses a recent libefivar for variable access just works". You have to put every program in the policy.

It's also quite a bit more code to implement FS_IOC_GETXATTR, FS_IOC_SETXATTR, FS_IOC_GETXATTRA, and FS_IOC_SETXATTRA. The immutable flag is just FS_IOC_SETXFLAGS and FS_IOC_GETXFLAGS, and it's just setting or clearing one bit in an int we're already carrying around.

On top of that, you'd either have to have a "default" label for the whole FS, or a series of them for various files in it - which means now we're making assumptions in the kernel about the policy userland will load. We're effectively adding limits on the selinux label namespace, which is kind of ugly from my perspective. It's also wildly non-backwards-compatible - current tools won't already be loaded into that policy. So as an OS erratum, it's a lot more packages than just the kernel. The alternative is that it'll have to be relabeled on every boot, which isn't that great either.

On top of that, there are distros that don't use SELinux.

It's not fundamentally impossible, and it may even be worthwhile to have labels on efivarfs, but it's more code, and there's no good way to have application compatibility with current userland for the normal use cases.

Bricking systems using rm

Posted Feb 12, 2016 4:15 UTC (Fri) by fandingo (guest, #67019) [Link]

Not on any default configuration of SELinux. Interactive sessions run unconfined_t, meaning they bypass all SELinux policy. Only system services or processes that explicitly confine themselves are in confined domains.

I suppose it's worth noting that unconfined_t, like any domain, has policies. A policy could be applied to unconfined_t to protect against this, but I doubt anyone would be interesting in applying any sort of denial policies to unconfined_t.

Bricking systems using rm

Posted Feb 11, 2016 19:16 UTC (Thu) by pjones (subscriber, #31722) [Link] (3 responses)

You do kind of have to wonder why "rm -r" doesn't enable --one-file-system and --preserve-root by default. You'd have to add --no-one-file-system as an option, and make those not subject to -f, but the man page doesn't indicate they should be, there's just an implicit /failure/ rather than prompting.

Anyway, seems like that'd prevent all kinds of accidents like this one.

Bricking systems using rm

Posted Feb 12, 2016 5:17 UTC (Fri) by josh (subscriber, #17465) [Link] (2 responses)

--preserve-root is the default.

--one-file-system isn't. It probably should be, but unlike --preserve-root, --one-file-system might actually break real scripts in the wild.

Bricking systems using rm

Posted Feb 22, 2016 12:54 UTC (Mon) by dvandeun (guest, #24273) [Link] (1 responses)

It would be a good start to provide a short option, e.g. -o, so that people would get into the habit of always typing rm -rfo when before they would have typed rm -rf. Doing the right thing should be convenient.

Bricking systems using rm

Posted Feb 27, 2016 2:22 UTC (Sat) by flussence (guest, #85566) [Link]

I'd suggest -x, since `du` already uses it as a short alias for --one-file-system.

Why is bootloader not turning off write permission?

Posted Feb 12, 2016 2:40 UTC (Fri) by bokr (guest, #58369) [Link] (3 responses)

I thought there was a bit for that, so only a bonafide reboot with reset
will re-enable write to UEFI nvram, which then will be accessible only
to the next booted image -- i.e., a UEFI-validated boot loader (unless
you choose to allow other images) .

If systemd needs to change some variables, it could leave a signed request
someplace persistent-across-boot known to the bootloader, which could then
be permissive or paranoid to taste, e.g. per policy in a signed config file,
or requiring other multi-factor authentication, in case that was missing, etc.

This would seem to restrict the attack surface to the UEFI-validated boot loader only,
which should be more secure than trusting a whole kernel with UEFI nvram writes,
I would think?
(Of course I agree that the UEFI implementation itself should be fixed).

Why is bootloader not turning off write permission?

Posted Feb 12, 2016 14:47 UTC (Fri) by pjones (subscriber, #31722) [Link]

No; there are variables that are "boot services" only, but that's from when you can /see/ them. Runtime variables are available for the duration. The part of the flash that contains the variable storage isn't block locked.

Why is bootloader not turning off write permission?

Posted Apr 12, 2019 9:44 UTC (Fri) by metux2 (guest, #131412) [Link] (1 responses)

> If systemd needs to change some variables,

I really wonder why the init system should ever want to modify bootloader variables.

--mtx

Why is bootloader not turning off write permission?

Posted Apr 12, 2019 9:52 UTC (Fri) by zdzichu (subscriber, #17118) [Link]

There are probably multitude of reasons. One I could think of is when the system is in boot-crash loop (unable to finish boot process), init may want to inform the firmware “hey, next time boot this recovery environment, not generic system”.
Or “reboot to firmware setup”. Pressing Del/F2 to enter firmware setup generally doesn't work on modern hardware, this need to be mediated by operating system.
Or even selecting which kernel to boot, this could be stored in firmware variables also.

Bricking systems using rm

Posted Feb 12, 2016 4:33 UTC (Fri) by pr1268 (guest, #24648) [Link] (5 responses)

Essentially, the user "9233" reported running:
    rm -rf --no-preserve-root /

I don't get it—why would anyone ever need to do that?

If you really want to wipe the filesystem, then boot from an optical disk or USB stick containing $DISTRO and use [c]fdisk(8) or parted(8). (If you're paranoid, like I am, then add in a dd if=/dev/zero of=$BLOCKDEVICE afterwards. And then go see a movie or take a really long lunch break.) ;-)

Bricking systems using rm

Posted Feb 12, 2016 8:38 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (4 responses)

You've never run destructive commands just to see what happens? (For Italians: "per vedere di nascosto l'effetto che fa")

curiosity killed the cat

Posted Feb 12, 2016 22:54 UTC (Fri) by johannbg (guest, #65743) [Link] (2 responses)

Have you ever pulled the pin of a grenade to see if it goes off or shot yourself in the foot just to know how it feels like?

I dont think people are in the habit of running destructive commands just to see what happens but if that's the general case then I have to ask...

Jon can we have a "Bricked ID-10T Award" content section which would be a tongue-in-cheek honour and recognition of individuals who have supposedly contributed to IT evolution by removing their online presence by their own command line actions. ;)

curiosity killed the cat

Posted Feb 13, 2016 16:07 UTC (Sat) by anselm (subscriber, #2796) [Link]

I dont think people are in the habit of running destructive commands just to see what happens

Maybe not “people”, but somebody should – to make sure that whatever check is put in to prevent the “destructive” command from bricking the computer actually works.

This reminds me of George Moore's stories about the (fly-by-wire) F-16, as related in RISKS 3.44:

“One of the first things the Air Force test pilots tried on an early F-16 was to tell the computer to raise the landing gear while standing still on the runway. Guess what happened? Scratch one F-16.”

curiosity killed the cat

Posted Feb 13, 2016 20:55 UTC (Sat) by raven667 (subscriber, #5198) [Link]

Gosh, this reminds me of a story from long ago, the Sysadmin I trained under was returning to inventory the Boss' old desktop, so was about to re-install it, and decided to see what would happen if we ran 'rm -rf /' as root. We watched it run for a while and then noticed that it didn't seem to be doing anything, it hadn't returned and the hard drive light wasn't flashing, so we tried to strace it to see what was happening. strace still worked and we saw a list of what it was deleting fly off the screen, one of the Developers standing next to me exclaimed "Those are my MP3 files!". We immediately ^C the rm and looked to see what happened, it turns out The Boss was a bit of a douchebag and thought he needed to poke around as local root in everyone's NFS mounted home directories so he mounted them all with no_root_squash, rather than use SSH and sudo, or talk to people if he had a question. Thankfully the Developer, whose music library we deleted, had a first name which started with an 'A' so the rm hadn't traversed any further than the letter 'M' in his home directory and we were able to restore everything from tape over the next few hours, so no harm was actually done, but it definitely was a stressful moment that gave us all a few grey hairs.

And now we know what happens when you rm -rf /, chaos.

Running destructive commands just to see what happens

Posted Feb 13, 2016 5:58 UTC (Sat) by pr1268 (guest, #24648) [Link]

You've never run destructive commands just to see what happens?

Well, yes I have. But rm -rf --no-preserve-root / seems so boring. And, I already know what happens there.

Some fun examples I've done (Kids, do not try this at home!):

  • In 2003-era Red Hat 9.0, open a shell terminal, change the preferences to black background and green text (to look like an old green monochrome display), then run % man $0. It's the Matrix!

  • When first learning Java (on a Windows NT 4.0 workstation with JDK version 1.2.2 or so, vintage 2000), while running a simple console program that printed a bunch of numbers in a loop, I tried closing the command-line window. Rather than closing, the JRE barfed out a whole lot of scary-looking verbiage about an "ApplicationUnownedException" or similar. But the program kept running! Clicking the close button repeatedly really made the JRE scream volumes of exception text. (Later version of Windows and/or Java didn't do this; either the JRE simply gave up, or the OS pulled the plug mercilessly.)

  • % echo "453827^691643" | bc -l — I don't remember actually waiting for the result, but it did heat up the room nicely.

  • main(){for(;;)fork();} (or its shell cousin, % :(){ :|:& };: ) — At least I had a per-user process limit set up (with ulimit -u).

  • Deluxe version of above: main(){for(;;)void*p=malloc(1<<24);memset(p,'9',1<<24);fork();} (the memset() ensured that the allocation was not deferred) — ulimit -u couldn't save me here.

Granted, I kinda felt like that mean bully kid who runs sadistic experiments on his toys in Toy Story... ;-)

Bricking systems using rm

Posted Feb 12, 2016 14:33 UTC (Fri) by ksandstr (guest, #60862) [Link] (13 responses)

>Some in the thread (and in discussions at Hacker News and reddit) tried to blame systemd for the problem, (...)

It may not be systemd's fault alone given that efivarfs has been mounted read-write in non-systemd installations as well, but it is in systemd's domain to ameliorate, and systemd requires the characteristic read-writeness which leads to an unexpected UEFI firmware-bricking. This much is evident in the article:

>Systemd needs the ability to write to EFI variables to reboot the system in some scenarios, so it mounts the efivarfs as a read-write filesystem on /sys/firmware/efi/efivars (...)

Clearly it's the case that programs requiring access to this freshly-dangerous filesystem should be wrapped in a pair of remounts, one making it read-write and the other restoring it to read-only. Given the severity of provoking a motherboard-bricking bug[0], and the extremely small number of such programs, this is absolutely the right solution. If systemd were built on (say) shell scripts, adding this functionality would be a matter of approx. three lines in a reboot script -- so not terribly onerous by any means.

The argument against the solution is that first, it's the motherboard manufacturer's fault[1]; second, making the filesystem read-only would prevent programs from writing EFI variables unannounced with a filesystem-like interface; and third, there's an alternative solution involving modification of /etc/fstab. While each portion of this argument would make sense if taken in isolation and at face value[2], in aggregate it is fatally flawed and made substantially worse by the stated development path of systemd overall.

My counterargument is that first, fault-assignment is factually immaterial to any user who's had their laptop bricked. Furthermore, leaving the bug to fester, while avoiding the occasional bricking here and there, requires great feats of education either toward manufacturers of UEFI boards[3] or, more likely, purchasers of hardware. Which is to say that bricking could've been prevented by prescient users, efivarfs authors[4], systemd authors, hardware manufacturers' subcontractors, and nobody else. In this situation, passing the buck is a solution akin to pissing your pants in cold weather: it'll make you warm for a short moment, but then you'll be not just cold but also wet and smelly.

Secondly, the alternative solution breaks those programs which do need access to EFI variables, as stated in the original argument. This is acceptable for installations where the bootloader, kernel, or other EFI-related pieces are never upgraded, or where the administrator is willing to manually remount efivarfs read-write for the upgrade's duration. However, there's a major snag: systemd will eventually do away with /etc/fstab completely in favour of a (very non-Unix) configuration file, or a set of defaults in its absence. These defaults will, because bricking is the manufacturer's fault, make efivarfs read-write from some future systemd upgrade onward, restoring vulnerability in susceptible systems.

Tl;dr: the solution the systemd authors have chosen doesn't work, and will never work; and the alternative solution breaks rebooting (in their own words) and incurs a planned regression by surprise sometime in the future. Both are directly caused by the systemd authors' longstanding inability to first, solve problems, and second, to fix their broken-ass shit. As such, the cancer that is systemd exposes all of its users to a potential damage-amplifying bug that turns the class of recursive-rm bugs[5] from ones that merely blow away all filesystems, to turning PCs into doorstops as well; and will shove this vulnerability down your throat sometime in the future whether you like it or not.

In closing, systemd should be destroyed.

[0] any more severe and we'd be talking about hardware literally exploding
[1] i.e. buck-passing, not atypical of the systemd sphere
[2] the stop-start logic we've come to expect in any discussion involving systemd's authorship
[3] who, like other intermediate BIOS consumers, generally repackage someone else's brain damage; thus being mostly helpless to do anything about future products, let alone those they've released and EOL'd
[4] and whatever crack they smoked that convinced them of filesystem operations' suitability wrt managing arbitrary firmware state, hazards and all! Anyone suggesting that "rm /dev/sda" should cause a security wipe on the first device in the system would be tarred and feathered.
[5] remember that proprietary software package from not long ago where the uninstall script would rm -rf everything when an environment variable evaluated to empty? Or was it an official distribution package? Either one will do in the future.

Bricking systems using rm

Posted Feb 12, 2016 14:42 UTC (Fri) by cortana (subscriber, #24596) [Link] (3 responses)

> systemd will eventually do away with /etc/fstab completely in favour of a (very non-Unix) configuration file, or a set of defaults in its absence.

Please provide evidence for this statement, or cease posting untruths.

Bricking systems using rm

Posted Feb 12, 2016 15:39 UTC (Fri) by ksandstr (guest, #60862) [Link] (2 responses)

Currently, systemd mounts filesystems based on either an existing /etc/fstab, or its own "unit files". In the future, systemd will do away with all of /etc in favour of a "stateless" design. This implies removal of /etc/fstab. QED.

As an aside, I'm quite sure it's also been stated explicitly that the "legacy" /etc/fstab is going to go away for reasons of dependency-tracking and suchlike, i.e. to fit in the "unit" mechanism favoured by systemd. This is an example of the various heresies of systemd.

Bricking systems using rm

Posted Feb 12, 2016 15:50 UTC (Fri) by cortana (subscriber, #24596) [Link] (1 responses)

> Currently, systemd mounts filesystems based on either an existing /etc/fstab, or its own "unit files". In the future, systemd will do away with all of /etc in favour of a "stateless" design. This implies removal of /etc/fstab. QED.

I think you need to take a course in logic.

> As an aside, I'm quite sure it's also been stated explicitly that the "legacy" /etc/fstab is going to go away for reasons of dependency-tracking and suchlike, i.e. to fit in the "unit" mechanism favoured by systemd.

You're 'quite' sure, but not sure enough to actually back your statement up with evidence?

When you signed off your prior post with 'systemd must be destroyed' I was half-convinced it was a work of satire... I'm disappointed to find out that this isn't so!

Bricking systems using rm

Posted Feb 12, 2016 16:13 UTC (Fri) by ksandstr (guest, #60862) [Link]

> I think you need to take a course in logic.

And I think your mother was a hamster -- but that, too, is immaterial. Would you like to argue the point instead of the person?

>I was half-convinced it was a work of satire...

Your half-conviction was exactly correct, but reaching the appropriate conclusion seems to have required a degree of culture that's unavailable to some.

Bricking systems using rm

Posted Feb 12, 2016 16:08 UTC (Fri) by fratti (guest, #105722) [Link] (8 responses)

Fine, I'll bite for the troll.

>It may not be systemd's fault alone given that efivarfs has been mounted read-write in non-systemd installations as well, but it is in systemd's domain to ameliorate

It is not in systemd's domain to do this, as it is a cross-init-system firmware-specific issue that should be addressed in the kernel, which it was.

>and systemd requires the characteristic read-writeness

Only if you actually use the function which it needs it for, which is to reboot into firmware setup. If you do not require this functionality, there's nothing stopping you from mounting it read-only, apart from all the userspace stuff unrelated to systemd that may break.

>Clearly it's the case that programs requiring access to this freshly-dangerous filesystem should be wrapped in a pair of remounts

Nobody knows how many tools - be it open-source or closed-source - require efivars to be writable, you cannot simply go around wrapping them all in remount commands. What *should* be done is that hardware vendors should fix their firmware. Since that's not a realistic approach, let's simply fix it in the kernel. That way, everybody is happy.

>and the extremely small number of such programs, this is absolutely the right solution

You're making assumptions which you've completely pulled out of your rear. Look at the kernel: Even if there's an interface that's only rarely used, the kernel does not break userspace. Because kernel developers know that they don't know.

>If systemd were built on (say) shell scripts, adding this functionality would be a matter of approx. three lines in a reboot script -- so not terribly onerous by any means.

Yes thank you we know that shell scripts are good for a quick hack.

>in aggregate it is fatally flawed and made substantially worse by the stated development path of systemd overall.

So you're not going to elaborate on that, but just leave this statement here expecting us that we share your delusions?

>Furthermore, leaving the bug to fester

But that's not what happened. It got fixed in the kernel. In a way that breaks nothing.

>Which is to say that bricking could've been prevented by prescient users, efivarfs authors[4], systemd authors, hardware manufacturers' subcontractors, and nobody else.

You, again, conveniently ignored the kernel, where this was actually fixed. However, thank you for sharing your experiences of what it is like to pee one's pants, you must be very experienced in that regard. Perhaps instead of getting into arguments about init systems in unrelated articles, you should focus more on your area of expertise.

>However, there's a major snag: systemd will eventually do away with /etc/fstab completely in favour of a (very non-Unix) configuration file, or a set of defaults in its absence.

That is some bullshit FUD, and the fact that you made sure to mention "very non-Unix" in this sentence clearly shows that you're thinking about software more religiously than a healthy human being should.

>These defaults will, because bricking is the manufacturer's fault, make efivarfs read-write from some future systemd upgrade onward, restoring vulnerability in susceptible systems.

You made a baseless claim, then you made a baseless assumption about said claim completely ignoring the role of distributions in this whose job it is to integrate software, and then you again completely ignored that this issue is now fixed in the kernel.

>and the alternative solution breaks rebooting (in their own words)

It breaks rebooting into UEFI firmware because you need write access to efivars for that. It does not break rebooting.

>and incurs a planned regression by surprise sometime in the future

See above, completely baseless claim + baseless assumptions + conveniently ignoring the fact that *this has been fixed*.

>Both are directly caused by the systemd authors' longstanding inability to first, solve problems, and second, to fix their broken-ass shit. As such, the cancer that is systemd exposes all of its users to a potential damage-amplifying bug that turns the class of recursive-rm bugs[5] from ones that merely blow away all filesystems, to turning PCs into doorstops as well; and will shove this vulnerability down your throat sometime in the future whether you like it or not.

Bla bla I'm from slashdot bla bla help I'm being forced to use software bla I'm going to complain without knowing anything about the software in question bla please pay attention to me getting replies on the Internet is all I have in life.

Bricking systems using rm

Posted Feb 12, 2016 17:41 UTC (Fri) by ksandstr (guest, #60862) [Link] (7 responses)

> It is not in systemd's domain to do this, as it is a cross-init-system firmware-specific issue that should be addressed in the kernel, which it was.

systemd is positioned as the only and final init system for GNU/Linux, as evidenced by its intrusion into udev -- an userspace component required by Linux. Disagreement aside, this doesn't change the fact that systemd could fix it, and moreover that it has expanded to take over mounting and unmounting of filesystems and that these things are therefore not just in its domain but also its responsibility.

>Only if you actually use the function which it needs it for, which is to reboot into firmware setup.

Yet mounting efivarfs read-only by default is unacceptable, per its authors' words. These peas don't fit in the same pod: either it's all-significant, or it isn't, or there's a degree of significance in proportion to some other knowable measure.

>Nobody knows how many tools - be it open-source or closed-source - require efivars to be writable, you cannot simply go around wrapping them all in remount commands. (...)

and later,

> (...) the role of distributions in this whose job it is to integrate software, (...)

So what's stopping a distribution from integrating these other efivarfs-writing programs by wrapping them in a pair of remounts (not unlike sudo or schroot do, albeit for different purposes)? Isn't this the distribution's role?

>Look at the kernel: Even if there's an interface that's only rarely used, the kernel does not break userspace.

Read-writeability of efivarfs is hardly equivalent to kernel ABI breakage given that the read-only option exists and is therefore part of the kernel ABI. On the contrary, userspace should account for the possibility that efivarfs is mounted read-only; either mainline versions or distribution-specific ports may use a read-write wrapper to get where they must. Consequently, programs written not to account for a read-only efivarfs were written irrespective of conditions that may occur. Indeed efivarfs could be mounted read-only in the event that e.g. the firmware says that it is in an inconsistent state which only permits safe reading, so this was a realistic case even before the most recent clusterfuck.

Even if efivarfs' always being read-write were a matter of kernel ABI, or akin to it, userspace tradition doesn't dictate what constitutes kernel ABI breakage. Many programs don't handle EINTR from read(2), for example, but they're no less invalid for it.

> Yes thank you we know that shell scripts are good for a quick hack.

An oneliner is a quick hack. Three lines records the prior state and adds (minimal) error response where none had existed before.

Even a quick hack demonstrates the lower limit of complexity to achieve a feature. Let's hope that this explanation shines a light on the meaning of such arguments in the future. On that note, you're welcome.

> But that's not what happened. It got fixed in the kernel. In a way that breaks nothing.

The fix itself is interim in nature, marking known bug-provoking files immutable and causing yet more unknowable failure in the innumerable proprietary or free programs not known to not to exist. Per your own criteria, this is as unacceptable as the wrapper solution.

>You, again, conveniently ignored the kernel, where this was actually fixed.

And what's under a more frequent update schedule, the kernel or systemd? Will the interim fix persist, or will it be deemed a thing that breaks userspace -- since read/writeability of efivarfs files can no longer be determined from whether the filesystem is mounted read-only -- and replaced with a different, subtly incompatible kernel fix?

>(...) the fact that you made sure to mention "very non-Unix" in this sentence clearly shows that you're thinking about software more religiously than a healthy human being should.

Ideas about health aside, there's already a stark difference between the formats of Windows-style .ini files (e.g. systemd's units) and tab-delimited configuration such as /etc/fstab. (in terms of function, only the latter can be parsed with scanf(3).) Additionally, /etc/fstab is present in the vast majority of modern vendor Unixes, reimplementations, and variants, and has remained constant for long enough to vote twice. If there's one thing that can be called Unixy, to separate it from non-Unix things, it is /etc/fstab.

> It breaks rebooting into UEFI firmware because you need write access to efivars for that. It does not break rebooting.

This point was not made clear in the article; thanks for pointing out an alternative viewpoint. I'll seek independent confirmation as necessary.

> Bla bla I'm from slashdot bla bla help I'm being forced to use software bla I'm going to complain without knowing anything about the software in question bla please pay attention to me getting replies on the Internet is all I have in life.

Can you do shadow puppets as well?

Bricking systems using rm

Posted Feb 12, 2016 17:56 UTC (Fri) by mjg59 (subscriber, #23239) [Link]

> this doesn't change the fact that systemd could fix it

Mounting efivarfs read-only would break existing userspace (and not just systemd). Fixing that userspace and pushing it into distributions would take time, during which we'd either leave systems vulnerable or broken. Even then, the remount, modify, remount dance is racy - if userspace crashes during that process you may end up with a read-write efivarfs, and even if it doesn't there's still a small window where something bad may happen.

Alternatively, we can just change the behaviour of the kernel in such a way that it doesn't break the vast majority of existing userspace. It'll take less time to get that into distributions (kernel updates are pushed regularly anyway) and it avoids the race conditions. Why wouldn't you want that?

Bricking systems using rm

Posted Feb 12, 2016 20:03 UTC (Fri) by anselm (subscriber, #2796) [Link] (5 responses)

If there's one thing that can be called Unixy, to separate it from non-Unix things, it is /etc/fstab.

Whatever. You may be able to use scanf(3) (not that scanf(3) is a library function that you should ever use in earnest) to locate the various fields but you're not done there, since there's no telling beforehand what is in those fields (it depends on exactly what options your implementation of mount(8) supports, which can vary wildly between systems), and they need to be parsed separately. There is no way for you to extend the file format if you have requirements that go beyond what the standard format provides. For that matter, /etc/fstab doesn't even seem to support spaces in directory names (which is so 1980s).

Systemd's configuration file format for mount units, on the other hand, is uniform, easily extensible, syntax-compatible to all other systemd configuration files (hence easier to understand, document, learn, and teach) as well as a lot safer and more straightforward to parse. Users of systemd's mount units aren't forced to mix configuration provided by the distribution and their own configuration settings in the way /etc/fstab requires, and aren't forced to carry their changes across updates of this file manually. Most programming languages of interest offer convenient library support for processing INI-style configuration files such as the ones used by systemd.

There isn't really a lot to be said in favour of /etc/fstab other than that people have been using it for a long time. It is part of a conglomerate of configuration files that were thrown together with no concern for a consistent underlying design, and that all differ from each other in their syntax. BTW, it's not as if Linux was the only Unix-like system to (possibly) deprecate /etc/fstab. AIX did away with it altogether 25 years ago and the world didn't end. Oh, and Solaris calls the file /etc/vfstab. In that sense systemd's approach is much more “Unixy” than that of these popular “real” Unixes because you still get to use the traditional syntax under the traditional name if you want to, and that functionality is unlikely to go away altogether even if at some point distributions stop using it by default.

Bricking systems using rm

Posted Feb 12, 2016 22:17 UTC (Fri) by jwilk (subscriber, #63328) [Link] (1 responses)

From the fstab(5) manpage:
> If the name of the mount point contains spaces these can be escaped as ‘\040’.
...
> The proper way to read records from fstab is to use the routines getmntent(3) or libmount.

Bricking systems using rm

Posted Feb 12, 2016 22:41 UTC (Fri) by anselm (subscriber, #2796) [Link]

If the name of the mount point contains spaces these can be escaped as ‘\040’.

Great, so you have to learn this stuff over and over again for all of the various file formats from /etc/fstab to /etc/crontab to /etc/inetd.conf to /etc/inittab and so on. With systemd, there is one configuration file format which is used for everything, and you only have to learn the basic syntax of that once.

The proper way to read records from fstab is to use the routines getmntent(3) or libmount.

The previous poster suggested that being able to use scanf(3) to parse /etc/fstab was desirable (I don't think so). Also, I'm not entirely convinced that it's a great idea to come up with a completely new set of get*ent() library calls for every configuration file in the system. All of that is extra code that needs to be designed, written, debugged, documented, maintained (for decades), programmed against and so on. It's much easier to have to deal with one single file format only.

Bricking systems using rm

Posted Feb 23, 2016 10:30 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

While I agree with almost everything you say, and ksandstr is clearly either engaging in parody or paranoid, one cannot say very many pleasant things about systemd's configuration file format. The only thing worse than it is the shell script mess it replaced. Its language is irregular, randomly extended whenever one of its limitations becomes too severe (hence the irregularity), and generally only slightly more pleasant to deal with than TeX. The fact that configuration is spread across so very many tiny files has caused me to actually paste those files together whenever I have to edit more than a few of them, edit them, then split them apart again, simply so that searching works without constantly opening file after file (most editors are... not very good at cross-file searches).

It is probably among the least pleasant configuration systems I have ever used: poorly designed for the purpose, overly restrictive, arcane (many interactions between various sorts of dependencies are obscure enough that you have to go digging in the source to figure out what they are), and far worse than a proper general-purpose programming language would have been. (Just... not /bin/sh.)

But at least it's not XML.

Bricking systems using rm

Posted Feb 23, 2016 10:35 UTC (Tue) by micka (subscriber, #38720) [Link] (1 responses)

You write

> The only thing worse than it is the shell script mess it replaced

but you yourself found other things worse :

> only slightly more pleasant to deal with than TeX

(OK, only slightly relevant as nobody (?) would use TeX as system config language)
and

> But at least it's not XML

well, I agree, XML is a kind of extreme anyway.

Bricking systems using rm

Posted Feb 27, 2016 2:31 UTC (Sat) by flussence (guest, #85566) [Link]

Any sufficiently parseable data interchange format can be perverted into a configuration syntax.

My favourite is mysql, which had a special-case optimization for 1-row-gazillion-column tables for this very use case (although I may be misremembering; it's been over 10 years since I last read the docs). Runner-up would be Microsoft Excel.

Bricking systems using rm

Posted Feb 12, 2016 15:17 UTC (Fri) by fandingo (guest, #67019) [Link] (6 responses)

I have to beg the question: Why represent EFI variables as a file system? They're not files and have to be mangled to appear so. There's all the problems of doing file IO properly.

I don't understand our obsession with representing interfaces as file systems. We don't need anything more than a getter and setter, so why isn't this a syscall interface?

Bricking systems using rm

Posted Feb 12, 2016 15:57 UTC (Fri) by ksandstr (guest, #60862) [Link]

For a guess: fashions at the time.

Certainly efivarfs doesn't itself manage the way its data is stored, because on some UEFI implementations it runs into the problem where storage allocated for deleted items is only released when the firmware performs a garbage collection cycle at boot[-1]. Writing into it may also provoke system-crashing UEFI bugs when the allocated space becomes full, which cannot be determined ahead-of-time or ensured against, both for the reason mentioned.

In this way efivarfs is an interface shim more than a traditional filesystem, or even a RAM disk. A simple CRUD interface would've likely been more appropriate, with semantics specified according to the minimum of what firmware is known to implement; instead, both users and applications that end up using efivarfs (for whatever purpose) must be aware of these potential quirks, and vendors must test their efivarfs-based features against hardware they're delivered on[0]. This abrogates most of the responsibilities of the operating system as an abstraction layer between userspace programs, hardware, and firmware.

For an ugly word, it's a clusterfuck.

[-1] instead of when the last reference to an open file goes away, as per POSIX semantics
[0] and I can't even begin to imagine the state space involved in such testing, so most likely such tests are run against the freshly-bought configuration and nothing else

Bricking systems using rm

Posted Feb 12, 2016 16:00 UTC (Fri) by cortana (subscriber, #24596) [Link]

<sarcasm>It's the UNIX way, duh!</sarcasm>

Bricking systems using rm

Posted Feb 12, 2016 19:01 UTC (Fri) by flussence (guest, #85566) [Link] (3 responses)

> Why represent EFI variables as a file system?

Want to hear a pedantically horrific answer? They're not *a* filesystem.

They're *two*! /sys/firmware/efi/{efivars,vars}/

The latter is a part of sysfs and uses a far more reasonable “echo an ID to one control file to create a var, another to delete one” interface. Given that it existed already, I'm not sure why an end-user-friendly way to scribble over system nvram was added in the first place.

Bricking systems using rm

Posted Feb 12, 2016 19:07 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (2 responses)

Because sysfs can't represent data that's larger than a page, and EFI variables can be (per spec) arbitrarily large.

Bricking systems using rm

Posted Feb 12, 2016 19:48 UTC (Fri) by gregkh (subscriber, #8) [Link] (1 responses)

sysfs can handle files that are larger than a page, just use the binary file interface, which for stuff like this (i.e. firmware), should be just fine.

Bricking systems using rm

Posted Feb 12, 2016 19:58 UTC (Fri) by mjg59 (subscriber, #23239) [Link]

Oh, sorry, I was misremembering what the problem was. The existing efivar ABI hardcoded an assumption that variables weren't more than 1K large, including the name as part of the size. At the time we had no reason to think that firmware would turn out to be so fragile around variables, so since we were going to have to implement a new ABI anyway, doing it as a filesystem seemed cleaner. Hindsight, etc.


Copyright © 2016, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds