|
|
Subscribe / Log in / New account

Blacklisting insecure filesystems in openSUSE

By Jonathan Corbet
February 8, 2019
The Linux kernel supports a wide variety of filesystem types, many of which have not seen significant use — or maintenance — in many years. Developers in the openSUSE project have concluded that many of these filesystem types are, at this point, more useful to attackers than to openSUSE users and are proposing to blacklist many of them by default. Such changes can be controversial, but it's probably still fair to say that few people expected the massive discussion that resulted, covering everything from the number of OS/2 users to how openSUSE fits into the distribution marketplace.

On January 30, Martin Wilck started the discussion with a proposal to add a blacklist preventing the automatic loading of a set of kernel modules implementing (mostly) old filesystems. These include filesystems like JFS, Minix, cramfs, AFFS, and F2FS. For most of these, the logic is that the filesystems are essentially unused and the modules implementing them have seen little maintenance in recent decades. But those modules can still be automatically loaded if a user inserts a removable drive containing one of those filesystem types. There are a number of fuzz-testing efforts underway in the kernel community, but it seems relatively unlikely that any of them are targeting, say, FreeVxFS filesystem images. So it is not unreasonable to suspect that there just might be exploitable bugs in those modules. Preventing modules for ancient, unmaintained filesystems from automatically loading may thus protect some users against flash-drive attacks.

If there were to be a fight over a proposal like this, one would ordinarily expect it to be concerned with the specific list of unwelcome modules. But there was relatively little of that. One possible exception is F2FS, the presence of which raised some eyebrows since it is under active development, having received 44 changes in the 5.0 development cycle, for example. Interestingly, it turns out that openSUSE stopped shipping F2FS in September. While the filesystem is being actively developed, it seems that, with rare exceptions, nobody is actively backporting fixes, and the filesystem also lacks a mechanism to prevent an old F2FS implementation from being confused by a filesystem created by a newer version. Rather than deal with these issues, openSUSE decided to just drop the filesystem altogether. As it happens, the blacklist proposal looks likely to allow F2FS to return to the distribution since it can be blacklisted by default.

The core of the debate, though, was over whether openSUSE should be blacklisting filesystem modules at all. Various participants made the claim that broad filesystem support was part of what attracted them to Linux in the first place, and that they would be upset if things stopped working. The plan is to continue to ship the blacklisted kernel modules, so getting support back would be a simple matter of (manually) deleting an entry from the list, but many users, it was argued, are not capable of making such a change. Blacklisting filesystem types, thus, makes the distribution less friendly for a certain class of users.

As others noted, though, that class of users is likely to be quite small. One would not expect to encounter many users who are running a system that dual-boots between Linux and, say, OS/2, but who also lack the skills to edit a configuration file. That is likely to be the case regardless of which Linux distribution is installed, but openSUSE in particular is not generally viewed as a beginner's Linux. On the other hand, blacklisting risky filesystems could potentially provide a layer of protection for millions of users — a rather larger group.

One commonly heard theme was that disabling these filesystems by default would put openSUSE at a competitive disadvantage with regard to other distributions that have not made this change. Users would note that some other distribution is able to mount their obscure filesystem while openSUSE is not (by default), so they would gravitate to competing distributions. This, Liam Proven said, would be bad:

To thrive, Linux distros have to attract users from other Linux distros. If you only get income from your existing customers, then you are going to die, because sometimes your customers will die. They will fail, or go bankrupt, or get bought out, or switch providers, or something.

This is a law of the market.

It is fair to say that many members of the openSUSE community do not see their situation that way and do not see maximizing the number of openSUSE users as being an important goal in its own right. Richard Brown asserted that "openSUSE is a community project, therefore one of our first concerns should always remain ensuring our Project is self-sustaining"; trying to support users whose interests diverge significantly from those of the community runs counter to that goal. Michal Kubecek agreed, saying:

We have limited resources and we have to weigh carefully what we use them for. Focusing on the kind of users you want to attract (unwilling to think about problems, unwilling to learn things, unwilling to invest their time and energy) means getting a lot of users who will need a lot of help even with the basic tasks. That means that skilled users will either spend a lot of time helping them or (more likely) will simply stop helping.

In the end, the argument that making this change would cause openSUSE to lose users did not carry the day. Another "helpful" participant repeatedly told the community to implement a microkernel model for filesystems so that security problems could be contained. As of this writing, that suggestion, too, has failed to find a critical mass of support.

The part of the discussion that did go somewhere had to do with what happens when a filesystem fails to mount because the requisite module has been blacklisted. In response, there has been some work done to improve the documentation and the messages emitted by the system when that happens so that users know what is going on and what to do about it. There were also concerns about existing systems that would fail to boot after an update — an outcome that all were able to agree is not entirely optimal. The solution to this problem appears to be a scheme whereby filesystem modules would be removed from the blacklist if they are loaded in the kernel that is running when the update is installed. So a system that is actively using one of the blacklisted filesystem types would continue to have access to it by default.

As of this writing, a pair of proposals for the actual changes are under consideration. One of them adds the blacklist and the mechanism described above; the other improves the output from a failed mount attempt, noting that the blacklist may be involved. The long discussion has mostly wound down with the proposal looking mostly like it did at the outset. In the end, it seems, the openSUSE community feels that protecting users against attacks on old and unmaintained code is more important than having ancient filesystems work out of the box.


to post comments

Blacklisting insecure filesystems in openSUSE

Posted Feb 8, 2019 19:03 UTC (Fri) by mangix (guest, #126006) [Link] (15 responses)

Both F2FS and Btrfs have very little testing under big endian platforms. I know as I recently tried using both under such a thing only to be presented with kernel panics and force remount to ro.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 0:49 UTC (Sat) by clump (subscriber, #27801) [Link] (7 responses)

If a distribution allows some feature, a user could reasonably expect that the feature is supported. I'd rather have something be unavailable than have a false impression of support or security updates.

Perhaps similar to your example, I'd tried compiling upstream kernels on Debian running on SPARC32 quite a while ago. Unfortunately for me, SPARC32 wasn't really well supported even when I wasn't doing unsupported things. My kernels failed to compile and I learned that nobody expected them to. But that's a different story than the distro itself providing something that nobody expects to work.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 21:54 UTC (Sat) by nix (subscriber, #2304) [Link] (6 responses)

Yeah, it's been over a decade since SPARC32 kernels really worked. Not terribly surprising, given that more or less the only SPARC32 processor in actual use any more is the LEON, and you're not going to be using one of those unless you're building a satellite.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 23:28 UTC (Sat) by bjacob (guest, #58566) [Link] (4 responses)

LEON is popular in more than just satellites. It is part of Intel's (acquired) Movidius hardware platform for embedded AI applications.
https://movidius.github.io/ncsdk/ncs.html

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 0:26 UTC (Mon) by nix (subscriber, #2304) [Link] (3 responses)

Interesting! (And it says something about the march of technology that this medium-big-iron architecture first found its way into satellites and now into *part* of a USB-key-sized neural-net contraption. I wonder why they chose LEON... low power requirements, probably. I can't believe they cared about radiation hardening :) )

Blacklisting insecure filesystems in openSUSE

Posted Feb 12, 2019 6:38 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

https://space.stackexchange.com/questions/729/why-did-the... has pointers.

The choice was made in 1991. What other open and mature ISA was there at the time?

Blacklisting insecure filesystems in openSUSE

Posted Feb 13, 2019 22:31 UTC (Wed) by k8to (guest, #15413) [Link]

Hmm, if you mean an arch with a neutral consortium etc guiding it, then I'm not aware of many arches that have met that bar in any point in time. Are there any besides Sparc and I guess hyper-v?

If you mean something with a lot of manufacturers where you could get parts in whatever form you wanted, I think MIPS had hit that stride by 1991. Don't think anything else was close.

Blacklisting insecure filesystems in openSUSE

Posted Feb 14, 2019 16:41 UTC (Thu) by nix (subscriber, #2304) [Link]

I was wondering why the NCSDK chose LEON, not why the ESA chose to make LEON a SPARC implementation. :)

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 18:44 UTC (Mon) by clump (subscriber, #27801) [Link]

I am really impressed developers got Linux working on things like SPARC32 hardware. I've run Linux distros on old SPARC machines, one PA-RISC machine, an SGI Indigo, and others. To the best of my knowledge, most of the porting and maintenance was volunteer work.

Nowadays Linux runs on just about everything, but there are strong economic reasons for this. I don't really have the desire or space for older hardware. A Raspberry Pi is remarkably inexpensive and makes a very capable, low-power Linux device.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 3:57 UTC (Sat) by jeffm (subscriber, #29341) [Link] (6 responses)

We ship btrfs as the default root file system for SLE12 and SLE15 on all architectures including s390x, which is big endian. The hiccups we have encountered aren't endian related, but rather in the way s390x handles page dirtying. I haven't seen an s390x-specific btrfs bug in some time and it's not for lack of deployments.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 6:15 UTC (Sat) by mangix (guest, #126006) [Link] (5 responses)

I have. From kernel 3.18 to 4.19.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 14:41 UTC (Sat) by mageta (subscriber, #89696) [Link] (4 responses)

Did you report them? I've been using btrfs with SLES12 and 15 on s390x for 3+ years now (as it is the default filesystem on SLES), and have not seen a single panic/warning, or at least I can't remember one. I didn't use any advanced features like raid and such, but at least snapshots worked also fine.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 19:46 UTC (Sat) by mangix (guest, #126006) [Link]

I've previously reported the f2fs error. No comment. The code actually has several endian related bugs from a quick glance at fsck output as well as the code.

Reported the btrfs issue just now. Given that my issue has probably been there for a long time, I'm not hopeful.

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 13:58 UTC (Mon) by khim (subscriber, #9252) [Link] (2 responses)

RAID56 is not just "advanced". It's red on Status page. And always was red. The fact that people are trying to use it (and are then burned) so often makes me wonder: why attempt to use it does not cause warning in CAPITAL LETTERS from kernel?

Blacklisting insecure filesystems in openSUSE

Posted Feb 15, 2019 16:46 UTC (Fri) by jhhaller (guest, #56103) [Link] (1 responses)

The btrfs status page gives a high level view of RAID56, the details are more subtle. There are a combination of issues related to the write hole potentially corrupting metadata, and no unclean dismount flags suggesting a log replay is needed. But if one uses RAID10 for metadata and RAID 5 for data, the problem should be no worse than other filesystems when there is an unclean dismount. RAID 6 doesn't really support multi-drive failure when the metadata is RAID 10, as one can only lose one drive before there is a possibility of data loss in the RAID 10 metadata. The most recent status is a thread in the email archives here: https://lore.kernel.org/linux-btrfs/5d7f63b2-d340-7c3a-67... - follow the thread.

Blacklisting insecure filesystems in openSUSE

Posted Feb 15, 2019 19:01 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

I have just lost data from a btrfs RAID5 NAS. We had power outages going and the external enclosure was b0rked by it, causing drives to appear and disappear. BTRFS has corrupted the filesystem beyond repair.

I do have a secondary cloud backup so it's all good.

However, there's no way I'm going to trust btrfs with ANYTHING important.

Blacklisting insecure filesystems in openSUSE

Posted Feb 8, 2019 20:00 UTC (Fri) by flewellyn (subscriber, #5047) [Link] (1 responses)

> "... it's probably still fair to say that few people expected the massive discussion that resulted..."

Really? It's been my experience that, no matter how ill-used a piece of code or a feature is, no matter how ancient or obsolete, no matter how bug-ridden and insecure, if a project considers deprecating it, much less removing it, there is going to be SOMEONE out there who insists that this will irrevocably disrupt their workflow. Probably multiple someones. And every single one of those someones is going to descend on the mailing list and make noise.

XKCD even has a comic about this: https://xkcd.com/1172/

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 20:44 UTC (Mon) by marcH (subscriber, #57642) [Link]

tl;dr : if your expectations for this mailing list were higher than for social media, lower them.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 4:06 UTC (Sat) by flussence (guest, #85566) [Link] (5 responses)

>To thrive, Linux distros have to attract users from other Linux distros.

I can't tell if this diatribe over a hypothetical mass migration between Linux On The Desktop™ distros by angry weirdos demanding slick graphical automount of flash drives containing filesystems from before flash drives existed is supposed to be satirical or not. On top of that, it's bizarre to see someone so openly talk about FOSS ecosystem projects infighting and cannibalising each other like it's a good thing when security is at stake.

Microsoft disabled autorun.inf ACE by default over a decade ago, IIRC. I doubt they lost even one millionth of a percent of market share to SUSE over that decision.

Blacklisting insecure filesystems in openSUSE

Posted Feb 10, 2019 5:31 UTC (Sun) by drag (guest, #31333) [Link] (4 responses)

There are a lot of people who have a zero-level grasp of economics.

One of the first really basic things to understand is that wealth is not a zero sum game. Just because one person gains wealth doesn't mean that somebody else must be made poorer. That is not now the world works. That isn't how wealth works.

It's also not how Open Source software works.

There are over 7 billion of people on the planet. There are hundreds of thousands of new people born every day. There are hundreds of millions of people approaching the age were they can start using computers in a meaningful way on their own. There are hundreds of millions of people getting to the point were they are technically aware and can have the ability to choose their own operating system.

For OpenSUSE to become more popular it does not require that it steals users from Debian or Ubuntu. It's not even desirable or useful to think of things in that way either. Ubuntu existing and being popular has made OpenSUSE a better operating system. Redhat succeeding and becoming popular in the enterprise has made OpenSUSE a better operating system.

You are really all on the same team here. Competition is good and diversity is good. But it competition doesn't mean that you are in a combat or war against a adversary. Diversity doesn't mean you need to engage in group politics and fear those that are different. You can gain from your competition even if they gain more then you do. Nobody has to lose anything.

For OpenSUSE to 'Win' all it needs to do is:

1. Create compelling reasons to use OpenSUSE.

2. Make sure that people are aware of these reasons.

3. Live up to the expectations they create in the minds of new users.

I expect, as far as the file system stuff goes, that new users expect that their brand new Linux operating system is secure enough that they can plug in that random USB key the found in the parking lot and not get immediately owned by a attacker.

Probably what is needed is to provide a desktop notification and dmesg output that states that the file system module has been blacklisted because it is known to be insecure. Then provide a way for users to learn a work around if they really really want to do it. You don't have to make it easy for people to screw themselves over, but you can't afford to make them frustrated and confused about what is going on.

Blacklisting insecure filesystems in openSUSE

Posted Feb 15, 2019 1:21 UTC (Fri) by kmweber (guest, #114635) [Link] (3 responses)

> Just because one person gains wealth doesn't mean that somebody else must be made poorer.

Well, that's not *necessarily* the case, but in practice it very often is.

Blacklisting insecure filesystems in openSUSE

Posted Feb 15, 2019 2:07 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> Well, that's not *necessarily* the case, but in practice it very often is.

What is the source of said belief?

Blacklisting insecure filesystems in openSUSE

Posted Apr 12, 2019 8:56 UTC (Fri) by Rudd-O (guest, #61155) [Link]

Communism, of course.

Blacklisting insecure filesystems in openSUSE

Posted Feb 22, 2019 19:15 UTC (Fri) by Wol (subscriber, #4433) [Link]

And in practice it very often isn't!

It's basically down to supply and demand. Why did we have the Renaissance? Because average *median* wealth rose dramatically as a result of the Black Death. That was what brought down the feudal system in a large chunk of Europe. We had the same effect after the First and Second World Wars.

Sadly now we have the opposite effect, where we have too many people chasing too little work, and even while the average *mean* wealth may be rising, the *median* wealth is falling. And politicians love to talk about the mean, while forgetting that in a skewed distribution like incomes, it has very little meaning.

Cheers,
Wol

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 6:35 UTC (Sat) by josh (subscriber, #17465) [Link] (9 responses)

I'd still love to see the long tail of filesystem drivers converted to FUSE filesystems, which would allow mounting them by default with a security-isolated driver. (The code could still come from the existing kernel driver.)

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 7:58 UTC (Sat) by dezgeg (subscriber, #92243) [Link] (3 responses)

lklfuse from https://github.com/lkl/linux allows this today.

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 9:24 UTC (Sat) by pabs (subscriber, #43278) [Link]

Is anyone working on merging LKL into Linux mainline these days?

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 13:30 UTC (Sat) by josh (subscriber, #17465) [Link] (1 responses)

Are any distributions using that by default, though?

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 22:49 UTC (Sat) by dezgeg (subscriber, #92243) [Link]

Well, I guess someone needs to be the first one. Sounds like a perfect time for someone from the openSUSE community to try ;)

Blacklisting insecure filesystems in openSUSE

Posted Feb 9, 2019 12:20 UTC (Sat) by warrax (subscriber, #103205) [Link] (2 responses)

Something something microkernels something :)

Blacklisting insecure filesystems in openSUSE

Posted Feb 17, 2019 14:03 UTC (Sun) by intgr (subscriber, #39733) [Link] (1 responses)

What?! Microkernels are from the stone age, clearly today the solution is to Rewrite it in Rust.

Blacklisting insecure filesystems in openSUSE

Posted Feb 18, 2019 0:21 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

And it actually works just fine most of the time, if Rust's current limitations are acceptable.

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 9:21 UTC (Mon) by viiru (subscriber, #53129) [Link] (1 responses)

> I'd still love to see the long tail of filesystem drivers converted to FUSE filesystems, which would allow mounting them by default with
> a security-isolated driver. (The code could still come from the existing kernel driver.)

Indeed, this is what I think the "helpful" suggestion "Another "helpful" participant repeatedly told the community to implement a microkernel model for filesystems so that security problems could be contained." should be read as. Filesystem implementations that are meant for enabling access to legacy media should not be in the kernel, they should be implemented using FUSE.

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 9:32 UTC (Mon) by mjthayer (guest, #39183) [Link]

> Filesystem implementations that are meant for enabling access to legacy media should not be in the kernel, they should be implemented using FUSE.

I could see that happening for more things in the kernel than just filesystems (someone mentioned Decnet and amateur radio protocols below). At the point where people decide its less work than keeping them at kernel-level security. And these days people are building up lots of experience with lots of forms of virtualisation. In the long term I could well see the Linux kernel becoming much more micro-kernel-like than it is today (though not in an academically clean way). Maybe.

Blacklisting insecure network protocols?

Posted Feb 11, 2019 4:47 UTC (Mon) by shemminger (subscriber, #5739) [Link] (1 responses)

What about Decnet, and all the amateur radio protocols?
Syskiller keeps finding holes in every one of the old protocols it touches.

Blacklisting insecure network protocols?

Posted Feb 11, 2019 12:34 UTC (Mon) by epa (subscriber, #39769) [Link]

Those have to be explicitly enabled. They don't get run when you insert a USB stick. Which is all the SuSE developers are proposing here -- an explicit step to enable these ancient filesystems.

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 9:35 UTC (Mon) by mjthayer (guest, #39183) [Link]

Linux distributions always have a few questions for the user at install time. How about adding a simple security-level question? "Do you want security or flexibility? If the second, do not use this system for anything which requires logging in to anything."

Blacklisting insecure filesystems in openSUSE

Posted Feb 11, 2019 21:22 UTC (Mon) by xorbe (guest, #3165) [Link]

Apparently a message was posted for 48 hours prior to F2FS being blacklisted. Turns out some users were unhappy about that. Who knew that 48 hours on a devel mailing list might have not reached a wide enough user base audience for comment? The mind boggles!

Blacklisting insecure filesystems in openSUSE

Posted Feb 12, 2019 22:32 UTC (Tue) by johannbg (guest, #65743) [Link] (2 responses)

Downstreams should not have to make the types of discussions and decisions.

Is this not failure on behalf of the kernel community?

I mean is it not it's responsability to remove insecure code and or drop it if they cant or if it's unmaintained and just shows that the upstream kernel community deprecation policies, aren't effective enough or non-existant?

How long does a module have to be unmaintained before it eventually gets removed from the kernel?

Which team is responable for the filesystem layer in the kernel community or the modules in question?

So fourth and so on...

Blacklisting insecure filesystems in openSUSE

Posted Feb 13, 2019 16:18 UTC (Wed) by roblucid (guest, #48964) [Link] (1 responses)

Broken code gets shipped so someone else fixes it :)

Blacklisting insecure filesystems in openSUSE

Posted Feb 13, 2019 19:36 UTC (Wed) by johannbg (guest, #65743) [Link]

Looks like it's getting disabled instead of fixed thou but if it builds ship it! ;)


Copyright © 2019, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds