SELinux kills multiboot [LWN.net]

From:		John Reiser <jreiser-AT-BitWagon.com>
To:		lwn-AT-lwn.net
Subject:		SELinux kills multiboot
Date:		Thu, 22 Dec 2005 10:11:44 -0800

Hi,

Security Enhanced Linux (SELinux), a large project that is working
thick and fast with Fedora Core, is creating compatibility problems
for "hobbiest" sysadmins, or anyone who multiboots and cross-mounts
multiple filesystems on the same box.

The latest manifestation can be seen in this thread on fedora-test-list:
http://www.redhat.com/archives/fedora-test-list/2005-Dece...
FC5test1 with SELinux is hazardous to any older ext3 root filesystem:
they become unbootable.

These compatibility problems seem to be even worse than the ones
that resulted from the xattr-on-symlink bugfix to ext3 more than
a year ago, when Fedora Core 2 zapped RedHat 9 and earlier ext3:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=152827

This is worthy of a short news item, if nothing else to spread
the word that you can zap yourself.

-- 
John Reiser, voice/fax +1 503 297 3754, jreiser@BitWagon.com

to post comments

SELinux kills multiboot

Posted Dec 24, 2005 8:56 UTC (Sat) by amacater (subscriber, #790) [Link] (7 responses)

This has, at least, shown me why RHEL3 and RHEL4 may have exhibited odd
behaviour [RHEL4 refusing to mount RHEL3 ext3]: from following the thread
I found that there was an e2fsck bug in reading older ext3. This latest
is, unfortunately, another reason why you should NOT use Fedora, in any
version, for business critical work (and should, perhaps, be circumspect
about updating RH Enterprise versions). Even with a good on-site technical
support, this sort of thing will bite you badly. Fedora moves too fast
and is too rough around the edges - deliberately, because it is a test bed
Unfortunately, Red Hat quality control and update policy are, perhaps, not
what they were : RH employs some of the very best Linux developers,though,
and contributes a great deal to the community. It may just be that far
fewer people run Enterprise Linux (and its derivatives like CentOS) than
ever ran earlier versions such that bugs aren't found until too late. It
may also be that bugs in Fedora are to be expected and largely run unfixed
because of time pressures and because Fedora Legacy can't keep up.

SELinux kills multiboot

Posted Dec 25, 2005 2:29 UTC (Sun) by dang (guest, #310) [Link] (6 responses)

As a desktop FC user and a sysadmin supporting EL3 and EL4, I just want to say that I have 0 clue what you are trolling about. Please stop.

SELinux kills multiboot

Posted Dec 26, 2005 15:18 UTC (Mon) by philips (guest, #937) [Link] (4 responses)

Do you use SELinux? I was trying FC2 (or 3?) specifically to see SELinux in action - with no
success at all. I mean I did make FC2 work with SELinux disabled - but with SELinux after some
manual operations system wasn't even booting. IIRC, next release FC disabled SELinux - it wasn't
me alone with the problems.

As much as I have tried FC - never touched RHEL - it would never be something good like RedHat
Linux 7.x. There are people who are still using it - and IIRC there are people who even port
security updates for it.

From mail list discussions I have understood that RH people do not use FC by themselves - they
only develop for it. FC is not something RH made for people - it is something RedHat made to
keep itself in open source development loop. So one cannot expect any production-like quality
from it.

P.S. But anyway I got some impression of FC & SELinux - but cannot say it is positive one. I do
not like yum - it is too slow compared to apt-get (ironically, before I tried I thought that apt-get
was slow). I didn't liked the way SELinux integrates into system: it depends too much on how OS
(FC in that case) sets security lines. And that bit was nowhere documented. I generally find
RedHat tools to be incomprehensive and poorly documented. And that SELinux situation was
100% stupid: some standard administration procedures were rendering system unbootable/
unusable.
To stop complaining - coward's choice - I stopped using FC. Might be that I'm using Debian &
SUSE for too long - it is easy to get used to good stuff.

SELinux kills multiboot

Posted Dec 26, 2005 17:09 UTC (Mon) by vonbrand (guest, #4458) [Link] (2 responses)

Let's go over these one by one...

Yes, my machines here run SELinux (targeted, in rawhide). Yes, SELinux is still very much in flux. It works passably now, if you are a bit careful. You can't just expect that a bizarre setup mounting the same filesystems with differing options and software versions (and different expectations as to security labelings) to "just work". SELinux is rather new, no wonder older kernels don't know what to do with a filesystem set up for it.

Yes, RH 7.3 was golden. But current Fedora (or RHEL) has a few years of furious development added on top of that... that means new functionality (and new eye candy ;-). Suit yourself.

What mailing lists gave you that impression? There are Red Hat developers (I'd guess they run either RHEL or Fedora, probably both), and Fedora developers (they do run Fedora). Fedora is not a dumping ground for half-backed RPMs to be fixed by the community (that would not go very far, now would it), it is a quite solid distribution. Sure, for some uses (mostly "server" type workloads) the fast upgrade cycle is a disadvantage, but for others (destop or hobbist usage, where "the latest version of everything" is being demanded) it is right on the money.

Can't comment on apt-get, but current yum is quite fast in its job (modulo network delays, which I do feel here). In any case, the time taken by yum itself is a fraction of the download and install time, so I don't know what you are complaining about.

Can't comment in detail about Debian or SUSE (last time I used either is too long ago to be relevant today), but precisely the beauty of OSS is that you can choose among a large variety of alternatives. No "One size has to fit all" here.

SELinux kills multiboot

Posted Dec 28, 2005 13:24 UTC (Wed) by rqosa (subscriber, #24136) [Link] (1 responses)

The thing that annoyed me most about yum is that, regardless of what operation it's doing, it always (IIRC) accesses the server to update the package list (the equivalent of "apt-get update"). With apt, several operations do not require accessing the server, such as "apt-get remove" or "apt-cache search".

SELinux kills multiboot

Posted Dec 28, 2005 14:36 UTC (Wed) by vonbrand (guest, #4458) [Link]

Most operations do need an up-to-date package list. In any case, you can force using just the chache with -C. Besides, the newer yum in rawhide has a timeout during which it just doesn't check for updates. No, it's far from ideal, but...

SELinux kills multiboot

Posted Dec 27, 2005 2:52 UTC (Tue) by dang (guest, #310) [Link]

One of thing things that Fedora has explicitly tried to do is make SELinux more usable with each release. Your out of the box experience with FC4 will be vastly better than FC3. Perfect? Eh, who knows. But evaluating the absolute worth of SELinux or any distro that supports it based on initial deployments makes no sense to me. What interests me is the fact that I'm currently comfortable deploying SELinux in datacenters where it matters. I wasn't in the past. Part of the difference is that I learned more and part of the difference is that distributions and the broader community have done a lot of work. This is a promising direction.

Have I actually used this stuff? Yes. I've also used GRSecurity, fwiw. None of it is perfect, but if you have a clear idea of what you need it to do , it can work. Trouble welding it onto an ancient distro? That is part of the equation when you choose to lag behind. And I'm not saying that you blew the equation; I completely understand the pressures that keep people stuck on ancient, "tried and true" distros. But if hyperthreading doesn't work, or IO ain't what you want it to be, or new features don't weld on neatly, well, you just can't carp.

I still don't get when a discusion of a bug in an RC1 ( however nasty it might be ) generates so much FUD. What, M$ is off on holidays so linux users have stick pins in their own eyes? One would hope that the discussion would center on root cause and path to remediation, or perhaps an ack on a useful heads up.

Bluh.

SELinux kills multiboot

Posted Dec 28, 2005 18:39 UTC (Wed) by amacater (subscriber, #790) [Link]

My first comment was not intended as a troll. As someone constrained to use
RHEL at work, I ran into a problem which my paid Red Hat support person on
site couldn't sort. A clean install of EL3 on the first of two disk
partitions was rendered unbootable by EL4 on the second. I'm well used to
Linux and don't consider RH Enterprise level distributions really well up
to scratch. Unfortunately, my unease is compounded by Fedora - yet I see
lots of people suggest it as a distribution on which to base mission
critical work. The fact that it's taken ten months for me to find a
potential reason [ext3 incompatibilities?] is annoying but not more than
that. If I had any message, it would be to distrust _all_ betas and,
potentially, every RH EL release - I'm also not happy that the EL is
not maintained as absolutely stable but that alpha quality components
can be released in a stable release [a thread here about GCC 4.0
probably in Feb or so when RH EL4 released refers.]

John Reiser says...

Posted Dec 24, 2005 10:50 UTC (Sat) by dpoon (guest, #27648) [Link] (1 responses)

So, Mr. Reiser tells us that there's a problem with ext3. This almost sounds like a prank. =)

John Reiser says...

Posted Dec 24, 2005 11:06 UTC (Sat) by rapunza (guest, #34728) [Link]

I think you're mistaking John Reiser with Hans Reiser from ReiserFS ?

that would be a prank :)

SELinux kills multiboot

Posted Dec 24, 2005 12:42 UTC (Sat) by dwmw2 (subscriber, #2063) [Link] (14 responses)

User installs prerelease test system; finds bug.

Film at 11.

Hell, I'm happy if I install a test release and it refrains from just eating all my filesystems completely. Using a new feature which is unfortunately not compatible with some older systems is a relatively minor problem in comparison with what the huge "THIS IS A TEST RELEASE; ARE YOU SURE YOU WANT TO DO THIS?" clickthrough box in the installer suggests might happen.

Nevertheless, I assume a fix or a workaround will be forthcoming before the real release of FC5 next year. On-medium compatibility is quite an important feature.

SELinux kills multiboot

Posted Dec 24, 2005 21:18 UTC (Sat) by jreiser (subscriber, #11027) [Link] (12 responses)

I assume a fix or a workaround will be forthcoming before the real release of FC5 next year...

Past experience suggests that you will be disappointed, except possibly for "workarounds" such as a warning "Don't do that!" or "backup all affected partitions and re-install." Fedora Legacy RedHat 9 has not fixed the xattr-on-symlinks compatibility issue, despite THREE official releases after the bugzilla #152827.

SELinux is important, ground-breaking work. But they've tended to concentrate on Functionality and Performance only, ignoring Usability, Reliability, and Supportability.

SELinux: Neither Important Nor Ground-breaking

Posted Dec 24, 2005 22:55 UTC (Sat) by AnswerGuy (guest, #1256) [Link] (6 responses)

Personally I think SELinux is horrendously complex ... to the point where no normal sysadmin should be expected to read and understand much less create a useable policy file.

Given this extreme complexity we can't expect any normal sysadmin to be able to audit an SELinux system configuration ... we'd have to rely on the distributin and application developers and packagers and this will inevitably lead to corner cases for any systems which don't fit neatly into a fully pre-packaged configuration. (For instance: add a package like MediaWiki to a web server node ... or worse different components to a web server front end, and your DBMS backend; and then try to make the whole thing work under SELinux while isolating it from the rest of the system).

I still think that SELinux is a giant step backwards in useability and I'm unconvinced that this results in any net benefit to security. I vastly prefer the far more lightweight and comprehensible systrace approach.

JimD

SELinux: Neither Important Nor Ground-breaking

Posted Dec 25, 2005 2:41 UTC (Sun) by dang (guest, #310) [Link]

I share the sentiment, but not as strongly as I once did. Distributions and the community are doing more to help you get policies that work for standard needs; and tools like audit2allow help one to move from "crap this isn't working" to some reasonable level of joy.

SELinux is probably not the tool that we'd most like to have, but it is becoming increasingly handy over time. Bill McCarty's book has been helpful to me ( but as with most books, borrow a copy or Safari it to see if presents the right information in a style that works for you ).

SELinux: Neither Important Nor Ground-breaking

Posted Dec 25, 2005 9:15 UTC (Sun) by danieldk (guest, #27876) [Link]

AOL, but as far as I know systrace is not really maintained for anything non-BSD. For instance, I haven't found any up to date kernel patches (although it is not too hard to rewrite the old patches for the latest 2.6 kernels, I tried it a few months ago).

SELinux: Neither Important Nor Ground-breaking

Posted Dec 25, 2005 19:26 UTC (Sun) by danielthaler (guest, #24764) [Link] (3 responses)

Personally I think SELinux is horrendously complex ... to the point where no normal sysadmin should be expected to read and understand much less create a useable policy file.

I disagree. I wrote a policy for mdadm for my (gentoo, not fc) system before the upstream one existed; it was no more complex than programming in any other language. Because as far as I'm concerned writing policies is essentially just another kind of programming.

Many simple programs only need a very narrow set of permissions and you can grant those easily (and without weakening your overall security) even if you have only a very basic understanding of policywriting

It doesn't seem excessive to assume that the admin of a SELinux system would learn to do that. After all, every sysadmin can write shellscripts which, while more familiar, is also almost certainly more complex

SELinux: Neither Important Nor Ground-breaking

Posted Dec 26, 2005 1:08 UTC (Mon) by mattdm (subscriber, #18) [Link] (2 responses)

> I disagree. I wrote a policy for mdadm for my (gentoo, not fc) system
> before the upstream one existed; it was no more complex than programming in
> any other language. Because as far as I'm concerned writing policies is
> essentially just another kind of programming.

Exactly. On the other hand, previous security policy mechanisms -- user accounts, groups, PAM config files, etc., aren't a kind of programming at all. SELinux needs to get to that point.

SELinux: Neither Important Nor Ground-breaking

Posted Dec 26, 2005 9:36 UTC (Mon) by danielthaler (guest, #24764) [Link] (1 responses)

A far more appropriate comparison is init scripts. Both a SELinux policy and the init scripts can be substantially different between distributions and are therefore (a simple kind of) programming.

The similarity even goes further than that: Init scripts usually offer configuration files so that you don't have to edit complicated bash scripts that you might not understand; SELinux has booleans (also in a separate file) that can achieve the same effect.

What makes them different is that init and scripts for it have been around forever, are present on 99% of all linux machines and have gotten to the point where they just work. SELinux is still getting to that point. I think it will, though.

SELinux: Neither Important Nor Ground-breaking

Posted Dec 26, 2005 15:37 UTC (Mon) by mattdm (subscriber, #18) [Link]

Also, init scripts tend to be packaged with the program they control -- not all in a lump.

SELinux kills multiboot

Posted Dec 25, 2005 3:21 UTC (Sun) by mattdm (subscriber, #18) [Link] (3 responses)

That's not a valid comparison. Changes like this -- essentially, a compatibility bugfix, as serious as it may be -- are outside of the scope of Fedora Legacy. Had you filed this bug during the original test phase for Red Hat Linux 9, the situation would have been quite different.

SELinux kills multiboot

Posted Dec 25, 2005 17:53 UTC (Sun) by error27 (subscriber, #8346) [Link] (1 responses)

Obviously it's a bit difficult to go back and change all the old kernels to the new version. The thing about the xattr bug is that instead of just refusing to mount the file system it let you mount it and the destroyed it as soon as you typed 'ls' on it.

It would have been better if they could have not made the filesystem incompatible with older kernels or failing that they could have made it unmountable so that the data was not destroyed.

ext3 is incompatible with ext3

Posted Dec 27, 2005 16:47 UTC (Tue) by jreiser (subscriber, #11027) [Link]

It would have been better if they could have not made the filesystem incompatible with older kernels or failing that they could have made it unmountable so that the data was not destroyed.

This is most certainly true. Data lives forever, and a sysadmin with a scrambled filesystem will remember for a long time that Linux+ext3 is unreliable.

Ext3 with a new feature had better interoperate with existing kernels that have "blessed" implementations (generally agreed as conforming: especially when promulgated by those who co-developed the specification) of "old" ext3. At worst, the older kernel must recognize the incompatible feature, then refuse to mount. The newer kernel must refuse to introduce the new feature to an older filesystem that lacks it, unless there is explicit confirmation. When a "bug" such as xattr-on-symlinks has existed for years and been deployed on hundreds of thousands of systems, then the "bug" has become a feature of the specification. Treat it as such, or else the fix must become "ext4."

Scope of Fedora Legacy

Posted Dec 27, 2005 16:15 UTC (Tue) by jreiser (subscriber, #11027) [Link]

All of the changes to Fedora Legacy 9 kernel since my bugzilla entry have been compatibility changes: fix "bitrot," that is, bugs that are revealed due to changes in the environment (namely, the security environment.) Fixing xattr-on-symlinks would be responding to changes in the environment seen by those who multiboot several generations: consultants supporting customers, testers of Fedora Core, etc. The patch is referenced in the bugzilla report, and it applied within the tolerance of rpmbuild. It should have been in the next released kernel for Fedora Legacy 9.

SELinux kills multiboot

Posted Dec 25, 2005 11:43 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

"Past experience suggests that you will be disappointed, except possibly for "workarounds" such as a warning "Don't do that!" or "backup all affected partitions and re-install." Fedora Legacy RedHat 9 has not fixed the xattr-on-symlinks compatibility issue, despite THREE official releases after the bugzilla #152827."

Fedora Legacy is entirely community driven project. If SELinux compatibility is important for you the way you can try to provide that is to produce patches. do QA, testing, packaging etc.

"SELinux is important, ground-breaking work. But they've tended to concentrate on Functionality and Performance only, ignoring Usability, Reliability, and Supportability."

Its a evolving security framework. Fedora Core 5 for example with include the reference policy from http://serefpolicy.sourceforge.net/ which is a policy designed from groundup to increase usability, reliability, supportability and compatibility between distributions that support SELinux. So all of these are being worked upon. As with any technology SELinux will go through a period of some rough edges, testing and wide deployments before getting more mature and usable.

Anyone can see a parallel between this and firewalls or Linux itself.

Also worth adding is that this newsitem in specific talks about a test/development release of Fedora and is not a Fedora specific bug. Several compatibility changes have been worked on in the upstream kernel. For example 2.6.15 will automatically assume default security contexts for filesystems where xattr support is unavailable thereby increasing interoperability between SELinux and non-SELinux systems to my understanding.

SELinux kills multiboot

Posted Dec 24, 2005 21:52 UTC (Sat) by error27 (subscriber, #8346) [Link]

It's not really a redhat bug, it's a kernel bug. It's not really a pre-release kernel either...

Plus I gotta feel that the xattr bug should get fixed first. It's been around since FC3. That bug has eaten a bunch of my filesystem...

Sounds familiar

Posted Dec 24, 2005 17:45 UTC (Sat) by ahz (guest, #27372) [Link]

This problem reminds me of a similar one. I was developing an embedded system (TS7250 with embedded Linux 2.4) and transferring files via USB disk between my desktop running Fedora Core 4 and the embedded system. When Fedora Core 4 mounted the disk, it would add SELinux attributes making the disk inaccessible to the TS7250. The solution was disabling SELinux on Fedora Core 4.

IIRC, the USB disk was ext3.

SELinux kills multiboot

Posted Dec 28, 2005 3:59 UTC (Wed) by gmaxwell (guest, #30048) [Link] (1 responses)

I could tell this one was available to the general public by the amount of trolling.

SELinux is still evolving, the lastest kernels added a feature which is not fully backwards compatible. Fedora failed to document this in their release notes for a test release. Boohoo.

If you multiboot, boot with SELinux disabled. Because of the differing security tags, SELinux will not be too useful in a multiboot across different distro versions. Alternatively, you could upgrade your kernel in the old systems to a new version which supports the new tagging before loading the newer distro up. I believe the new kernel now in FC4 updates, for example, supports the new SELinux tags.

SELinux kills multiboot

Posted Feb 4, 2006 1:05 UTC (Sat) by ChuckW (guest, #16682) [Link]

What you really mean is that the selinux API (as it were) isn't stable. There's nothing wrong with that, but you don't need to be such a "boohoo" snob about it. Those "trolls" you whine about are (or were) your customers.