|
|
Subscribe / Log in / New account

Removing run-time disabling for SELinux in Fedora

By Jake Edge
September 23, 2020

Disabling SELinux is, perhaps sadly in some ways, a time-honored tradition for users of Fedora, RHEL, and other distributions that feature the security mechanism. Over the years, SELinux has gotten easier to tolerate due to the hard work of its developers and the distributions, but there are still third-party packages that recommend or require disabling SELinux in order to function. Up until fairly recently, the kernel has supported disabling SELinux at run time, but that mechanism has been deprecated—in part due to another kernel security feature. Now Fedora is planning to eliminate the ability to disable SELinux at run time in Fedora 34, which sparked some discussion in its devel mailing list.

SELinux is a Linux Security Module (LSM) for enforcing mandatory access control (MAC) rules. But the "module" part of the LSM name has been a misnomer since a 2007 change to make the interface static and remove the option to load LSMs at run time. So kernels are built with a list of supported LSMs, and they can be enabled or disabled at boot time using kernel command-line options. Certain architectures had bootloaders that made it difficult for users to add parameters to the command line, though, so the SELinux developers added a way to disable it at run time. The need for that functionality has faded, and removing it will allow another kernel hardening feature to be used.

The post-init read-only memory feature provides a way to mark certain kernel data structures as read-only after the kernel has initialized them. The idea is that various data structures are prime targets for kernel exploits; function-pointer structures, like those used by the LSM hooks, are of particular interest. So the LSM hooks were protected that way. However, that hardening is only enabled if the ability to disable SELinux at run time is not present in the kernel. The presence of the SELinux feature is governed by the CONFIG_SECURITY_SELINUX_DISABLE kernel build option.

In order to get that hardening feature, Ben Cotton posted a proposal for Fedora 34 to remove the support for disabling SELinux at run time. The proposal is owned by Petr Lautrbach and Ondrej Mosnacek; it would migrate users to the selinux=0 command-line option if they are currently disabling SELinux via the SELINUX=disabled setting in /etc/selinux/config. The proposal, which has been updated on the Fedora wiki based on feedback, would not change the ability to switch SELinux between enforcing and permissive modes at run time using setenforce

The 5.6 kernel deprecated the run-time-disable feature for SELinux. The kernel currently prints a message to that effect, but there are plans to make using it even more painful by sleeping for five seconds when it is used. It may get even more obnoxious over time; eventually the plan is to remove it altogether. Red Hat distributions (Fedora, CentOS, RHEL) are the only known users of the feature at this point, so once they have all moved away, the feature can be removed from the kernel. RHEL and CentOS systems will stick around for a lot longer than Fedora systems, since it is only supported for a bit over year. But Red Hat will just continue to maintain the feature in the RHEL/CentOS kernels; removing the run-time disable from Fedora presumably means that the next RHEL/CentOS major release will no longer support it either.

The proposal seeks to smooth the path for users who upgrade but have SELinux already disabled via the config file. The kernel command-line parameter will not be automatically added, but it will be documented as the only real way to disable SELinux. Systems without selinux=0 at boot, but that disable it in the config file, will simply get a system that has SELinux in it, but without any policy loaded, so the run-time impact should be minimal. In addition, the SELinux filesystem will not be mounted at /sys/fs/selinuxfs.

In general, the reaction was positive, though there were some concerns. James Cassell wondered about forcing systems that have SELinux disabled in user space, but not in the kernel, to loudly fail. He was concerned that the performance impact of SELinux without any policy loaded could affect "certain real-time use cases", as he had found with SELinux in permissive mode. But Mosnacek said that the impact should be minimal. He did think it would be useful to alert the system administrator to that situation, but was not sure how to go about it beyond just documenting it in the release notes. Lautrbach suggested possibly adding a systemd unit that would warn users once if that situation was found, but he also pointed out that the existing description in the comments of /etc/selinux/config accurately describe the situation:

#     disabled - No SELinux policy is loaded.
^^ this is exactly what will happen when this change is accepted. SELinux will be enabled internally in kernel but no policy will be loaded and as it was already explained for [users] this situation should be almost the same as SELinux disabled.

There were also some suggestions for improvements to the proposal document from Vít Ondruch and Michal Schorm. In particular, Schorm wanted to see more clarity in the document that the change does not affect the ability to switch to SELinux permissive mode at run time. Those changes have been made and the Fedora Engineering Steering Committee (FESCo) ticket shows five "+1" votes at the time of this writing, so its adoption is all-but-assured.

Silent denials

An interesting sidelight came up in the discussion, though. Richard Hughes asked about SELinux denials that do not generate audit messages. Typically, when SELinux denies an operation, it puts out an audit log entry (known as an access vector cache, or AVC, denial), but sometimes that does not happen.

Speaking from personal experience, I've wasted days over the last decade trying to debug a locally installed system service that was not working where there were no messages in any of the logs (e.g. no AVCs) -- and turning off selinux at runtime magically fixed the problem.

Whilst I'm of course in favour of fixing the lockdown issue, would it also be fair to say that any selinux regression not triggering an AVC (which is fixed using selinux=0) would block this kind of proposal?

It turns out that there is a class of SELinux denials that are marked as "dontaudit", Mosnacek said:

They are used for when the impact of a failed access check doesn't prevent the application from functioning and we don't want to allow that access vector (e.g. when the application just checks to see if it can do some privileged operation and if not, it continues with some fallback). Unfortunately, it can happen sometimes that such a rule hides some denial that actually does break something.

It should not require rebooting to turn off SELinux in order to test whether that is occurring; switching to permissive mode should be sufficient. He also noted that the semodule command can be used to disable the dontaudit rules (and re-enable them). That should at least provide a useful error message, which should help Hughes (and others) figure out mysterious failures from SELinux:

There's sometimes no log output anywhere obvious that a syscall or something was blocked. It's the reason I turn off selinux on my work development machine, and I've often wasted *hours* of my life on code "doing something impossible" over the last decade until a neuron at the back of my brain remembers "you've not yet turned off selinux" and then when I "sudo setenforce 0" it works, and I can't actually file a bug as there's no indication of what selinux actually blocked or why.

Neal Gompa helped fill in some of the background for how these dontaudit rules get added. Based on conversations that he has witnessed, Red Hat customers are often the driving force:

Because Red Hat customers put the SELinux policy developers into no-win situations: they complain about AVC denials that don't actually significantly break anything in *their* app and often just disable SELinux in those scenarios. Red Hat wants customers to use it and not freak out all the time, so these kinds of things get added because it is very hard to come up with the right rules for all cases and there's not enough time to work on that.

"Hiding" denials is certainly unfortunate—and counterproductive, at least for Fedora. The distribution should perhaps consider its developer audience and disable the dontaudit rules by default. That may risk swamping the audit log with harmless denials, as described by Mosnacek and in this section of the Red Hat SELinux documentation, but it can be seriously maddening to try to debug a problem where SELinux is quietly denying some action. Raising the profile of dontaudit rules is clearly called for; that may help avoid more sudden baldness in frustrated users.

The ability to disable SELinux at run time would seem to be something that is unlikely to be missed. There are other, better ways to accomplish the same goals; removing it paves the way for Fedora systems to get more hardening in the form of read-only protection on the LSM hooks.


Index entries for this article
SecurityDistribution security
SecuritySecurity Enhanced Linux (SELinux)


to post comments

Removing run-time disabling for SELinux in Fedora

Posted Sep 23, 2020 18:38 UTC (Wed) by sub2LWN (subscriber, #134200) [Link] (3 responses)

"dontaudit" has made its way into manpages on Fedora 32 and RHEL 8: man 8 semanage-dontaudit; Programs which tell users to disable SELinux are like ones which recommend disabling any firewall instead of adjusting rules. Counter to "dontaudit", I wonder if anyone has proposed a "loudlyaudit" for denials too important to merely have in the audit log or the AVC pop-up in the GUI case. :-)

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 5:24 UTC (Thu) by madhatter (subscriber, #4665) [Link]

I don't know that we'll get as far as systems shipping with an "SELinux foghorn", to be sounded when important denies occur. But when in their manual (linked above) RH write:

The downside of dontaudit is that, although SELinux denies access, denial messages are not logged, making troubleshooting more difficult.

I think they rather understate the issue ("The downside of using explosives for surface preparation is that the building may well collapse, making repainting the interior walls more difficult").

Removing run-time disabling for SELinux in Fedora

Posted Sep 25, 2020 16:43 UTC (Fri) by mvdwege (guest, #113583) [Link] (1 responses)

Oh nice, they added a manpage. That really helps if you don't know that dontaudit exists. /sarcasm

(I ran into dontaudit when I wanted to use blackhole routing in fail2ban, instead of the default firewalling. Turns out failtoban calls the ip program, and it starts out not having permission to transition to ifconfig_exec_t. Ok, I'm a seasoned admin, I grep the audit log to make sure I only have fail2ban messages, pipe it to audit2allow, and i get a nice 2 line policy module. Still doesn't work. 3 days of banging my head against the wall, I finally find dontaudit. I disable it, and it turns out I need permission to do writes in a domain belonging to netlink. 3 line policy now, and it works.

But I see why they disable it. There is lots of code that does the right thing, try something and do something else on failure, instead of test-then-modify, but under SELinux that generates a *hell* of a lot of spurious audit messages)

Removing run-time disabling for SELinux in Fedora

Posted Sep 26, 2020 6:18 UTC (Sat) by sub2LWN (subscriber, #134200) [Link]

I agree, it's too obscure as a default currently. Oops, I was mistaken too, the centos 7 system I was looking at didn't have the policycoreutils-python-2.5 package. I guess the default is that selinux is enabled but there aren't any tools to manage it. :o) after updating mandb now that audit2allow is installed, $ cat /etc/redhat-release; whatis audit2allow
CentOS Linux release 7.8.2003 (Core)
audit2allow (1) - generate SELinux policy allow/dontaudit rules from logs of denied operations

Maybe they should call that command "audit2allow-or-dontaudit" since it doesn't generate "allow" rules, it generates "allow/dontaudit" rules. My first post was going to be "it's not even in the manpages!" but then I checked newer systems and it was there, and it's there for some slightly older systems too. Not sure how long the audit2allow manpage has been titled that way.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 6:44 UTC (Thu) by ragnar (guest, #139237) [Link] (14 responses)

The difficulty of debugging issues is a big problem of SELinux. But an even bigger problem with SELinux is that when you find an issue there doesn't seem to be any way to fix it other than disabling SELinux unless you are a seasoned SELinux expert. At least I haven't found any material on how to easily fix rules myself and the article indicates that most other people also just disable SELinux at first sign of an issue.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 8:03 UTC (Thu) by rwmj (subscriber, #5474) [Link] (10 responses)

For future reference, use audit2allow: https://docs.fedoraproject.org/en-US/Fedora/22/html/SELin...

It's actually pretty straightforward to at least find out what rule needs to be added, and you can either add it temporarily to your machine or submit a BZ to get it permanently fixed in the distribution.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 9:08 UTC (Thu) by MKesper (subscriber, #38539) [Link] (2 responses)

That's still much work, though, sadly.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 10:41 UTC (Thu) by james (subscriber, #1325) [Link] (1 responses)

Couple of minutes' work, maybe? And, in my experience, the resulting rules are comprehensible, showing you what you are allowing and giving you a chance to check that the program does actually need that access.

The example given in the documentation is

allow certwatch_t var_t:dir write;
which is pretty clear even if you aren't very familiar with SELinux.

Also, SELinux in Fedora/CentOS mostly restricts sensitive OS-type programs: big user programs like LibreOffice are extremely unlikely to encounter problems.

(I wonder how many counter-examples I'm going to get...)

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 12:56 UTC (Thu) by jmclnx (guest, #72456) [Link]

But for people just getting exposed to SELinux, it is not a few minutes work.

I have spent 3+ days trying to get vnstatd active using "audit2why", "audit2allow", "checkmodule", "semodule_package" and "semodule" (plus many others) and it still fails.

I am NOT asking for help on this, but I am pointing out how hard it is to work with SELinux. The documentation is extremely complex and seems one must spend many weeks reading and re-reading docs to even figure out how to do the simplest task.

I really want to keep it active, at least I know it can be very useful, but getting items to work is quite hard.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 9:30 UTC (Thu) by ragnar (guest, #139237) [Link] (4 responses)

That is at least encouraging that it is possible to fix problems. But from the examples on that page it is still really difficult to understand more than "access was denied".

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 12:21 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

I would recommend watching

https://www.youtube.com/watch?v=_WOKRaM-HI4

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 12:40 UTC (Thu) by rwmj (subscriber, #5474) [Link] (2 responses)

What I do is run audit2allow in a terminal, then copy and paste the AVC (SELinux error message) straight into that terminal. It will print the rule that is needed to allow the access, and from there it's usually fairly straightforward to understand what failed. The AVCs can be found in either /var/log/audit/... or ausearch -m avc.

As an example this AVC was produced by abrt a month ago:

type=AVC msg=audit(1597179780.871:66254): avc:  denied  { setattr }
for  pid=2188952 comm="abrt-action-sav" name="rpmdb.sqlite-shm"
dev="dm-1" ino=1051653 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
and simply pasting that into audit2allow gives me:
#============= abrt_t ==============
allow abrt_t var_lib_t:file setattr;
which means that abrt wasn't able to setattr (change the permissions) on a file with the var_lib_t label, which I guess is something in /var/lib.

Note that I am definitely no expert on SELinux. If I was really concerned about this I'd file a BZ against selinux-policy and get the experts to look into it.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 12:59 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

A few tips:

Ensure that setroubleshoot-server and policycoreutils-python-utils are installed on your system and take advantage of sealert -l "*"

https://access.redhat.com/documentation/en-us/red_hat_ent...

If you have setroubleshoot-server installed before any denials happen, you will get an easy to understand log in /var/log/messages

https://access.redhat.com/documentation/en-us/red_hat_ent...

example:

setroubleshoot: SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket. For complete SELinux messages. run sealert -l 8c123656-5dda-4e5d-8791-9e3bd03786b7

Running the suggested sealert command will tell you exactly what to do to resolve the problem. In many cases, it is something as simple as toggling a boolean or one liner policy change

Removing run-time disabling for SELinux in Fedora

Posted Oct 1, 2020 9:03 UTC (Thu) by eduperez (guest, #11232) [Link]

I am afraid I did not understand any of what you wrote, I am not a SELinux developer. I could be granting full permissions to a piece of malware, and I would not know the difference.

Removing run-time disabling for SELinux in Fedora

Posted Sep 25, 2020 8:02 UTC (Fri) by jbenc (subscriber, #40051) [Link] (1 responses)

Except for the cases when it doesn't work. I did exactly that, installed the generated rule, got no further AVC messages, yet the software in question (a poorly written proprietary cups filter) was still failing. Disabling selinux solved the issue.

Now I learned about the dontaudit misfeature from this article (I wonder why it's not mentioned in bold in all of those selinux tutorials I found), so maybe it's the culprit.

I echo the sentiment, debugging selinux is impossible, unless one is a seasoned selinux expert.

Removing run-time disabling for SELinux in Fedora

Posted Oct 1, 2020 9:10 UTC (Thu) by eduperez (guest, #11232) [Link]

> I echo the sentiment, debugging selinux is impossible, unless one is a seasoned selinux expert.

Part of the issue, IMHO, is that debugging SELinux is often considered as something that any administrator should do: "is some software failing? oh, just add some new rule".

A piece of software that fails because of SELlinux should be considered as a critical bug, in either SELinux or the software.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 9:22 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (2 responses)

You should run it in permissive mode and then add rules to all the audits that were generated and then disable permissive mode.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 9:36 UTC (Thu) by ragnar (guest, #139237) [Link] (1 responses)

So if there are some hidden malware running on my machine I have now given it permission to do whatever nefarious things it wants to do anyway? How is that better than just disabling SELinux?

It is also still pretty user hostile for people not well versed in SELinux.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 11:16 UTC (Thu) by michich (guest, #17902) [Link]

You don't have to make the whole system permissive. You can set permissive domains - https://access.redhat.com/documentation/en-us/red_hat_ent...

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 10:25 UTC (Thu) by hei8483j (guest, #124709) [Link]

If you still can turn on permissive mode, then it is a non-issue. I have never had to turn off SELinux. Permissive mode is mainly needed when creating new rules for proprietary software that doesn't behave.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 12:40 UTC (Thu) by bkw1a (subscriber, #4101) [Link] (2 responses)

I've gone through this many times (basically with every new CentOS version): I start out with the intention of leaving selinux on. Something is broken, so I use the logs or audit2allow to make adjustments to the selinux configuation. Things work again, until I run into another problem and repeat. But eventually I get to something that's broken and gives me no log messages that point to any problem. After banging my head against the wall for a while, it eventually occurs to me that I should try disabling selinux. I do, and the problem goes away.

If I could rely on selinux always giving me useful log information when it blocks something, I'd be fine with using it. But as long as it continues to inexplicably break things sometimes, I can't trust it in production.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 13:44 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

If you disable dontaudit, you should always get something in the log

https://access.redhat.com/documentation/en-us/red_hat_ent...

Removing run-time disabling for SELinux in Fedora

Posted Oct 1, 2020 8:59 UTC (Thu) by eduperez (guest, #11232) [Link]

His is exactly my experience, I could not have said it better.

Removing run-time disabling for SELinux in Fedora

Posted Sep 24, 2020 17:21 UTC (Thu) by darwi (subscriber, #131202) [Link]

> The kernel currently prints a message to that effect, but there are plans to make using it even more painful by sleeping for five seconds when it is used.

This is one of the most childish cr*p I’ve read in a while. Production kernels don’t behave like that.

Following down the thread, It’s obvious that it was mentioned that Linus might rightfully get angry about this.


Copyright © 2020, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds