Removing run-time disabling for SELinux in Fedora
Disabling SELinux is, perhaps sadly in some ways, a time-honored tradition for users of Fedora, RHEL, and other distributions that feature the security mechanism. Over the years, SELinux has gotten easier to tolerate due to the hard work of its developers and the distributions, but there are still third-party packages that recommend or require disabling SELinux in order to function. Up until fairly recently, the kernel has supported disabling SELinux at run time, but that mechanism has been deprecated—in part due to another kernel security feature. Now Fedora is planning to eliminate the ability to disable SELinux at run time in Fedora 34, which sparked some discussion in its devel mailing list.
SELinux is a Linux Security Module (LSM) for enforcing mandatory access control (MAC) rules. But the "module" part of the LSM name has been a misnomer since a 2007 change to make the interface static and remove the option to load LSMs at run time. So kernels are built with a list of supported LSMs, and they can be enabled or disabled at boot time using kernel command-line options. Certain architectures had bootloaders that made it difficult for users to add parameters to the command line, though, so the SELinux developers added a way to disable it at run time. The need for that functionality has faded, and removing it will allow another kernel hardening feature to be used.
The post-init read-only memory feature provides a way to mark certain kernel data structures as read-only after the kernel has initialized them. The idea is that various data structures are prime targets for kernel exploits; function-pointer structures, like those used by the LSM hooks, are of particular interest. So the LSM hooks were protected that way. However, that hardening is only enabled if the ability to disable SELinux at run time is not present in the kernel. The presence of the SELinux feature is governed by the CONFIG_SECURITY_SELINUX_DISABLE kernel build option.
In order to get that hardening feature, Ben Cotton posted a proposal for Fedora 34 to remove the support for disabling SELinux at run time. The proposal is owned by Petr Lautrbach and Ondrej Mosnacek; it would migrate users to the selinux=0 command-line option if they are currently disabling SELinux via the SELINUX=disabled setting in /etc/selinux/config. The proposal, which has been updated on the Fedora wiki based on feedback, would not change the ability to switch SELinux between enforcing and permissive modes at run time using setenforce
The 5.6 kernel deprecated the run-time-disable feature for SELinux. The kernel currently prints a message to that effect, but there are plans to make using it even more painful by sleeping for five seconds when it is used. It may get even more obnoxious over time; eventually the plan is to remove it altogether. Red Hat distributions (Fedora, CentOS, RHEL) are the only known users of the feature at this point, so once they have all moved away, the feature can be removed from the kernel. RHEL and CentOS systems will stick around for a lot longer than Fedora systems, since it is only supported for a bit over year. But Red Hat will just continue to maintain the feature in the RHEL/CentOS kernels; removing the run-time disable from Fedora presumably means that the next RHEL/CentOS major release will no longer support it either.
The proposal seeks to smooth the path for users who upgrade but have SELinux already disabled via the config file. The kernel command-line parameter will not be automatically added, but it will be documented as the only real way to disable SELinux. Systems without selinux=0 at boot, but that disable it in the config file, will simply get a system that has SELinux in it, but without any policy loaded, so the run-time impact should be minimal. In addition, the SELinux filesystem will not be mounted at /sys/fs/selinuxfs.
In general, the reaction was positive, though there were some concerns.
James Cassell wondered
about forcing systems that have SELinux disabled in user space, but not in
the kernel, to loudly fail. He was concerned that the performance impact
of SELinux without any policy loaded could affect "certain real-time
use cases
", as he had found with SELinux in permissive mode.
But Mosnacek said
that the impact should be minimal. He did think
it would be useful to alert the system administrator to that situation, but
was not sure how to go about it beyond just documenting it in the release
notes. Lautrbach suggested
possibly adding
a systemd unit that would warn users once if that situation was found, but
he also pointed out that the existing description in the comments of
/etc/selinux/config accurately describe the situation:
# disabled - No SELinux policy is loaded.^^ this is exactly what will happen when this change is accepted. SELinux will be enabled internally in kernel but no policy will be loaded and as it was already explained for [users] this situation should be almost the same as SELinux disabled.
There were also some suggestions for improvements to the proposal document from Vít Ondruch and Michal Schorm. In particular, Schorm wanted to see more clarity in the document that the change does not affect the ability to switch to SELinux permissive mode at run time. Those changes have been made and the Fedora Engineering Steering Committee (FESCo) ticket shows five "+1" votes at the time of this writing, so its adoption is all-but-assured.
Silent denials
An interesting sidelight came up in the discussion, though. Richard Hughes asked about SELinux denials that do not generate audit messages. Typically, when SELinux denies an operation, it puts out an audit log entry (known as an access vector cache, or AVC, denial), but sometimes that does not happen.
Whilst I'm of course in favour of fixing the lockdown issue, would it also be fair to say that any selinux regression not triggering an AVC (which is fixed using selinux=0) would block this kind of proposal?
It turns out that there is a class of SELinux denials that are marked as "dontaudit", Mosnacek said:
It should not require rebooting to turn off SELinux in order to test whether that is occurring; switching to permissive mode should be sufficient. He also noted that the semodule command can be used to disable the dontaudit rules (and re-enable them). That should at least provide a useful error message, which should help Hughes (and others) figure out mysterious failures from SELinux:
Neal Gompa helped fill in some of the background for how these dontaudit rules get added. Based on conversations that he has witnessed, Red Hat customers are often the driving force:
"Hiding" denials is certainly unfortunate—and counterproductive, at least for Fedora. The distribution should perhaps consider its developer audience and disable the dontaudit rules by default. That may risk swamping the audit log with harmless denials, as described by Mosnacek and in this section of the Red Hat SELinux documentation, but it can be seriously maddening to try to debug a problem where SELinux is quietly denying some action. Raising the profile of dontaudit rules is clearly called for; that may help avoid more sudden baldness in frustrated users.
The ability to disable SELinux at run time would seem to be something that is unlikely to be missed. There are other, better ways to accomplish the same goals; removing it paves the way for Fedora systems to get more hardening in the form of read-only protection on the LSM hooks.
Index entries for this article | |
---|---|
Security | Distribution security |
Security | Security Enhanced Linux (SELinux) |
Posted Sep 23, 2020 18:38 UTC (Wed)
by sub2LWN (subscriber, #134200)
[Link] (3 responses)
Posted Sep 24, 2020 5:24 UTC (Thu)
by madhatter (subscriber, #4665)
[Link]
The downside of dontaudit is that, although SELinux denies access, denial messages are not logged, making troubleshooting more difficult.
I think they rather understate the issue ("The downside of using explosives for surface preparation is that the building may well collapse, making repainting the interior walls more difficult").
Posted Sep 25, 2020 16:43 UTC (Fri)
by mvdwege (guest, #113583)
[Link] (1 responses)
(I ran into dontaudit when I wanted to use blackhole routing in fail2ban, instead of the default firewalling. Turns out failtoban calls the ip program, and it starts out not having permission to transition to ifconfig_exec_t. Ok, I'm a seasoned admin, I grep the audit log to make sure I only have fail2ban messages, pipe it to audit2allow, and i get a nice 2 line policy module. Still doesn't work. 3 days of banging my head against the wall, I finally find dontaudit. I disable it, and it turns out I need permission to do writes in a domain belonging to netlink. 3 line policy now, and it works.
But I see why they disable it. There is lots of code that does the right thing, try something and do something else on failure, instead of test-then-modify, but under SELinux that generates a *hell* of a lot of spurious audit messages)
Posted Sep 26, 2020 6:18 UTC (Sat)
by sub2LWN (subscriber, #134200)
[Link]
Maybe they should call that command "audit2allow-or-dontaudit" since it doesn't generate "allow" rules, it generates "allow/dontaudit" rules. My first post was going to be "it's not even in the manpages!" but then I checked newer systems and it was there, and it's there for some slightly older systems too. Not sure how long the audit2allow manpage has been titled that way.
Posted Sep 24, 2020 6:44 UTC (Thu)
by ragnar (guest, #139237)
[Link] (14 responses)
Posted Sep 24, 2020 8:03 UTC (Thu)
by rwmj (subscriber, #5474)
[Link] (10 responses)
It's actually pretty straightforward to at least find out what rule needs to be added, and you can either add it temporarily to your machine or submit a BZ to get it permanently fixed in the distribution.
Posted Sep 24, 2020 9:08 UTC (Thu)
by MKesper (subscriber, #38539)
[Link] (2 responses)
Posted Sep 24, 2020 10:41 UTC (Thu)
by james (subscriber, #1325)
[Link] (1 responses)
The example given in the documentation is
Also, SELinux in Fedora/CentOS mostly restricts sensitive OS-type programs: big user programs like LibreOffice are extremely unlikely to encounter problems.
(I wonder how many counter-examples I'm going to get...)
Posted Sep 24, 2020 12:56 UTC (Thu)
by jmclnx (guest, #72456)
[Link]
I have spent 3+ days trying to get vnstatd active using "audit2why", "audit2allow", "checkmodule", "semodule_package" and "semodule" (plus many others) and it still fails.
I am NOT asking for help on this, but I am pointing out how hard it is to work with SELinux. The documentation is extremely complex and seems one must spend many weeks reading and re-reading docs to even figure out how to do the simplest task.
I really want to keep it active, at least I know it can be very useful, but getting items to work is quite hard.
Posted Sep 24, 2020 9:30 UTC (Thu)
by ragnar (guest, #139237)
[Link] (4 responses)
Posted Sep 24, 2020 12:21 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
Posted Sep 24, 2020 12:40 UTC (Thu)
by rwmj (subscriber, #5474)
[Link] (2 responses)
As an example this AVC was produced by abrt a month ago:
Note that I am definitely no expert on SELinux. If I was really concerned about this I'd file a BZ against selinux-policy and get the experts to look into it.
Posted Sep 24, 2020 12:59 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
Ensure that setroubleshoot-server and policycoreutils-python-utils are installed on your system and take advantage of sealert -l "*"
https://access.redhat.com/documentation/en-us/red_hat_ent...
If you have setroubleshoot-server installed before any denials happen, you will get an easy to understand log in /var/log/messages
https://access.redhat.com/documentation/en-us/red_hat_ent...
example:
setroubleshoot: SELinux is preventing /usr/sbin/httpd from name_bind access on the tcp_socket. For complete SELinux messages. run sealert -l 8c123656-5dda-4e5d-8791-9e3bd03786b7
Running the suggested sealert command will tell you exactly what to do to resolve the problem. In many cases, it is something as simple as toggling a boolean or one liner policy change
Posted Oct 1, 2020 9:03 UTC (Thu)
by eduperez (guest, #11232)
[Link]
Posted Sep 25, 2020 8:02 UTC (Fri)
by jbenc (subscriber, #40051)
[Link] (1 responses)
Now I learned about the dontaudit misfeature from this article (I wonder why it's not mentioned in bold in all of those selinux tutorials I found), so maybe it's the culprit.
I echo the sentiment, debugging selinux is impossible, unless one is a seasoned selinux expert.
Posted Oct 1, 2020 9:10 UTC (Thu)
by eduperez (guest, #11232)
[Link]
Part of the issue, IMHO, is that debugging SELinux is often considered as something that any administrator should do: "is some software failing? oh, just add some new rule".
A piece of software that fails because of SELlinux should be considered as a critical bug, in either SELinux or the software.
Posted Sep 24, 2020 9:22 UTC (Thu)
by LtWorf (subscriber, #124958)
[Link] (2 responses)
Posted Sep 24, 2020 9:36 UTC (Thu)
by ragnar (guest, #139237)
[Link] (1 responses)
It is also still pretty user hostile for people not well versed in SELinux.
Posted Sep 24, 2020 11:16 UTC (Thu)
by michich (guest, #17902)
[Link]
Posted Sep 24, 2020 10:25 UTC (Thu)
by hei8483j (guest, #124709)
[Link]
Posted Sep 24, 2020 12:40 UTC (Thu)
by bkw1a (subscriber, #4101)
[Link] (2 responses)
If I could rely on selinux always giving me useful log information when it blocks something, I'd be fine with using it. But as long as it continues to inexplicably break things sometimes, I can't trust it in production.
Posted Sep 24, 2020 13:44 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
https://access.redhat.com/documentation/en-us/red_hat_ent...
Posted Oct 1, 2020 8:59 UTC (Thu)
by eduperez (guest, #11232)
[Link]
Posted Sep 24, 2020 17:21 UTC (Thu)
by darwi (subscriber, #131202)
[Link]
This is one of the most childish cr*p I’ve read in a while. Production kernels don’t behave like that.
Following down the thread, It’s obvious that it was mentioned that Linus might rightfully get angry about this.
Removing run-time disabling for SELinux in Fedora
I don't know that we'll get as far as systems shipping with an "SELinux foghorn", to be sounded when important denies occur. But when in their manual (linked above) RH write:
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
CentOS Linux release 7.8.2003 (Core)
audit2allow (1) - generate SELinux policy allow/dontaudit rules from logs of denied operations
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Couple of minutes' work, maybe? And, in my experience, the resulting rules are comprehensible, showing you what you are allowing and giving you a chance to check that the program does actually need that access.
Removing run-time disabling for SELinux in Fedora
allow certwatch_t var_t:dir write;
which is pretty clear even if you aren't very familiar with SELinux.
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
What I do is run Removing run-time disabling for SELinux in Fedora
audit2allow
in a terminal, then copy and paste the AVC (SELinux error message) straight into that terminal. It will print the rule that is needed to allow the access, and from there it's usually fairly straightforward to understand what failed. The AVCs can be found in either /var/log/audit/...
or ausearch -m avc
.
type=AVC msg=audit(1597179780.871:66254): avc: denied { setattr }
for pid=2188952 comm="abrt-action-sav" name="rpmdb.sqlite-shm"
dev="dm-1" ino=1051653 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0
and simply pasting that into audit2allow gives me:
#============= abrt_t ==============
allow abrt_t var_lib_t:file setattr;
which means that abrt wasn't able to setattr
(change the permissions) on a file with the var_lib_t
label, which I guess is something in /var/lib
.
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora
Removing run-time disabling for SELinux in Fedora