Features no excuse

adding SELinux defiantly slows things down, and if you run a benchmark on a system running a kernel compiled with SELinux you will get lower results than if you run the same benchmark on the same kernel without SELinux

so to not use the feature of SELinux you would compile a kernel without it.

the same thing goes for many fetures, turning them on at compile time increases the cache footprint and therefor slows the system, even if you don't use the feature. but you (usually) do have the option to not compile the code into the kernel to really avoid the runtime cost of them.