An important set of stable kernel updates

Posted Oct 21, 2016 13:43 UTC (Fri) by pizza (subscriber, #46)
In reply to: An important set of stable kernel updates by ppel512
Parent article: An important set of stable kernel updates

> Call them whatever you want. My team and I just spent the last 18 hours patching over 45,000 environments (mix of VM & baremetal). We had to disrupt our customers with unplanned maintenance and burn the candle from both ends which pretty much destroys any hope of productivity for the actual work day today.

We have an expression for that sort of thing: Shit happens.

You deal with it, you move on.

> The answer is no. It's not acceptable.

Then perhaps you should demand a refund and/or switch software suppliers.

An important set of stable kernel updates

Posted Oct 21, 2016 13:46 UTC (Fri) by ppel512 (guest, #111882) [Link] (4 responses)

If this security posture continues we may indeed have to do just that. (review other *nix options). Either way, I don't see how 'Shit happens' is an acceptable response to an avoidable wide scale security issue like this.

Quite frankly, this kind of attitude is how the cyber criminals of the world get to put food on the table at our expense.

An important set of stable kernel updates

Posted Oct 21, 2016 14:33 UTC (Fri) by pizza (subscriber, #46) [Link]

No, what I'm saying is "if I'm in the line of work of providing 45K instances of essentially the same system, it behoves me to plan for unexpected problems coming along that could impact all of them at once."

The details of that "unexpected problem" don't actually matter. You only know that *something* will go wrong, and try to architect mitigations into your systems and processes to handle it.

As already mentioned elsewhere, if you find the current status quo so unacceptable, why weren't you already running grsecurity/PaX/etc on those 45K instances? But even grsecurtiy/etc have flaws, to say nothing of the actual hardware it's all running on..

An important set of stable kernel updates

Posted Oct 21, 2016 20:12 UTC (Fri) by patrick_g (subscriber, #44470) [Link] (1 responses)

> an avoidable wide scale security issue like this

How was this avoidable? I thought even Grsecurity was vulnerable to this bug?

An important set of stable kernel updates

Posted Oct 22, 2016 11:19 UTC (Sat) by Wol (subscriber, #4433) [Link]

So if the OP was actually running grsecurity, he'd have to patch 50K instances instead of 45K, to make up for grsecurity's performance impact?

As pointed out elsewhere, people have concerns other than security. A hell of a lot of linux instances are supercomputers, where even a 1% performance impact on the kernel, has a massive impact on the performance of the computer as a whole.

Oh - and since I've been following the linux-raid list, I've come to the conclusion that kernel programming is like making sausages - if you like the end product, don't go looking at how it's made. That or actually get in there and get your hands dirty :-)

Cheers,
Wol

An important set of stable kernel updates

Posted Oct 22, 2016 7:26 UTC (Sat) by Lionel_Debroux (subscriber, #30014) [Link]

Unfortunately, the other *nix options aren't that good when it comes to security either. spender has been lambasting them for years as well, especially OpenBSD, which prides itself about security and tries to make people believe it's so great, but isn't...

Let's just look at ASLR. It was invented by the PaX Team in 2001. ASLR broke a number of direct exploits based on predictable memory addresses. Even nowadays, it still raises the cost for attackers, who have to use infoleaks to get around it, so I'd say it's still an effective component of defense in depth. PaX has a plugin for sanitizing the kernel stack, as well as better allocated / freed memory sanitization, thereby removing a whole class of infoleaks, BTW.
* Linux grew PIE support in 2003 and other ASLR aspects in 2005. Even KASLR in 2014, but that barely provides any security in return for its complexity, as predicted by PaXTeam and spender from the get go, and as shown by the many KASLR bypasses of the past two years;
* OpenBSD implemented some ASLR features in 2003, but didn't implement PIE until 2008;
* NetBSD didn't enable ASLR by default until 2016 (!);
* FreeBSD still doesn't have ASLR because some of its maintainers refuse it because "it's not really effective". The HardenedBSD derivative has it, for good reason, though;
* for completeness, though it's not an option for your 45K Linux servers (cost): Solaris recently grew ASLR support, but it wasn't there in the discontinued OpenSolaris, and AFAICT, its derivatives don't have it;
* not an option either for you (special hardware requirements, cost): MacOS X (and iOS) got ASLR after Linux did.

In 2016, OpenBSD was trounced by kernel fuzzers, just like Linux, finding easy DoS and LPE.
OpenBSD reinvents a subset of PaX/grsecurity's wheels in NIH mode: look at what the PaX Team and spender wrote and said about W^X.

Overall, deploying grsec-patched Linux kernels with MEMORY_UDEREF, KERNEXEC, RAP, CONSTIFY, RANDSTRUCT and the dozens of PaX/grsec goodies onto some of your many servers is pretty much the best you can do to protect them. None of the other *nix options come anywhere close to the level of protection offered by grsec. Obviously, there will be a performance hit.
You should also consider buying (having management buy, I mean...) commercial support from grsecurity. For a corporation which has 45K Linux servers, it's a tiny drop in the ocean. All the more you'll save money (besides saving your sleep and sanity) every time you don't _have_ to reboot your servers *absolutely right now* (but can choose to do so tomorrow or on Monday) to fix yet another critical issue, thanks to the PaX/grsecurity defenses making an issue completely unexploitable, e.g. because the corresponding ops structures targeted by the exploit were constified, because the refcount overflow protection kicks in, or for other reasons.