Encouraging a wider view

For his keynote at the 2013 Linux Security Summit, Ted Ts'o weaved current events and longstanding security problem areas together. He encouraged the assembled kernel security developers to look beyond the kernel and keep the "bigger picture" in mind. His talk kicked off the summit, which was co-located with the Linux Plumbers Conference (and others) in New Orleans, Louisiana.

Adversaries

Ts'o began by looking at the adversaries we face today, starting with the secret services of various governments—our own and foreign governments, no matter where we live. Beyond that, though, there are organized cyber-criminals who maintain botnets and other services for hire. He noted that there is a web service available for solving CAPTCHAs, where a rural farmer with no knowledge of English (or even Roman characters, perhaps) will solve one in realtime. "Isn't capitalism wonderful?", he asked.

The historic assumptions made about the budgets of our adversaries may not be accurate, he said. Many in the room will know about the problems he is describing, but the general public does not. How do we get the rest of the world to understand these issues, he asked.

Beyond criminals, we have also seen the rise of cyber-anarchists recently. These folks are causing trouble "for the fun of it". They have different motivations than other kinds of attackers. No matter what you might think of their politics, he said, they can cause a lot of problems for "systems we care about".

Ts'o related several quotes from Robert Morris, who was the chief scientist at the US National Security Agency (NSA)—and father of Robert T. Morris of Morris worm "fame". Morris was also an early Multics and Unix developer, who was responsible for the crypt() function used for passwords. The upshot of Morris's statements was that there is more than one way to attack security and that underestimating the "time, expense, and effort" an adversary will expend is foolhardy. Morris's words were targeted at cryptography, but are just as applicable to security. In addition, it is fallible humans who have to use security software, so Morris's admonition to "exploit your opponent's weaknesses" can be turned on its head: our opponents may have vast resources, but developers need to "beware the stupid stuff", Ts'o said.

The CA problem

In May, Ts'o and his Google team were at a hotel in Yosemite for a team-building event where he encountered some kind of man-in-the-middle attack that highlighted the problems in the current SSL certificate system. While trying to update his local IMAP mail cache, which uses a static set of certificates rather than trust the certificate authority (CA) root certificates, his fetch failed because the po14.mit.edu certificate had, seemingly, changed—to a certificate self-signed by Fortinet. That company makes man-in-the-middle proxy hosts to enable internet surveillance by companies and governments.

He dug further, trying other sites such as Gmail and Harvard University, but those were not being intercepted. In addition, requesting a certificate for the MIT host from elsewhere on the internet showed that the certificate had not actually changed. Something was targeting traffic from the hotel (and, perhaps, other places as well) to MIT email hosts for reasons unknown. The bogus certificate was self-signed, which would hopefully raise red flags in most email clients, but the problem persisted for the weekend he was there—at least.

As people in the room are aware, but, again, the rest of the world isn't, the CA system is broken, Ts'o said. He referred to a Defcon 19 presentation [YouTube] by Moxie Marlinspike about the problems inherent in SSL and the CA system. While Marlinspike's solution may not be workable, his description of the problem is quite good, Ts'o said.

It comes down to the problem that some certificate issuers are "too big to jail", so that punishing them by banning their root certificates is unworkable. Marlinspike estimated that banning Comodo (which famously allowed fraudulent certificates to be issued) would have caused 20-25% of HTTPS servers on the internet to go dark. Comodo got to that level of popularity by being one of the cheapest available providers, of course. There are some 650 root authorities that are currently blindly trusted to run a tight ship, with no way to punish them if they don't, Ts'o said.

There are some solutions like certificate pinning, which Google started and various browser vendors have adopted, but that solution doesn't scale. Many have claimed that DNSSEC is a solution, but Marlinspike has argued otherwise—the actors are different, but the economic incentives are the same. Instead of trusting a bunch of CAs, Verisign will have to be trusted instead.

Ts'o doesn't know how to solve the CA problem, but did have a selfish request: he would like to see certificates be cached and warnings be issued when those certificates change. Unfortunately, it won't work for the average non-technical person, nor would it be all that easy because OpenSSL and libraries that call it are typically unconnected from the user interface, but it would make him happier.

Linux security solutions

A short program that just did setuid(0) and spawned a shell led to Ts'o's question of "when is a setuid program not a setuid program?". He showed that the program wasn't owned by root with the setuid bit set, yet it gave a root shell. It worked because the file had CAP_SETUID set in its file capabilities—something that all of the security scanning tools he looked at completely ignored. File capabilities have been around since 2.6.30, but no one is paying attention, which is "kind of embarrassing". Worse yet, there is no way to disable file capabilities in the kernel, he said.

Linux capabilities are meant to split up root's powers into discrete chunks, but their adoption has been slow. The idea is that capabilities are by default not inherited by children, so parents need the right to pass on their capabilities, and the child executable has to have the right to accept them. But there is a "compatibility mode" that has been created where root-spawned processes inherit all of the parent's capabilities. This is done so that running shell scripts as root continues to work, but that mode leads to another problem.

Of the 30 or so powers granted by capabilities, over half can (sometimes) be used to gain full root privileges. You must be able to use those capabilities in an "unrestricted way", which may or may not be true depending on how the system is set up. But many would not be a privilege-escalation problem at all if it weren't for the compatibility mode.

So, why not use SELinux instead, he asked. It can do all of the things that capabilities were intended to do, although the policy has to be set up correctly. Unfortunately, the policy is several megabytes of source that is difficult to understand, change, or use.

As it turns out, though, things have "gotten a lot better" in the SELinux world, according to Ts'o. Every few years, he turns on SELinux to see how well it is working. "Usually, it screws me to the wall" and he has to immediately disable it. In one case, he even had to reinstall his system because of it. But when he tried it just prior to the summit, it mostly worked for him.

The audit2allow program, which looks at the SELinux denials and generates new policy, is "a huge win". On his system, it generated 400 lines of local policy to make things work. Overall, it is much better and he will probably leave it running on his system. There is still a ways to go, particularly in the area of documentation. There is plenty of beginner documentation and expert documentation (i.e. the source code), but information for intermediate users is lacking. That leads to those users just turning off SELinux. The problems he ran into (which were fewer than his earlier tries, but still present) may have been partly due to the SELinux policy packages for Debian testing; perhaps Fedora users would have had a better time, he said.

His experiment with SELinux showed another problem, though. He now gets email every two hours from logcheck with a vast number of complaints. It is clear that his logcheck configuration files are out of sync with the SELinux installation. How to handle security policy and configuration with respect to various kinds of distribution packages is a difficult problem. Right now, the SELinux policy package maintainers and logcheck package maintainers would need to coordinate, but that doesn't scale well. Does logcheck also need to coordinate with AppArmor as well, or should the policy packages be handling the configuration needed for logcheck? There is no obvious solution to that problem, but perhaps automated tools a la audit2allow might help, he said.

Wrapping up

Turning to the summit itself, Ts'o noted all of the different example topics listed in the call for participation, which included ideas like system hardening, virtualization, cryptography, and so on. The program committee did a good job on that list, he said, but what ended up on the schedule? An update to Linux Security Module (LSM) A, a change to LSM B, a new feature for LSM C, and composing (i.e. stacking) LSMs. That's not completely fair, Ts'o said, as there are other topics on the list like kernel address space layout randomization (ASLR) and embedded Linux security, but his point was clear.

He encouraged Linux security developers to think more widely. The program committee can only choose from the topics that are submitted and people submit what they can get funding to work on. The executives of the companies they work for only fund those things that users really care about, so how can we get users to care about security, he asked.

It turns out that perhaps "NSA" is part of the answer, he said—to widespread laughs. But the best outcome from the Snowden revelations is that people are talking about security again. According to Ts'o, US President Obama has been quoted as saying "never let a good crisis go to waste". Security researchers and developers should follow that advice, he said.

A business case needs to be made for better Linux security, Ts'o said. After the kernel.org compromise, some companies were interested in funding Linux security work, but after two months or so, that interest all dried up. It may be that the NSA surveillance story also dies away, but Glenn Greenwald is something of an expert at dribbling out the details from Snowden. That may give this particular crisis longer legs.

Security folks need to find a way for security countermeasures to take advantage of the power of scale, he said. Both Google and the NSA have figured out that if you can invest a large amount into fixed costs and bring the incremental costs way down, you can service a lot of users. Cyber-criminals have also figured this out; the security community needs to do so as well.

In the kernel developers' panel that had been held at LinuxCon the day before, Linus Torvalds suggested that he would be willing to lose some of the best kernel developers if they would export kernel culture to various user-space projects. The same applies to security, Ts'o said. The security of the libraries needs to improve, hardware support for random number generation needs to be more widely available, and so on. Though there have been concerns about the RDRAND instruction in Intel processors because it is not auditable, Ts'o said he would much rather have it available than not.

Similarly, the trusted platform module (TPM) available in most systems is generally not used. Some TPM implementations are suspect, but there is no incentive for manufacturers to improve them since they aren't really used. It is hard enough to get a manufacturer to add $0.25 to the bill of materials (BOM) for a device; without a business case (i.e. users), it is likely impossible.

Security technology is not useful unless it gets used. In fact, as the file capabilities example showed, it can actually be actively harmful if it isn't used.

Ts'o concluded by suggesting that the assembled developers think about a "slightly bigger picture" than LSMs and the composition of LSMs. Those topics are important, but there is far more out there that needs fixing. As he noted, though, it will take a push from users to get the needed funding to address many of these issues.

[ I would like to thank LWN subscribers for travel assistance to New Orleans for the Linux Security Summit. ]