User: Password:
Subscribe / Log in / New account

CloudLinux busted — twice

Benefits for LWN subscribers

The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

By Jonathan Corbet
June 27, 2012
Proprietary kernel modules have always had an unclear status in the kernel development community. Most (but not all) developers seem to agree that it is possible to make a module that is not a derived product of the kernel itself (and, thus, not subject to the requirements of the GPL); the Andrew filesystem, ported from Unix in the early days, has been cited as an example of this kind of module. Most developers also seem to agree that many other types of modules cannot be written as an independent work. There is far less agreement on where the line should be drawn, but there is a consensus around the idea that modules should not lie about their licensing status to gain closer access to kernel internals. The story of a company that was recently caught shipping that sort of dishonest module shows the kind of reaction that can be expected—and why closed-source modules are a bad idea in general.

Loadable kernel modules can only access kernel symbols that have been explicitly exported for that use. Two types of declarations are used to export these symbols. EXPORT_SYMBOL_GPL() is an indication that any module using the symbol is reaching so deeply into the kernel that it is, by necessity, a derived product of the kernel. Instead, EXPORT_SYMBOL() is used for symbols that only might indicate a derived-product status; it is also used for symbols that were already exported before the GPL-only mechanism was added.

The kernel's module loader includes an enforcement mechanism that prevents any module without a GPL-compatible license from accessing explicitly GPL-only symbols. To function, this mechanism clearly must know what the license for a specific module is. That is the weak point of the entire scheme: the kernel uses the honor system. Loadable modules include a special declaration that tell the kernel which license applies; the module loader looks at that declaration and decides whether the module will have access to GPL-only symbols or not. If the module lies about its license, the kernel will normally trust it.

That is where CloudLinux comes in. This company offers a CentOS-like distribution aimed at the needs of cloud computing providers. Essentially, CloudLinux has taken a RHEL clone, added its own container mechanism, and created a business around providing updates and support. The container mechanism is at least partially implemented by the company's "lve" kernel module; lve includes an access control and resource limit mechanism that, seemingly, is implemented independently of the kernel's control group subsystem. It can place limits on CPU usage, I/O bandwidth usage, and memory consumption, among other things; all this is managed through an administrator-friendly XML file.

Matthew Garrett recently became aware that, while the lve module claims to be GPL-licensed, there is not any source available for it; he responded with a patch causing the kernel to recognize and blacklist this module. Since lve uses GPL-only symbols, this change will prevent it from loading, essentially breaking it. This change has yet to be pushed into the mainline, but it has been accepted by the modules maintainer (Rusty Russell) and marked for stable updates as well. Unless something changes, it will soon become impossible to load the lve module into any current kernels.

The immediate effect of this change on CloudLinux is likely to be limited. Chances are that few users download the binary lve module and load it into their own kernels; almost all lve users are probably running the entire CloudLinux distribution. CloudLinux could easily patch its own kernel to remove the check for the lve module and restore the previous functionality. But such a move would almost certainly be noticed, and there is a distinct possibility that one or more kernel developers would react in an unpleasant way. It would not be an advisable course of action.

What CloudLinux did instead was to apologize and plead for time:

We planned to close source the module, and we will do it later on. Yet, it looks like one of our developers missed the point -- and did things incorrectly.

Please, give us two-three weeks to straighten things out. By the end of three weeks I plan to have source RPMs with the GPLed version of the modules available in our source repositories.

Later on we will have new module that is not GPL released.

This response was not entirely well received; it is not at all clear why CloudLinux is unable to immediately ship the source corresponding to the binary module it is distributing. That said, the community will probably limit itself to grumbling as long as CloudLinux follows through and comes back into compliance with the kernel's licensing. Nobody likes a delay in a source release, but short delays are usually tolerated, grudgingly.

So the licensing part of this episode is, probably, resolved, but it is also interesting to look at why the lve module needs GPL-only symbols in the first place. The specific symbols it needs relate to the creation of binary attributes (essentially unstructured sysfs files) in the sysfs virtual filesystem. Much of the internal sysfs interface is GPL-only, and the functions for the management of binary attributes (which are generally discouraged) are certainly so. It turns out that lve creates binary attributes for a reason that would certainly never get through a proper review:

We do a "hack", which is not a pretty one, populating /sys with .htaccess files. This is really needed only by shared hosters, where one of the end users on the server, could be a hacker and could create symlinks that would later be followed by apache to read privileged information.

As a comprehensive security solution, this implementation leaves just a little bit to be desired.

CloudLinux is trying to reduce the amount of damage that could be done should an attacker successfully compromise a daemon running on the system. Kernel developers have been working for years to mitigate just this sort of threat; the results of that work take the form of security modules, namespaces, and more. Had CloudLinux done its work in the open, it would certainly have been directed toward solutions that have support from others and years of experience in actual use. Solutions, in other words, that might actually deliver some additional security. Instead, they got a one-off "not pretty hack" that could never get upstream.

If CloudLinux follows through on its plan to close the module entirely, it will have to cease use of GPL-only symbols and, thus, will not be able to implement this particular mechanism. That may be enough to convince kernel developers to leave it alone. But it should leave others wondering what other surprises may be lurking inside—surprises that cannot be investigated, since the source for the module is not available.

There may well be a viable business in the creation of a well-supported distribution aimed at the needs of cloud providers. With solid security support and some top-tier user-space management tools, a company like CloudLinux could well produce an offering that others are willing to pay for. But the most robust and trustworthy server distribution will come from using (and improving) the work that many developers have done for years to create a useful containers implementation. It is hard to imagine that a proprietary solution put together by a small company can work as well. Compared to that, the risk of licensing troubles seems like a case of adding insult to injury.

(Log in to post comments)

CloudLinux busted — twice

Posted Jun 28, 2012 9:13 UTC (Thu) by epa (subscriber, #39769) [Link]

They haven't heard of the FollowSymlinks option in the Apache config file?

CloudLinux busted — twice

Posted Jun 28, 2012 11:29 UTC (Thu) by nix (subscriber, #2304) [Link]

The description of an XML configuration file as 'administrator-friendly' was an example of the trademark Corbet dry wit, right?

(And, .htaccess in /sys. Stunning, just stunning. What's the *rest* of their closed-source module like, I wonder? It could hardly be *more* embarrassing than that...)

CloudLinux busted — twice

Posted Jun 28, 2012 11:55 UTC (Thu) by wazoox (subscriber, #69624) [Link]

Is it me, or does the whole CloudLinux marketing blurb make it looks like a sham? Increase stability by enforcing limits through a poorly written proprietary kernel module? What is the technical logic supporting that?

CloudLinux busted — twice

Posted Jul 5, 2012 19:05 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]

They are following the reliable methodology of security through obscurity. If you hide the source code for the module, no-one can work out how to subvert it. Simple! We should apply this to all the LSMs too!

CloudLinux busted — twice

Posted Jul 6, 2012 5:53 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

SELinux people have an even better idea!

They give you away the source code, but make sure that once you start reading it you go mad before you even start thinking about subverting it.

CloudLinux busted — twice

Posted Jul 6, 2012 11:58 UTC (Fri) by spender (subscriber, #23067) [Link]

Seems someone forgot the history of LSM and Crispin Cowan ;) That's exactly what WireX wanted to do with LSM, a goal that influenced its design.

In case you forgot how that one went down, much to Crispin's chagrin:


CloudLinux busted — twice

Posted Jun 28, 2012 13:48 UTC (Thu) by dowdle (subscriber, #659) [Link]

I also have to wonder just why they are trying to implement yet another flavor of containers for Linux. The mainline has LXC but many people see that as a bunch of zombie parts people are making a monster out of. In the out-of-tree category there are Linux-VServer and OpenVZ... which are both very complete and mature container implementations that are released under a GPLv2 license. Both Linux-VServer and OpenVZ started life outside of the mainline kernel and grew and developed to big and mature such that they are never going to make their way into the mainline.

Anyway, I'm not sure what CloudLinux uses... but I would imagine bits and pieces of at least one of those, right? Or have they really created a completely new flavor of containers?

CloudLinux busted — twice

Posted Jun 28, 2012 19:50 UTC (Thu) by geofft (subscriber, #59789) [Link]

We do a "hack", which is not a pretty one, populating /sys with .htaccess files. This is really needed only by shared hosters, where one of the end users on the server, could be a hacker and could create symlinks that would later be followed by apache to read privileged information.

So, uh, why can't they just write their own file system that just exposes a .htaccess in every single directory, and union-mount / overlayfs / aufs it on top of /sys?

CloudLinux busted — twice

Posted Jun 29, 2012 7:10 UTC (Fri) by ekj (guest, #1524) [Link]

Rube Goldberg would be proud !

A separate file-system, union-mounted on top of every filesystem in the machine, all in order to avoid actually configuring Apache correctly.

CloudLinux busted — twice

Posted Jun 29, 2012 11:25 UTC (Fri) by robbe (guest, #16131) [Link]

What about a kernel module that on every open() looks if the file has "<VirtualHost" somewhere in it, and rewrites the configuration to something safe on read() in this case? This wonderful piece of technology would need to be closed-source, of course.

CloudLinux busted — twice

Posted Jul 5, 2012 14:51 UTC (Thu) by njs (guest, #40338) [Link]

Of course. I know that if *I* put work into programming such a unique feature, I'd certainly not want anyone else to be able to see.

CloudLinux busted — twice

Posted Jul 8, 2012 3:49 UTC (Sun) by cloudlinux (subscriber, #81222) [Link]

At least now I know what it takes to be noticed by LWN :)

My name is Igor Seletskiy and I am CEO of CloudLinux.
First of all let me start by apologizing to kernel & Linux communities. What happened was internal company miscommunication. Source RPMs for the modules are available from our source repositories.
Just in case -- I am not trying to say that it I told my guys to deploy it to source repository all along. I was just sure that we are not using GPL symbols in the module - as the issue of licensing and using GPL symbols was discussed a few times internally.

Now, if you don't mind -- let me respond to few questions raised by the article:
1. Everyone who uses our module also uses our kernel. Our kernel is slightly modified version of OpenVZ kernel. The mainstream kernel just doesn't have everything that we need.
2. We don't compete with OpenVZ/V-Server -- our client base is shared hosting providers - not VPS/cloud/etc.... Think of a server with 500 to 5,000 clients on it, each having a few websites. We did implement new "flavor" of linux containers specific for this case. No rocket science here -- just serving a need of specific market niche

3. Our revenue is not based on support, while we do provide 24/7 support, our clients are very technical -- and usually don't need it. Our revenue is from the software licensing / features that we provide.
4. Our marketing might look like total BS if you think about generic use of server. If you think of shared hosting, where you have hundreds of users that you only know by credit card number they provided, running code that was never vetted on your server -- and making sure that one will not bring down the whole server for everyone else -- you will see how it might help. It actually quite effective, and we have 8000+ servers licensed for that reason. I know it is a tiny number, but the market is small, and this is all paying customers.
5. Yes, we heard about FollowSymLinks. The problem is that user1, who might turn out a hacker would just do this:
ln -s ~user2/public_html/config.php read1.html
ln -s ~user2/pbulic_html/configure.php read2.html
And then read them -- often hitting config files for wordpress and other apps.
FollowSymLinksIfOwnerMatch is no help as well, as it can be overwritten via .htaccess
There are patches for apache to prevent that - but there is a race condition that circumvents that protection anyway -- so it doesn't really help.
Anyway, we did solve that part, but what we didn't solve was part of reading directory info via something like:
ln -s /etc/valiases 1.html
And that would give user all the domains on the server.
We decided it would be easier for us at that moment to add .htaccess to some places, and protect /proc & other directories via kernel module.
Please, note that many things went into decision, including:
a) A believe that we needed solution soon
b) The complexity of pushing changes to apache (as it is controlled by control panel vendors -- and it really takes time until they include our patch)
Was it a best solution - defiantly not. We knew that, but we didn't go for the best - we were going for something that we could do in short term, and that would provide protection against that particular type of attacks.

6. I don't know why we chose to do it via module instead of creating another filesystem. Maybe it would be better solution.

7. SELinux -- FUNNY :)

Once again -- my apologies to anyone who felt offended.

CloudLinux busted — twice

Posted Jul 8, 2012 13:00 UTC (Sun) by spender (subscriber, #23067) [Link]

Speaking of FollowSymlinksIfOwner, on Sunday (of last week) I developed a proper race-free, group-based kernel implementation. As of yesterday, it was also pushed to both stable branches of grsecurity (so we have implementations for 2.6.32, 3.2, and 3.4). My checks are performed correctly against every link followed while walking a path (and against the actual targets used along the way to produce the final resolved path). I've noted that it's not possible to implement an equivalent protection by anything but GPL code, as you need to interpose at a location deep in VFS internals. The only exported GPL'd LSM API even is called prior to the necessary interception point.

So I have to question the merit of using closed source to do a weakened form of something that's already done correctly, for free under the GPL ;)


CloudLinux busted — twice

Posted Jul 8, 2012 18:42 UTC (Sun) by cloudlinux (subscriber, #81222) [Link]


Did you do the check on open or on creation of the symlink?
We explored both ways -- and rejected them.
If you are doing check on open -- it must carry a price tag. On a shared hosting server, open is pretty common operation, and you just made it costly.
Also, to properly defend against race conditions it has to be atomic, which makes it even costlier.
We really didn't want to make performance worse, if we didn't have to.

If you did it on creation of symlink (we actually know about hosting company who did it that way) -- it wouldn't work for us as well, as it breaks some installation scripts, like WHMSonic ShoutCast Admin Pro (the script makes a link as a user, for each user, to sc_serv binary that is owned by root). I was pretty sure there other apps like that - and I didn't want to deal with all the support questions.

Or am I missing something?

CloudLinux busted — twice

Posted Jul 8, 2012 20:56 UTC (Sun) by spender (subscriber, #23067) [Link]

It's on following of a symlink and only involves a simple uid check, no rebuilding of paths or anything else that would have to be done via a module hack. It's even simpler and less "expensive" than the symlink restrictions in grsecurity (that have been copied upstream). It does not impact performance at all. It doesn't have to use atomic operations (if that's what you're implying) -- it simply has to check the same inodes the kernel resolves when following a link.


CloudLinux busted — twice

Posted Jul 8, 2012 21:16 UTC (Sun) by spender (subscriber, #23067) [Link]

BTW the mention of "open" or "create" as the check points is troubling. Path walking code is used by many more areas of the kernel than just through the "open" syscall. I'm making a prediction here, but it sounds like maybe you created a module that hooked a syscall and as a result it was slow and in fact just reproduced the same userland race (and made it even worse).

By hooking "open" you would have performed the same copying from userland of the opened filename (or maybe just accessed it directly!), then probably did an lstat and stat from the kernel against the same name (with the associated slowness and race). This would have introduced an additional race: the attacker wouldn't have to touch the filesystem to exploit a race, but could just replace the filename string with a different one between when your "open" hook and the real kernel's "open" executed.

Like I said, this is a feature that needs to be implemented deep in VFS internals to be correct -- a closed-source implementation will either be weaker or introduce additional races.

This is the patch:


CloudLinux busted — twice

Posted Jul 9, 2012 14:50 UTC (Mon) by cloudlinux (subscriber, #81222) [Link]


We patch a lot in the kernel, and it really doesn't matter for us open source / closed source for most of the minor things.

I guess if it is portable to 2.6.18 kernel as well, and we can port kernel changes to prevent creation of hard links to files owned by someone else to that kernel -- it might work for us.

What I would need to research first though -- if any control panel installs symbolic links to files owned by root into virtual hosts that are not for end user (like for webmail, etc...). If they do -- this solution would break it.

Anyway -- it does look very good, thank you!

Copyright © 2012, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds