CloudLinux busted — twice [LWN.net]

CloudLinux busted — twice

Posted Jul 8, 2012 3:49 UTC (Sun) by cloudlinux (subscriber, #81222)
Parent article: CloudLinux busted — twice

At least now I know what it takes to be noticed by LWN :)

My name is Igor Seletskiy and I am CEO of CloudLinux.
First of all let me start by apologizing to kernel & Linux communities. What happened was internal company miscommunication. Source RPMs for the modules are available from our source repositories.
http://repo.cloudlinux.com/cloudlinux/
Just in case -- I am not trying to say that it I told my guys to deploy it to source repository all along. I was just sure that we are not using GPL symbols in the module - as the issue of licensing and using GPL symbols was discussed a few times internally.

Now, if you don't mind -- let me respond to few questions raised by the article:
1. Everyone who uses our module also uses our kernel. Our kernel is slightly modified version of OpenVZ kernel. The mainstream kernel just doesn't have everything that we need.
2. We don't compete with OpenVZ/V-Server -- our client base is shared hosting providers - not VPS/cloud/etc.... Think of a server with 500 to 5,000 clients on it, each having a few websites. We did implement new "flavor" of linux containers specific for this case. No rocket science here -- just serving a need of specific market niche

3. Our revenue is not based on support, while we do provide 24/7 support, our clients are very technical -- and usually don't need it. Our revenue is from the software licensing / features that we provide.
4. Our marketing might look like total BS if you think about generic use of server. If you think of shared hosting, where you have hundreds of users that you only know by credit card number they provided, running code that was never vetted on your server -- and making sure that one will not bring down the whole server for everyone else -- you will see how it might help. It actually quite effective, and we have 8000+ servers licensed for that reason. I know it is a tiny number, but the market is small, and this is all paying customers.
5. Yes, we heard about FollowSymLinks. The problem is that user1, who might turn out a hacker would just do this:
ln -s ~user2/public_html/config.php read1.html
ln -s ~user2/pbulic_html/configure.php read2.html
...
Etc
And then read them -- often hitting config files for wordpress and other apps.
FollowSymLinksIfOwnerMatch is no help as well, as it can be overwritten via .htaccess
There are patches for apache to prevent that - but there is a race condition that circumvents that protection anyway -- so it doesn't really help.
Anyway, we did solve that part, but what we didn't solve was part of reading directory info via something like:
ln -s /etc/valiases 1.html
And that would give user all the domains on the server.
We decided it would be easier for us at that moment to add .htaccess to some places, and protect /proc & other directories via kernel module.
Please, note that many things went into decision, including:
a) A believe that we needed solution soon
b) The complexity of pushing changes to apache (as it is controlled by control panel vendors -- and it really takes time until they include our patch)
Was it a best solution - defiantly not. We knew that, but we didn't go for the best - we were going for something that we could do in short term, and that would provide protection against that particular type of attacks.

6. I don't know why we chose to do it via module instead of creating another filesystem. Maybe it would be better solution.

7. SELinux -- FUNNY :)

Once again -- my apologies to anyone who felt offended.

(Log in to post comments)

CloudLinux busted — twice

Posted Jul 8, 2012 13:00 UTC (Sun) by spender (subscriber, #23067) [Link]

Speaking of FollowSymlinksIfOwner, on Sunday (of last week) I developed a proper race-free, group-based kernel implementation. As of yesterday, it was also pushed to both stable branches of grsecurity (so we have implementations for 2.6.32, 3.2, and 3.4). My checks are performed correctly against every link followed while walking a path (and against the actual targets used along the way to produce the final resolved path). I've noted that it's not possible to implement an equivalent protection by anything but GPL code, as you need to interpose at a location deep in VFS internals. The only exported GPL'd LSM API even is called prior to the necessary interception point.

So I have to question the merit of using closed source to do a weakened form of something that's already done correctly, for free under the GPL ;)

-Brad

CloudLinux busted — twice

Posted Jul 8, 2012 18:42 UTC (Sun) by cloudlinux (subscriber, #81222) [Link]

Brad,

Did you do the check on open or on creation of the symlink?
We explored both ways -- and rejected them.
If you are doing check on open -- it must carry a price tag. On a shared hosting server, open is pretty common operation, and you just made it costly.
Also, to properly defend against race conditions it has to be atomic, which makes it even costlier.
We really didn't want to make performance worse, if we didn't have to.

If you did it on creation of symlink (we actually know about hosting company who did it that way) -- it wouldn't work for us as well, as it breaks some installation scripts, like WHMSonic ShoutCast Admin Pro (the script makes a link as a user, for each user, to sc_serv binary that is owned by root). I was pretty sure there other apps like that - and I didn't want to deal with all the support questions.

Or am I missing something?

CloudLinux busted — twice

Posted Jul 8, 2012 20:56 UTC (Sun) by spender (subscriber, #23067) [Link]

It's on following of a symlink and only involves a simple uid check, no rebuilding of paths or anything else that would have to be done via a module hack. It's even simpler and less "expensive" than the symlink restrictions in grsecurity (that have been copied upstream). It does not impact performance at all. It doesn't have to use atomic operations (if that's what you're implying) -- it simply has to check the same inodes the kernel resolves when following a link.

-Brad

CloudLinux busted — twice

Posted Jul 8, 2012 21:16 UTC (Sun) by spender (subscriber, #23067) [Link]

BTW the mention of "open" or "create" as the check points is troubling. Path walking code is used by many more areas of the kernel than just through the "open" syscall. I'm making a prediction here, but it sounds like maybe you created a module that hooked a syscall and as a result it was slow and in fact just reproduced the same userland race (and made it even worse).

By hooking "open" you would have performed the same copying from userland of the opened filename (or maybe just accessed it directly!), then probably did an lstat and stat from the kernel against the same name (with the associated slowness and race). This would have introduced an additional race: the attacker wouldn't have to touch the filesystem to exploit a race, but could just replace the filename string with a different one between when your "open" hook and the real kernel's "open" executed.

Like I said, this is a feature that needs to be implemented deep in VFS internals to be correct -- a closed-source implementation will either be weaker or introduce additional races.

This is the patch:
http://grsecurity.net/~spender/symlinkown.diff

-Brad

CloudLinux busted — twice

Posted Jul 9, 2012 14:50 UTC (Mon) by cloudlinux (subscriber, #81222) [Link]

Brad,

We patch a lot in the kernel, and it really doesn't matter for us open source / closed source for most of the minor things.

I guess if it is portable to 2.6.18 kernel as well, and we can port kernel changes to prevent creation of hard links to files owned by someone else to that kernel -- it might work for us.

What I would need to research first though -- if any control panel installs symbolic links to files owned by root into virtual hosts that are not for end user (like for webmail, etc...). If they do -- this solution would break it.

Anyway -- it does look very good, thank you!