Not logged in
Log in now
Create an account
Subscribe to LWN
LWN.net Weekly Edition for May 23, 2013
An "enum" for Python 3
An unexpected perf feature
LWN.net Weekly Edition for May 16, 2013
A look at the PyPy 2.0 release
This message brought to you by the "Winning Friends and Influencing People Network."
poor social estrategy
Posted Jan 7, 2005 20:16 UTC (Fri) by hmh (subscriber, #3838)
OTOH, sending patches of any sort to Linus or A.M. if you are not one of their trusted senders is not very effective. Sending it to the security teams of Debian, Fedora or SuSE would probably have been a much faster path.
And, as you said, the tone of the email doesn't improve the chances of a better direct channel to Linus and A.M. any :-)
Posted Jan 7, 2005 21:24 UTC (Fri) by Psychopath (guest, #4501)
If one of them makes the decision whether to include a security fix or not include it dependant on the tone of the reporting email I wouldn't want to use this product :)
Posted Jan 7, 2005 22:44 UTC (Fri) by sbergman27 (guest, #10767)
Perhaps some sort of directory service is needed:
Caller: Operator, I need to know who to report this security vulnerability to.
Operator: Please explain the nature of the vulnerability.
Caller: <Fill in the blank>
Operator: Hold for the email address, please...
Posted Jan 8, 2005 14:33 UTC (Sat) by PaXTeam (subscriber, #24616)
the grsecurity announcement you can see here is *not*, i repeat, *not* the same as the bugreports that Spender or me sent (and kept resending) Linus and/or Andrew. the announcement quotes my own mail verbatim towards the end (it even says so... did anyone bother to read that at all?), and i don't think it had anything offensive in it. my personal gripe is that for 3 weeks not a single acknowledgement arrived in my mailbox, i don't think that's the way the chief developers are supposed to handle security issues (however small or irrelevant they may have been in this case - it takes a one liner to tell us so). as for going to other persons - who would that be? i don't want to talk to anyone i don't personally trust, and this immediately excludes the vendor-sec subscribers (observe the uselib() bug leak). anyone else left? with that said, i personally didn't agree with the chastising of their procedure (or rather, lack thereof) in public, but then it wasn't my announcement either.
as for Andrew's comment about anyone being able to DoS a linux box with malloc/memset... that attitude of downplaying bugs is pretty sad if he really meant it (and if true, that's a pretty sad state of affairs for linux). for one, we have CONGIG_SWAP, vm accounting and the OOM killer for a reason, second, there's no built-in recovery mechanism for the mlock/expand_down bug, so it is more serious than this malloc/memset issue.
Posted Jan 8, 2005 16:22 UTC (Sat) by sbergman27 (guest, #10767)
But it is important to understand that one can't just pick up the "Bat Phone" and have Linus or Andrew on the other end. Those days are gone.
I'm sure many would appreciate reporters of vulnerabilities following the prescribed procedures before subjecting the rest of us to increased (albeit temporary) risk.
None of this is to imply that most in the community do not appreciate the reporting of vulnerabilities, of course.
Posted Jan 8, 2005 19:31 UTC (Sat) by PaXTeam (subscriber, #24616)
second, you're suggesting contacting the subsystem maintainer(s). to the best of my knowledge, the VM (to which expand_stack/mlockall belong, i think) has no such person, nothing relevant turns up in MAINTAINERS at least. so that leaves Linus/Andrew. do i have a 'cause to complain' now?
third, i beg to differ on your assertion that i was the cause for your increased risk. if anyone subjected you to risk then it is the person(s) who didn't bother to deal with the bugreports (you realize that i even provided a patch which is now included in -ac verbatim). or as in the case of the uselib() bug, didn't do so in a timely manner.
Posted Jan 8, 2005 22:45 UTC (Sat) by sbergman27 (guest, #10767)
That's just off the top of my head. Being an armchair bug reporter, I can't be sure that the email address is current. I think it is, though. Get to know the procedures and relevant people. How can you guys maintain "grsecurity" and not know all this already?
On a side note, have you any relation to D. J. Bernstein?
There is a family resemblance.
Posted Jan 8, 2005 23:05 UTC (Sat) by PaXTeam (subscriber, #24616)
as for DJB: DJB didn't give any time for the authors he notified (well, in case of nasm it was one day, but that was probably the exception, not the rule). contrast that to my 2-3 weeks and several emails to establish contact before going public. pick a better example next time.
Posted Jan 8, 2005 23:36 UTC (Sat) by sbergman27 (guest, #10767)
"vm maintainer" "linux kernel"
(and you can just hit "I'm feeling lucky" because it is the first hit),
you will get a "kernel trap" article.
You can just type "www.kerneltrap.org", though, because this particular example happens to be on the front page of today's kerneltrap.org, thought the article is from Dec 27, 2004.
And it mentions that:
"An interesting dicussion on the lkml examined the efficiency of the inode cache in the 2.4 Linux kernel [forum], discussing several tunables primarily helpful to systems serving large NFS or Samba mounts. In particular, a slowdown was reported on such a system easily reproducible by doing a find / while cat'ing large files to /dev/null. In a discussion between 2.4 maintainer Marcelo Tosatti [interview], 2.6 maintainer Andrew Morton [interview] and VM maintainer Andrea Arcangeli [interview], it was decided that this was likely due to too small of an inode cache hash table resulting in a large number of collisions. For the work case in question, some tunables looked to prove helpful. Going forward, effort might be made in 2.6 or beyond to improve the inode cache."
As to my ditching your original question. do you subscribe to LKML? Do you read it?
That's where I learned this stuff, and you should too.
You ditched my question, however. Why do the grsecurity maintainers, who just released exploit code in the wild without following prescribed procedures, putting us all at risk, not already know all this?(!) How much can we trust you?
Posted Jan 9, 2005 0:18 UTC (Sun) by PaXTeam (subscriber, #24616)
as for lkml, i'm not subscribed but sometimes i scan it for interesting posts/topics.
you keep accusing me (btw, i'm not a grsecurity developer, i develop PaX only) of not following 'prescribed procedures' yet you *still* haven't shown a *single* reference to said procedure. please, stop this generic mumbo-jumbo and just post the URL to the document that clearly and unambiguously describes the 'official' procedure for reporting linux security bugs (and which doesn't involve Linus/Andrew and prescribes >3 weeks of waiting time, else you'd just prove my point). short of that, you have no basis for your claims (how can i not follow something you don't seem to know yourself either?).
we'll talk about being responsible when you manage to answer the question above.
Posted Jan 9, 2005 0:51 UTC (Sun) by sbergman27 (guest, #10767)
I will, however, take this opportunity to agree with you that if such a document does not exist, it very much should. People should *not* have to read LKML to know these things. If grsecurity had not released exploit code to the world, I would even be sympathetic. However, when an organization actually releases an exploit, it needs to be held to higher standards than the average Joe. The research on proper proceedure needs to be done, and thoroughly.
If you wish, we can continue this discussion via private email. I'm at steve_AT_rueb.com.
Live Long and Prosper,
Posted Jan 9, 2005 17:13 UTC (Sun) by PaXTeam (subscriber, #24616)
keep the pissing contests for IRC
Posted Jan 9, 2005 22:07 UTC (Sun) by dw (subscriber, #12017)
Posted Jan 10, 2005 12:13 UTC (Mon) by coolian (guest, #14818)
Posted Jan 10, 2005 12:29 UTC (Mon) by zorgan (guest, #4016)
Posted Jan 11, 2005 8:11 UTC (Tue) by Wol (guest, #4433)
After all, it was *Andrea* that *wrote* the thing in the first place, not Linus...
Posted Jan 11, 2005 20:11 UTC (Tue) by zorgan (guest, #4016)
Posted Jan 10, 2005 13:07 UTC (Mon) by philips (guest, #937)
I've being hitting this bug (as bug, but not securinty hole) several times before.
I hope that now it will be fixed.
Posted Jan 9, 2005 1:36 UTC (Sun) by zorgan (guest, #4016)
Posted Jan 9, 2005 12:48 UTC (Sun) by PaXTeam (subscriber, #24616)
For what it is worth...
Posted Jan 9, 2005 2:33 UTC (Sun) by jd (guest, #26381)
I understand politics, the fights over who has to deal with what, etc. I understand them and I detest them. Which is easier for a person to do? Hit the "delete" button, or hit a "forward" button to get the message to the right person? Seems to me, it's still one button. So arguing "did it go to the right person" is largely missing the very point of such an argument. If it did NOT go to the right person, but DID go to someone who has to know who the right person was, why did it NOT get to the right person?
Secondly, are we interested in who-did-what/who-didn't-do-what fights, or are we interested in Linux being the best OS that ever was and ever will be? If we are more interested in the latter, then someone should include the patches! It's that simple. At the end of the day, there's only one sure way of making progress, and that's to move forward.
Lastly, do I think some of the security developers tend to be rough around the edges? Well, yes. I had a few fights on my hands, over including patches into FOLK that others wanted to keep to themselves. I included them anyway. Nobody, but NOBODY, is going to tell me that Linux users deserve second-best, should live in ignorance of what's out there, or should have the difficulties inherent in merging multiple patches.
I wanted an easy showcase for technology, and that's exactly what I produced, whether others liked it or not. It was my opinion then, and still mine today, that knowledge is exactly like power - something to be distributed as widely as humanly possible, for the betterment of all.
Sure, I'm crazy. Sure, if Linux were to include everything that was important to a substantial fraction of it's users, the kernel would be nearly twice the size it is now. But I can survive being crazy, and those who download source can survive a little longer download time. Neither of these will kill. If substantial, wholly justified, scares into Linux' security or stability hit the community, then those might very well kill Tux.
I think the potential consequences make a LITTLE bit of effort by SOMEBODY in the Linux Kernel Development environ into getting Linus to install these patches, and in full, well worth any likely price. We can always back things out, if they turn out to be an error or become obsolete. Intermezzo, the original Ext FS, and a few other favourites have gone. DevFS is going to. Patches have been introduced with resistance from Linus - IIRC, he once said Linux would never run on anything but an ix86.
All of these just show that nobody is perfect. But they also show that those who aren't perfect should allow themselves to be steered by others in areas they're not so skilled on.
Posted Jan 10, 2005 8:33 UTC (Mon) by Wol (guest, #4433)
Simple. And also, what PaxTeam appear to be missing.
Linus is ONE person. There are only 24 hours in a day. There is a *high* probability that the PaxTeam message never got to Linus' eyeballs...
THAT is why there is all this maintainers/lieutenants business. To reduce the workload on Linus to the point where it is manageable.
PaxTeam isn't subscribed to LKML. Why? Because "there's too much"? Bearing in mind Linus probably gets a hell of a lot of mail addressed to him personally, then he has to keep an eye on LKML, then he actually has a job to do, then he has to discuss things with his lieutenants... I'm afraid a mesage from a total unknown has low priority. And that fact that it claims to report a security vulnerability is quite likely to get it classified as "crying wolf" - I bet loads of people do cry wolf - intentionally or down to their own incompetence.
At the end of the day, the "proper channels" are there for a reason - in other words the system would collapse if they weren't there. And the fact PaxTeam don't take LKML says just about everything else - Oh and if they don't know Andrea Arcangeli and/or Rik van Riel are the people to talk to about VM, then they haven't been watching kernel development. Who remembers the VM-wars of 2001? :-) Hint to PaxTeam - at the very least, read Kernel Traffic *in* *depth*.
Posted Jan 10, 2005 11:26 UTC (Mon) by PaXTeam (subscriber, #24616)
now that you know some background, tell me again, 1. how much more we should have waited, 2. why we shouldn't have contacted Linus/Andrew in the first place, 3. why we should have contacted Alan first (who is explicitly not the security contact anymore), 4. why we should have contacted a VM hacker first (none of whom is a security contact either, not even for their respective employer, let alone linux/VM in general).
see, i've been in the security industry for some number of years now, and i know quite well what best practices are (everyone's got his own, but there're some common elements):
rule 1: you contact the explicit security contact first. for linux this used to be Alan himself, nowadays it's vendor-sec (yes, that means you're not supposed to deal with individual distros, that's why vendor-sec was established in the first place). except they proved to unreliable, not to mention that it's *impossible* to contact them in a secure way (they don't have a PGP key).
rule 2: short of such a security contact, you begin contacting the 'people in control', from top to down, not the other way around. for companies that's relevant because the chain of control also represents the chain of responsibility. you can argue that open source/free software projects are free of chain of control, but they're not free of responsibility. i believed and still believe that we did the right thing when we began contacting Linus, then Andrew and were about to contact Alan when external events intervened.
> THAT is why there is all this maintainers/lieutenants business.
except the VM has no explicitly listed maintainer. but yes, i can guess who the main contributors are, but that doesn't make them a security contact (remember, we only wanted to get feedback, be told what to do next, and *not* to force Linus or anyone to actually manage the issue). it makes them the right person to actually fix the bug, but that's only the second step after the initial contact.
> PaxTeam isn't subscribed to LKML. Why? Because "there's too much"?
correct, i have a day job (unrelated to linux), family and friends, i can't handle that email load (and there's more in my world than lkml). i don't know where you got that i didn't like lkml, if i wasn't sympathetic to linux, i would have posted everything to bugtraq a month ago (contrast that to the recent DJB case).
> And that fact that it claims to report a security vulnerability is quite
> likely to get it classified as "crying wolf"
i provided a proof of concept exploit (which you would know if you had actually read the announcement and posts here).
Posted Jan 10, 2005 18:41 UTC (Mon) by geomon (guest, #27127)
From my perspective, you are trying to defend yourself when you shouldn't have to.
"understand please that we (well, spender at least) already had had a working two-way email connection with Linus. during the holidays..."
That is a problem. You have, of course, identified the source of the problem and have already recommended a solution.
"1. how much more we should have waited..."
For a legitimate bug? Not long I would hope (I am a user!).
"2. why we shouldn't have contacted Linus/Andrew in the first place"
You should have if there isn't an appropriate point-of-contact already established. That is the root cause of the problem.
"3. why we should have contacted Alan first (who is explicitly not the security contact anymore)"
You shouldn't have to. If Alan is not *the* person for security matters, that would be inappropriate as well.
"4. why we should have contacted a VM hacker first (none of whom is a security contact either, not even for their respective employer, let alone linux/VM in general)."
I've got to agree with this one too. Why should I go to the grocery store to get my car's front end aligned?
"see, i've been in the security industry for some number of years now, and i know quite well what best practices are (everyone's got his own, but there're some common elements)"
You are projecting defensiveness again. Give it a rest - you've made your point.
"rule 1: you contact the explicit security contact first."
An explict security contact should have been established *first*. That has either not happened yet, or the point-of-contact has changed. In either case, if the information is not readily available, then NO credible process exists for submitting security patches.
I share your shock at that prospect.
"for linux this used to be Alan himself, nowadays it's vendor-sec (yes, that means you're not supposed to deal with individual distros, that's why vendor-sec was established in the first place)."
That may work for individual vendors. How about establishing a Linux security working group that is composed of security contacts from the vendors?
What is missing in this discussion is a single point-of-contact, regardless of how it is composed, with contact information posted at kernel.org, kerneltrap.org, linux.org, or lwn.net.
"rule 2: short of such a security contact, you begin contacting the 'people in control"
See Rule 1.
"> PaxTeam isn't subscribed to LKML. Why? Because "there's too much"?
correct, i have a day job (unrelated to linux),"
And you shouldn't HAVE to subscribe to a mailing list to get a point-of-contact. That is pure stupidity.
That's why web pages were invented: "http://groups-beta.google.com/group/alt.hypertext/msg/395..."
"> And that fact that it claims to report a security vulnerability is quite likely to get it classified as "crying wolf"
i provided a proof of concept exploit (which you would know if you had actually read the announcement and posts here)."
The fact is, every event of security should be treated as a serious condition. It should not be the job of the submitter to determine whether the issue is serious or not; they may not be security experts but probably have noted a serious condition that they cannot explain by themselves.
This issue is a confidence buster if the community cannot produce a credible notification scheme. One of the key arguments that Linux advocates have used for years in defending the security of its products has been the claim that "many eyes" are better than code obfuscation. I have observed how this submitter has been treated and would question why ANYONE would submit a security concern to the community at this point.
Continuing to blame the person who submits the bug report, regardless of how they did it, is unacceptable. That smacks of the same arrogance that drove us to use Linux in the first place. Linux developers need to provide certainty to the user community that their concerns will be addressed and not arbitrarily dismissed.
Posted Jan 10, 2005 19:49 UTC (Mon) by jd (guest, #26381)
There is very little value in a system that does everything superbly, with absolutely the minimum possible latency, total stability, yadda yadda yadda, if the next day some idiot turns you finely-honed silicon masterpiece into a spammer zombie.
Yes, I think tempers may be running a little high, and that's probably not helping the situation. To me, there are only two issues at stake - how to get ALL of the fixes in place, ASAP, and how to ensure that future problems are addressed as fast as humanly possible.
Posted Jan 10, 2005 21:30 UTC (Mon) by Ross (subscriber, #4065)
As for reporting, the grsecurity team followed what they thought
were the best ways to report the problems. The fact they guessed "wrong"
(which is not actually a fact ... just an unsupported assertion on
several people's part) is not their fault. I can't fault them for it.
I'm still unclear as to what the "correct" way would have been. There
was absolutely no documentation on how to report such problems. You
claim they "should have known" that Andrea is the right contact based on
years-old information. Using that reasoning Alan Cox would be the right
contact... but we know that is incorrect.
Sure Linus gets a lot of email. He probaly doesn't read a lot of it.
The point of them explaining they had "established two-way link"
hat Linus had read and responded to the original message ... but not
fixed the problem once the bug became exploitable. You are complaining
that they didn't read the lkml ... but what, exactly, would that have
accomplished? Posting to the lkml is every bit as public as this
disclosure. Or are you saying you aren't allowed to report bugs without
They were very patient. People are claiming this is like DJB's arrogant
security reports. It is not even close. Several people have claimed
these issues were released as punishment and that the use_lib() thing is
unrelated. The point that had been made is that these issues were being
released because someone else leaked the use_lib() bug... otherwise it
would have all be taken care of together. Once the information is out it
is in everyone's best interest that is is fully and widely disclosed.
Secrecy is only useful for so long... other people may find the bug and
information tends to leak out. A month and change seems more than
Someone else said that they shouldn't have mailed the kernel people but
one or more of the distribution maintainers. I can't disagree more.
Report security bugs to the upstream. They will be fixed in a single
place, in the right way -- not to mention the advantage of limiting the
number of people the information is sent to.
Why can't MAINTAINERS have an entry for kernel security bug reports? Why
not setup a separate mailing list or alias which will magically go to the
right people (not including a bunch of distro maintainers)? Why not
create a PGP key for it to accept encrypted messages? I think everyone
agrees there is a problem. We may not agree on who is at fault but
aren't there _very_ easy ways to fix it?
Posted Jan 8, 2005 3:08 UTC (Sat) by Junior_Samples (guest, #26737)
Nevertheless, Linux probably needs a "kernel security officer" who would be the initial
contact point when a vulnerability is discovered. It is shameful that
these reports went unanswered for almost a month.
Posted Jan 8, 2005 11:15 UTC (Sat) by pointwood (guest, #2814)
That's what I was thinking too, unless they already have some kind of process for that?
Posted Jan 8, 2005 17:18 UTC (Sat) by wolfrider (guest, #3105)
grsecurity 2.1.0 and kernel vulnerabilities
Posted Jan 7, 2005 21:20 UTC (Fri) by uravanbob (subscriber, #4050)
Posted Jan 8, 2005 0:37 UTC (Sat) by mmarq (guest, #2332)
And the key word here, i belive, is "development" !!... Linux were always in a constant state of development. There never were a relly true 'stable version' out of the initial series of 'stable marked' kernels. But that is my opinion. Also, IMO, the best solution for these problems is to open another tree, after a new *wild* unstable serie is open...
...lets say a 'stable-final', where only bug fixing and security holes *should* be mended, and not as we have today where pretty major features changes are made on stable series...
...there shoudn't be any clash among a wild 'unstable' and a 'stable-final' flowing in simultaneos, since they are so different in purpouse.
And that is not asking for to much, i belive, since the it is in all the interest of the commercial party of Linux world, to have a 'Titanium' solid product,... and they have the resources to make it rock.
Posted Jan 8, 2005 15:21 UTC (Sat) by Klavs (subscriber, #10563)
Posted Jan 8, 2005 18:41 UTC (Sat) by mmarq (guest, #2332)
So 2.4 is a real evidence that a stable serie are just better continued versions of the unstable ones.
"IMHO a real "stable" kernel, will result in it lacking features..."
How is that ?... i mean, if 2.4.25 was at the point where there wasent any more features added, if efficiently decided it should had been 2.4.20 or 21, then afterwards the tree could became 2.4.(??) stable-final, that is, no more features besides bug features and security holes mended(dont know but guess more features were added after 2.4.25). So what's wrong with this view ?
I mean, you should have had a closed stable serie, and instead had a live *real* stable-final for only bug, security holes, and drivers, and at the same time have the normal 2.5 or 2.6(stable)... If people wanted more features they should had waited for 2.6, and had helped more in its development, instead of trying pushing it to 2.4 where they fealt more save.
So in conclusion i belive is obvious, that a *real* stable-final, 'IF WELL MANAGED', will help in the general development of Linux, because there is a sure place for things like the ones discussed in this article, and more skilled and inventive developers will be pushed to the next *real development* tree.
And the well managed part, intentionaly in bold(sorry), means that a stable-final(ex:2.4.50-FINAL) au contrary of the development tree, should be very very carefully planned, and should have only *ONE* final version number, after the necessary " pre " ;... and the process shoul only be repeated, after a lot of consideration(ex:2.4.55-FINAL) if results were not at all satisfactory.
I belive the community has enough room now, and Linux kernel enough features(well above average for servers, not quit enough for desktops) for it to go smoodly, without vacating any of the trees out of developers.
Posted Jan 8, 2005 18:02 UTC (Sat) by bluefoxicy (guest, #25366)
In this blog entry I talk about the problems with the 2.6 development model being used for a 'stable' tree, opening up by discussing PaX finally catching up on all the VM changes like it shouldn't have to until 2.8.
I then took a page out of whatever book the Ubuntu guys are going for and structured a low-maintenance, long support cycle release mechanism around the 2.6 development model, simply by shifting the model to the odd number branches (2.7) and using the even branches for bugfixes only (security holes are bugs).
Driver backporting can be invasive, and is more work than just bugfixes. Drivers requiring invasive changes (Reiser4 hacked up the FS core a lot) would be forbidden for "Stable" in my design. Drivers that are just drop-in and modify a Makefile and Kconfig slightly are ok, but remember that somebody has to maintain that.
My goal is a release cycle that produces long-term support for stable kernels and timely releases, without stacking too many stable releases on maintainers at once or too much maintenance on a stable release. Maintaining a stable kernel should take about 10-15 minutes of your time per day in net. That means that an hour or two spent on your weekends to merge in a backported security patch is the target.
Unfortunately it seems that I didn't produce a model that would make everybody happy. At the very least I have no support; but that's ok, because I have no experience designing development models, so I'm probably way out of line.
Still, get the damn ball rolling and come up with something.
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds