| |||||||||||||
poor social estrategypoor social estrategyPosted Jan 7, 2005 21:24 UTC (Fri) by Psychopath (guest, #4501)In reply to: poor social estrategy by hmh Parent article: grsecurity 2.1.0 and kernel vulnerabilities
"the tone of the email doesn't improve the chances of a better direct channel to Linus and A.M. any :-)"
If one of them makes the decision whether to include a security fix or not include it dependant on the tone of the reporting email I wouldn't want to use this product :)
(Log in to post comments)
poor social estrategy Posted Jan 7, 2005 22:44 UTC (Fri) by sbergman27 (guest, #10767) [Link] He did not say that "the decision whether to include a security fix or not include is [sic] dependant on the tone of the reporting email". He said that it would not help them to establish a direct channel to Linus or Andrew. A single person, even two, do not scale well, and the rest of the "contributors" need to learn who to report problems to. :-)
Perhaps some sort of directory service is needed:
Caller: Operator, I need to know who to report this security vulnerability to.
Operator: Please explain the nature of the vulnerability.
Caller: <Fill in the blank>
Operator: Hold for the email address, please...
poor social estrategy Posted Jan 8, 2005 14:33 UTC (Sat) by PaXTeam (subscriber, #24616) [Link] a bit late, but for the record:
the grsecurity announcement you can see here is *not*, i repeat, *not* the same as the bugreports that Spender or me sent (and kept resending) Linus and/or Andrew. the announcement quotes my own mail verbatim towards the end (it even says so... did anyone bother to read that at all?), and i don't think it had anything offensive in it. my personal gripe is that for 3 weeks not a single acknowledgement arrived in my mailbox, i don't think that's the way the chief developers are supposed to handle security issues (however small or irrelevant they may have been in this case - it takes a one liner to tell us so). as for going to other persons - who would that be? i don't want to talk to anyone i don't personally trust, and this immediately excludes the vendor-sec subscribers (observe the uselib() bug leak). anyone else left? with that said, i personally didn't agree with the chastising of their procedure (or rather, lack thereof) in public, but then it wasn't my announcement either.
as for Andrew's comment about anyone being able to DoS a linux box with malloc/memset... that attitude of downplaying bugs is pretty sad if he really meant it (and if true, that's a pretty sad state of affairs for linux). for one, we have CONGIG_SWAP, vm accounting and the OOM killer for a reason, second, there's no built-in recovery mechanism for the mlock/expand_down bug, so it is more serious than this malloc/memset issue.
poor social estrategy Posted Jan 8, 2005 16:22 UTC (Sat) by sbergman27 (guest, #10767) [Link] The proper procedure has been well known for some time. Determine the official maintainers for the parts of the kernel in question and report the bugs or vulnerabilities to them. Linus and Andrew depend upon them heavily for the sake of their own scalability. (This was all hashed out back during the "Linus Scalability Crisis" of a few years ago.) If Linus or Andrew happen to be the official maintainers for the relevant part of the kernel, then the reporter has cause to complain.
But it is important to understand that one can't just pick up the "Bat Phone" and have Linus or Andrew on the other end. Those days are gone.
I'm sure many would appreciate reporters of vulnerabilities following the prescribed procedures before subjecting the rest of us to increased (albeit temporary) risk.
None of this is to imply that most in the community do not appreciate the reporting of vulnerabilities, of course.
poor social estrategy Posted Jan 8, 2005 19:31 UTC (Sat) by PaXTeam (subscriber, #24616) [Link] i've got to love armchair bug reporters. where's the 'proper procedure' you're talking about? saying it exists doesn't answer my question, i want to know the details, URLs, etc. CREDITS says that Alan Cox was the former security contact point and vendor-sec is the new one which i don't trust for very good reasons (i'll remind you again of the uselib() bug leak).
second, you're suggesting contacting the subsystem maintainer(s). to the best of my knowledge, the VM (to which expand_stack/mlockall belong, i think) has no such person, nothing relevant turns up in MAINTAINERS at least. so that leaves Linus/Andrew. do i have a 'cause to complain' now?
third, i beg to differ on your assertion that i was the cause for your increased risk. if anyone subjected you to risk then it is the person(s) who didn't bother to deal with the bugreports (you realize that i even provided a patch which is now included in -ac verbatim). or as in the case of the uselib() bug, didn't do so in a timely manner.
poor social estrategy Posted Jan 8, 2005 22:45 UTC (Sat) by sbergman27 (guest, #10767) [Link] VM? Andrea Arcangeli. ( andrea@suse.de )
That's just off the top of my head. Being an armchair bug reporter, I can't be sure that the email address is current. I think it is, though. Get to know the procedures and relevant people. How can you guys maintain "grsecurity" and not know all this already?
On a side note, have you any relation to D. J. Bernstein?
There is a family resemblance.
poor social estrategy Posted Jan 8, 2005 23:05 UTC (Sat) by PaXTeam (subscriber, #24616) [Link] Andrea is a kernel hacker, he's not a security contact nor is he the maintainer of the VM (if you know otherwise, show me the proof). and you ditched my original question, so let me ask it again: what is the 'proper procedure' *you* were talking about?
as for DJB: DJB didn't give any time for the authors he notified (well, in case of nasm it was one day, but that was probably the exception, not the rule). contrast that to my 2-3 weeks and several emails to establish contact before going public. pick a better example next time.
poor social estrategy Posted Jan 8, 2005 23:36 UTC (Sat) by sbergman27 (guest, #10767) [Link] If you go to google (it's at http://www.google.com) and do a search on:
"vm maintainer" "linux kernel"
(and you can just hit "I'm feeling lucky" because it is the first hit),
you will get a "kernel trap" article.
You can just type "www.kerneltrap.org", though, because this particular example happens to be on the front page of today's kerneltrap.org, thought the article is from Dec 27, 2004.
And it mentions that:
"An interesting dicussion on the lkml examined the efficiency of the inode cache in the 2.4 Linux kernel [forum], discussing several tunables primarily helpful to systems serving large NFS or Samba mounts. In particular, a slowdown was reported on such a system easily reproducible by doing a find / while cat'ing large files to /dev/null. In a discussion between 2.4 maintainer Marcelo Tosatti [interview], 2.6 maintainer Andrew Morton [interview] and VM maintainer Andrea Arcangeli [interview], it was decided that this was likely due to too small of an inode cache hash table resulting in a large number of collisions. For the work case in question, some tunables looked to prove helpful. Going forward, effort might be made in 2.6 or beyond to improve the inode cache."
As to my ditching your original question. do you subscribe to LKML? Do you read it?
That's where I learned this stuff, and you should too.
You ditched my question, however. Why do the grsecurity maintainers, who just released exploit code in the wild without following prescribed procedures, putting us all at risk, not already know all this?(!) How much can we trust you?
poor social estrategy Posted Jan 9, 2005 0:18 UTC (Sun) by PaXTeam (subscriber, #24616) [Link] thanks for educating me on how to use google, but what you failed to answer is how i was supposed to figure out that the 'proper procedure' for reporting a *security bug* in the VM means a google search for specific keywords and accepting the first hit (it made me smile no end, i suggest you post this to the bugtraq or dailydave lists and see what the real security community thinks about it). so try again. it also makes me think, would you really blindly accept the first hit as your point of contact for a security matter? i'm glad you don't get to make that call.
as for lkml, i'm not subscribed but sometimes i scan it for interesting posts/topics.
you keep accusing me (btw, i'm not a grsecurity developer, i develop PaX only) of not following 'prescribed procedures' yet you *still* haven't shown a *single* reference to said procedure. please, stop this generic mumbo-jumbo and just post the URL to the document that clearly and unambiguously describes the 'official' procedure for reporting linux security bugs (and which doesn't involve Linus/Andrew and prescribes >3 weeks of waiting time, else you'd just prove my point). short of that, you have no basis for your claims (how can i not follow something you don't seem to know yourself either?).
we'll talk about being responsible when you manage to answer the question above.
poor social estrategy Posted Jan 9, 2005 0:51 UTC (Sun) by sbergman27 (guest, #10767) [Link] No need. You have already stated your postition and mindset quite clearly enough for everyone to understand. And yes, I am exiting this thread, as I rather dislike participating in useless flamewars, which is what this has become.
I will, however, take this opportunity to agree with you that if such a document does not exist, it very much should. People should *not* have to read LKML to know these things. If grsecurity had not released exploit code to the world, I would even be sympathetic. However, when an organization actually releases an exploit, it needs to be held to higher standards than the average Joe. The research on proper proceedure needs to be done, and thoroughly.
If you wish, we can continue this discussion via private email. I'm at steve_AT_rueb.com.
Live Long and Prosper,
poor social estrategy Posted Jan 9, 2005 17:13 UTC (Sun) by PaXTeam (subscriber, #24616) [Link] everything i discussed here very much belongs to the public and it shall stay so. i see you are quick to accuse others of being irresponsible but don't actually have the guts to apologize when your claims prove to be without merit. you also haven't realized that there are bigger issues here than the fate of a handful bugs, which is the whole point why we tested the waters with them only, not more critical stuff. some questions for you and the community to answer: why can the BSDs have a designated security officer and linux not? why can you communicate with said officers using proper encryption whereas you cannot with vendor-sec? why did nothing happen after the do_brk() bug/exploit leak more than a year ago so just in time history could repeat itself with the uselib() bug/exploit? why didn't you, Steve Bergman, complain about *that* yet? why is it acceptable that isec can release local root exploits with their advisories (which are anything but simple to understand and hence reproduce) but a few liner in assembly (trivial to reproduce) makes you scream 'irresponsible'? hipocrisy abound and you talk about holding others to higher standards.
keep the pissing contests for IRC Posted Jan 9, 2005 22:07 UTC (Sun) by dw (subscriber, #12017) [Link] If there is still a problem here, take it to e-mail, or IRC. LWN comment postings are not the place for it - remember your comments will probably exist here long after you cease to.
keep the pissing contests for IRC Posted Jan 10, 2005 12:13 UTC (Mon) by coolian (guest, #14818) [Link] Just remember, the Pax guy is trying to do a good thing, and wants itfixed. He may have the social graces of a walrus, but he's at least *doing* something about a problem. Stop ripping the guy and acting like armchair quarterbacks.
keep the pissing contests for IRC Posted Jan 10, 2005 12:29 UTC (Mon) by zorgan (guest, #4016) [Link] May I politely request that everybody stops confusing the PaX team withthe grsecurity developers? I haven't seem anybody of the PaX team "behaving like a social walrus" (not that I think this term would be fair to Brad Spengler, either). Anyway, the suggestion that Andrea Arcangeli would currently be more of a VM maintainer than Linus Torvalds or Andrew Morton is so funny that I don't understand why anybody took sbergman27 seriously. He has done important development on the VM a couple of times, but he has never adopted anything like a maintainer's role, AFAIK. But I think the main point remains (and that's why this discussion makes sense): If Linus and akpm get too much flooded with e-mails that they cannot even reply to a local DoS report within 3 weeks, then maybe they should appoint someone to be a security contact person? Someone who is willing to look into such reports, can judge their severity, and contact the relevant maintainers to review proposed patches etc.?
keep the pissing contests for IRC Posted Jan 11, 2005 8:11 UTC (Tue) by Wol (guest, #4433) [Link] Why's it funny that "Andrea would be more of a VM mainainer than Linus"?
After all, it was *Andrea* that *wrote* the thing in the first place, not Linus...
Cheers,
keep the pissing contests for IRC Posted Jan 11, 2005 20:11 UTC (Tue) by zorgan (guest, #4016) [Link] Because *writing* != *maintaining*.Maintaining means reviewing other people's patches, forwarding them to tree maintainers, making sure the code stays clean and well-documented, etc. Andrea has not even bothered much sending his own to Linus/Marcelo.
poor social estrategy Posted Jan 10, 2005 13:07 UTC (Mon) by philips (guest, #937) [Link] Thanks for you work, PaX.
I've being hitting this bug (as bug, but not securinty hole) several times before.
I hope that now it will be fixed.
Alan Cox? Posted Jan 9, 2005 1:36 UTC (Sun) by zorgan (guest, #4016) [Link] It always seemed to me that Alan Cox is very responsible with respect tosecurity problems. Now that he is maintaining a stable kernel again, he might be a good contact point.
Alan Cox? Posted Jan 9, 2005 12:48 UTC (Sun) by PaXTeam (subscriber, #24616) [Link] yes, Alan used to be the security contact person, but according to MAINTAINERS, vendor-sec took over that role (for the worse, as the facts show). with that said, we did consider contacting him after having waited for weeks in vain, but unfortunately the sudden leak of the uselib() bug and exploit made it necessary to release a new grsecurity version (which was otherwise being finalized for release anyway).
For what it is worth... Posted Jan 9, 2005 2:33 UTC (Sun) by jd (guest, #26381) [Link] Personally, I don't like getting into spitting-matches. I've singed myself on WAY too many flame-wars in the past, and my brain can't take it any more. However, that being said, I'll trow in my two units of local currency.I understand politics, the fights over who has to deal with what, etc. I understand them and I detest them. Which is easier for a person to do? Hit the "delete" button, or hit a "forward" button to get the message to the right person? Seems to me, it's still one button. So arguing "did it go to the right person" is largely missing the very point of such an argument. If it did NOT go to the right person, but DID go to someone who has to know who the right person was, why did it NOT get to the right person? Secondly, are we interested in who-did-what/who-didn't-do-what fights, or are we interested in Linux being the best OS that ever was and ever will be? If we are more interested in the latter, then someone should include the patches! It's that simple. At the end of the day, there's only one sure way of making progress, and that's to move forward. Lastly, do I think some of the security developers tend to be rough around the edges? Well, yes. I had a few fights on my hands, over including patches into FOLK that others wanted to keep to themselves. I included them anyway. Nobody, but NOBODY, is going to tell me that Linux users deserve second-best, should live in ignorance of what's out there, or should have the difficulties inherent in merging multiple patches. I wanted an easy showcase for technology, and that's exactly what I produced, whether others liked it or not. It was my opinion then, and still mine today, that knowledge is exactly like power - something to be distributed as widely as humanly possible, for the betterment of all. Sure, I'm crazy. Sure, if Linux were to include everything that was important to a substantial fraction of it's users, the kernel would be nearly twice the size it is now. But I can survive being crazy, and those who download source can survive a little longer download time. Neither of these will kill. If substantial, wholly justified, scares into Linux' security or stability hit the community, then those might very well kill Tux. I think the potential consequences make a LITTLE bit of effort by SOMEBODY in the Linux Kernel Development environ into getting Linus to install these patches, and in full, well worth any likely price. We can always back things out, if they turn out to be an error or become obsolete. Intermezzo, the original Ext FS, and a few other favourites have gone. DevFS is going to. Patches have been introduced with resistance from Linus - IIRC, he once said Linux would never run on anything but an ix86. All of these just show that nobody is perfect. But they also show that those who aren't perfect should allow themselves to be steered by others in areas they're not so skilled on.
For what it is worth... Posted Jan 10, 2005 8:33 UTC (Mon) by Wol (guest, #4433) [Link] "So arguing "did it go to the right person" is largely missing the very point of such an argument. If it did NOT go to the right person, but DID go to someone who has to know who the right person was, why did it NOT get to the right person?"
Simple. And also, what PaxTeam appear to be missing.
Linus is ONE person. There are only 24 hours in a day. There is a *high* probability that the PaxTeam message never got to Linus' eyeballs...
THAT is why there is all this maintainers/lieutenants business. To reduce the workload on Linus to the point where it is manageable.
PaxTeam isn't subscribed to LKML. Why? Because "there's too much"? Bearing in mind Linus probably gets a hell of a lot of mail addressed to him personally, then he has to keep an eye on LKML, then he actually has a job to do, then he has to discuss things with his lieutenants... I'm afraid a mesage from a total unknown has low priority. And that fact that it claims to report a security vulnerability is quite likely to get it classified as "crying wolf" - I bet loads of people do cry wolf - intentionally or down to their own incompetence.
At the end of the day, the "proper channels" are there for a reason - in other words the system would collapse if they weren't there. And the fact PaxTeam don't take LKML says just about everything else - Oh and if they don't know Andrea Arcangeli and/or Rik van Riel are the people to talk to about VM, then they haven't been watching kernel development. Who remembers the VM-wars of 2001? :-) Hint to PaxTeam - at the very least, read Kernel Traffic *in* *depth*.
Cheers,
For what it is worth... Posted Jan 10, 2005 11:26 UTC (Mon) by PaXTeam (subscriber, #24616) [Link] lots of speculation so let's see the actual timeline a bit. spender emailed Linus sometime early december about the few issues he had found. he also mentioned some of the fixes that were in PaX, the result of one of them was this commit: http://linux.bkbits.net:8080/linux-2.6/cset@41bc900azV2y9... . understand please that we (well, spender at least) already had had a working two-way email connection with Linus. during the holidays i had finally time to work on the forward port of PaX (last supported version was 2.6.7) and that's when i realized the change in status of the expand_down() bug as since 2.6.9 it became exploitable by unprivileged users as well. so i emailed Linus about it (of the importance, not the bug itself, he had already known about it from spender, although he had never replied back on that one). one week later, which is early this year i resent the mail to Linus and Andrew as well, and the next day spender forwarded the mail himself to them (as i said, he had a known working email route to Linus at least). nothing happened except spender was preparing the next grsecurity release and it became more and more urgent to get some feedback on these issues. we were considering emailing Alan Cox (the week of waiting allotted to Andrew as well wasn't over yet) when the uselib() exploit suddenly hit the net and everyone entered forced release mode, we couldn't delay it either.
now that you know some background, tell me again, 1. how much more we should have waited, 2. why we shouldn't have contacted Linus/Andrew in the first place, 3. why we should have contacted Alan first (who is explicitly not the security contact anymore), 4. why we should have contacted a VM hacker first (none of whom is a security contact either, not even for their respective employer, let alone linux/VM in general).
see, i've been in the security industry for some number of years now, and i know quite well what best practices are (everyone's got his own, but there're some common elements):
rule 1: you contact the explicit security contact first. for linux this used to be Alan himself, nowadays it's vendor-sec (yes, that means you're not supposed to deal with individual distros, that's why vendor-sec was established in the first place). except they proved to unreliable, not to mention that it's *impossible* to contact them in a secure way (they don't have a PGP key).
rule 2: short of such a security contact, you begin contacting the 'people in control', from top to down, not the other way around. for companies that's relevant because the chain of control also represents the chain of responsibility. you can argue that open source/free software projects are free of chain of control, but they're not free of responsibility. i believed and still believe that we did the right thing when we began contacting Linus, then Andrew and were about to contact Alan when external events intervened.
> THAT is why there is all this maintainers/lieutenants business.
except the VM has no explicitly listed maintainer. but yes, i can guess who the main contributors are, but that doesn't make them a security contact (remember, we only wanted to get feedback, be told what to do next, and *not* to force Linus or anyone to actually manage the issue). it makes them the right person to actually fix the bug, but that's only the second step after the initial contact.
> PaxTeam isn't subscribed to LKML. Why? Because "there's too much"?
correct, i have a day job (unrelated to linux), family and friends, i can't handle that email load (and there's more in my world than lkml). i don't know where you got that i didn't like lkml, if i wasn't sympathetic to linux, i would have posted everything to bugtraq a month ago (contrast that to the recent DJB case).
> And that fact that it claims to report a security vulnerability is quite
i provided a proof of concept exploit (which you would know if you had actually read the announcement and posts here).
For what it is worth... Posted Jan 10, 2005 18:41 UTC (Mon) by geomon (guest, #27127) [Link] "lots of speculation so let's..."
From my perspective, you are trying to defend yourself when you shouldn't have to.
"understand please that we (well, spender at least) already had had a working two-way email connection with Linus. during the holidays..."
That is a problem. You have, of course, identified the source of the problem and have already recommended a solution.
"1. how much more we should have waited..."
For a legitimate bug? Not long I would hope (I am a user!).
"2. why we shouldn't have contacted Linus/Andrew in the first place"
You should have if there isn't an appropriate point-of-contact already established. That is the root cause of the problem.
"3. why we should have contacted Alan first (who is explicitly not the security contact anymore)"
You shouldn't have to. If Alan is not *the* person for security matters, that would be inappropriate as well.
"4. why we should have contacted a VM hacker first (none of whom is a security contact either, not even for their respective employer, let alone linux/VM in general)."
I've got to agree with this one too. Why should I go to the grocery store to get my car's front end aligned?
"see, i've been in the security industry for some number of years now, and i know quite well what best practices are (everyone's got his own, but there're some common elements)"
You are projecting defensiveness again. Give it a rest - you've made your point.
"rule 1: you contact the explicit security contact first."
WRONG!
An explict security contact should have been established *first*. That has either not happened yet, or the point-of-contact has changed. In either case, if the information is not readily available, then NO credible process exists for submitting security patches.
I share your shock at that prospect.
"for linux this used to be Alan himself, nowadays it's vendor-sec (yes, that means you're not supposed to deal with individual distros, that's why vendor-sec was established in the first place)."
That may work for individual vendors. How about establishing a Linux security working group that is composed of security contacts from the vendors?
What is missing in this discussion is a single point-of-contact, regardless of how it is composed, with contact information posted at kernel.org, kerneltrap.org, linux.org, or lwn.net.
"rule 2: short of such a security contact, you begin contacting the 'people in control"
WRONG.
See Rule 1.
"> PaxTeam isn't subscribed to LKML. Why? Because "there's too much"?
correct, i have a day job (unrelated to linux),"
And you shouldn't HAVE to subscribe to a mailing list to get a point-of-contact. That is pure stupidity.
That's why web pages were invented: "http://groups-beta.google.com/group/alt.hypertext/msg/395..."
"> And that fact that it claims to report a security vulnerability is quite likely to get it classified as "crying wolf"
i provided a proof of concept exploit (which you would know if you had actually read the announcement and posts here)."
The fact is, every event of security should be treated as a serious condition. It should not be the job of the submitter to determine whether the issue is serious or not; they may not be security experts but probably have noted a serious condition that they cannot explain by themselves.
This issue is a confidence buster if the community cannot produce a credible notification scheme. One of the key arguments that Linux advocates have used for years in defending the security of its products has been the claim that "many eyes" are better than code obfuscation. I have observed how this submitter has been treated and would question why ANYONE would submit a security concern to the community at this point.
Continuing to blame the person who submits the bug report, regardless of how they did it, is unacceptable. That smacks of the same arrogance that drove us to use Linux in the first place. Linux developers need to provide certainty to the user community that their concerns will be addressed and not arbitrarily dismissed.
Thoughts Posted Jan 10, 2005 19:49 UTC (Mon) by jd (guest, #26381) [Link] I hope that I'm not speculating too much, myself. The last thing I want to do is add to the general confusion, here. My personal opinion is that a bug is a bug is a bug, and needs to be fixed. Where that bug is a security hole, as is the case here, it needs to be fixed almost before any other concern, come hell or high water.There is very little value in a system that does everything superbly, with absolutely the minimum possible latency, total stability, yadda yadda yadda, if the next day some idiot turns you finely-honed silicon masterpiece into a spammer zombie. Yes, I think tempers may be running a little high, and that's probably not helping the situation. To me, there are only two issues at stake - how to get ALL of the fixes in place, ASAP, and how to ensure that future problems are addressed as fast as humanly possible.
For what it is worth... Posted Jan 10, 2005 21:30 UTC (Mon) by Ross (subscriber, #4065) [Link] There was no crying wolf and there were no "proper channels". If thesetype of bug reports are not taken seriously then I am shocked. It wasn't a subtle or complex set of bugs. These are the kinds of problems that show up all the time. The source code snippits should have been more than enough to "prove" their case.
As for reporting, the grsecurity team followed what they thought
Sure Linus gets a lot of email. He probaly doesn't read a lot of it.
They were very patient. People are claiming this is like DJB's arrogant
Someone else said that they shouldn't have mailed the kernel people but
Why can't MAINTAINERS have an entry for kernel security bug reports? Why
poor social estrategy Posted Jan 8, 2005 3:08 UTC (Sat) by Junior_Samples (guest, #26737) [Link] I wouldn't worry about the "tone" at all. Linus isn't some frail petunia. In fact, Linus freely admits that he is one of the most brusque, if not out and out rude, individuals you will ever meet among Linux developers. In his position it's an asset; in this role, someone else with fragile sensibilities would be quickly steamrolled or suffer a nervous breakdown. Linus can dish it out. He can take it just as well.Nevertheless, Linux probably needs a "kernel security officer" who would be the initial contact point when a vulnerability is discovered. It is shameful that these reports went unanswered for almost a month.
poor social estrategy Posted Jan 8, 2005 11:15 UTC (Sat) by pointwood (guest, #2814) [Link] Quote: Nevertheless, Linux probably needs a "kernel security officer" who would be the initial contact point when a vulnerability is discovered. It is shameful that these reports went unanswered for almost a month.
That's what I was thinking too, unless they already have some kind of process for that?
poor social estrategy Posted Jan 8, 2005 17:18 UTC (Sat) by wolfrider (guest, #3105) [Link] In all fairness to Linus, neither one of their Subject lines really stood out as an important thing to check RIGHTNOW... Linus is so swamped with work that they prolly just fell by the wayside, and it took this kick to the ass to get him to notice.
|
|||||||||||||