CentOS on the systems intrusions at Red Hat
| From: | Karanbir Singh <kbsingh-AT-centos.org> | |
| To: | CentOS-Announce <centos-announce-AT-centos.org>, CentOS mailing list <centos-AT-centos.org> | |
| Subject: | [CentOS-announce] CentOS position on systems intrusion at Red Hat | |
| Date: | Fri, 22 Aug 2008 23:15:29 +0100 | |
| Message-ID: | <48AF3A81.3090903@centos.org> |
Earlier in the day today Red Hat made an announcement [1] that there had been an intrusion into some of their computer systems last week. In the same announcement they mention that some of the packages for OpenSSH on RHEL-4 ( i386 and x86_64 ) as well as RHEL-5 ( x86_64 ) were signed by the intruder. In their announcement they also clarified that they were confident that none of these, potentially compromised, packages made their way into or through RHN to client and customer machines. As a security measure a script [3] was made available along with a semi-detailed description of the issue [2]. We take security issues very seriously, and as soon as we were made aware of the situation I undertook a complete audit of the entire CentOS4/5 Build and Signing infrastructure. We can now assure everyone that no compromise has taken place anywhere within the CentOS Infrastructure. Our entire setup is located behind multiple firewalls, and only accessible from a very small number of places, by only a few people. Also included in this audit were all entry points to the build services, signing machines, primary release machines and connectivity between all these hosts. Since OpenSSH is a critical component of any Linux machine, we considered it essential to audit the last two released package sets ( openssh-4.3p2-26.el5.src.rpm, openssh-4.3p2-26.el5_2.1.src.rpm ). I have just finished this code audit, and can assure everyone that there is no compromised code included in either of these packages. A similar check is also being done for the CentOS-4 sources. Packages released today, by upstream, ( based on : openssh-4.3p2-26.el5_2.1.src.rpm, openssh-3.9p1-11.el4_7.src.rpm ) address two issues. Firstly they contain a fix for http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4752 . And secondly, in the remote event that someone had indeed got compromised packages via RHN, their packages would get updated to a known good state. We wanted to get these packages out right away to address the first issue, and also to cover users converting non updated RHEL installs to CentOS in the next few weeks/months. Release of these packages into the mirror.centos.org network does *not* imply that CentOS users are affected by the intrusion at Red Hat. Finally, while we feel confident that there is no possibility of this compromise having been passed onto the CentOS userbase, we still encourage users to verify their packages independently using whatever resources they might have available. -- [1]: https://rhn.redhat.com/errata/RHSA-2008-0855.html [2]: http://www.redhat.com/security/data/openssh-blacklist.html [3]: https://www.redhat.com/security/data/openssh-blacklist-1.... :Its important to note that this script *only* checks for packages built within Red Hat, and will *not* be a reliable source of verification on CentOS since we rebuild from sources, using no Red Hat binary. -- Karanbir Singh CentOS Project { http://www.centos.org/ } irc: z00dax, #centos@irc.freenode.net _______________________________________________ CentOS-announce mailing list CentOS-announce@centos.org http://lists.centos.org/mailman/listinfo/centos-announce
Posted Aug 23, 2008 4:38 UTC (Sat)
by sbergman27 (guest, #10767)
[Link]
Posted Aug 23, 2008 9:57 UTC (Sat)
by gvy (guest, #11981)
[Link] (12 responses)
Posted Aug 23, 2008 12:44 UTC (Sat)
by epa (subscriber, #39769)
[Link]
Posted Aug 23, 2008 15:04 UTC (Sat)
by dowdle (subscriber, #659)
[Link] (5 responses)
You don't seem to understand CentOS. While they might have a bug tracking system... I don't think their goal is to track general package bugs. Their goal is to provide as close of an experience to upstream as possible... bugs and all. The only bugs that should be in CentOS' bug tracking system are bugs that relate to packages that they alter from upstream... or packages in their additional, optional repositories. There are only a small handful of packages that CentOS modifies.
That means that kernel bugs and package bugs (those unmodified by CentOS) should really be reported to Red Hat as it is the upstream source of the CentOS packages... although it is better to have some sort of business relationship with Red Hat when wanting to use them for support. That's only polite. Over the years Red Hat has seemed to have warmed up to the CentOS project and realized that they are helping to keep people in the Red Hat fold... and some CentOS users, when their needs expand beyond the support model offered by the CentOS project, will become Red Hat customers. I don't really have a reference to give you on this but it is my understanding that, being what it is, CentOS has several million users... and while it doesn't get a lot of attention on sites like distrowatch.com, it is one of the most popular distributions for server usage.
To the best of my knowledge, CentOS doesn't have any kernel developers as it is a completely volunteer effort. If ALT Linux has some kernel developers, I'm guessing it is sponsored by some commercial entity who can afford to pay them? It is kind of hard to find kernel developers who will maintain kernel code for a distribution over a long period of time for free... as if they are competent they can likely get big bucks elsewhere.
It is a bit tacky of you to criticize a volunteer who is in a position of leadership and obviously gives a lot of his time to the community. The bug in question is quite old and I'd be surprised if it hasn't been fixed long ago in a kernel update from upstream.
Posted Aug 23, 2008 15:11 UTC (Sat)
by paragw (guest, #45306)
[Link]
Posted Aug 23, 2008 17:19 UTC (Sat)
by gvy (guest, #11981)
[Link] (3 responses)
If RH clones had brains, they might try and come up with workflow like:
This way, there might occur reasonable development sense in a project, not just an "affordable repackaging" one (am I closer to "understanding CentOS" with this statement? ;-).
> [...] CentOS doesn't have any kernel developers
If a volunteer effort would pair with another volunteer effort (like kernelnewbies), I bet that those growing up with more or less simple situations would find even better job soon. Should definitely do no harm to efforts as well.
But then again, I've only done one silly little kernel patch nine years ago, and it was such an obvious re-hardwiring that I didn't even bother anyone (it got fixed properly with CONFIG_ handle that year anyways).
Back to bugzilla, we have quite a few bugs assigned to nobody@ which does help with finding something to wrestle in one's spare time. Assigning lots of bugs *manually* to one's self just to forget them isn't going to work but rather leads to "buried alive under a pile of bugreports".
> It is a bit tacky of you to criticize a volunteer who is in
Thanks for your way better tempered comment, still when I would have written replies like "get lost" and people published them and got me to understand _my_ fault I found myself thankful to them in the longer run.
Back to the original matter, I still don't trust any claim of "complete audit" by volunteers who don't have time to close kernel bugs at least as WONTFIX; and that was largely the point, not just insulting folks around.
Hope that intruders were too busy with RH/Fedora and didn't get around to try their hand at the third target, whatever they used... had to redo one system from scratch after 2003's openssh adventure, no joy.
Posted Aug 23, 2008 21:05 UTC (Sat)
by louie (guest, #3285)
[Link]
Posted Aug 23, 2008 22:09 UTC (Sat)
by dowdle (subscriber, #659)
[Link] (1 responses)
Given the popularity of CentOS, it's good reputation because it has a goal and it sticks to it, the large number of distros that are based on it (including a few prominent pay ones), and it's lasting success, I don't think they really need any advice from you... but I'm just a user. What do I know?
There are a number of clones I know less about... some of them even charge for timely updates. Perhaps you could advise them?
Posted Aug 26, 2008 13:10 UTC (Tue)
by gvy (guest, #11981)
[Link]
> Given the popularity of CentOS, it's good reputation
> I don't think they really need any advice from you...
> Perhaps you could advise them?
Posted Aug 23, 2008 15:41 UTC (Sat)
by bfields (subscriber, #19510)
[Link] (4 responses)
It's normal for developers working on high-profile projects to have "loads of open/stuck bugs" assigned to them.
While I think we should all strive to be polite, I suspect the proportion of developers that have been a little brusque with someone in private email is also pretty high.
Also, to drift more off-topic: please keep comments on any bug report focused on information that makes progress towards fixing the bug. It's not the place for opinions on the general quality of the project or the bug assignee; if your goal is to criticize either, a narrow focus on the technical issue at hand would be a more effective way to do that. Similarly, you'd make a more convincing advertisement for the competing project by sticking to specific technical information found by that project's developers, and omitting broad claims about the competitor's kernel team being "world-class" or CentOS being "crap".
Posted Aug 23, 2008 16:46 UTC (Sat)
by gvy (guest, #11981)
[Link] (3 responses)
Thanks for your advice -- as you might have seen, I've been actually trying to help with testing/debugging as I had that particular hardware at hand, knowing how it feels without a sample at hand. Guess being rude and rejecting such help from peers is part of being a high-profile developer of yet another RH clone... (and not forgetting/forgiving part of me :-/)
So, my sincere wishes of luck to users of this cent-class project (have quite a few friends among those, btw) :-)
Posted Aug 24, 2008 21:28 UTC (Sun)
by sbergman27 (guest, #10767)
[Link] (2 responses)
There are those in this forum who would tell me that I have done that with Fedora. And they would be right.;-)
Posted Aug 26, 2008 13:21 UTC (Tue)
by gvy (guest, #11981)
[Link] (1 responses)
Of course, those who prefer living with known bugs because they also have known workarounds are quite in their will to cope with things this way, it's absolutely no blame for a sysadmin whose job is working systems.
Maybe they were calm children not wielding a screwdriver for no reason and always using their toys as prescribed. :)
Posted Aug 26, 2008 21:27 UTC (Tue)
by sbergman27 (guest, #10767)
[Link]
So make a commitment to do that. I'm sure that CentOS and Red Hat both would welcome such a valuable contribution from you.
-Steve
Posted Aug 26, 2008 14:04 UTC (Tue)
by dowdle (subscriber, #659)
[Link]
Karanbir Singh wrote:
CentOS on the systems intrusions at Red Hat
kbsingh@? hm
I wouldn't trust "complete audit" by a person who has loads of open/stuck bugs assigned to him in centos bugzilla, and who was handling feedback on some of them in especially delightful way (e.g. #2177). Not even a complete audit of GNU hello.
The GNU hello source code is pretty complex. It would certainly be a day or so's work to audit it all.
kbsingh@? hm
kbsingh@? hm
kbsingh@? hm
better clone workflow | volunteering | "complete audit"
Indeed. I'm stupid enough project manager, you see.
- own bugtracker
- cloned (stable) repo
- unstable/proposals repo which might have packages diverging from original
- process of proposing patches/spec fixes/version updates via bugzilla.redhat.com
Kernel developers don't jump out of thin air, they grow up (just like with any profession). In this particular example, figuring out which of the patches in similarly 2.6.18-based but working kernel might be reasonable exercise for a kernel hacker wannabe (in a good sense :-) -- given that hardware access off a separate HDD/LiveCD could be organized quite easily.
> a position of leadership and obviously gives a lot of his time
> to the community.
Heh, I'm a volunteer in a sort of leadership with quite some time spent exactly on connecting creative people in community with opportunities to grow up, that is bugs and not-yet-done stuff :-) And yes, guiding people who might write a good bugreport in a mailing list to post it to bugzilla.
better clone workflow
better clone workflow | volunteering | "complete audit"
[wontfix] better clone workflow
> similar to the one you recommend.
Having repos and a bugtracker isn't enough for having a workflow... (btw I rather "suggested" rather than "recommended" that, since it usually takes some time of doing/using something myself to recommend that)
> because it has a goal and it sticks to it
I'd rather say its popularity/reputation is >99% a job of redhat.com folks (development, packaging, QA, security response) and <1% a job of centos.org folks. Not to put down those who finally managed to create "the" RH clone but to put things in perspective.
I don't think they'll take any.
I'd probably better care for 32 ALT bugs assigned to myself (looks like that's almost minimum of them usually) than chasing clones and advising them to be a bit more like humans :) rather ranting... I don't consider clones a viable point of considerable development having watched ASP Linux for many years with my 5-years-old predictions coming true one after another, and having begun my Linux practice with WGS Linux Pro back in RH4 days. :)
kbsingh@? hm
that particular one
> to have "loads of open/stuck bugs" assigned to them.
I sorta know...
that particular one
that particular one
It's perfectly clear... it was proposed in comments above that a route for bugreports and bugfixes _could_ be attempted _if_ there are desire and resources for that. This way clone-generated bugreports might come with proposed (and probably even tested in overlay repo) fixes to have somewhat higher value in upstream bugzilla. Still such fixes would only have one way of getting into the main repo or updates: via upstream distro.
that particular one
...it was proposed in comments above that a route for bugreports and bugfixes _could_ be attempted _if_ there are desire and resources for that.
"""
Red Hat Bugzilla adds CentOS issue number support
> hi Guys,
>
> bugzilla.redhat.com now has direct support for cross referencing a
> CentOS Bug/Issue number. So when filing issues upstream that were
> reported on bugs.centos.org, check:
>
> Add External Bug: CentOS