Weekly Edition
Return to the Distributions page
| ||||||||||||||||
An update on Fedora's "issues"
The Fedora Infrastructure team continues to work on the issues we discovered earlier this week. Right now, we're getting the account system restored to service, along with some of the application servers. We're also taking advantage of the outages to upgrade a few systems at the same time. Some services such as the Account System and the wiki should return to normal over the weekend, but we expect outages to continue for some other systems. Please be patient as we continue to work the problem. -- Paul W. Frields gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://paul.frields.org/ - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-announce-list mailing list fedora-announce-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-announce-list (Log in to post comments)
An update on Fedora's "issues" Posted Aug 16, 2008 21:50 UTC (Sat) by ESRI (guest, #52806) [Link] Lots of speculation going on. The one thing I'm sure of is that this must be a very serious issue. Were it anything less, the Fedora Project would likely not risk the potential negative fallout from how the issue is being handled (ie only a very few know what is going on). I think Fedora has earned enough good will with its users and development community that people trust that there is a very good reason for things being handled the way they are. Most everyone is antsy, but able to hang on and wait a little longer for information.
An update on Fedora's "issues" Posted Aug 16, 2008 22:16 UTC (Sat) by me@jasonclinton.com (subscriber, #52701) [Link] We'll really just have to wait until they disclose what happened. If this turns out to be a security issue, I don't think anyone will ever trust them again (given the lack of disclosure). OTOH, it could just be failed hardware. Hard to tell without some actual information.
An update on Fedora's "issues" Posted Aug 17, 2008 0:35 UTC (Sun) by ofeeley (guest, #36105) [Link] That depends on what the security issue actually is and who is affected by it. Full disclosure is attractive when I'm the one affected, but if the problem solely affects the Fedora Project entity then they have a right to restrict information as they see fit. However, it's pointless to speculate at this stage (and believe me I've been searching the lists and bug-tickets for possible clues!) and may be something mundane involving running out of resources for crucial infrastructure servers or completely broken packages being made available for updates. It's hard to guess why such causes would not simply be stated as a reason though.
An update on Fedora's "issues" Posted Aug 17, 2008 14:14 UTC (Sun) by AlexHudson (guest, #41828) [Link] Full disclosure only works when you are able to disclose a work-around or some kind of other fix. If you disclose and there's nothing (or only very painful solutions) that people can implement, that's a pretty bad idea.
An update on Fedora's "issues" Posted Aug 17, 2008 14:54 UTC (Sun) by jengelh (subscriber, #33263) [Link]
>We'll really just have to wait until they disclose what happened. If this turns out to be a
security issue, I don't think anyone will ever trust them again (given the lack of
disclosure). OTOH, it could just be failed hardware. Hard to tell without some actual
information.
"Please don't update" with failed hardware? No, if the hardware failed they'd just let it go
("server is busted, please use a mirror"). This smells much like the Debian intrusion in July
2006.
An update on Fedora's "issues" Posted Aug 17, 2008 2:04 UTC (Sun) by zlynx (subscriber, #2285) [Link] Maybe they don't *know* what the problem is yet. They could be running forensics, trying to find if there was a hardware or software failure or if they had a security failure.
Mystery problems in complex systems Posted Aug 17, 2008 16:01 UTC (Sun) by tialaramex (subscriber, #21167) [Link] Yeah. Definitely a possibility. We had a machine built (with CentOS 5) a little while back, handed over to us as working. We discovered it didn't have a compiler installed (perils of letting non-developers specify the machine). So 'yum install gcc' and Yum immediately segfaults. Tried manually updating the kernel because someone suggested a recent AMD bugfix could be related. Still segfaults. Well, that was bad news, so I tried replacing Yum RPMs manually, got nothing. Eventually we found that RPM libraries were corrupted somehow, replaced those, and Yum was working, but soon after installing GCC I ran 'less' and that segfaulted. While investigating this, the disk became read only and the kernel reported serious ext3 corruption. So we turned it off and handed it back over as faulty hardware, suspected RAM or disk failure. But the hardware guys ran every diagnostic they could think of, found nothing and re-installed CentOS. And now it seems fine (we have some fairly hard-core correctness tests still to run on the finished system of software + hardware). So what happened there? Corrupted install media? Cosmic rays? Some new type of malware? Just good old PEBCAK? Normally I would want to understand, but in this case hunting for the answers seems likely to be fruitless. If the Fedora people have some mystery symptoms it would be nice if they set themselves (and told us) a deadline where they'll declare it just another unsolved mystery, like a headless torso washed up on a beach with no identifying marks.
Mystery problems in complex systems Posted Aug 17, 2008 16:38 UTC (Sun) by vonbrand (subscriber, #4458) [Link] Those symthoms are very similar to what I had way back with a Western Digital disk + DMA: The filesystem got slowly corrupted, and in the end nothing worked. Also when in a machine the IDE cable was wrapped around the power cables. Bad disk, controller? Bad RAM? CPU overheats (bad fan)?
Mystery problems in complex systems Posted Aug 17, 2008 23:10 UTC (Sun) by jengelh (subscriber, #33263) [Link] >Also when in a machine the IDE cable was wrapped around the power cables. Bad disk, controller? Bad RAM? CPU overheats (bad fan)? Sounds like bad spirits and particles. 80-pin cables have at least 40 grounds just to combat the crosstalk. And then there is also the magnetic field around the Molex line. The combination sounds hardly good.
An update on Fedora's "issues" Posted Aug 17, 2008 14:49 UTC (Sun) by sbergman27 (guest, #10767) [Link] """ I think Fedora has earned enough good will with its users and development community that people trust that there is a very good reason for things being handled the way they are. """ That is very much a matter of personal opinion. I've had too many systems bitten by Fedora's playing fast and loose with updates, and am not particularly surprised at the fact that they are having this problem (which is almost certainly security related, else they would not be so tight-lipped) or by the way they are handling it. In the mean time, I continue my planned migrations of existing Fedora servers to CentOS and Ubuntu Server.
An update on Fedora's "issues" Posted Aug 17, 2008 22:12 UTC (Sun) by BeS (subscriber, #43108) [Link] >I've had too many systems bitten by Fedora's playing fast and loose with updates That's probably the downside of having a bleeding-edge distribution. Personally i enjoy this character of Fedora on my personal desktop systems. Sure from time to time a update can break something but in my experience this is really seldom and happen to my more often with Debian testing (which i used before) But... >In the mean time, I continue my planned migrations of existing Fedora servers ... imho you should never use Fedora for a server because both the bleeding-edge characteristic and the short support cycle is not really suitable for a server. For a server i would always choose CentOS or RedHat if i want something like "Fedora for the server" or Debian if it could/should be something complete different than Fedora.
An update on Fedora's "issues" Posted Aug 18, 2008 13:22 UTC (Mon) by vonbrand (subscriber, #4458) [Link] In the end, it is better to stay with one line of distributuions for desktop + server, be it Debian(ish) or Fedora + CentOS/RHEL. The trouble you get into because you don't remember how to handle some configuration in one or the other, or subtle inconsistencies due to different software strains, just isn't worth it.
Full disclosure Posted Aug 17, 2008 4:05 UTC (Sun) by dwheeler (guest, #1216) [Link] One of primary reasons many people switched to full disclosure was that when vulnerability reporters told vendors, "You have a serious problem", the vendors would try to gag the reporter, pretend there was no problem, and avoid doing anything to fix it.That is not what seems to be happening here. There are no reports as to whether this is a security problem or not (they may not know!), but clearly they are not trying to pretend there is no problem, and they seem to be working hard to fix it. "Full disclosure" has its own problems. The big problem with full disclosure is that if the attack isn't already known publicly, it creates a window of time where users often cannot protect themselves effectively (because no fix is available), yet attackers who wouldn't have known about the vulnerability otherwise can now use it. I think you ought to give the supplier a short time to fix the problem, unless or until the supplier has demonstrated an unwillingness to react quickly on security vulnerability reports. At which point, full disclosure may very well be necessary, since it's hard to know when attackers will find the vulnerability... so at that point it's best to provide customers with that information. Since Fedora does not seem to be pretending there's no problem, I don't think a call for full disclosure is necessary; it would just expose its users unnecessarily. I realize that some people think full disclosure is the one true way, and whatever I say is unlikely to change their minds. But I think there are two modes, "full disclosure" and "delayed disclosure". The latter is often called "responsible disclosure", but I hate that phrase; if the supplier persists in ignoring security issues, then long-delayed reporting is actually irresponsible. I think it's better to select modes for each vendor, depending on how well they respond to vulnerability reports. So please, I think we should cut Fedora a break. They seem to be trying to do the right thing, instead of ignoring a problem of some kind.
Full disclosure Posted Aug 17, 2008 7:48 UTC (Sun) by fatherted (guest, #33354) [Link] i think they CAN'T pretend there is no problem, given that everything they've got is/has been down. the problem is the secrecy. that's really been the fedora way ever since the beginning -- a small in-bunch, and then the rest of us unwashed masses. its very hard to trust when secrecy is all that seems to happen with them. if it weren't a security issue, it would have been easy to say something about the problem. if it is, then all the unwashed masses need to know if we're affected or not. seems to me that fedora has very much not earned much credibility. and has mostly lost what little it had because of the way this is being handled.
Full disclosure Posted Aug 17, 2008 9:55 UTC (Sun) by tajyrink (subscriber, #2750) [Link] One cannot judge them before the details are known. Of course more knowledge would be nice, but it may be the best way the situation could be handled is being done. I think this is a bit same than around here (probably everywhere) when there is some tragic accident, some people always shout that "people need to know!!" just because they are curious and want to read everything terrible from tabloids, even though actually minimizing the tragedy and trying to investigate could rather need secrecy. If they do not know the exact problem, and have a reason to believe hostile people could theoretically do something bad if they would let some non-full knowledge to the public, secrecy is the only way to try to minimize risks. If it turns out more information should have been let out earlier to minimize risks, then they chose wrong. Let's see.
Full disclosure Posted Aug 17, 2008 12:59 UTC (Sun) by salimma (subscriber, #34460) [Link] Your judgment seems rather harsh. Pre-existing anti-Red Hat bias? The Fedora project is actually one of the more openly-run community Linux distributions -- apart from Debian, I'm hard-pressed to name another project that is as open to community input (and in fact, it is easier to become a Fedora contributor than to be a Debian developer). As others have said, this is probably a Fedora-specific vulnerability. If it affects no other service providers, and Fedora has warned its users not to use their services from the time being, how is a full disclosure the more responsible thing to do? You'd be providing more information about possible attack vectors, without any legitimate use. Disclaimer: I am a Fedora contributor myself
Full disclosure Posted Aug 17, 2008 9:07 UTC (Sun) by gdt (subscriber, #6284) [Link] So please, I think we should cut Fedora a break. They seem to be trying to do the right thing, instead of ignoring a problem of some kind. You've ignored why full disclosure is important to system administrators. Full disclosure allows system owners to make their own assessment of the risk of subversion of their fielded computers. Red Hat has not made any statement about risk beyond "as a precaution, we recommend you not download or update any additional packages on your Fedora systems". That's a fine initial request. But enough time has passed for more information about the risk to fielded systems to have been produced. Starting with the date of the suspected subversion of Fedora's systems. That would at least indicate to system administrators the date from which all IDS logs, NetFlow data and so on should be retained for future analysis if needed. A high-security site might well choose to turn off all Fedora systems updated after that date until more information becomes available. Hopefully soon we will also be told if recently-updated Fedora systems should be treated as compromised or not, if the risks are equivalent for Fedora 8 and 9, and so on. What about mirror sites, should they be preventing access for Fedora updates downloaded after a particular date to lessen the risk to their clients? With the current dearth of information, they can't make a decision. If this had just occurred, then I'd say "yes, give them a break, give them time to work the problem". But enough time has now passed and the most recent "information" provided was just pathetic.
Full disclosure Posted Aug 17, 2008 13:17 UTC (Sun) by cmc (subscriber, #16767) [Link] What about mirror sites, should they be preventing access for Fedora updates downloaded after a particular date to lessen the risk to their clients? With the current dearth of information, they can't make a decision. The official word so far is no. There is a (closed) mailing list for mirror admins that is used to keep us apprised of various issues related to mirror maintenance. If there was some issue that required the mirrors to be shut down, they would say so. Of course they can't force compliance, but they could remove or replace material on the masters and those removals would trickle down to most regularly updated mirrors unless their maintainers took special steps to retain that material.
Full disclosure Posted Aug 17, 2008 20:23 UTC (Sun) by danieldk (guest, #27876) [Link] Of course they can't force compliance, but they could remove or replace material on the masters and those removals would trickle down to most regularly updated mirrors unless their maintainers took special steps to retain that material. They can kinda force it, since most people use yum with (Fedora-provided) mirrorlists. If necessary, they could just remove unwanted mirrors from the mirrorlist. In fact, CentOS automatically returns mirrors close to the machine location (according to GeoIP), and leaves out mirrors that are not in sync.
Full disclosure Posted Aug 17, 2008 14:50 UTC (Sun) by ofeeley (guest, #36105) [Link] Red Hat has not made any statement about riskDon't expect any statement from Red Hat. Fedora is governed and administered by the Fedora Project Board of which Paul Frields is the chair and Fedora Project Leader. We've already had two statements from him as reported here and while they don't contain a lot of information I think we can assume that its as much as can be said right now. Although Red Hat contributes generously and strongly to the Fedora Project it would be inappropriate for them to make any statement about this independent (albeit with very strong ties) project.
My take Posted Aug 17, 2008 13:34 UTC (Sun) by pizza (subscriber, #46) [Link] Call it an educated guess. I suspect that someone's account on an ImportantFedoraMachine (eg the authentication server) was compromised, and as such, they have to treat all the data on that machine as possibly compromised too. (eg local exploits become possible) This trickles down to all other machines that depend on the first; they have to ensure that they haven't been "compromised" (via info possibly obtained from the first -- eg passphraseless ssh keys) and their data messed with too. As the affected machines include the master fedora distro mirrors, this means that *all* packages ever released need to be validated (eg SHA1sum) to ensure nothing's been tampered with. As such, this isn't a problem with Fedora per se; it just happened to be The Fedora Project's servers that got hit. It's the sort of thing that could happen to anyone. Granted, if the (possibly) compromised servers hadn't included a master distro mirrors, nobody (outside of the Fedora Admins) would have really cared. But, hey, I could be completely wrong. Take this with a massive dose of salt.
OFF-TOPIC but relevant Posted Aug 17, 2008 18:16 UTC (Sun) by mgb (guest, #3226) [Link] It would be really useful if one could optionally display LWN comments in (reverse) chronological order, in order to find the recent posts.
OFF-TOPIC but relevant Posted Aug 17, 2008 18:22 UTC (Sun) by corbet (editor, #1) [Link] This page does exactly that. It may not quite be what you want, though, in that it shows all comments, not just those associated with a specific article.If all goes well, we'll have a stronger server soon and will be able to contemplate adding some smarter features to the site.
OFF-TOPIC but relevant Posted Aug 17, 2008 18:56 UTC (Sun) by mgb (guest, #3226) [Link] Thanks Jon, that helps a lot. While we're in wish-list mode: It would be nice to have the ability to optionally receive emails for each comment to some articles rather than only receiving replies to my posts.
OFF-TOPIC but relevant Posted Aug 17, 2008 18:59 UTC (Sun) by mosfet (guest, #45339) [Link] It may not quite be what you wantRight, it's better :)
OFF-TOPIC but relevant Posted Aug 17, 2008 20:44 UTC (Sun) by janfrode (subscriber, #244) [Link] Another nuisance is the lack of automatic/sensible linewrapping in comments. All the articles are quite easy to read on my cellphone's browser (Nokia E71), but comments are typically displayed with horrendously long lines, forcing lots of horizontal scrolling.
OFF-TOPIC but relevant Posted Aug 18, 2008 10:52 UTC (Mon) by rsidd (subscriber, #2582) [Link] I agree, the text width should depend on the width of the browser and the size of the font, unless it's plain text being reproduced verbatim (like a raw e-mail). Horizontal scrollbars are annoying even on a desktop browser. On the other hand I think mobile browsers should work around this, which is hardly a unique issue to LWN. LWN pages look fine on Opera Mini, on my mobile phone.
OFF-TOPIC but relevant Posted Aug 18, 2008 11:14 UTC (Mon) by janfrode (subscriber, #244) [Link] In opera mini I can read about 40 characters before I need to scroll. Your comment used more than 90 characters per line. So more than two pages to scroll sideways back and forth for every line to read.I try to optimize it by reading two lines at the time, but it gets confusing :-) I posted this as "HTML" instead of "Plain text". wonder if that will make any difference..
OFF-TOPIC but relevant Posted Aug 18, 2008 15:37 UTC (Mon) by paulj (subscriber, #341) [Link] Yep it does.. The parent to your post has his comment in 'pre' tags. Yours uses regular HTML formatting. Its very noticeable in the Comments RSS feed..
OFF-TOPIC but relevant Posted Aug 18, 2008 16:01 UTC (Mon) by corbet (editor, #1) [Link] FWIW, the problems with the current formatting of "plain text" comments are well understood; it's on my list to do something smarter. But the previous attempt (which tended to render as a bunch of long-line/short-line pairs) wasn't very nice either. I'll probably end up doing something simpler which will not try so hard to preserve formatting in the plain text mode.
OFF-TOPIC but relevant Posted Aug 18, 2008 15:39 UTC (Mon) by janfrode (subscriber, #244) [Link] Yes, my "HTML"-posting above was perfectly readable on my cellphone.
OFF-TOPIC but relevant Posted Aug 19, 2008 8:11 UTC (Tue) by lamikr (guest, #2289) [Link] And yet another nice feature would be to have "show preferred topics in all" option to select which topics (front page, security, kernel development, ...) to show in same page. This is usefull if I sometime wants to print the lwn for reading it later on train for example. In such cases I am not always interested in from all some topics which takes lot of pages.
An update on Fedora's "issues" Posted Aug 17, 2008 21:44 UTC (Sun) by luya (subscriber, #50741) [Link] yum is now working as I am currently update Rawhide. Some infrastructures like git repository don't work at the moment.
An update on Fedora's "issues" Posted Aug 17, 2008 23:32 UTC (Sun) by Lovechild (guest, #3592) [Link] And perfectly timed, no I am not the slightest bit disappointed that there first was a call get major changes done before the beta freeze and then the build system gets taken down without the slightest bit of information. If we are looking at a security issue, fine, but so far nobody knows as we are not being told. Limited disclosure would be helpful, the current situation feels very un-fedoraish and closed. Also I am worried as to how this might affect the schedule for F10 and the deadline for features. Paul and I spend time tested Mono 2.0 preview 1 to beat the deadline (even if it would cause limited breakage), it was submitted and right after the buildsystem was taken down making us miss the deadline. I surely hope we will giving feature owners another week or two to get everything to settle down otherwise the beta will suffer greatly as well as our final feature list.
An update on Fedora's "issues" Posted Aug 18, 2008 14:12 UTC (Mon) by mattdm (subscriber, #18) [Link] The schedule has been pushed back for lesser things, so I wouldn't worry about that aspect. I agree, however, that it seems quite un-fedora-ish. Something really serious must be going on.
|
||||||||||||||||