This isn't new

Posted Jul 16, 2012 17:17 UTC (Mon) by dwmw2 (subscriber, #2063)
Parent article: Left by Rawhide

I'm not sure I'd agree with the assertion that the 'No Frozen Rawhide' policy has "changed the nature of the Rawhide distribution in fundamental ways.".

What goes into Rawhide does go into the next Fedora release. It's not like linux-next where stuff is supposed to be present only if it's going to be sent to Linus in the next merge window; with rawhide it actually is true. What's pushed to Rawhide now will be in Fedora 18, unless it's subsequently updated again.

The only thing that changed is that in the past, Rawhide was frozen while a Fedora release was being stabilised. During that period, there was nowhere that people could commit the shiny new exciting stuff. This lead to a huge amount of changes hitting Rawhide when the floodgates were opened just after a release.

One of the things that the 'No Frozen Rawhide' policy was supposed to do was remove that damming effect, and improve stability in the "immediate post-release period" that you mention.

Also, users have always been told that "if it breaks in Rawhide you get to keep both pieces"; that isn't new either. And it's always been variable — some developers really don't care about their packages in Rawhide until a new release is imminent, while others do try to keep it working.

I believe that Fedora policy does forbid having a package in the stable distro which is newer than the one in Rawhide. It's the same policy that forbids Fedora N from having a new version of a package than Fedora N+1. Upgrades should, well, upgrade.

I don't think there's anything really new here. You take your life in your hands when you run Rawhide, and you've been fairly lucky if it's been consistently usable for you for a number of years. I note your 'exploded' link actually refers to a mail describing how to fix things after the problematic update, thus implying that a full restore from backup shouldn't have been necessary?

to post comments

This isn't new

Posted Jul 16, 2012 17:32 UTC (Mon) by corbet (editor, #1) [Link] (25 responses)

The thing that I think has changed is that developers are not actually running Rawhide and are less concerned with whether it works or not. It has changed in my experience. It's not that things break, one expects that. It's what happens afterward that is different.

And no, the instructions to fix the system did not work for me. My life does not currently allow time to figure out what else was hosed; I really had to just give up on it.

This isn't new

Posted Jul 16, 2012 18:49 UTC (Mon) by jspaleta (subscriber, #50639) [Link] (3 responses)

Have you looked at using the following scenario.

run N release
run N+1 pre-release branch when it gets branched from rawhide
stay on N+1 pre-release branch through official N+1 release
jump to N+2 pre-release branch when it gets branched from rawhide

Would this be closer to matching your expectations?

-jef

This isn't new

Posted Jul 16, 2012 19:00 UTC (Mon) by corbet (editor, #1) [Link] (1 responses)

That possibility is mentioned in the conclusion. Certainly it would be the easiest change to make from running full frontal Rawhide.

This isn't new

Posted Jul 16, 2012 19:06 UTC (Mon) by jspaleta (subscriber, #50639) [Link]

bah sorry about that, reading is hard.

-jef"ELACKOFCOFFEE"spaleta

This isn't new

Posted Jul 17, 2012 6:40 UTC (Tue) by ghane (guest, #1805) [Link]

I am doing this with Ubuntu. Soon after 11.04 was released, I shifted to 11.10 pre-alphas, updating daily; and so on.

I am now on quantal 12.10. Things break every couple of weeks, but logging into console and running an update again has fixed those.

--
Sanjeev

This isn't new

Posted Jul 16, 2012 20:14 UTC (Mon) by dwmw2 (subscriber, #2063) [Link] (13 responses)

You may be right that what happens after a breakage is different — but that was always extremely variable from one developer to another anyway. So you'd need a reasonable sample size, over time, to really be sure of that.

I'm also not sure that such a change, if it's real, could necessarily be blamed on the no-frozen-rawhide thing. There have been other changes in Fedora packaging over time. There's much more of a focus on "packagers" these days, and much less on "package maintainers" who actually get their hands dirty and work on the code. This naturally translates into a much less happy experience for those who file bugs.

This isn't new

Posted Jul 16, 2012 21:42 UTC (Mon) by johannbg (guest, #65743) [Link] (3 responses)

That's my assessment as well as in that quality of the components in the distribution has been thrown overboard for the quantity of components in the distribution...

This isn't new

Posted Jul 17, 2012 9:24 UTC (Tue) by drag (guest, #31333) [Link] (2 responses)

Which is yet another reason why distributions need to keep their scope limited, maintain standardized APIs across distributions, and have the upstream developers themselves package the software.

There is no way that you can expect a OS to bundle everything that everybody would possibly want run on it. Extensibility is important and users lack the knowledge, time, and drive to port and compile the software they want to use themselves.

This isn't new

Posted Jul 17, 2012 19:40 UTC (Tue) by epa (subscriber, #39769) [Link]

But equally, you can't expect an upstream developer to package the program for every possible distribution and architecture. Even expecting the upstream developer to support two is a losing proposition, since most people will have one distro that they use regularly.

Is there some way the distros and the upstreams can meet halfway? Source code could be distributed in a standard source package format, which as well as the usual configure script contains metadata about what files are built, what libraries are required and so on. This in turn can generate source packages for rpm, dpkg, Gentoo and so on using translator tools maintained by each distro. This would not be appropriate for core packages, which usually have a lot of distro-specific hacking, but for the long tail of applications and libraries it could bridge the gap between 'distro must package all possible programs' and 'upstream muat package for all possible distros'.

This isn't new

Posted Jul 20, 2012 10:57 UTC (Fri) by misc (subscriber, #73730) [Link]

You never tried to say "no" to a packager. Most of them all go with a frenzy phase of "let's add as much stuff we can to the distro, because users ask for it and threathen to go elsewhere". Then, that go to "we need more packager to take care of that", then "we need more QA and people doing the boring bits", except that boring stuff are not done, because that's boring.

A solution would be to take in account ressources as a whole, ie, you cannot upload a package unless there is enough people to take care for QA and updates.

But such view are far from being popular IMHO ( and most users do not care, like most do not care about sustainability ).

This isn't new

Posted Jul 17, 2012 17:14 UTC (Tue) by nim-nim (subscriber, #34454) [Link] (8 responses)

This time (and all the previous times Rawhide spectacularly failed) the breakage came from Red Hat-written code pushed in Rawhide by the people who wrote it.

It's not pure packagers which are the problem. They are actually very careful about what they push precisely because they're not sure they'll be able to fix it if it breaks. The problem is people who write code and package it themselves, and feel they won't make any mistake ever, and anyway if it breaks they owe nothing to Rawhide guinea pigs which are only there to help them meet coding deadlines (gross generalization, dracut people are more aware than most of the risks though they do have to contend with systemd abrupt changes nowadays)

That's the typical problem you get in any project where you have devs that perform operational jobs, if you didn't make sure beforehand that they understand operations means they have real users and have to fix the problems they cause as soon as they cause them (unlike broken code that can languish in a VCS for days without consequences)

This isn't new

Posted Jul 18, 2012 5:08 UTC (Wed) by nim-nim (subscriber, #34454) [Link]

Case in point:

* Tue Jul 17 2012 020-96.git20120717 ← dracut working again
- disabled systemd in the initramfs, until it works correctly

* Wed Jul 11 2012 020-84.git20120711
- add back "--force" to switch-root, otherwise systemd umounts /run

* Wed Jul 11 2012 020-83.git20120711
- more systemd journal fixes
- nfs module fix
- install also /lib/modprobe.d/*
- fixed dracut-shutdown service
- safeguards for dracut-install
- for --include also copy symlinks

* Tue Jul 10 2012 020-72.git20120710
- stop journal rather than restart
- copy over dracut services to /run/systemd/system

* Tue Jul 10 2012 020-70.git20120710
- more systemd unit fixups
- restart systemd-journald in switch-root post
- fixed dracut-install loader ldd error message

* Mon Jul 09 2012 020-64.git20120709
- fixed plymouth install
- fixed resume
- fixed dhcp
- no dracut systemd services installed in the system

* Mon Jul 09 2012 020-57.git20120709
- more fixups for systemd-udevd unit renaming

* Mon Jul 09 2012 020-55.git20120709
- require systemd >= 186
- more fixups for systemd-udevd unit renaming

* Mon Jul 09 2012 020-52.git20120709
- fixed prefix in 01-dist.conf

* Fri Jul 06 2012 020-51.git20120706 ← start of huge breakage
- cope with systemd-udevd unit renaming
- fixed network renaming
- removed dash module

And:

* Tue Jul 03 2012 Lennart Poettering - 186-1
- New upstream release

This isn't new

Posted Jul 18, 2012 19:17 UTC (Wed) by nix (subscriber, #2304) [Link] (6 responses)

Ah, software developer optimism. You gotta love it.

(Of course *I* would never suffer from such a delusion. My code simply has no bugs. I know because I fixed all the bugs I found, and now I can find no more bugs so I know there must be no more bugs. Also, the Earth has a green sky and six retrograde moons.)

This isn't new

Posted Jul 20, 2012 3:27 UTC (Fri) by doogie (guest, #2445) [Link] (3 responses)

The only code that is bug free is that which is not yet written.

This isn't new

Posted Jul 20, 2012 8:47 UTC (Fri) by liw (subscriber, #6379) [Link] (2 responses)

Code not yet written doesn't run, which is a bug. ;-)

This isn't new

Posted Jul 20, 2012 9:24 UTC (Fri) by nim-nim (subscriber, #34454) [Link] (1 responses)

Code not written has no users and thus by definition it is perfect.

That's the uncertainty principle applied to software: bugs not observed do not exist (yet)

This isn't new

Posted Jul 20, 2012 12:41 UTC (Fri) by nix (subscriber, #2304) [Link]

Not so. I currently suspect there is at least one bug in part of my code that has never run, but I am attacking it later, when other bugs are fixed so code flow can reach that bug and prove its existence. This bug is in an indeterminate state. :)

Schneier's law of software?

Posted Jul 22, 2012 4:40 UTC (Sun) by Max.Hyre (subscriber, #1054) [Link] (1 responses)

Sounds as if we have a programming equivalent of Schneier's Law:

Any programmer can write a program so good he can't find any bugs in it.

Schneier's law of software?

Posted Jul 23, 2012 17:45 UTC (Mon) by Tet (subscriber, #5433) [Link]

It pretty much already exists, and long predates Schneier's:

"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"
-- Brian Kernighan, "The Elements of Programming Style"

This isn't new

Posted Jul 16, 2012 21:29 UTC (Mon) by johannbg (guest, #65743) [Link] (1 responses)

I disagree with the statement that we have fewer maintainers running rawhide then we did a while back but argue instead we have roughly the same amount ( and probably more or less the same maintainers ) but the number of packages has increase which might give the impression that the number of maintainers running rawhide has decreased.

This isn't new

Posted Jul 17, 2012 13:33 UTC (Tue) by dwmw2 (subscriber, #2063) [Link]

FWIW I almost never run Rawhide, and I almost never did. The no-frozen-rawhide made no difference to me. The only time in history that I recall any of my machines spending much time running rawhide was when we were doing the PowerPC port to new machines. And that was mostly installer testing.

This isn't new

Posted Jul 16, 2012 21:49 UTC (Mon) by ceplm (subscriber, #41334) [Link] (3 responses)

I think the point is not that Rawhide is now broken (after all, it is supposed to eat your babies, right?), but that nobody uses it is so often broken for a long long time, because not many developers uses it as their production machine. Old-time Rawhide used to be broken often (I was told that "there are days when Rawhide actually boots up" when I was asking about its usefulness in approx. FC6 time frame), but it was also rapidly fixed because the developer's own machine was broken as well.

Not saying that it is good or bad (yes, I was using that "update in time of Alpha policy" as well when I was still using Fedora as my main system ;now I have to use RHEL for the work-related reasons), just clarifying what I read in the original article.

Matěj

This isn't new

Posted Jul 17, 2012 17:02 UTC (Tue) by nim-nim (subscriber, #34454) [Link] (2 responses)

Over the years, some Fedora developers have successfully lobbied to impress the 'Rawhide eat babies' and 'so what' if it's broken, it's Rawhide' on Fedora users.

It wasn't always the case. A decade ago there was some pride in keeping rawhide usable (and using it oneself). It's on (then rhl) list I've read first the 'dogfoodable' term (about rawhide).

It's sad that in their eagerness not to be held accountable for what they put in rawhide some packagers are poisoning the well from which Fedora itself (and derivatives) come.

This isn't new

Posted Jul 17, 2012 20:40 UTC (Tue) by dwmw2 (subscriber, #2063) [Link] (1 responses)

I agree. I'm not trying to defend this situation — just pointing out that I don't think it's caused by the 'No Frozen Rawhide' thing.

This isn't new

Posted Jul 18, 2012 7:25 UTC (Wed) by ceplm (subscriber, #41334) [Link]

I would even go further ... even if Rawhide quality really went down (I don't use Rawhide myself, so I cannot testify about that), and even if such decrease in quality was caused by the current Fedora development work-flow (and that's a big if), it still doesn't follow that such change in work-flow wasn't worthy of possible increase in overall quality of released Fedoras (if that happened). Maybe we just have to adjust our upgrading policy to the new reality and the total may be still worthy.

And yes, nevertheless I am still longing after those days when men were men, women were women, and both of them (maybe helping each other) used Rawhide. ;)

This isn't new

Posted Jul 17, 2012 19:24 UTC (Tue) by nwnk (guest, #52271) [Link]

I don't think that's entirely fair. The class of xserver ABI bug cited in the "Is Rawhide Supposed To Be Useful" thread you linked pretty much can't happen anymore, like, ever again. If I didn't care, I wouldn't have fixed it. (I will happily admit it was not fixed on a timescale I feel was acceptable, but I had this whole other point in the thread about that, that nobody bit on then and I don't expect will now.)

I don't think it's reasonable to just toss rawhide to the "it will eat your data" wolves. But I also don't think it's reasonable to expect it to be your daily driver if you're not willing to do some of your own dirty work (and others'). That's what releases are for. That's the entire point of calling them releases: they are things you can expect to consume without needing to think or worry about.

I can see why rawhide isn't meeting your needs, but I'm a bit unsure what would.

Definitely not new; definitely always been an awful idea

Posted Jul 16, 2012 21:25 UTC (Mon) by louie (guest, #3285) [Link]

Also, users have always been told that "if it breaks in Rawhide you get to keep both pieces"; that isn't new either. And it's always been variable — some developers really don't care about their packages in Rawhide until a new release is imminent, while others do try to keep it working.

While it is true that this has always been the policy, it has also always been an awful policy, pretty much guaranteeing that only the clinically insane, like our beloved editor ;), perform QA on Rawhide.

Perhaps more importantly, it also has the awful side-effect of supporting waterfall-y (read: bad) developer habits. All developers should always be encouraged, by hook and by crook, to keep HEAD buildable and runnable. Explicitly telling users "HEAD will frequently not be runnable" implicitly tells developers that it is perfectly fine to leave HEAD in a non-runnable state. That kind of attitude leads to worse software.

[Not that you don't sometimes need to break HEAD; but it should be something done only when the other options are very bad, and ideally after much testing on a branch. Same should apply to Rawhide.]