| |||||||||||||
Packaging and patchingPackaging and patchingPosted Aug 22, 2009 0:16 UTC (Sat) by rlk (guest, #47505)Parent article: On properly packaging perl
It's frustrating when distributors change things without discussion with upstream developers, even when it's a bugfix.
A number of distributions patched Gutenprint 5.0.0 (which project I lead) to add margins for certain paper sizes (Gutenprint 5.0.0 with CUPS simply chopped off anything that wouldn't fit in the printable area of the page, and the way it was explained to me by one distributor was that their management got frustrated with printing letters that were chopped off) -- specifically, this patch forced margins for A4, letter, and legal size paper. This patch had the unfortunate side effect of making it impossible to print borderless to these page sizes even on printers that had no problem with it.
The problem is that they didn't consult with us first. We weren't aware of the level of impact this was having (arguably we should have, but it just happened that what I was printing wasn't causing me problems severe enough for me to care about). We only found out about it when people started complaining to us about not being able to print borderless; I had to look at the source RPM to figure out what was going on.
We wound up fixing this issue in a very different way in 5.0.1, and we release noted it extensively. Unfortunately, distributions never bothered removing the patch, so the broken behavior continued.
Finally, in 5.2, as part of a source code rearrangement, we were able to do something about it: we moved the file to a different location in the source tree, and I inserted some comments into the offending parts of the XML source that guaranteed that any attempt to apply the patch would break, forcing someone to take a look at it and (hopefully) reading our instructions that this was a really bad thing to do, and in any event, the problem underlying this was solved.
We've always had a section in our README specifically targeted to packagers and tried to make it very visible at the top of the file that they should read that section. Among other things, we request that distributors join our development mailing list and discuss changes with us. In this case, it would have raised the urgency of a real fix, and even if distributors decided to temporarily go with the fix, we would have known what was going on, would have known who to redirect bugs to, and would have told them very clearly that in 5.0.1 that patch should go away. I tried beefing it up. I don't know what more we can do. When users complain about problems on prepackaged bits that look fishy to me, I often do tell them to install from source to see if the problem goes away -- not too different in principle from what Joerg Schilling did -- because I don't want to try to debug someone else's patches. I don't agree with everything that he's done, but I certainly sympathize with the kind of frustration this can engender.
The reality is that under the GPL (and most other FOSS licenses), there is nothing legally we could do (I believe -- IANAL) to prevent this, and people do have a right to modify the code. However, from a *pragmatic* point of view, distributors should work with upstream developers and try to understand why developers make the decisions we do. That would also help developers understand the issues distributors face.
(Log in to post comments)
Re: Packaging and patching Posted Aug 24, 2009 6:03 UTC (Mon) by nevyn (subscriber, #33129) [Link] It's frustrating when distributors change things without discussion with upstream developers, even when it's a bugfix. Packagers don't live in an ivory tower, and the vast majority will happily accept at least co-maintainers from upstream (if not just give them the package). We wound up fixing this issue in a very different way in 5.0.1, and we release noted it extensively. Unfortunately, distributions never bothered removing the patch, so the broken behavior continued. So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".
Re: Packaging and patching Posted Aug 24, 2009 10:56 UTC (Mon) by rlk (guest, #47505) [Link]
There are far more distributions than there are developers. We release Gutenprint as a source package. We don't know everyone who distributes Gutenprint, nor what patches they apply, until somebody complains. Meanwhile, everyone who doesn't complain thinks Gutenprint is buggy because of things like this.
Re: Packaging and patching Posted Aug 24, 2009 15:26 UTC (Mon) by mcopple (guest, #2920) [Link]
quoth:
So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".
-------
It seems to me that the correct workflow would be for downstream packagers / maintainers to discipline themselves to check for upstream changes on a regular basis, rather than relying on upstream to figure out who is packaging their product and notify.
It is incumbent on the upstream developer to have channels available to get the word out -- a developer's list, a bug tracking system, release notes, etc. But it is not reasonable to expect upstream to check the bug trackers of all of the scores of active distributors out there.
Re: Packaging and patching Posted Aug 29, 2009 19:30 UTC (Sat) by dirtyepic (subscriber, #30178) [Link]
> It seems to me that the correct workflow would be for downstream packagers / maintainers to discipline themselves to check for upstream changes on a regular basis, rather than relying on upstream to figure out who is packaging their product and notify.
At the very least they should find the time to browse the release notes.
Re: Packaging and patching Posted Aug 24, 2009 17:20 UTC (Mon) by jimi (guest, #6655) [Link]
Quote:
So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".
This is absolutely the wrong attitude and workflow pattern. The software developers should have no responsibility at all to file bugs to various distributors' mailing lists. The job of a distributor is to package the original source and optionally patch *and notify the upstream developers* of any bugs. This notification must go from distributor to developer, not the other way around. The article notes that Red Hat would have preferred that the perl guys had filed a bug in the Red Hat bug tracking system. That is completely the wrong expectation. It is Red Hat who should possibly be filing bugs, not the other way around.
There is a possible exception to this rule: when the upstream developers are unreachable. In that case the distributor may choose to become the new maintainers of the software.
Re: Packaging and patching Posted Aug 24, 2009 17:52 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
among other things, you can't expect the upstream developers to run every distro that exists to be able to test them.
Re: Packaging and patching Posted Aug 24, 2009 18:41 UTC (Mon) by nevyn (subscriber, #33129) [Link] So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".This is absolutely the wrong attitude and workflow pattern. The software developers should have no responsibility at all to file bugs to various distributors' mailing lists. The job of a distributor is to package the original source and optionally patch *and notify the upstream developers* of any bugs. This notification must go from distributor to developer, not the other way around. FFS, read what I wrote. Upstream had already seen the patch the distro. did ... and added release notes about how they'd fixed it differently (to stop other problems). But couldn't spend the couple of minutes directly telling the distro? ... and this is the distros. fault for "not doing it's job". I think not. But in the general case ... Yes, in a software developers perfect world they'd release upstream code full of "features". Distros. would then fix everything for them, and spend huge amounts of time rewritting those patches to make the developers happy while they spend their time adding more features. Back in the real world packagers are often only spending a fraction of their time doing packaging, often among a bunch of packages, so that "blah install random-crack" works. They aren't unpaid upstream minions, and they sure as hell aren't paid for their "job". Yes, they are hopefully following development in some way, but they are still unpaid humans and can easily miss things. On the other side upstream developers are often spending a majority of their time on a 1-3 projects. So those upstream developers spending an hour a month or so to make sure the packaging in (at least) the major distros. isn't broken in some way, doesn't seem like too much to ask IMNSO. And, yeh, for larger projects (like perl) you might expect that the distros. have more resources to put towards upstream ... but at the same time upstream will often have lots more resources to look downstream.
Re: Packaging and patching Posted Aug 25, 2009 0:10 UTC (Tue) by rlk (guest, #47505) [Link]
In this particular case (the Gutenprint issue) I did report it to my contact at the particular distribution I use. He ack'ed it, but it still didn't get changed. I only found out much later that this particular hack got propagated to other distributions. It wasn't a packaging problem, by the way. It was a badly conceived patch -- more like the Debian OpenSSL debacle than the Perl issue.
Checking all of the major distributions -- let's say, Fedora, Ubuntu, OpenSUSE, Debian unstable, Mandriva -- is a lot of work. It also means having the disks to do a lot of installations, the time to spend downloading the latest bits, the time to run tests, and so forth. No, this isn't practical.
Our automated test suite wouldn't have caught this in any case, since the problem was that the definition of letter and A4 paper was changed in a way that's legal within Gutenprint (and would therefore not fail functional and regression testing) but which would yield unexpected results.
The interaction I would have preferred is:
* Distributor files a bug/complains on the -devel mailing list that the behavior is unacceptable.
* We reply that it's behaving as designed.
* Distributor says "well, maybe so, but but it's still broken and we need to do something about it."
* Maybe we figure out the ultimate solution and get a fix in time for the next release (in which case everyone's happy) or maybe not, and distributor proposes the particular hack they actually did. More likely the latter, since the ultimate fix needed quite a bit of testing to get right.
* Distributor probably decides, after discussion, to accept that borderless printing won't work to those paper sizes and tells us that they're going to release with it. We suggest that they release note what's going on and keep checking back for a real fix.
* We get the fix out a bit quicker than we ultimately did, since we realize how critical it is in the real world; distributor picks it up and either patches the current release or at least makes sure the next release has the real fix (without applying the bogus patch again).
Re: Packaging and patching Posted Aug 27, 2009 7:42 UTC (Thu) by job (guest, #670) [Link]
I expect a package maintainer to have a clue what changes went into the package they maintain. At least for security reasons, so when upstream gets hacked and/or goes crazy and inserts malware in the code it gets caught.
Otherwise all security that surround distributions is for nothing if they package whatever comes upstream (via unsecured FTP sites and whatever).
Not even reading the release notes is in my opinion very far from having a clue what changes are applied!
Packaging and patching Posted Aug 24, 2009 6:17 UTC (Mon) by dgm (subscriber, #49227) [Link]
This is a serious argument against distro packagers playing developers.
In my opinion, a packager should NEVER, EVER patch something. If it doesn't work, complain to upstream until it's fixed. If it's not fixed, drop the package or revert to a previous version. The case of security fixes is not different. But that's just my opinion, of course.
Packaging and patching Posted Aug 24, 2009 8:44 UTC (Mon) by Trou.fr (subscriber, #26289) [Link]
and when upstream is not replying ? Not ready to accept any patch ? When some packaging requirements are only met with patches ?
And regarding security issues, i'd rather have a quick patch by the packager than a week late patch by upstream.
Packaging and patching Posted Aug 24, 2009 12:19 UTC (Mon) by dgm (subscriber, #49227) [Link]
Just drop the damn thing. Or find someone to fork it.
A quick fix made by a packager that doesn't understand the implications can create more problems that it solves (Debian OpenSSL anyone?) Also your arguments is based on the *false* assumption that packagers do release fixes faster than upstream. And now that we're at it, instead of just packaging everything under the sun, why not just talk to upstream regarding their stance behind the code _before_ creating a package for it?
Packaging and patching Posted Aug 24, 2009 14:53 UTC (Mon) by foom (subscriber, #14868) [Link] Why? Because I like distros that have everything under the sun, and work as an intermediary between possibly-uncooperative upstreams and me. And who do the work that most upstreams don't do to make the software work together easily and consistently.There have been a couple disastrous changes, indeed, but the vast majority of minor modifications I greatly appreciate having.
Packaging and patching Posted Aug 25, 2009 7:04 UTC (Tue) by dgm (subscriber, #49227) [Link]
I also did like them, back when there was not much OSS software. But it's not 1995 any more, and now we have loads and loads of OSS packages for almost everything.
I tend to think that the 80/20 rule also applies to software (yes, all software), thus much of what is there is crap. A process that sinks crap and lets the good stuff stay afloat is really needed to ensure a sane evolution of the good software. Some distributions (Debian and Ubuntu at least) let this up to the user, in the form of a popularity contest. Wrong. This is _the_ task that the packager should be doing. All those "minor modifications" just make things _unnecessarily_ harder for everybody.
Packaging and patching Posted Aug 25, 2009 15:38 UTC (Tue) by foom (subscriber, #14868) [Link] Well, I like my software modified and configured to work better as a whole. And I like that no matter what software I want (even if you consider it crap), it's almost certainly already available in Debian.There's space in the world for multiple distributions with different goals, and I bet there's at least one which doesn't modify upstream source code that you can use.
Packaging and patching Posted Aug 27, 2009 23:16 UTC (Thu) by branden (subscriber, #7029) [Link]
Your citation of the Debian OpenSSL fiasco is a bullshit example.
As was widely reported at the time, Debian's package maintainer *did* take
There was lots of fail in the Debian OpenSSL situation, but failure to run
Packaging and patching Posted Aug 28, 2009 0:12 UTC (Fri) by rlk (guest, #47505) [Link]
That was my recollection about what happened; thank you for correcting the details.
In the Gutenprint example, nobody ever attempted to contact us (we have a development mailing list that's noted on our web site that's open subscription), even to notify us that they were making the change in question. We were left to find out about it from complaints from unhappy users.
Packaging and patching Posted Aug 29, 2009 20:51 UTC (Sat) by dirtyepic (subscriber, #30178) [Link]
Um, what? Packagers _do_ release fixes faster than upstream. They even release _upstream_ fixes faster than upstream. If they didn't, there wouldn't be a reason to patch in the first place.
I can patch a freetype bug in Gentoo and have it out to users sometime within the next hour (whenever the next rsync mirror update is). I don't think they can make a release that fast. In fact, we patched a security vulnerability in freetype-2.3.9 back in May, and as of now there still hasn't been a new upstream release. So, should we drop freetype or fork it? (note: i don't mean to pick on freetype here, it's just a package i happen to maintain)
And by "release" I mean get the fix out to the people encountering the bug, not fix it in the repo for whenever the next release may be. If you have some other definition then please share it.
We also recently patched fontconfig because recent upstream changes were causing problems with our sandboxed build environment (packages calling fc-cache during `make install` would fail because fc-cache now runs chmod on /var/cache/fontconfig and we don't allow packages to change permissions on files or directories they don't own outside of the DESTDIR during install). I did let upstream know about the issue, but ultimately I fixed it locally because it was our policy causing the issue, not upstream. Debian also modifies most packages to comply with their policies. Expecting upstream to accommodate the (sometimes conflicting) packaging policies of every major distro is ridiculous.
Trust me, we would love it if everything worked the way we wanted out of the box. We don't go through the trouble of maintaining large patchsets for the sheer enjoyment it gives us.
Packaging and patching Posted Aug 24, 2009 15:32 UTC (Mon) by mcopple (guest, #2920) [Link]
It doesn't have to be an either/or proposition.
When downstream discovers the problem, open a bug for it with upstream. If upstream doesn't have it fixed by the time the distributor needs it, fix it yourself, display it prominently in the release notes, notify upstream you've done it.
If upstream still doesn't do anything about the problem, then it is upstream's problem; you keep the patch in place and your customers are satisfied. If upstream resolves the issue to the distributor's satisfaction, use upstream and quietly drop the downstream patch.
Packaging and patching Posted Aug 24, 2009 18:46 UTC (Mon) by nevyn (subscriber, #33129) [Link] Sure, Fedora should just fork python or drop it completely (upstream have refused to take multilib seriously). Not to mention what awesome support you'd get off all upstreams for everything in RHEL-5 atm. You seem to have awesome ideas, do you produce a newsletter I can subscribe to?
|
|||||||||||||