Weekly Edition
Return to the Development page
| |||||||||||||
On properly packaging perl
The perl5-porters recently saw a rather acrimonious discussion on how the
Red Hat and Fedora distributions choose to package the Perl language and
associated modules. Things have calmed down (the parties have essentially
agreed to disagree), but an interesting issue remains: what can development
projects do if they're unhappy with how distributors are treating their
code?
When Tom Christiansen gets irritated with somebody, one can generally be assured that they will know about it. In this case, Tom let the world know that he was not happy with the way Red Hat packages and distributes Perl. The complaint is that, if one installs "perl," one does not get the entire Perl 5.10 distribution. Tom says:
Of these, the most egregious omission is CPAN.pm itself, as its
absence precludes the easy fix of using CPAN to grab what Redhat
forgot. Other pieces notably missing include Text::Harness and
h2xs -- although h2ph *is* included! Go figger.
As it happens, it is possible to get the entire 5.10 distribution; one simply needs to install "perl-core" instead of plain "perl." Tom asserts that this arrangement is confusing; "perl-core" should not be a superset of "perl"; the package called simply "perl" (which is what most users will install) should be the thing that the Perl developers shipped. He says that Red Hat's arrangement causes confusion, with users not knowing if they actually have "Perl" or not. Beyond that, it goes against the developers' intent, which included providing users with all the basic modules they needed from the outset. Many users, it seems, will not (or cannot) add extension modules to their systems; the Perl developers tried to ensure that these users would have a minimally-functioning system available to them. But, alas:
But why did we bother? For all that effort is now undermined, even
unravelled, if now vendors choose to strip down the real Perl
distribution by paring away pieces that we've decided to ship. I
don't envy them their positioning themselves as forkers, doomed to
winnow and weed and forever maintain, but that's their own
problem. Worse is when they present this stripped minidistro as the
real thing. It's a misleading and confusing state of affairs which
should be discouraged.
Tom "spot" Callaway explained Red Hat's packaging decisions:
At the time of the split, we had a LOT of packages which depended
on simply "perl". We also had a lot of people who wanted individual
perl modules updated that live within the "perl tarball", which is
a complicated task to undertake. By splitting the perl modules out
into separate subpackages we were at least able to allow people to
build newer RPMs. In addition, the split had the added benefit that
people who did not need nor want all of the components built with
perl could get a system without them. This lowered the default
Fedora installation footprint.
Tom says that the package naming could be changed, but that would have a number of unfortunate effects on users. That said, there are some possibilities for improving the situation, but the best course is not entirely clear. Tom Christiansen was not completely happy with the explanation, but he also seemed to understand the pressures which led to Red Hat's way of doing things.
In the near term, it looks like things will Upstream developers are harder to convince; they have released the software in the form that they think is best, so it can be discouraging to see others messing with it. Most developers suffer in stoic silence, taking comfort in the fact that their software is finding a wider audience. But others have taken various types of action in an attempt to influence how distributors treat their code. Consider some examples. The kernel developers changed their release process dramatically with the (successful) goal of reducing the number of patches applied by distributors. Some developers do their own packaging. Jeff Waugh has suggested that much of the packaging role of distributions could be "disintermediated" entirely, with users routinely getting their software directly from its developers. Jörg Schilling inserted (widely ignored) statements into his code stating the alteration of some parts of the program would be a copyright violation. Firefox makes aggressive use of its trademarks to control the changes made by distributors. Daniel Bernstein took things to a (non-free) extreme, only allowing qmail to be distributed if it had not been modified at all; this restriction inhibited qmail development for years until the code was finally released into the public domain. In the end, releasing code under a free license means giving up control over what is done with it. So free software developers will always be at the mercy of distributors, who will always have the right to make the changes they think are necessary. The occasional grumble notwithstanding, the system works pretty well; all of the parties involved share an interest in having the software work as well as possible for their users. (Log in to post comments)
On properly packaging perl Posted Aug 21, 2009 21:15 UTC (Fri) by Baylink (subscriber, #755) [Link]
I can understand both sides of this argument; CPAN vs RPM has always been a problem...
but TomC's right: they should have included CPAN itself in the base package, and perl-core is a bad choice of name, sounding like a *subset* of 'perl', not the superset it apparently is.
But then, Fedora's not a "real" distro, either, right?
On properly packaging perl Posted Aug 21, 2009 21:50 UTC (Fri) by spot (subscriber, #15640) [Link]
Really? Fedora's not a real distro? This is news to me. :P
On properly packaging perl Posted Aug 21, 2009 21:55 UTC (Fri) by Baylink (subscriber, #755) [Link]
I'd always had the impression that Fedora was supposed to be treated as a test sandbox for Red Hat, not something you depended on for Real Work.
Did *that* change while I wasn't looking, too? Damn. :-)
On properly packaging perl Posted Aug 21, 2009 22:06 UTC (Fri) by dowdle (subscriber, #659) [Link]
Depends on what you call "real work". Is desktop use real work? Fedora is my preferred distro for desktop usage because I like the bleeding edge stuff, don't mind the constant stream of updates, and don't mind occasional (almost always only minor) breakage. Makes me check bugzilla more often and contribute. :)
I don't often run Fedora on servers (or server virtual machines / containers) unless I have a need for something newer than RHEL5/CentOS5 and that isn't very often.
On properly packaging perl Posted Aug 21, 2009 22:21 UTC (Fri) by quotemstr (subscriber, #45331) [Link] I don't often run Fedora on servers (or server virtual machines / containers) unless I have a need for something newer than RHEL5/CentOS5 and that isn't very often.I regularly backport Fedora packages to RHEL when I need a newer version of a particular program or library. It works wonderfully (though of course, I have to manually monitor security lists for that package from that point onward.)
On properly packaging perl Posted Aug 23, 2009 19:41 UTC (Sun) by mmahut (guest, #45550) [Link]
What about EPEL?
On properly packaging perl Posted Aug 23, 2009 19:44 UTC (Sun) by quotemstr (subscriber, #45331) [Link]
EPEL is great, but doesn't have everything. For example, the RHEL5 version of sudo spits out an infuriating error message if auditing is not available; the bug is fixed in the Fedora 11 version, which builds and runs fine on a RHEL5 machine.
On properly packaging perl Posted Aug 24, 2009 14:18 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]
Just to clarify, EPEL is a add-on repository and by policy, wouldn't have updated versions of sudo since that would conflict or override packages in the base OS. What it would have instead, is packages not available in RHEL.
On properly packaging perl Posted Aug 21, 2009 23:57 UTC (Fri) by Baylink (subscriber, #755) [Link]
This was roughly what I was talking about, yes. When they blew it on 8, and then decided to go commercial with 9, I switched to SuSE, and I now use it for everything. Except Zimbra. They don't support it commercially, so I use CentOS 5 and lie.
On properly packaging perl Posted Aug 21, 2009 22:06 UTC (Fri) by smoogen (subscriber, #97) [Link]
It is in the same way Debian is the test sandbox for Ubuntu (or vice versa)... :) :)
On properly packaging perl Posted Aug 21, 2009 23:03 UTC (Fri) by BenHutchings (subscriber, #37955) [Link]
Debian unstable is the test sandbox for Debian and Ubuntu releases.
On properly packaging perl Posted Aug 22, 2009 5:28 UTC (Sat) by drag (subscriber, #31333) [Link]
and like Fedora it makes a kick-ass workstation/desktop OS.
It's just something that won't ever be mainstream and is dubious for production environment.
On properly packaging perl Posted Aug 22, 2009 8:03 UTC (Sat) by yoe (subscriber, #25743) [Link]
You are aware that Debian is used on 80.000 systems in the Extramadura region in Spain, right?
You are aware that Debian is used on several thousand systems in Munich too, right?
You are aware that the servers of Second Life all run Debian, right?
These are just the three first examples I could come up with off the top of my head. They all sound pretty 'mainstream' and 'production' to me.
I'm not saying Debian Rulez the World, but saying "won't ever be mainstream and is dubious for production" is just trolling.
On properly packaging perl Posted Aug 22, 2009 8:13 UTC (Sat) by xanni (subscriber, #361) [Link]
Did you miss the word "unstable" after "Debian"?
On properly packaging perl Posted Aug 29, 2009 18:55 UTC (Sat) by dirtyepic (subscriber, #30178) [Link]
sssh. he's busy aggrandizing.
On properly packaging perl Posted Aug 22, 2009 8:15 UTC (Sat) by Sho (subscriber, #8956) [Link]
He meant Debian's *unstable* branch, afaict.
On properly packaging perl Posted Aug 21, 2009 21:48 UTC (Fri) by spot (subscriber, #15640) [Link]
So, I disagree with Jonathan's assessment that nothing is going to change. We (the Fedora perl maintainers) listened to the arguments and we're going to redo the subpackages in the Fedora 13 (and RHEL 6) release timelines so that there is:
perl: depends on all subpackages provided in upstream tarball
(To preempt the "why not in Fedora 12", we decided to hold off on making major perl changes during this short release cycle, since we have several big changes in addition to this to make, including the brown paper bag -DDEBUGGING fix)
Honestly, I would have preferred it if Tom Christiansen had filed a bug against our perl packages, to give us the opportunity to discuss this before jumping to huge conclusions and breathing fire, but since all of us are watching the p5p mailing list (if perhaps, not daily, at least weekly), we were able to respond to the issue in a reasonable amount of time.
Most of our followup discussion has happened on the fedora-perl-devel-list, you can see it here:
https://www.redhat.com/archives/fedora-perl-devel-list/20...
On properly packaging perl Posted Aug 21, 2009 22:03 UTC (Fri) by paravoid (subscriber, #32869) [Link]
FWIW, Debian & distros based on it has perl-base and perl/perl-modules (the first being architecture dependent while the latter being architecture neutral).
It may be better to lean on -base for consistency.
Perl Progress Posted Aug 21, 2009 23:41 UTC (Fri) by corbet (editor, #1) [Link] OK, so I missed the discussion on the Fedora-specific list. Guess I don't watch enough mailing lists...I stand corrected, sorry.
Perl Progress Posted Aug 22, 2009 1:15 UTC (Sat) by mmcgrath (subscriber, #44906) [Link]
> OK, so I missed the discussion on the Fedora-specific list. Guess I don't watch enough mailing lists...I stand corrected, sorry.
:: cough cough :: perhaps it's time to talk to people before writing about them and their intentions?
Perl Progress Posted Aug 22, 2009 2:55 UTC (Sat) by sbergman27 (guest, #10767) [Link]
Perhaps you would like to donate your time to help? Please try to keep criticisms constructive.
Perl Progress Posted Aug 22, 2009 2:59 UTC (Sat) by mmcgrath (subscriber, #44906) [Link]
> Perhaps you would like to donate your time to help? Please try to keep criticisms constructive.
Yeah, that was constructive criticism. Had he done it (I suggested he do the same thing in one of his last columns), he wouldn't have had to retract one of his conclusions. I don't have time to donate so instead I donate my money.
RHEL6 based on Fedora 13? Posted Aug 22, 2009 23:05 UTC (Sat) by gurulabs (subscriber, #10753) [Link]
Do I read this to mean that RHEL 6 is going to based off of Fedora 13?
RHEL 6 coming in May/June 2010?
RHEL 6? Posted Aug 22, 2009 23:06 UTC (Sat) by gurulabs (subscriber, #10753) [Link]
Do I read this to mean that RHEL 6 is going to based off of Fedora 13?
RHEL 6 coming in May/June 2010?
RHEL 6? Posted Aug 23, 2009 17:36 UTC (Sun) by spot (subscriber, #15640) [Link]
I would not draw either conclusion.
All I can say is that the Fedora perl maintainers contain the same folks who will maintain it for RHEL 6, and that this change will make it into RHEL 6.
On properly packaging perl Posted Aug 21, 2009 23:40 UTC (Fri) by MattPerry (guest, #46341) [Link]
This is another example of why I feel that software providers should provide packages and the distros should use them. Then you know you're getting the real McCoy and it will work, and be supported, as designed.
On properly packaging perl Posted Aug 22, 2009 0:07 UTC (Sat) by khc (subscriber, #45209) [Link]
except then upstream would need to know how to create packages for dozens of distros.
On properly packaging perl Posted Aug 22, 2009 1:58 UTC (Sat) by MattPerry (guest, #46341) [Link]
So? That sounds like a problem that could be fixed.
On properly packaging perl Posted Aug 22, 2009 6:18 UTC (Sat) by bojan (subscriber, #14302) [Link]
Not likely. The moment upstream does something like link in a library that some distro doesn't like, you'd have flame war on your hands. Or the opposite - if it doesn't link it in. Etc., etc.
The upstream packages are raw materials. Distros peruse them to make a meaningful components for their system.
On properly packaging perl Posted Aug 24, 2009 6:20 UTC (Mon) by nevyn (subscriber, #33129) [Link]
So fix it then, if it's such an easy problem.
Except you might want to search for zero-install/autopackage/etc. where people have tried already ... and failed.
Or ask upstream developers why they don't ask for maintainership of their packages in (at least) Debian/Fedora[/EPEL].
On properly packaging perl Posted Aug 29, 2009 19:05 UTC (Sat) by dirtyepic (subscriber, #30178) [Link]
It is fixed. Upstreams already have multiple teams of dedicated people packaging their software for distros. They're called distros.
On properly packaging perl Posted Aug 22, 2009 1:32 UTC (Sat) by bojan (subscriber, #14302) [Link]
Bad plan. A modern Linux distro is much more then a collection of packages.
On properly packaging perl Posted Aug 22, 2009 2:15 UTC (Sat) by MattPerry (guest, #46341) [Link]
> Bad plan.
Please elaborate.
On properly packaging perl Posted Aug 22, 2009 6:15 UTC (Sat) by bojan (subscriber, #14302) [Link]
Integration between various pieces of software is key. Take Fedora, for example. With every spin, new things are turned on, for which other things need to be turned on etc. You cannot do that with generic packages shipped by upstream.
It also allows each distro to bring their own ideas forward. Some stick, some don't, but this is how they get proven (or not).
On properly packaging perl Posted Aug 23, 2009 1:54 UTC (Sun) by drag (subscriber, #31333) [Link]
That does not mean that you have to do the packages separately for each and every distro. Also it does not mean that the majority of the packages should be customized to that degree.
There are a few things going on here...
Maybe quite a few application customizations that are compile-time options can be made to be runtime configuration options. That way you can do things upstream and still get the customization you need downstream. There probably has to be a understanding of what is a 'application' and what is a core system.
Even things that need to be compile time that does not mean that you still can't do packages upstream. Every decent package management system is perfectly capable of working on the source code level. Both Debs and RPMs have 'src' verions and tools.
The vast majority of upstream projects already package their software. Of course that package format is usually just tarballs. Even if binary compatibility is out of the question for some things it still does not mean packaging can't be done in a way that it is shared among all distros.
-------------------
As a end user one of the things that I always hate is having to give up Debian's packages. A few years ago trying to use CentOS or Fedora was hell for me because functionality and things I took for granted couldn't be done without huge amount of work.
Nowadays Fedora is closer to Debian, but that is taking a huge amount of effort and a huge amount of time on the part of the Fedora folks. It's years and it's still not quite up to the same level. I applaud the hard work and the dedication.. but essentially a lot of what they are doing is just doing work that was already done.
And it goes the other way also.
Debian could benefit heavily from the work Redhat and Fedora on putting into things like Virt-Manager and 389 Directory server. Sure Debian has Virt-Manager stuff packaged, but it's not really up to date and sucks compared to Fedora's version. There are a few differences that need to be taken into account... like Debian does not have the default firewall configuration that is used for Virt-Manager.. but that's not really that big of deal.
And 389 doesn't exist at all for Debian users as far as I can tell. If I was running a organization that already had LDAP and Kerberos intergration based around Debian systems (which is done in a very excellent way for Debian, btw. Does Fedora have things down to little details like Kerberos compatibility packages for OpenAFS so that K5-based users can still authenticaiton to use AFS?) and I want to take a look at 389 it would either mean I would be stuck building my own packages from scratch or switching out my operating system.
There is a huge amount of software that never really is that easily accessable to end users because it's never been packaged and tested very well. At work I am not in a position to spend time working out how to compile, package, and test applications... I have a busy enough time dealing with production issues and evaluating software that is already built and packaged by experts.
On properly packaging perl Posted Aug 23, 2009 21:52 UTC (Sun) by bojan (subscriber, #14302) [Link]
> That does not mean that you have to do the packages separately for each and every distro. Also it does not mean that the majority of the packages should be customized to that degree.
Example: Recently we had a mass rebuild of packages for F-12, because the target will be i686, not i586 like in F-11. Now, if things were done the way you are suggesting, a decision like that would have to be coordinated among all distros, so that upstream starts shipping that at some magical point.
There is about zero chance of that happening.
Re: On properly packaging perl Posted Aug 24, 2009 5:43 UTC (Mon) by nevyn (subscriber, #33129) [Link] There probably has to be a understanding of what is a 'application' and what is a core system. No, there doesn't. On a very superficial level, yeh, "everyone" knows glibc is part of the core system and gnucash is an application. But what about say python? And how do you split python, and why do you do those splits. This is all to do with the higher level applications and their deps. (much like the perl splits) ... which are completely different among distros. (or they'd be identical). Realistically there's just not enough agreement to do "generic distro." packages, even if upstream was qualified/motivated to do them (which they aren't, or they'd be Fedora/Debian/etc. packagers already).
On properly packaging perl Posted Aug 28, 2009 16:02 UTC (Fri) by stevem (subscriber, #1512) [Link]
And you'll end up with a mess of conflicting software rather than an operating system that works consistently. Critical parts of it in random directories under /opt/SUNW/foo, /usr/local/bar/bin/ etc. No agreement over uids, permissions. No thanks...
On properly packaging perl Posted Aug 22, 2009 0:13 UTC (Sat) by eparis123 (guest, #59739) [Link]
I really liked the author's smart comparison between the kernel developers and Jörg Schilling in dealing with distributions modifications.
Really, no matter how smart of right someone is, the one who just destructively screams around loses.
Makes me remember the Reiser4 case.
Packaging and patching Posted Aug 22, 2009 0:16 UTC (Sat) by rlk (guest, #47505) [Link]
It's frustrating when distributors change things without discussion with upstream developers, even when it's a bugfix.
A number of distributions patched Gutenprint 5.0.0 (which project I lead) to add margins for certain paper sizes (Gutenprint 5.0.0 with CUPS simply chopped off anything that wouldn't fit in the printable area of the page, and the way it was explained to me by one distributor was that their management got frustrated with printing letters that were chopped off) -- specifically, this patch forced margins for A4, letter, and legal size paper. This patch had the unfortunate side effect of making it impossible to print borderless to these page sizes even on printers that had no problem with it.
The problem is that they didn't consult with us first. We weren't aware of the level of impact this was having (arguably we should have, but it just happened that what I was printing wasn't causing me problems severe enough for me to care about). We only found out about it when people started complaining to us about not being able to print borderless; I had to look at the source RPM to figure out what was going on.
We wound up fixing this issue in a very different way in 5.0.1, and we release noted it extensively. Unfortunately, distributions never bothered removing the patch, so the broken behavior continued.
Finally, in 5.2, as part of a source code rearrangement, we were able to do something about it: we moved the file to a different location in the source tree, and I inserted some comments into the offending parts of the XML source that guaranteed that any attempt to apply the patch would break, forcing someone to take a look at it and (hopefully) reading our instructions that this was a really bad thing to do, and in any event, the problem underlying this was solved.
We've always had a section in our README specifically targeted to packagers and tried to make it very visible at the top of the file that they should read that section. Among other things, we request that distributors join our development mailing list and discuss changes with us. In this case, it would have raised the urgency of a real fix, and even if distributors decided to temporarily go with the fix, we would have known what was going on, would have known who to redirect bugs to, and would have told them very clearly that in 5.0.1 that patch should go away. I tried beefing it up. I don't know what more we can do. When users complain about problems on prepackaged bits that look fishy to me, I often do tell them to install from source to see if the problem goes away -- not too different in principle from what Joerg Schilling did -- because I don't want to try to debug someone else's patches. I don't agree with everything that he's done, but I certainly sympathize with the kind of frustration this can engender.
The reality is that under the GPL (and most other FOSS licenses), there is nothing legally we could do (I believe -- IANAL) to prevent this, and people do have a right to modify the code. However, from a *pragmatic* point of view, distributors should work with upstream developers and try to understand why developers make the decisions we do. That would also help developers understand the issues distributors face.
Re: Packaging and patching Posted Aug 24, 2009 6:03 UTC (Mon) by nevyn (subscriber, #33129) [Link] It's frustrating when distributors change things without discussion with upstream developers, even when it's a bugfix. Packagers don't live in an ivory tower, and the vast majority will happily accept at least co-maintainers from upstream (if not just give them the package). We wound up fixing this issue in a very different way in 5.0.1, and we release noted it extensively. Unfortunately, distributions never bothered removing the patch, so the broken behavior continued. So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".
Re: Packaging and patching Posted Aug 24, 2009 10:56 UTC (Mon) by rlk (guest, #47505) [Link]
There are far more distributions than there are developers. We release Gutenprint as a source package. We don't know everyone who distributes Gutenprint, nor what patches they apply, until somebody complains. Meanwhile, everyone who doesn't complain thinks Gutenprint is buggy because of things like this.
Re: Packaging and patching Posted Aug 24, 2009 15:26 UTC (Mon) by mcopple (guest, #2920) [Link]
quoth:
So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".
-------
It seems to me that the correct workflow would be for downstream packagers / maintainers to discipline themselves to check for upstream changes on a regular basis, rather than relying on upstream to figure out who is packaging their product and notify.
It is incumbent on the upstream developer to have channels available to get the word out -- a developer's list, a bug tracking system, release notes, etc. But it is not reasonable to expect upstream to check the bug trackers of all of the scores of active distributors out there.
Re: Packaging and patching Posted Aug 29, 2009 19:30 UTC (Sat) by dirtyepic (subscriber, #30178) [Link]
> It seems to me that the correct workflow would be for downstream packagers / maintainers to discipline themselves to check for upstream changes on a regular basis, rather than relying on upstream to figure out who is packaging their product and notify.
At the very least they should find the time to browse the release notes.
Re: Packaging and patching Posted Aug 24, 2009 17:20 UTC (Mon) by jimi (guest, #6655) [Link]
Quote:
So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".
This is absolutely the wrong attitude and workflow pattern. The software developers should have no responsibility at all to file bugs to various distributors' mailing lists. The job of a distributor is to package the original source and optionally patch *and notify the upstream developers* of any bugs. This notification must go from distributor to developer, not the other way around. The article notes that Red Hat would have preferred that the perl guys had filed a bug in the Red Hat bug tracking system. That is completely the wrong expectation. It is Red Hat who should possibly be filing bugs, not the other way around.
There is a possible exception to this rule: when the upstream developers are unreachable. In that case the distributor may choose to become the new maintainers of the software.
Re: Packaging and patching Posted Aug 24, 2009 17:52 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
among other things, you can't expect the upstream developers to run every distro that exists to be able to test them.
Re: Packaging and patching Posted Aug 24, 2009 18:41 UTC (Mon) by nevyn (subscriber, #33129) [Link] So you go to all the effort of finding the bug, and documenting the correct fix etc. ... but don't bother to spend 5 minutes opening a distro. bug saying "X is the wrong fix, this is the upstream fix".This is absolutely the wrong attitude and workflow pattern. The software developers should have no responsibility at all to file bugs to various distributors' mailing lists. The job of a distributor is to package the original source and optionally patch *and notify the upstream developers* of any bugs. This notification must go from distributor to developer, not the other way around. FFS, read what I wrote. Upstream had already seen the patch the distro. did ... and added release notes about how they'd fixed it differently (to stop other problems). But couldn't spend the couple of minutes directly telling the distro? ... and this is the distros. fault for "not doing it's job". I think not. But in the general case ... Yes, in a software developers perfect world they'd release upstream code full of "features". Distros. would then fix everything for them, and spend huge amounts of time rewritting those patches to make the developers happy while they spend their time adding more features. Back in the real world packagers are often only spending a fraction of their time doing packaging, often among a bunch of packages, so that "blah install random-crack" works. They aren't unpaid upstream minions, and they sure as hell aren't paid for their "job". Yes, they are hopefully following development in some way, but they are still unpaid humans and can easily miss things. On the other side upstream developers are often spending a majority of their time on a 1-3 projects. So those upstream developers spending an hour a month or so to make sure the packaging in (at least) the major distros. isn't broken in some way, doesn't seem like too much to ask IMNSO. And, yeh, for larger projects (like perl) you might expect that the distros. have more resources to put towards upstream ... but at the same time upstream will often have lots more resources to look downstream.
Re: Packaging and patching Posted Aug 25, 2009 0:10 UTC (Tue) by rlk (guest, #47505) [Link]
In this particular case (the Gutenprint issue) I did report it to my contact at the particular distribution I use. He ack'ed it, but it still didn't get changed. I only found out much later that this particular hack got propagated to other distributions. It wasn't a packaging problem, by the way. It was a badly conceived patch -- more like the Debian OpenSSL debacle than the Perl issue.
Checking all of the major distributions -- let's say, Fedora, Ubuntu, OpenSUSE, Debian unstable, Mandriva -- is a lot of work. It also means having the disks to do a lot of installations, the time to spend downloading the latest bits, the time to run tests, and so forth. No, this isn't practical.
Our automated test suite wouldn't have caught this in any case, since the problem was that the definition of letter and A4 paper was changed in a way that's legal within Gutenprint (and would therefore not fail functional and regression testing) but which would yield unexpected results.
The interaction I would have preferred is:
* Distributor files a bug/complains on the -devel mailing list that the behavior is unacceptable.
* We reply that it's behaving as designed.
* Distributor says "well, maybe so, but but it's still broken and we need to do something about it."
* Maybe we figure out the ultimate solution and get a fix in time for the next release (in which case everyone's happy) or maybe not, and distributor proposes the particular hack they actually did. More likely the latter, since the ultimate fix needed quite a bit of testing to get right.
* Distributor probably decides, after discussion, to accept that borderless printing won't work to those paper sizes and tells us that they're going to release with it. We suggest that they release note what's going on and keep checking back for a real fix.
* We get the fix out a bit quicker than we ultimately did, since we realize how critical it is in the real world; distributor picks it up and either patches the current release or at least makes sure the next release has the real fix (without applying the bogus patch again).
Re: Packaging and patching Posted Aug 27, 2009 7:42 UTC (Thu) by job (guest, #670) [Link]
I expect a package maintainer to have a clue what changes went into the package they maintain. At least for security reasons, so when upstream gets hacked and/or goes crazy and inserts malware in the code it gets caught.
Otherwise all security that surround distributions is for nothing if they package whatever comes upstream (via unsecured FTP sites and whatever).
Not even reading the release notes is in my opinion very far from having a clue what changes are applied!
Packaging and patching Posted Aug 24, 2009 6:17 UTC (Mon) by dgm (subscriber, #49227) [Link]
This is a serious argument against distro packagers playing developers.
In my opinion, a packager should NEVER, EVER patch something. If it doesn't work, complain to upstream until it's fixed. If it's not fixed, drop the package or revert to a previous version. The case of security fixes is not different. But that's just my opinion, of course.
Packaging and patching Posted Aug 24, 2009 8:44 UTC (Mon) by Trou.fr (subscriber, #26289) [Link]
and when upstream is not replying ? Not ready to accept any patch ? When some packaging requirements are only met with patches ?
And regarding security issues, i'd rather have a quick patch by the packager than a week late patch by upstream.
Packaging and patching Posted Aug 24, 2009 12:19 UTC (Mon) by dgm (subscriber, #49227) [Link]
Just drop the damn thing. Or find someone to fork it.
A quick fix made by a packager that doesn't understand the implications can create more problems that it solves (Debian OpenSSL anyone?) Also your arguments is based on the *false* assumption that packagers do release fixes faster than upstream. And now that we're at it, instead of just packaging everything under the sun, why not just talk to upstream regarding their stance behind the code _before_ creating a package for it?
Packaging and patching Posted Aug 24, 2009 14:53 UTC (Mon) by foom (subscriber, #14868) [Link] Why? Because I like distros that have everything under the sun, and work as an intermediary between possibly-uncooperative upstreams and me. And who do the work that most upstreams don't do to make the software work together easily and consistently.There have been a couple disastrous changes, indeed, but the vast majority of minor modifications I greatly appreciate having.
Packaging and patching Posted Aug 25, 2009 7:04 UTC (Tue) by dgm (subscriber, #49227) [Link]
I also did like them, back when there was not much OSS software. But it's not 1995 any more, and now we have loads and loads of OSS packages for almost everything.
I tend to think that the 80/20 rule also applies to software (yes, all software), thus much of what is there is crap. A process that sinks crap and lets the good stuff stay afloat is really needed to ensure a sane evolution of the good software. Some distributions (Debian and Ubuntu at least) let this up to the user, in the form of a popularity contest. Wrong. This is _the_ task that the packager should be doing. All those "minor modifications" just make things _unnecessarily_ harder for everybody.
Packaging and patching Posted Aug 25, 2009 15:38 UTC (Tue) by foom (subscriber, #14868) [Link] Well, I like my software modified and configured to work better as a whole. And I like that no matter what software I want (even if you consider it crap), it's almost certainly already available in Debian.There's space in the world for multiple distributions with different goals, and I bet there's at least one which doesn't modify upstream source code that you can use.
Packaging and patching Posted Aug 27, 2009 23:16 UTC (Thu) by branden (subscriber, #7029) [Link]
Your citation of the Debian OpenSSL fiasco is a bullshit example.
As was widely reported at the time, Debian's package maintainer *did* take
There was lots of fail in the Debian OpenSSL situation, but failure to run
Packaging and patching Posted Aug 28, 2009 0:12 UTC (Fri) by rlk (guest, #47505) [Link]
That was my recollection about what happened; thank you for correcting the details.
In the Gutenprint example, nobody ever attempted to contact us (we have a development mailing list that's noted on our web site that's open subscription), even to notify us that they were making the change in question. We were left to find out about it from complaints from unhappy users.
Packaging and patching Posted Aug 29, 2009 20:51 UTC (Sat) by dirtyepic (subscriber, #30178) [Link]
Um, what? Packagers _do_ release fixes faster than upstream. They even release _upstream_ fixes faster than upstream. If they didn't, there wouldn't be a reason to patch in the first place.
I can patch a freetype bug in Gentoo and have it out to users sometime within the next hour (whenever the next rsync mirror update is). I don't think they can make a release that fast. In fact, we patched a security vulnerability in freetype-2.3.9 back in May, and as of now there still hasn't been a new upstream release. So, should we drop freetype or fork it? (note: i don't mean to pick on freetype here, it's just a package i happen to maintain)
And by "release" I mean get the fix out to the people encountering the bug, not fix it in the repo for whenever the next release may be. If you have some other definition then please share it.
We also recently patched fontconfig because recent upstream changes were causing problems with our sandboxed build environment (packages calling fc-cache during `make install` would fail because fc-cache now runs chmod on /var/cache/fontconfig and we don't allow packages to change permissions on files or directories they don't own outside of the DESTDIR during install). I did let upstream know about the issue, but ultimately I fixed it locally because it was our policy causing the issue, not upstream. Debian also modifies most packages to comply with their policies. Expecting upstream to accommodate the (sometimes conflicting) packaging policies of every major distro is ridiculous.
Trust me, we would love it if everything worked the way we wanted out of the box. We don't go through the trouble of maintaining large patchsets for the sheer enjoyment it gives us.
Packaging and patching Posted Aug 24, 2009 15:32 UTC (Mon) by mcopple (guest, #2920) [Link]
It doesn't have to be an either/or proposition.
When downstream discovers the problem, open a bug for it with upstream. If upstream doesn't have it fixed by the time the distributor needs it, fix it yourself, display it prominently in the release notes, notify upstream you've done it.
If upstream still doesn't do anything about the problem, then it is upstream's problem; you keep the patch in place and your customers are satisfied. If upstream resolves the issue to the distributor's satisfaction, use upstream and quietly drop the downstream patch.
Packaging and patching Posted Aug 24, 2009 18:46 UTC (Mon) by nevyn (subscriber, #33129) [Link] Sure, Fedora should just fork python or drop it completely (upstream have refused to take multilib seriously). Not to mention what awesome support you'd get off all upstreams for everything in RHEL-5 atm. You seem to have awesome ideas, do you produce a newsletter I can subscribe to?
On properly packaging perl Posted Aug 22, 2009 0:59 UTC (Sat) by dwheeler (guest, #1216) [Link] Although it wouldn't have helped this case, one thing that would help would be if FLOSS packages followed the usual conventions for source releases. Stuff like, support ./configure --prefix=/usr ; make ; make install, and support the DESTDIR convention (which both RPM and deb essentially require). So many source packages fail to support the DESTDIR convention that I ended up having to create Auto-DESTDIR, a program to automatically add DESTDIR support to programs that don't support it natively. When creating the package takes a thousand lines, it's easy to justify five more, and it's easy to miss a "bad" change later. In contrast, when the packaging work is only a few lines long, local patches are a lot more glaring - and in my experience, less likely.
On properly packaging perl Posted Aug 22, 2009 6:24 UTC (Sat) by pabs (subscriber, #43278) [Link]
There are more such tips about distro-friendly projects here:
On properly packaging perl Posted Aug 22, 2009 22:30 UTC (Sat) by RobWilco (guest, #40828) [Link]
It took me some time to identify the prerequisites for an upstream tarball to be easily packaged. It is good to insist, thanks for the link.
On a related thought, I see no good page for helping python projects to be friendly toward distributions. I can see the python buildout system getting momentum among developers but no standard way to transform a buildout into a debian package.
On properly packaging python Posted Aug 23, 2009 5:39 UTC (Sun) by zooko (subscriber, #2589) [Link]
There's http://github.com/astraw/stdeb/tree/master . It is a nice tool to automate some of the "manual labor" of generating a Debian package from your Python package. It can't do everything, but it is a start. There's also http://pypi.python.org/pypi/van.pydeb and bdist_deb. I haven't tried them.
On properly packaging perl Posted Aug 23, 2009 8:28 UTC (Sun) by Cato (subscriber, #7643) [Link]
Thanks for the link. I now understand why checkinstall has problems with some packages and requires fstrans to be turned off. It would be really nice if you could get checkinstall patched to include your Auto-DESTDIR model, as that would cover most cases, it seems.
Auto-DESTDIR Posted Aug 23, 2009 12:18 UTC (Sun) by dwheeler (guest, #1216) [Link] I think so, too. I've sent some emails to people who people I thought would be interested in integrating it into their systems. I've started the process of trying to get auto-destdir packaged into Fedora; that doesn't integrate it into checkinstall, but it makes it more accessible to a few. If others would like to package it for other distros, or help patch it into other programs, that'd be great.
On properly packaging perl - Auto-DESTDIR now in Fedora repository Posted Sep 14, 2009 15:28 UTC (Mon) by dwheeler (guest, #1216) [Link] Auto-DESTDIR is now in the standard (yum) repository of Fedora 10, Fedora 11, and the upcoming Fedora 12 (aka rawhide). Barring catastrophe, it'll even be on the Fedora 12 DVDs/CDs. I intend to create EPEL packages, too, so Red Hat Enterprise Linux and CentOS users can easily use them too. I'd love to see packaging for Debian, Ubuntu, SuSE, etc - volunteers welcome! Hopefully this will help package and install programs.
On properly packaging perl Posted Aug 27, 2009 23:27 UTC (Thu) by branden (subscriber, #7029) [Link]
There you go, ruining a perfectly good festival of hatred for distributors
with a slab of advice counseling original developers to avoid cultivating ignorance and idiosyncrasy.
An essential aspect of software engineering is learning how to retain
Some upstream developers actively monitor the bug-tracking lists of major
On properly packaging perl Posted Aug 24, 2009 13:26 UTC (Mon) by zsouthboy (subscriber, #60295) [Link]
I'm not familiar with perl - can someone explain why they would bother stripping modules out of the standard distribution of perl?
I don't know CPAN, etc., but this sounds like the equivalent of leaving out a standard part of the language, right? Like packaging python (which I am familiar with) without, say, the Math library?
How can you expect your perl script to run anywhere if the default libraries can't be depended on?
On properly packaging perl Posted Aug 24, 2009 14:38 UTC (Mon) by djpig (subscriber, #18768) [Link] I'm not familiar with perl - can someone explain why they would bother stripping modules out of the standard distribution of perl?Here on my Debian lenny system perl + perl-modules + perl-base currently use about 36MB, of which perl-base uses 4.6 MB. And perl-base gives you a fully functional perl interpreter... Sure, on a modern server or desktop 36MB is not a lot, but Debian tries to support as many use cases as possible, and so it makes sense to define a basic set of perl modules that are always available, so people can use them in e.g. package scripts without needing to depend on the full Perl. (The split in perl and perl-modules I can't explain, but they depend on each other so you can't install one without the other anyway)
On properly packaging perl Posted Aug 27, 2009 2:50 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]
"perl-modules" is architecture-independent whereas "perl" is architecture-dependent since it contains native code. Separating the two saves space in the archive and its mirrors, and on multi-arch installation media.
On properly packaging perl Posted Aug 24, 2009 15:22 UTC (Mon) by bferrell (subscriber, #624) [Link]
How about python without setuptools. Suse 11.1 does that... And I'm a Suse fan!
On properly packaging perl Posted Aug 28, 2009 19:39 UTC (Fri) by oak (subscriber, #2786) [Link]
> I don't know CPAN, etc., but this sounds like the equivalent of leaving
> out a standard part of the language, right? Like packaging python > (which I am familiar with) without, say, the Math library?
Python's actually similarly split in Debian based distros.
[1] like perl-base is on all Debian derivatives. I just wonder why one is
Mixing CPAN and packaging Posted Aug 24, 2009 15:22 UTC (Mon) by zlynx (subscriber, #2285) [Link]
I think leaving CPAN out of the RPM is a *very good idea*.
I cannot begin to tell you how many problems CPAN has caused when mixed with RPM packages coming from the distro and/or from alternate repositories.
CPAN wants to upgrade packages it has no business touching. It wants to install things into locations that disagree with RPM. It leads to multiple copies of the same thing installed into slightly different places making it impossible to tell exactly what copy of SSL is crashing your app.
Use RPM *or* CPAN but not both.
Mixing CPAN and packaging Posted Aug 24, 2009 22:33 UTC (Mon) by Cato (subscriber, #7643) [Link]
CPAN is quite flexible, and can be used by non-root users from their home directories, so I'm sure it's possible to set it up so that it only touches non-distro managed directory trees such as /usr/local - it even has an option to do upgrades into a new directory without overwriting the system version.
CPAN is required for all third party software Posted Aug 27, 2009 7:52 UTC (Thu) by job (guest, #670) [Link]
Not packaging CPAN is a mistake. It is the normal way of distributing program dependencies, even for programs that are not distributed via CPAN themselves.
Every program written in Perl that you would want to install manually comes with install instructions that includes using CPAN to get the required libraries installed. That's the culture and it has served well for the past fifteen years or so. The alternative would be to include all dependencies in all programs, which would be worse.
A system without CPAN would leave many beginning system administrators very confused on how to proceed and give Perl programs an unfounded reputation of being hard to install. There should at least be a stub installed that brings in the real stuff when called for or at least explains the situation.
CPAN is required for all third party software Posted Aug 27, 2009 8:21 UTC (Thu) by Cato (subscriber, #7643) [Link]
The best option would be to package CPAN and also include a default configuration for it that avoids it trampling on distro managed directories, e.g. use /usr/local/CPAN for all CPAN-installed (or upgraded) packages, and a suitable Perl library path so that everything uses this tree first. Then the Perl user can use CPAN and know that anything installed by CPAN takes priority over the version in the distro, while still being able to upgrade the distro's Perl-based packages.
What would be even nicer is a simple tool that, given a CPAN package name, first tries the distro's packaged versions of CPAN packages, and only if they don't exist tries CPAN directly.
CPAN is required for all third party software Posted Aug 28, 2009 15:41 UTC (Fri) by epa (subscriber, #39769) [Link]
...better still, first tries the distro's own repository, then attempts to
get the package from CPAN and build a package of it using a tool like cpanflute2, cpan2rpm, makerpm.pl, etc etc.
On properly packaging perl Posted Sep 10, 2009 20:35 UTC (Thu) by philips (guest, #937) [Link]
> In the end, releasing code under a free license means giving up control over what is done with it.
You are right in general.
But you are wrong in the context. We are not talking about a package and random modification to it. We are talking about a programming language.
Perl 5 doesn't have any specification. There are not distro types. Perl 5 is a Perl 5 only when the whole shebangs from tarball is installed. Otherwise you run into the possibility that 3rd party script depending on standard (present in tarball) module will not run - because RH decided that it's not needed.
To exemplify. Imagine that RH had installed GCC but decided to exclude /usr/include/stdio.h. Yes, they have very right to do it. It's all open source. But it would break compilation of more or less every C/C++ program.
Perl 6 which has proper specification (separating mandatory modules from extras) shouldn't have the problem. But for Perl 5 - the tarball is the specification. Deviating from it means only pains for its users.
|
|||||||||||||