Bitten by the Red Hat Perl bug (InfoWorld) [LWN.net]

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 28, 2008 16:19 UTC (Thu) by Sutoka (guest, #43890) [Link]

On the other hand, theres probably at least as many cases where the Distro *fixes* problems that upstream hadn't dealt with, or maybe only in a version that can't be shipped *cough*Firefox*cough*.

I think the /real/ lesson to take from this is to try to work as closely with upstream as possible, trying to get any of your patchs upstream so you can stop worrying about them.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 28, 2008 16:37 UTC (Thu) by MattPerry (subscriber, #46341) [Link]

On the other hand, theres probably at least as many cases where the Distro *fixes* problems that upstream hadn't dealt with, or maybe only in a version that can't be shipped *cough*Firefox*cough*.

I agree, but I'd much rather see those patches make it into the upstream repository instead. Don't put the patches into the distro if they aren't in the upstream.

As for Firefox, Mozilla should be encouraged to create their own debs and rpms. Then all users need to do is add the Mozilla repository to their sources list and apt-get (or yum) to install. It's already that easy for Windows users. I would hope that Linux users would have it just as easy in the future.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 28, 2008 17:21 UTC (Thu) by vmole (subscriber, #111) [Link]

As for Firefox, Mozilla should be encouraged to create their own debs and rpms.

PLEASE GHOD NO NO NO! There. I feel better.

Upstream supplied .debs are almost always policy-ignoring disasters (excluding those where upstream is *also* a Debian developer). You may think Debian Policy is a lot of useless bureaucracy, but it's why several thousand random packages can work together.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 28, 2008 17:59 UTC (Thu) by MattPerry (subscriber, #46341) [Link]

Debs can be installed on more systems than just Debian. Could you elaborate on the policy-ignoring disasters portion? I'm not familiar with how independent debs have worked out in the past.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 6:07 UTC (Fri) by jengelh (subscriber, #33263) [Link]

Like 64-bit libraries put into /usr/lib instead of /usr/lib64. Nvidia's manual installer likes to do that sort of crap for example.

Or stuffing things in hilarious locations, like H-Sphere RPMs do.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 8:57 UTC (Fri) by spot (subscriber, #15640) [Link]

You know, like FHS says to do? Oh wait, I forgot Debian is continuing to pretend that part of the FHS doesn't exist.

http://www.pathname.com/fhs/pub/fhs-2.3.html#LIB64

P.S. It's not optional. It's only optional for that directory to exist on an architecture that doesn't need it.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 11:40 UTC (Fri) by vmole (subscriber, #111) [Link]

Well, let's see:

/lib64 and /lib32 : 64/32-bit libraries (architecture dependent)

The 64-bit architectures PPC64, s390x, sparc64 and AMD64 must place 64-bit libraries in /lib64, and 32-bit (or 31-bit on s390) libraries in /lib.

The 64-bit architecture IA64 must place 64-bit libraries in /lib.

It's not even self-consistent: the title says "/lib32" but the text says "/lib". Anyway, the Debian folks are trying to get multiarch implemented so that we have a general long-term solution. In the meantime, the assumption is that Debian is about free software that can be packaged for a specific architecture.

multiarch is vaporware

Posted Aug 29, 2008 13:55 UTC (Fri) by dlang (subscriber, #313) [Link]

we are about to hit the second major debian release since multiarch was trotted out as "no, we won't implement things the way everyone else does, we will do this instead", and as far as I can see it's still not fully defined, let alone implemented anywhere. it's not even in the unstable branch, which means that it's extremely unlikly to be in the next stable series after Lenny (which means that it won't matter to most people for several years more)

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 31, 2008 17:24 UTC (Sun) by salimma (subscriber, #34460) [Link]

The current multiarch implementation is rather unfortunate. The GNUstep-esque solution is much nicer: have a sub-directory for each architecture under /usr/lib. Do the same for /usr/bin. And while at it, go the full distance and allow for application and framework (library) bundles as well.

The more self-contained applications can be, the less can go wrong with upstream-created packages.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Sep 4, 2008 8:21 UTC (Thu) by pj (subscriber, #4506) [Link]

How 'GNUstep-esque' is Gobolinux?

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 11:54 UTC (Fri) by vmole (subscriber, #111) [Link]

Could you elaborate on the policy-ignoring disasters portion?

Just a a random sample:

Missing and incorrect dependencies
Using non-standard uids and gids
Undocumented Conflicts
Installation into non-standard directories, like /usr/local or /opt
Hacking on other packages config files

Which isn't to say that "official" Debian packages are always perfect, but they get fixed or dropped.

My point is that not that upstreams are stupid or evil, they're not. They're just busy, and proper packaging for a distribution is more than just "the package installs on my system okay."

Everybody (mostly) agrees that it's better to get your driver into the main kernel tree rather than keep it separate. Likewise, better to get your package into the main distribution archives.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Sep 8, 2008 12:17 UTC (Mon) by cortana (subscriber, #24596) [Link]

Random example off the top of my head: Plesk. It's debs contain monstrous, 2000-line maintainer scripts that are hideously unreliable; making it pretty much impossible to ever remove them after the system has been touched by them.

Policy-respecting .debs?

Posted Aug 28, 2008 20:02 UTC (Thu) by GreyWizard (subscriber, #1026) [Link]

As someone daft enough to try creating a policy-respecting Debian package I'm not surprised. Debian's policy guide is somniferous, the messages produced by the policy lint tool are inscrutable and the changes required are arcane. I find it hard to fathom how these elaborate rules actually help, especially now that Fedora is catching up in package count without comparable fuss.

Policy-respecting .debs?

Posted Aug 30, 2008 11:34 UTC (Sat) by tzafrir (subscriber, #11501) [Link]

Could you please give an example to such a case where lintian gives messages you cannot understand?

Can you give example of an arcane rule that has actually bothered you?

Policy-respecting .debs?

Posted Sep 1, 2008 12:22 UTC (Mon) by GreyWizard (subscriber, #1026) [Link]

Something is inscrutable when it is not readily understood, not merely when it's utterly incomprehensible. So I can't give you an example of a lintian error or warning I'm not able to comprehend after enough research. But I can give you examples that aren't obvious to someone who is not a Debian expert but is capable of building software packages:

E: <pkgname>: no-shlibs-control-file usr/lib/lib<pkgname>.so.0.0.0
E: <pkgname>: non-dev-pkg-with-shlib-symlink usr/lib/lib<pkgname>.so.0.0.0 usr/lib/lib<pkgname>.so

Digging up the meaning of these is possible, and I've done it, but it's rather a lot of bother compared to the process of creating an RPM.

As for examples of the arcane I hardly know where to begin. Why should I need a shlibs control file for a simple libtool shared library? Aren't there reasonable default assumptions that could be made? Why is it forbidden to set the executable bit on such libraries? That's the way they come out of the build process after all and somehow RPM based distributions survive that calamity. Why is it wrong to ship .pyc files? Why should my .py files be executable when they are meant to be used only as libraries (I feed the files to python directly when running built-in unit tests)?

I'm sure there are answers to all of these questions for those with the fortitude to dig through the Debian Policy Guide -- and do enlighten me if you happen to know them. But the notion that such exhaustive policy is necessary to make a large distribution coherent just isn't credible.

Policy-respecting .debs?

Posted Sep 1, 2008 14:44 UTC (Mon) by vmole (subscriber, #111) [Link]

Digging up the meaning of these is possible, and I've done it, but it's rather a lot of bother compared to the process of creating an RPM.

So the fact that Debian provides tools that point out mistakes in packaging is bad. Hmmm. Do you turn off compiler warnings that you don't understand?

I can't speak to the shared library issues, but the reason you don't ship .pyc files is so that the install system can build the python version specific .pyc files according to which versions are on the user's system. Also, you don't ship preprocessed man pages, right? And I don't know why you think python library files have to be executable; the ones on my system aren't. Finally, if these are real issues, you should discuss them on the Debian python list, where the experts are.

Finally, it should be noted that pretty much every item in Debian Policy is there for reason, for solving a real problem. They don't add stuff just to be difficult. Sometimes it's "do X because Y is wrong", and sometimes it's "We can do X or Y, either would be right, but we choose X." Consistency is worth something.

Policy-respecting .debs?

Posted Sep 2, 2008 9:50 UTC (Tue) by GreyWizard (subscriber, #1026) [Link]

Where do you find evidence to support the absurd conclusion that I would prefer to have policy problems hidden from me? I'll go through this again using small words: the policy is complex, the lint tool mumbles and the documentation is poorly organized. Your explanation about .pyc files makes sense and I would thank you for it if not for the tone of the rest of your post. I don't know why .py files should be executable. All I know is that lintian complains when mine aren't. Perhaps there is some arcane incantation I haven't mastered which makes that acceptable. And yes, I know there are reasons for the policy -- in fact I said exactly that the message you replied to. (Consider reading posts before you reply to them in the future.)

But having a reason is not the same as having a good reason. From the perspective of an outsider who just wants to create a policy compliant package the documentation, tools and (at least if you are a representative example) developers are decidedly unfriendly. In light of that the fact that packages created by people who aren't committed Debian developers are "policy-ignoring disasters" is no surprise.

Policy-respecting .debs?

Posted Sep 2, 2008 11:25 UTC (Tue) by vmole (subscriber, #111) [Link]

And we are back to my original point: upstream developers often do not have the time or interest to create proper distribution packages. That's fine. Leave it to the experts. That's the whole point of a distribution: combine hundreds or thousands of disparate pieces of software into a coherent whole. Why would you expect that to be simpler than a writing a single program?

Policy-respecting .debs?

Posted Sep 2, 2008 12:13 UTC (Tue) by GreyWizard (subscriber, #1026) [Link]

Attempting to create a policy compliant package (and even arguing about the obstacles) is a way of trying to contribute in a small way. You seem to be suggesting that anyone who isn't willing to make the large investment of time and effort to actually become a Debian developer shouldn't be bothering at all. I can live with that. It doesn't cost me anything to stop running lintian. But giving up community good will for .so files without executable bits seems like a poor trade to me.

Policy-respecting .debs?

Posted Sep 2, 2008 12:49 UTC (Tue) by vmole (subscriber, #111) [Link]

No, I'm suggesting that anyone who isn't willing to read the relevant parts of Policy, and fix the relevant lintian errors, should let someone else build the distro-specific packages. Supplying broken packages is NOT contributing. If you want to contribute to the Linux Kernel, then you are expected to understand how the parts work, and to follow the established coding conventions, even if you don't see the point of them.

And just to be perfectly clear: I don't think you are a bad person for not wanting to invest the time. Lots of people can (and do) create distro packages. They're good at it. You should spend your time doing whatever it is you want to do, and do it well. Better you fix a bug in your code, or add a feature, than worry about Debian shared library permissions.

Policy-respecting .debs?

Posted Sep 2, 2008 13:45 UTC (Tue) by GreyWizard (subscriber, #1026) [Link]

Why would I run lintian at all if I thought supplying broken packages was acceptable? I complained about a few specific cases where the policy is arcane because I was explicitly asked to do so above, not because I believe I shouldn't have to follow the established conventions.

I'm touched by your concern for how I spend my time, but arguing against points I haven't actually advanced is probably not a good use of yours.

Policy-respecting .debs?

Posted Sep 2, 2008 3:58 UTC (Tue) by tzafrir (subscriber, #11501) [Link]

In that case you should run lintian with the option -i to provide you more information about those errors/warnings , as well as a reference to thew relevant section in the policy manual.

no-shlibs-control-file:

Although the package includes a shared library, the package does not
have a shlibs control file. If this is intentional, please override this
error.

Section 8.6: http://www.debian.org/doc/debian-policy/ch-sharedlibs.htm...

non-dev-pkg-with-shlib-symlink:

Although this package is not a `-dev' package, it installs a
`libsomething.so' symbolic link referencing the corresponding shared
library. When the link doesn't include the version number, it is used by
the linker when other programs are built against this shared library.

Shared libraries are supposed to place such symbolic links in their
respective `-dev' packages, so it is a bug to include it with the main
library package.

However, if this is a small package which includes the runtime and the
development libraries, this is not a bug. In the latter case, please
override this warning.

Section 8.4: http://www.debian.org/doc/debian-policy/ch-sharedlibs.htm...

The executable bit should only be set where it is needed. For scripts. Shared libraries don't need to be executable. If you use the recommended toolchain this will mostly be done for you. I can't think of a specific problem caused by this, but this is the general rule here. Don't mistake /usr/lib for /usr/libexec .

As for the other issues you have raised:

I'm not sure I can easily answer all of your questions. But here's an example: you don't ship .pyc files because you can't tell in advance with what version of python (and potentially: with what other libraries) the code will build. So those are likely to change. See the python policy manual for the exact details:
http://www.debian.org/doc/packaging-manuals/python-policy/

Also, if the file is has a '#!', it is probably intended to be executable. If you have some python code that is only intended to be used as a module and not as an executable, remove the '#!' from the first line. Chances are you merely forgot to set a script as executable (note that lintian here gives you a warning rather thna an error). Actually python modules are often have a small main for a simple test driver.

Policy-respecting .debs?

Posted Sep 2, 2008 10:02 UTC (Tue) by GreyWizard (subscriber, #1026) [Link]

That's quite helpful, thank you. Still, the point I'm trying to make is that all this is quite hard to master. There are surely things Debian can do to be more friendly to people who want to create packages that conform to their policies. One concrete suggestion is to make the more informative mode the default for lintian. Experts can set whatever switch is necessary to get more concise output.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 3:04 UTC (Fri) by epa (subscriber, #39769) [Link]

Well RPM version 3 is the LSB standard package format and doesn't carry the Debian-policy baggage, so surely Mozilla should just use that? Built against some suitably ancient distribution to make sure it works everywhere.

The one big black mark against using RPMs for distributing third-party software is that there is no easy way for a user to install the software in their own home directory. You have to be root.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 8:59 UTC (Fri) by spot (subscriber, #15640) [Link]

That's not entirely true. You can install to a rpm root in your homedir, it just won't be able to resolve dependencies sanely, and you'd need to add directories to your path.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Sep 8, 2008 14:34 UTC (Mon) by epa (subscriber, #39769) [Link]

Yes... no easy way. And all RPMs, in practice, are built to install into / and nowhere else. It is a shame they are not relocatable.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 28, 2008 19:00 UTC (Thu) by Sutoka (guest, #43890) [Link]

>Don't put the patches into the distro if they aren't in the upstream.

So the distro will have to wait for a new release of the package before they could include it?

>As for Firefox, Mozilla should be encouraged to create their own
>debs and rpms. Then all users need to do is add the Mozilla
>repository to their sources list and apt-get (or yum) to install.
>It's already that easy for Windows users. I would hope that
>Linux users would have it just as easy in the future.

How is an extra step *easier*? I can't think of any distro thats not extremely niche that doesn't include Firefox in their default repositories. Mozilla also has a tendency to not support older versions of Firefox that some distros ship.

I can't possibly understand how having people that don't know much about your distro make packages for it, force you to manually add their repository to your lists, and then manually fetch it (post install) would be better than the people that know what they're doing making the package.

IMO, saying upstream should provide all packages (hell, for the application I write I don't even provide RPMs for the distro I run!) or banning distros from patching is simply a knee-jerk reaction to a quite rare occurrence.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 8:55 UTC (Fri) by MattPerry (subscriber, #46341) [Link]

> So the distro will have to wait for a new release of the package before
> they could include it?

Exactly. Ideally, distros shouldn't be shipping the software at all. If the vendor packages it, and you have the vendor's repository in your software update list, then you can update that package (or not) independent of the distro's time line.

> How is an extra step *easier*? I can't think of any distro thats not
> extremely niche that doesn't include Firefox in their default
> repositories. Mozilla also has a tendency to not support older versions
> of Firefox that some distros ship.

In the following, let us assume that Firefox could be any application.

What is easier is that one can upgrade their operating system or the application independent of each other without having to learn the details of system administration. Right now, if I want a new version of Firefox, I have to upgrade my entire operating system which I do not want to do. This will also upgrade all of the other packages on my system. That's a big change just to get a new version of one program.

I could install Firefox from source, or install the binary from Mozilla, but then I am venturing into system administration territory. I have to open a shell and type commands, figure out where to put the files, adjust paths, etc.

Contrast this with Windows users who have the ease of just double-clicking the icon to start the installer. Windows users can upgrade software without having to upgrade everything else on their computer. Likewise, they can upgrade their OS and then reinstall older versions of application software with the same ease as installing the new.

Modern linux distros assume that everyone wants to run the latest versions of software on their systems. For people such as myself who may want to run a mix of software versions there is no easy way to do so. No current Linux distros cater to my desktop needs.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 9:28 UTC (Fri) by Sutoka (guest, #43890) [Link]

>Exactly. Ideally, distros shouldn't be shipping the software at
>all.

So we'd be abolishing Linux distros and relying on upstream to package their software against the magical combination of libraries on your system? Yeah, good luck with that happening.

>If the vendor packages it, and you have the vendor's
>repository in your software update list, then you can update
>that package (or not) independent of the distro's time line.

You'd need all the repositories for all of the dependencies. You'd also need the same repositories as upstream used, otherwise ABI incompatibilies might prevent it from working at all. You'd also have no way to be sure that your packages won't collide with each other, as theres no central authority making the packages to prevent that from happening.

Basically, it'd be an absolute nightmare.

>Modern linux distros assume that everyone wants to run the
>latest versions of software on their systems. For people
>such as myself who may want to run a mix of software versions
>there is no easy way to do so. No current Linux distros cater
>to my desktop needs.

Distros often include a backports repository for stuff like that. And just because the current distros don't fit *your* needs, doesn't mean they don't fit the needs of the most people as far as packaging is concerned.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 9:35 UTC (Fri) by MattPerry (subscriber, #46341) [Link]

> So we'd be abolishing Linux distros and relying on upstream to package
> their software against the magical combination of libraries on your
> system? Yeah, good luck with that happening.

If a distro is LSB compliant and the package is built to work on LSB compliant system, then there won't be a problem.

> You'd need all the repositories for all of the dependencies. You'd also
> need the same repositories as upstream used, otherwise ABI
> incompatibilies might prevent it from working at all. You'd also have no
> way to be sure that your packages won't collide with each other, as
> theres no central authority making the packages to prevent that from
> happening.

Again, LSB compliance. The packages are built to a standard and the distros support this standard. This isn't rocket science.

> Distros often include a backports repository for stuff like that.

So I have to depend on the distro maintainers to support older versions? Are there backports of every package in the repository? Are the repositories in the sources list by default?

> And just because the current distros don't fit *your* needs, doesn't
> mean they don't fit the needs of the most people as far as packaging is
> concerned.

Just because they do fit your needs doesn't mean they fit the needs of most people as far as packaging is concerned.

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 15:11 UTC (Fri) by Sutoka (guest, #43890) [Link]

>If a distro is LSB compliant and the package is built
>to work on LSB compliant system, then there won't be a problem.

So now no software can have dependencies on software not included in the LSB? Or are all non-LSB dependencies going to have to be statically compiled in or shipped with the package in some way? Neither of those are a very nice solution... If you don't do either, you just push the problem up slightly higher.

>Just because they do fit your needs doesn't mean they fit
>the needs of most people as far as packaging is concerned.

Quite few people complain about the current system, and lots of people tout the current system as being one of Linux's big advantages. This is while lots of people complain about how it works on Windows, with most software not having any auto update mechanism, and the developers that do each have their own implementation (Windows Update vs Adobe Updater vs Steam vs Firefox's vs etc).

Bitten by the Red Hat Perl bug (InfoWorld)

Posted Aug 29, 2008 11:18 UTC (Fri) by vonbrand (subscriber, #4458) [Link]

What is easier is that one can upgrade their operating system or the application independent of each other without having to learn the details of system administration. Right now, if I want a new version of Firefox, I have to upgrade my entire operating system which I do not want to do. This will also upgrade all of the other packages on my system. That's a big change just to get a new version of one program.

That's called "dependencies", i.e., shiny new <foo> package needs new glibc's additional/fixed/... system calls, which require a new kernel, which in turn needs new administrative tools, and those...

The price you pay for fast, solid progress is that old code is just left behind. Just like a stable API in the kernel is nonsense, it applies in userland.