Shuttleworth urges Linux patch and bug collaboration (Linux-Watch) [LWN.net]

What are you smoking ?

Posted Jun 16, 2007 7:23 UTC (Sat) by khim (subscriber, #9252) [Link]

Never seen anything more idiotic. What is the package ? It's program with distribution-specific changes (SELinux rules, correct install scripts for this distribution, etc). Everything else must go to "upstream". If your idea is to remove all changes between distributions in term of packaging standards (everyone will be using SELinux and everyone will use one fixed set of init scripts, etc) then it's obvious to see why this idea will not fly. If your idea is to simply forget about differences and hope for the best then it's easy to see why this willl not fly also. And finally if your plan is to introduce "independent package maintaines" and still keep guys who will adopt packages for particular distribution... the the question arises: why the hell this independenp packages are needed and we can not move their work in official tarball ???

Remember: Git does not work without knowleadgeable person who does pulls. In kernel world they are called maintainers, in the world of Linux distributions they are called package maintainers... And Linux distributions always used "the Git model" - this goes without saying. May be some way to make patches exchange simpler will be good, of course, but to decouple package maintainers from the distribution they are suppoosed to maintain... that's just stupid...

What are you smoking ?

Posted Jun 16, 2007 11:20 UTC (Sat) by drag (subscriber, #31333) [Link]

> Never seen anything more idiotic. What is the package ? It's program with distribution-specific changes (SELinux rules, correct install scripts for this distribution, etc).

A package is a deb, tar, or rpm format which end users use to conviently install software. Almost nobody gives a crap about the minor differences in init scripts and such things.

> Everything else must go to "upstream". If your idea is to remove all changes between distributions in term of packaging standards (everyone will be using SELinux and everyone will use one fixed set of init scripts, etc) then it's obvious to see why this idea will not fly. If your idea is to simply forget about differences and hope for the best then it's easy to see why this willl not fly also.

To "Ignore the differences and hope for the best" approach in is superior to what we have now were the SAME EXACT SOFTWARE is kept intentionally incompatable between distributions for little to no good reason. Through things as stupid as differences in versioning number policies.

Right now we have the most innefficient way to manage packaged software imaginable. Each distribition keeps it's own (to what pretty much amounts to their own) fork of the SAME EXACT software that everybody else in the world maintains.

I sincerly doubt that the version of OO.org that Fedora supports is so much more advanced in functionality and stability then what Debian Unstable (or visa versa) ships that it realy justifies the duplication of effort and division of resources that maintaining them requires.

> And finally if your plan is to introduce "independent package maintaines" and still keep guys who will adopt packages for particular distribution... the the question arises: why the hell this independenp packages are needed and we can not move their work in official tarball ???

A tarball is definately already a software packaging system. Most support the generic "./configure ;make; make install" dance. Using a system like that it does basic dependancy checking and other such things, but it's not realy usefull for end users.

Take a deb-src package from Debian Unstable, for example. From that package I can take it and compile binary packages for Unstable, testing, Etch, or Sarge without much to-do. It's not something I do all the time, but it's often enough that it's not unusual for me to do that.

With only a little more effort I can take it and probably make packages for Ubuntu, Mepis, Xandros, Lindows, and other Debian-based distributions.

If there was better coordination between RPM-land and Deb-land (like as if they were compatable in some way) then I would of been able to handle Fedora or Mandriva also.

With that I can create custom versions with different compile-time options and different init scripts or whatever I need or want. Add my own SELinux whatever.

Of course that won't work for well enough for everything nowadays because of stupid and pointless things like differences in version numbering scemes.

So what I am saying is that Distributions need to figure out a way to make it possible to do the equivelent of deb-src, but with a more upstream/universal bent. Let the actuall application developers themselves use their own configuration and compiling methods, but have the package makers take those guy's tarballs and cvs snapshots and turn them into something that can be taken by distributions and easily integrated.

Let the package makers coordinate different number versioning scemes, different locations for files, and other such things. Let them concentrate on the software they have a reason for concentrating on.

No big trial-by-fire to become a 'official this-or-that' for a distribution.

Distributions will pull down packages from the people they trust. From people that are good and smart and follow that distribution's guidelines.

Then distributions add on their own little spices and differences. And of course distrkibutions will maintain their own special configuration, installation, and default configurations. The stuff that realy matters, not weither or not one distribution likes to say a package is 1.0.3 were another says that 1-3 is better way of doing counting on the SAME EXACT peice of software.

What are you smoking ?

Posted Jun 16, 2007 12:14 UTC (Sat) by khim (subscriber, #9252) [Link]

Almost nobody gives a crap about the minor differences in init scripts and such things.

Unless they actually need to run the software, right ? If init scripts don't adhere to the distribution's standard then your daemon will not run, if you don't have correct SELinux rules for your package then your server will be unaccessible on Fedora and if you've compiled packages with ELF hash table format then you can not use them on systems with GlibC older then 2.5!

In short: there a lot of differences between distributions. Difference in version numbers are minor thing (they can be easily fixed with a simple table), the real differences go way deeper.

OO.org in Fedora is compiled with ELF hash tables - this reduces startup time (sometimes as much as 50%) but makes it incompatible with Debian's package (Debian Unstable will probably run the beast, but who will do Q&A if there are no package maintainer for Debian?).

Unfortunatelly there are more differences between distributions then just version numbers. Guys who did AutoPackage tried to solve this problem and did it so well than I usually just refuse to try to help poor sods who tried to use this monstrosity and killed their system: usually it's easier to just reinstall then to fix this mess.

If there was better coordination between RPM-land and Deb-land (like as if they were compatable in some way) then I would of been able to handle Fedora or Mandriva also.

There are no RPM-land. Often it's easier to port package from Debian to Fedora then from SUSE to Fedora. Each major distribution creates it's own land by setting the policies. Unless everyone agree to play by the same rules we'll never have compatibility between distributions.

With that I can create custom versions with different compile-time options and different init scripts or whatever I need or want. Add my own SELinux whatever.

Usually you need proper init stripts and proper SELinux policies to even run program (unless it's simple self-contained program which can be just put in /opt/<programname> like Rar). Who will produce them ?

And who will do things like Kerberos integration, RBAC support or SELinux policies and so on ? Most "package maintaines" will not care because they don't need them but distribution users will care. Who should produce the things ? They must be done across the system to be useful. This means we'll have distribution-specific package maintaines anyway and then why will we need independent ones?

No big trial-by-fire to become a 'official this-or-that' for a distribution.

You can create such package today (and a lot of people actually do) - you only need to become 'official this-or-that' if you want some endorsment from the distribution - and I fail to see how anyone can avoid this: package either supports things distribution creators care about or not. If not - then it'll not be used by distribution anyway, if yes - then why not get the official approval ?

Distributions will pull down packages from the people they trust. From people that are good and smart and follow that distribution's guidelines.

If they follow the distribution's then what's the problem ? And how these packages will help other distributions ? They have different distribution's guidelines...

The stuff that realy matters, not weither or not one distribution likes to say a package is 1.0.3 were another says that 1-3 is better way of doing counting on the SAME EXACT peice of software.

It's not "SAME EXACT peice of software". It's different. May be it's compiled with Cairo support and your distribution does not include Cairo. Or may it's compiled with java 1.6 and your distribution only supports java 1.5. When you have "SAME EXACT peice of software" you usually have "SAME EXACT versioning number policy" - because it only happens if distribution are coordinating policies and versioning number is actually the easies thing to agree upon.

What are you smoking ?

Posted Jun 16, 2007 12:53 UTC (Sat) by drag (subscriber, #31333) [Link]

> It's not "SAME EXACT peice of software". It's different. May be it's compiled with Cairo support and your distribution does not include Cairo. Or may it's compiled with java 1.6 and your distribution only supports java 1.5. When you have "SAME EXACT peice of software" you usually have "SAME EXACT versioning number policy" - because it only happens if distribution are coordinating policies and versioning number is actually the easies thing to agree upon.

Kerberos intigration is done by the original application authors anyways. The application has to be substantially modified to support something like gssapi. If a application supports kerberos and individual distributions choose to compile it with or without support, then having a deb-src style package isn't going to change that. It's very easy to modify one line in one text file in a package to disable or enable something like that.

If it's more then that then your looking at massively modifying the source code of the package and in that case each distribution would have to maintain their own fork seperate from upstream.

And SELinux rules for Apache or whatnot. SELinux is just another set of system permissions, just insanely complex ones. Tarballs provided by upstream already do default permissions. Having a deb-src package and changing the default permissions for the finished binary package isn't difficult. If distributions want to add on their own custom SELinux rules then nothing I am saying would prevent them from doing this, in fact instead of reinventing the wheel with doing basic packaging and testing tasks they can spend more time getting good SELinux rules.

I've taken mplayer packages from debian and recompiled them with jackd support, for example. It's easy to do. Hell I accidently added support for a bunch of other stuff missing simply because I had extra *-dev packages installed.

Then entire Ubuntu distribution is more-or-less a recompiled and slightly modified version of Debian unstable. With a different init system and updated kernel/gnome stuff. All of multiverse and universe is essentially recompiled Debian packages.

>OO.org in Fedora is compiled with ELF hash tables - this reduces startup time (sometimes as much as 50%) but makes it incompatible with Debian's package (Debian Unstable will probably run the beast, but who will do Q&A if there are no package maintainer for Debian?).

So what? Is this that different that it requires massive changes to the source code of OO.org? Is Fedora maintaining their own private OO.org fork?

If it doesn't require massive changes then it's just a compile-time option. That can easily be done when you compile the package for your distribution.

If Fedora is maintaining it's own OO.org fork then that's it, it's not worth bothering with compatability.

Init script is simple to change. In fact a lot of source code tarballs provided by upstream already has debian or redhat-specific stuff in it like that.

If you need support for stuff newer then glibc2.5 then compile the packages with newer versions of that libs, if not then don't.

If you haven't before then take a good look at apt-build and related stuff. It's very easy to substantially modify how a application is compiled and such things without actually changing anything much in the actual deb-src package.

It's sort of like Gentoo's portage.

That's essentially what I am proposing. A more sophisticated form of tarball with "./configure;make;make install", or a more generic form of deb-src that all distributions can use.

Let the upstream developers use whatever they'd like, but then have people that setup repositories of applications stored in packaged source form that can easily be compiled into custom binary packages. Required and optional dependancies are noted and provided in a way that provides usefull automation for checking that sort of thing.

And not some big monolythic with-every-package-in-the-world thing, but repositories grouped by software project or related projects.

Distributions pull from those places, compile them and provide their own monolythic repositories like the 'multiverse' or 'universe' repos. The users will still be presented with binary packages pre-compiled with specific options for their use by the distribution based on whatever criteria.

If users have a paticular need for bleeding edge package or whatnot and are adventurous then they can pull from 'upstream source package' themselves and compile it and have it integrated into their existing package management system.

There is no reason why every time you want a create a binary package for a distribution you have to be forced to run to upstream, grab a cvs snapshot or tarball, and go it alone..

I am NOT proposing a form of 'Autopackage' were you try to create generic binary packages that work across all distributions and pull down software independant of any sort of real package management software.

What are you smoking ?

Posted Jun 16, 2007 13:01 UTC (Sat) by drag (subscriber, #31333) [Link]

And this doesn't preclude distributions adding their own small patches. That is also something that is quite easy to do with dealing with deb-src packages, and then compile them into binary packages.

We need something like that, but more generic, and have it done upstream from the distributions.

What are you smoking ?

Posted Jun 17, 2007 5:35 UTC (Sun) by khim (subscriber, #9252) [Link]

If it integrates into their existing package management system then it'll break the distribution sooner or later (depends on difference between packager ideals and distroibution ideals), if it does not - then how it can help distribution creators ?

I've seen such schemes introduced again and again but they just don't work. Either there are single version of package (produced by upstream) so anyone can be sure "it's ugly, but it has this and that configuration options" or you have distribution versions (again - fully known options). Anything else just introduces complexity, creates a lot of problems and forces end-user to fix them. Gentoo just does not work for Joe Average... and if it works for you then why just not switch to Gentoo ?

What are you smoking ?

Posted Jun 17, 2007 19:32 UTC (Sun) by vonbrand (subscriber, #4458) [Link]

What you are proposing is having one distribution. Hasn't happened, and (hopefully) won't happen.

Besides, there are other (non-Linux) users too: Solaris, MacOS, various BSD flavors, even Windows, for the exact same software. Are the poor maintainers will have to cater to all that, just because you say so? Most just would give up in despair.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 16, 2007 13:13 UTC (Sat) by copsewood (subscriber, #199) [Link]

The fact that the kernel supports various hardware architectures using an efficient process suggests to me that a package should similarly be able to contain customisations in its source tree to enable it to be able to build on different distributions. But I think that for this to work in practice would require better standardisation, particularly on how things are named. I think a difficult issue here that would definitely require naming and versioning standardisation would be how the dependencies work. For example, I just locally fixed a bug and sent an Ubuntu patch which concerned how a script within Python Twisted failed to build a correct .deb package for a server I was customising. The reason it broke was down to dependency package and version naming, presumably between Debian and Ubuntu.

For translations of user interface strings to enable packages to handle multiple languages not to break between versions, this requires the upstream source package to contain all user interface strings within a database and be able to recognise that when a new version comes along only the new user interface strings need to be retranslated. Having a standard form of source code documentation decorator between programming languages to indicate the presence of a user-interface string requiring translation would help populate and update the translation database.

So I think there are some major difficulties to be overcome, but I don't see this as beyond the wit of man and woman. Perhaps Mark Shuttleworth could usefully consider sponsoring a cross distribution and language packagers standardisation conference.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 17, 2007 14:12 UTC (Sun) by tzafrir (subscriber, #11501) [Link]

The changes between distributions are not like the architectures. Hardware architectures are there, whther we like it or not. Distributions, OTOH, have their own policies and their own release schedules.

Fedora releases a new version on October. On the month or two before that its package maintainer wants the package to freeze: to only change for bugfixes.

When should upstream apply changes to the tree? To the Fedora branch? When can the Fedora branch of the Upstream tree be considered "stable"?

Now add 6 other distributions with their own release schedules.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 17, 2007 19:45 UTC (Sun) by vonbrand (subscriber, #4458) [Link]

The fact that the kernel supports various hardware architectures using an efficient process suggests to me that a package should similarly be able to contain customisations in its source tree to enable it to be able to build on different distributions.

But the software being discussed here does work (with small patches and configuration tweaks) on assorted Linux distributions and other systems! The "customizations" don't belong in the upstream package at all, that's the customer's (downstream distribution) job.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 16, 2007 19:52 UTC (Sat) by talex (guest, #19139) [Link]

I agree with what you're saying.

Most of the time upstream already produces packages that work on many distributions (our development version can't get proper testing if we have to wait for all distributions to package it!).

The model you propose (distributed parties produce packages and distributions pull the ones they want) is the same model used by Zero Install:

http://0install.net

i.e. the distribution provides QA sign-off and branding because users want a relationship with a single distribution ("Ubuntu"), not thousands of individual packagers.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 17, 2007 5:25 UTC (Sun) by khim (subscriber, #9252) [Link]

The model you propose (distributed parties produce packages and distributions pull the ones they want) is the same model used by Zero Install

Not really. 0install is kind of "distribution in a distribution" (like CygWin is "distribution in a distribution"), it works like a typical distribution, but it breaks much faster then package system in a typical distribution (because you have no control over packages supplied by others). It's useful, yes, but a replacement to a packages from a distribution it's not...

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 17, 2007 9:28 UTC (Sun) by talex (guest, #19139) [Link]

Why would it break 'faster'?

If Ubuntu QA signs off to say that Zero Install package "X" meets their standards, then that version should continue working indefinitely. Remember that there are no 'forced upgrades' in Zero Install. You can roll back individual libraries on a per-program basis. So if Ubuntu ships a new GTK that breaks package X, then someone* just needs to annotate X to make it require the previous version, and everything keeps working.

* Normally upstream does this, but it could be a user (if the problem has only just been discovered) or the distribution (if upstream are slow to respond for some reason).

Distributions do have control over Zero Install packages, it's just that the default behaviour is to use upstream unmodified. Which is actually normally what you want, because generally the upstream authors know far more about the program that the distribution's packagers.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 18, 2007 6:28 UTC (Mon) by khim (subscriber, #9252) [Link]

Why would it break 'faster'?

Because there are noone who'll fix broken things.

If Ubuntu QA signs off to say that Zero Install package "X" meets their standards, then that version should continue working indefinitely.

Nope - it'll work till you upgrade to the next version of Ubuntu. Or may be even if you'll install some security patch. Microsoft tried this - it does not work. It kind-of-work-if-you-are-lucky-enough. For a limited time.

So if Ubuntu ships a new GTK that breaks package X, then someone* just needs to annotate X to make it require the previous version, and everything keeps working.

...unless library itself is broken - and then you need to fix library. And if this library can not be fixed without fixing some other libary... WHO will fix all this ? 0install works only for things which don't require new libraries - if you only use old well-debugged technologies. Even then I've seen enough packages incompatible with NPTL - and glibc 2.5 does not even include LinuxThreads anymore! So you will be forced to upgrade sooner or later if your program is using bunch of libraries and if it does not use anything except kernel... what's wrong with tarball and /opt ?

* Normally upstream does this, but it could be a user (if the problem has only just been discovered) or the distribution (if upstream are slow to respond for some reason).

Normally upsteam care about exactly two versions: latest-released one and not-yet-released one (sometimes about designated beta or alpha version too). Answer to any complains about previous versions are simple: upgrade.

Yup. That's why distributions will just ignore 0install packages while upstream will abandon old version (it's not as exciting as development of new versions and is looks stupid for any upstream devloper to try to keep old package working if the new one is much better) - you are back in square one. In short: 0install is made to be more robust then typical distribution package system but when (not if, but when) it breaks - there are noone who's in charge of fixing it. That's why it's Ok to use it for "try a program for a time", but if you plan to use it - it'll be better to included it in distribution's repository (even if you'll end up maintaining it).

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 18, 2007 11:41 UTC (Mon) by Tom2 (guest, #43780) [Link]

There are noone who'll fix broken things.

There's no evidence to support this. In your world, there are lots of people who must fix a package that breaks due to a library upgrade: the Debian packager, the Ubuntu packager, the Fedora packager, the Gentoo packager... on and on. If all these people can exist, why can't a single person exist to fix the upstream version?

In any case, this is missing the point. You don't need a new release. You just need someone to put a message on the web-site saying This version only works with GTK versions before 2.6.8.

Most upstream authors would be happy to do that. With Zero Install it's the same thing, except that you write the message on your web-site in XML rather than English. Zero Install will see that the program is incompatible with the distribution's new library version, and will download a version of the library that does work, install it out of the way (to avoid affecting any other programs) and run the program that wasn't working with that.

Microsoft tried this - it does not work. It kind-of-work-if-you-are-lucky-enough. For a limited time.

I'm not aware of Microsoft ever trying this. Do you have a link?

That's why it's Ok to use it for "try a program for a time", but if you plan to use it - it'll be better to included it in distribution's repository (even if you'll end up maintaining it).

It sounds like what you want is a support contract, not (necessarily) a distribution. Of course you always need someone who can take over a program you rely on, but with Zero Install the distribution (or whoever) only needs to take over when there's an actual problem, not all the time.

Shuttleworth urges Linux patch and bug collaboration (Linux-Watch)

Posted Jun 17, 2007 10:30 UTC (Sun) by MathFox (subscriber, #6104) [Link]

Linus also spoke about building a "network of trust". He accepts patches (pulls diffs via git) from a dozen lieutenants, without reviewing each and every line of code. Similarly, each of his lieutenants has his network of trusted developers. Git helps to move patches over more efficiently, so that the grunt work can be done more efficiently, but building a network of trust takes time.
One of the main issues I see is that the various bug reporting systems don't make it easy to track bugs up- and downstream. When I report a Gnome bug at Gentoo and Gentoo decides it's an upstream bug, it has to be manually entered at Gnome and updates on the Gnome bug site are not tracked by the Gentoo bugzilla. Furthermore, bug reporting and version control systems are not coupled... It would be great when one could automatically track a fix from "patch received" via "accepted" and "tested" to "in release".