Applications and bundled libraries [LWN.net]

Applications and bundled libraries

Posted Mar 17, 2010 23:51 UTC (Wed) by shahms (subscriber, #8877) [Link] (7 responses)

Frankly, the centralized control does anything but hurt developers and
users. It might make developers' lives slightly more difficult by requiring
them to actually think about deployment instead of wrapping up their
/usr/lib and pushing it out in one friggin' huge binary, but the end result
is a *better* experience for users as they get regular bug fixes, security
updates and (hopefully) tested upstream changes rather than whatever "fixes"
the application developers thought expedient to cobble together so they
could get the app out the door.

And it's not like the distros are iron-fisted dictators. All of the major
ones do development in the open and are generally welcoming of input,
patches, etc.

Applications and bundled libraries

Posted Mar 18, 2010 2:11 UTC (Thu) by drag (guest, #31333) [Link] (6 responses)

Still though packaging should be treated as part of the developer's responsibility and not something that is 'downstream'.

It simply is not realistic to depend on third parties to know the proper way to build everything and know exactly the right combination of dependency versioning and compile flags that you (as a developer) have intended and tested against.

It can work for a few hundred packages easily enough to do it the 'apt-get way'. It can scale upwards to several thousands. But to be on par with something like what Windows provides you have to scale to millions and you have a shitload of programs that nobody in the Debian project (or Fedora or Redhat or anybody else) will never be aware of, much less know enough to package and build them for end users.

There is just simply no chance in hell that a centrally managed, closed system like the current package management status quo can ever hope to scale and meet the diverse needs of the average population.

It has to be a distributed system.. And the only logical way (that I see it) is to do a distributed packaging system is by having it be treated as part of the programming of software and have all packaging happen 'upstream'. "Make install" should not drop binaries onto /usr/local, it should produce a 'deb' or 'rpm' file. Software then should not be distributed through tarballs or central source code repositories, but through built packages.

Distributers would, then, work with upstream package creation and aid and correct problems as they come up and then collect as many popular packages as possible for the convience of the end users.

Distributions cannot behave that they are the sole source of the software and libraries people are going to want and require.

To do that it's going to require substantial changes in how Linux, as a operating system, is managed and coordinated. Centralized repositories are stop-gap at best and only exist through pure brute force of volenteers. It won't continue for ever. People will get burned out doing the same old thing again and again.

The quicker people realize this and be willing to abandon technically excellent solutions for real- world practical/useful ones, the better off all of us are going to be.

Applications and bundled libraries

Posted Mar 18, 2010 3:11 UTC (Thu) by thedevil (guest, #32913) [Link] (1 responses)

No, no, no, no.

I don't consider myself "average". Why should I welcome a product explcitly targeted at "the average population"? I don't care about world domination one bit. Windows can have 99% of the market for all I care. I want something that serves me (a programmer) well. Windows isn't it, and Linux which works just like Windows (hi, Gnome!) wouldn't be it, either.

Besides, you contradict yourself:

"... diverse needs of the average population."

"... the better off *all* of us are going to be."
(emphasis mine)

It can't very well be both.

Applications and bundled libraries

Posted Mar 18, 2010 9:14 UTC (Thu) by dgm (subscriber, #49227) [Link]

OK, so you're a programmer and you're soo special. Fine, so do I. But do you realize that that you run a UNIX like system on cheap PC hardware thanks to the MILLIONS of people that buy the same kind of hardware, don't you?

The same logic applies to software. If we hide in the proverbial Ivory Tower, the OS that so wonderfully works for us will languish and die.

The moral being, be little more humble my friend.

Applications and bundled libraries

Posted Mar 18, 2010 18:47 UTC (Thu) by vonbrand (subscriber, #4458) [Link]

Strongly disagree.

First, developers are interested in developing, let them do that. Let others take over the packaging. lest you loose developers (ouch!).

Second, not all distributions are equal. They do have different aims: There are "enterprise" distributions, who are commited to UI/API/ABI stability at almost any cost; there are "end user distributions" commited to making the ordinary user's life as simple as possible; then there are the "technology showcase" ones, always shipping the latest&greatest; and "source code only" distributions which take pride in shipping (almost) unchanged upstream sources, mostly from the bleeding edge. Asking any developer to ship packages for each of those use cases means driving them to sheer madness.

Third, just shipping some "standard package" doesn't help a bit. Witness what became of LSB, which aimed at providing a common base for shipping binaries. The stuff they standardized on was too minimal and too soon outdated to be of any relevance.

Fourth, in FLOSS we have the option to fix the source, and package that. Most closed source applications are bought once, and users would scream if they had to pay again just because the operating system vendor decided to update some random library. And this works both ways, the applications are forced to aim at a low common denominator (or ship their own environment), while the vendor has to bend over backwards so as to minimize breakage (the story of the Microsoft "fix" for a bug in Simcity is just a case in point). And support the resulting mess for 10 or more years (take Windows XP, which refuses to die to this day, after two successor systems shipped) and even longer (there are "backward compatibility" layers in current Windows systems dating back who knows when).

Lastly, one of the strengths of Linux (and Unixy systems in general) is precisely their diversity. Your comment talks only of Linux distributions, and predicts there will only be one distribution in a short while. Sorry to disappoint you, this has been predicted often during the last 20 or so Linux years, and if anything is farther from reality now than ever before. And then you have to figure in other systems like Solaris, Apple's OSX, the swarm of BSDs, and even stuff like CygWin, not to mention closed and/or obsolescent systems, not to mention the sprouting of embedded systems due to ever growing capacities of "small" systems (my cellphone is way more of a computer than the first machine I programmed ever was...). Think of it as a sort of darwinian breeding ground for software: If it is able to survive in many of those environments, it is probably fit for human consumption. Sure enough, it is also our greatest curse...

Applications and bundled libraries

Posted Mar 21, 2010 18:31 UTC (Sun) by ikm (guest, #493) [Link] (1 responses)

> packaging happen 'upstream'. "Make install" should not drop binaries onto /usr/local, it should produce a 'deb' or 'rpm' file. Software then should not be distributed through tarballs or central source code repositories, but through built packages.

Funnily enough, that's exactly what happens under Windows. When you release a piece of software for Windows, all you have to provide is an .exe installer. Why? Because that's what end users expect and because it's quite easy to do (you only have one single platform and packaging format). And everyone's happy! Yes, libraries get bundled, they waste space, have bugs, but this doesn't seem like a major issue for most programs.

In contrast, under Linux you just can't provide 1) debs for three flavors of Debian and two flavors of Ubuntu, 2) rpms for each of the existing RH-derivatives, 3) ebuilds for gentoo and gentoo-derived distros, 4) whatever else other formats are out there. This is just crazy! Thus the only reliable and easy form is to provide compilable source. And let all those zillions of distros do the rest.

So what plagues linux, in my opinion, is that hailed "diversity". It's just too diverse to provide an easy way to install a piece of software. Linux needs its own Microsoft to make something a standard at last.

Applications and bundled libraries

Posted Mar 22, 2010 16:06 UTC (Mon) by Frej (guest, #4165) [Link]

Just for the record, the windows way is also a bit crappy from a user standpoint.
With OSX it's better. For many apps you just drag the 'icon' to wherever you want. This simplicity is for users the same as when we want everything to be a file. It can be expressed as 'everything is just an object' for users. Of course this can't cover all cases, but the simplicity is attractive.
Why isn't a program just a file?

In short this way normal users actually have a chance of
1) Locating the program after 'installing it' (Since they decided the location)
2) Uninstall is just dragging same file (program) to trash.

But i agree, if you want to manage your computer (i think it's fun...) linux is for you. But if you don't, package systems are pretty annoying.

Applications and bundled libraries

Posted Mar 28, 2010 11:11 UTC (Sun) by cas (guest, #52554) [Link]

what you are describing here is not a system, it is a random, chaotic mess.

distributions are made by "big picture" systems people. they want the ENTIRE system to work smoothly as an integrated whole. i.e. they're mostly systems administrators rather than programmers (although there's a lot of crossover there's also a very obvious distinction between the two).

applications like firefox, chrome, etc are made by programmers. their focus is far narrower, all they really care about is their application - even at the expense of the larger system that it will be installed on.

this is not to say one kind of developer or the other is "better" - they're not, and BOTH are absolutely essential. but software works best when programmers and sysadmins work together, rather than try to work around each other.

Applications and bundled libraries

Posted Mar 18, 2010 0:11 UTC (Thu) by cmccabe (guest, #60281) [Link] (2 responses)

> The big advantage is app developer control. Imagine being able to update
> your customers/users application without being dependent on N different
> distributions somewhat random selection of software, it also avoids "Oh to
> get our next update, you need to update your distro" which is completely
> unacceptible from an administration point of view.

Giving developers more power is almost never what you want to do. You want to give power to the users and system administrators.

Some system administrators are conservative. They just don't want to apply any patches except security updates. They might use RHEL 5 or something like this. They ought to be able to follow this policy without interference from developers.

If developers have to add an #ifdef somewhere in the code to make this happen, it's a small price to pay for stability and security.

> Distro's tend to want monolithic control over everything, even if it
> potentially hurts users and developers. The problem with security updates
> wouldn't be the distro's responsibility if they didn't have control every
> bit of software on your computer.

Users shouldn't have to manually update every piece of software on their computer. If it weren't the distro's responsibility, security would fall on to the users and system administrators-- another burden.

Microsoft would love to have a single update button that you could press to update all the software on your Windows PC. They've made that a reality for all the software they directly control. But they can't do it for third party software.

Applications and bundled libraries

Posted Mar 18, 2010 8:40 UTC (Thu) by Frej (guest, #4165) [Link] (1 responses)

>> The big advantage is app developer control. Imagine being able to update
>> your customers/users application without being dependent on N different
>> distributions somewhat random selection of software, it also avoids "Oh to
>> get our next update, you need to update your distro" which is completely
>> unacceptible from an administration point of view.
>Giving developers more power is almost never what you want to do. You want to give power
>to the users and system administrators.
>Some system administrators are conservative. They just don't want to apply any patches
>except security updates. They might use RHEL 5 or something like this.
>They ought to be able to follow this policy without interference from developers. If
>developers have to add an #ifdef somewhere in the code to make this happen, it's
>a small price to pay for stability and security.

Good point and I agree. The current system is great for sysadmins. But we need a system that
works for both users,devs and sysadmins. Currently it works for companies (they pay), where
there are people dediated to support the system. You don't have that on a personal computer
;).

But the problem isn't just ifdefs. It's a about shipping software to end-users.

>> Distro's tend to want monolithic control over everything, even if it
>> potentially hurts users and developers. The problem with security updates
>> wouldn't be the distro's responsibility if they didn't have control every
>> bit of software on your computer.
>Users shouldn't have to manually update every piece of software on their computer. If it
>weren't the distro's responsibility, security would fall on to the users and system
>administrators-- another burden.

I never said anyone should update manually. Some software can update itself ;). And True, the
N different updaters aren't a good solution either. It doesn't really matter for the home user,
but the sysadmin would hate it.

But it should be possible to to create a system that can solve both, we just need to
seperate mechanism (fetching new code) from policy (per app lib or system lib, where to
check for updates). It's not a simpler system, but it could solve different needs for more than
just sysadmins.

I'm aware that money pour in by keeping sysadmins happy, and thus the system is designed
for them. Changing that, is difficult to from a business point of view, but that is ok!

>Microsoft would love to have a single update button that you could press to update all the
>software on your Windows PC. They've made that a reality for all the software they directly
>control. But they can't do it for third party software.

I can't argue with what what you think MS would love to do.

Applications and bundled libraries

Posted Mar 20, 2010 23:10 UTC (Sat) by cmccabe (guest, #60281) [Link]

> Good point and I agree. The current system is great for sysadmins. But we
> need a system that works for both users,devs and sysadmins. Currently it
> works for companies (they pay), where there are people dediated to support
> the system. You don't have that on a personal computer ;).

I use Fedora Core 12 and I turned on automatic updates.

That is "a system that works for me" and I didn't need to pay or hire anyone to support the system.

There's a lot of areas where the Linux desktop is behind Windows. But in the area of automatic updates, Linux is way ahead. This matters not only because you get nice features, but because updating regularly is an important part of securing your system.

Firefox and Chrome rolled their own update system because their main audience is Windows. There is no system-wide update on Windows, except for Microsoft's code. They could nicely strip out all their updater code on Linux, and cooperate with upstream, but it's more work than just doing things the same way on both platforms.

The bundled libraries issue is the same problem. On Windows, you have to bundle all your libraries with your app, because there's no dependency management framework. You can't really trust the DLLs in the Windows folder because someone else might have put a different version there. And you can't put your version there because you might break somebody else who needs to use the earlier version.

Anyway, web browsers are kind of special. They've almost grown into mini operating systems over the last decade. Unfortunately, most of the wheels they've reinvented were rounder the first time around. At least it's an open platform, by and large.

Applications and bundled libraries

Posted Mar 18, 2010 3:26 UTC (Thu) by blitzkrieg3 (guest, #57873) [Link] (10 responses)

> The big advantage is app developer control. Imagine being able to update
> your customers/users napplication without being dependent on N different
> distributions somewhat random selection of software, it also avoids "Oh to > get our next update, you need to update your distro" which is
> completely unacceptible from an administration point of view.

First of all, this is the job of the system packager, not the app dev. Second of all, major distros have package dependencies with automatic resolution these days. I know on my system it is 'yum update firefox'. Why is it better to have a dedicated application installer pulling in /usr/local/lib/libpng.so and having the dependency resolver pull in an update to libpng?

Applications and bundled libraries

Posted Mar 18, 2010 8:07 UTC (Thu) by Frej (guest, #4165) [Link] (9 responses)

>First of all, this is the job of the system packager, not the app dev. Second of all, major
distros
>have package dependencies with automatic resolution these days. I know on my system it is
>'yum update firefox'. Why is it better to have a dedicated application installer pulling in
>/usr/local/lib/libpng.so and having the dependency resolver pull in an update to libpng?

Why do we even have a system packager?
It's a source of contention, say you just released important update 1.4.0, but no users can get
it!
And you have no control of when they get it. It may even be they never get it, because the
distro
think it's too big a change.

Every software must be packaged N times for N distros, and the system packager will always
say "NO!" to any 1.2->2.0 update for a stable dist. Even if the developer want it and some of
the users wan't it.

For a (private) laptop user and the app developer it's a mess. The the distro in complete
control
of your laptop, but nobody really needs that. But the the current complete control/handling of
your system is only great for admins who wan't to control a multiuser system or a webserver.

The point of a self-controlled application is being able to update the actual app, not libpng.
Who
cares about libpng. And it's better because the app developers is in control of their own app
and in direct contact with it's users.

PS: Anything that requires the terminal is not a real solution for 95% of the world ;)

Applications and bundled libraries

Posted Mar 18, 2010 11:53 UTC (Thu) by tzafrir (subscriber, #11501) [Link] (1 responses)

Because otherwise the developer will need to test N! combinations of interacting versions.

Applications and bundled libraries

Posted Mar 18, 2010 20:14 UTC (Thu) by Frej (guest, #4165) [Link]

Exactly and (M-K)! for K libraries that are bundled ;).
Note that the app developer can choose which the subset K to bundle so the remaining part, M-
K are those such that they are less likely to cause problems.

But the point is that the app developer can fix the error (including version incompatabilities) and
push an update, fixing any problems without the middle-layer of a distro.

I'm not saying this is the best solution for everybody!

Applications and bundled libraries

Posted Mar 18, 2010 16:51 UTC (Thu) by viro (subscriber, #7872) [Link] (1 responses)

And when somebody finds an overflow in aforementioned libpng, they are supposed to try and catch every duhveloper that happens to use said library, explain to each why multiplication overflow is a bad thing (starting with "what's multiplication?" in a lot of cases), then convince them to update their stuff, then upgrade every such crapplication on every box out there.
Including laptops. _Wonderful_.

Applications and bundled libraries

Posted Mar 18, 2010 20:34 UTC (Thu) by Frej (guest, #4165) [Link]

Sure under the assumption of a crapplication and duhveloper the implication is that bad things
happen. But i don't agree the assumption is always true ;).

It a question of trust, why force the user to only trust the distro? Sure other models have the
problem you state - but why can't we build a system where it doesn't happen? just because
applications bundle stuff, they can still hook in to the same updater the admin/user runs.

A distro letting go of control does not mean that the that sysadmin has less control/more work..
We can do both, it's just harder and new territory.

Applications and bundled libraries

Posted Mar 18, 2010 19:02 UTC (Thu) by vonbrand (subscriber, #4458) [Link] (3 responses)

You are picking the wrong distro, methinks. If you want the absolute latest source from the developer's keyboard, grab something like Gentoo or Arch. If you want to run something tested to death and guaranteed stable, pick some Enterprise distribution.

As a end user (and sometime sysadmin, and developer at times) I don't want some random developer dictating what version of their package I should run. I'm happy to run the development snapshots of some stuff at my own risk for non-critical use, but only thoroughly wetted and QA'd software on a enterprise distribution for real-world uses.

Sometimes I might decide that what the distro ships is too outdated, and (carefully counting in the extra cost of keeping track of upstream myself) replace selected packages with newer versions. But never across the board.

Applications and bundled libraries

Posted Mar 18, 2010 20:45 UTC (Thu) by Frej (guest, #4165) [Link] (2 responses)

Good questions, but ideally choice of distro shouldn't matter.

>You are picking the wrong distro, methinks. If you want the absolute latest source from the
>developer's keyboard, grab something like Gentoo or Arch. If you want to run something
>tested to death and guaranteed stable, pick some Enterprise distribution.
What if i want stable distro/system but with newest app a,b and c? And developer of a b or c
doesn't want to support N distros? They can just bundle the libs they need.

>As a end user (and sometime sysadmin, and developer at times) I don't want some random
>developer dictating what version of their package I should run. I'm happy to run the
>development snapshots of some stuff at my own risk for non-critical use, but only thoroughly
>wetted and QA'd software on a enterprise distribution for real-world uses.

I'm sure a sensible system would allow a user to refuse appliation updates.

>Sometimes I might decide that what the distro ships is too outdated, and (carefully counting
>in the extra cost of keeping track of upstream myself) replace selected packages with newer
>versions. But never across the board.

Nobody can argue with that, but the app developer still has to wait/hope for N
distros/packagers.

Applications and bundled libraries

Posted Mar 19, 2010 18:16 UTC (Fri) by vonbrand (subscriber, #4458) [Link] (1 responses)

I'm sorry, but "stable distro" and "latest versions of A, B, C" just don't jibe.

Whenever I really needed installing non-official software like that, after much looking around I usually decided not to do it. And when I did, the "application A" for which I truly, really, no-other-option-works, had to get a later version it was something very localized (not exactly (pieces off) newest Gnome or KDE), and I installed that from source (and created a package for simple installation/update). The "dependency hell problems" mentioned mostly just weren't.

Where I did install a larger set of stuff was when we had Suns with Solaris, where many pieces were almost useless (like the infamous cc or its klunky sh, or its bloated beyond recognition version of X) . There the first step was what somebody called GNU > /usr/local (which did include X Windows, TeX and an assortment of other pieces). But it was still done carefully and as limited as reasonable.

Applications and bundled libraries

Posted Mar 21, 2010 0:13 UTC (Sun) by nye (subscriber, #51576) [Link]

>I'm sorry, but "stable distro" and "latest versions of A, B, C" just don't jibe.

That is because in your mindset every application is inextricably part of The System. That isn't the way anyone thinks outside of the Linux ecosystem, and it's frustratingly difficult for one side to understand the other.

The average user wants to continue with the same stable system (with the appropriate fixes if they're towards the higher end of the average), but with the option of whatever software versions they choose. A Windows user doesn't expect that updating Firefox may require them to reinstall every other application on their system to support a complex web of interdependencies - the idea would be beyond ludicrous. This highly desirable goal is currently achieved by bundling libraries - perhaps it always will be.

It doesn't have to be that way, but the [overly IMO] rapid pace of Linux distribution releases means that having separate system and applications package trees would rapidly lead to massive combinatorial explosion.

Applications and bundled libraries

Posted Mar 19, 2010 20:58 UTC (Fri) by dirtyepic (guest, #30178) [Link]

> Why do we even have a system packager?

You've answered your own question. Because every app developer thinks their app is so important that users need to be using the latest version the day it's released, with no thought to system stability or security. Who cares about libpng indeed.