LWN: Comments on "Applications and bundled libraries"

Applications and bundled libraries

dlang — Fri, 09 Apr 2010 18:46:05 +0000

Ok, your systems is stripped down more than mine is :-)

if you don't do an apt-get clean after doing the update debian will keep the .deb files of the packages that you downloaded around (in /var/cache/apt/archives)

if you are wanting your product to update directly from debian's public servers then you need to plan to support the large package list (any way for them to split the package list is going to put _someone's_ critical package in an optional repository), but you can run your own repository and only put packages in it that you want to make available to your product. This will also save you time in the updates.

a minor point, I'll also point out that most of this space is in /var, which is not what most people would think of when you said it was in the rootfs

Applications and bundled libraries

wookey — Fri, 09 Apr 2010 13:20:27 +0000

That stat comes from the fact that the (emdebian-grip) rootfs for the product I am currently working on is about 60Mb unpacked, and after you do your first apt-get update at which point it is about 100Mb. Those numbers are from memory, but the point is that on a small system the overhead is significant. Debian's package list alone is 25Mb, and the package cache is another 13Mb on this box I have to hand. This desktop box has 60Mb of apt cache and package lists. Clearly 60Mb on a 4Gb system is not a big deal at all, and even on a 60Mb system we still find it useful enough that we choose to use it. We used to use ipkg which is enormously slimmer, and necessary if you don't have the storage to support apt/dpkg, but apt/dpkg do work a lot better when things get a little complex.

Applications and bundled libraries

dlang — Thu, 08 Apr 2010 18:41:20 +0000

I build pretty stripped down debian systems, and I don't see anywhere near the '40% of rootfs is package management' overhead that you describe. people who install fatter systems would see even less as much of the overhead is constant (there are a handful of files per package, but they are tiny compared to the rest of the package)

Applications and bundled libraries

wookey — Thu, 08 Apr 2010 18:27:07 +0000

Whilst I am a huge fan of 'the Debian way' of central package management, it really isn't true to say that the current (large) pile covers most areas. There are big gaps in no doubt many areas; I've recently noticed them in two: energy monitoring/control and building design. There are an awful lot of handy Windows apps for things like calculating beam loadings in buildings, and many more for specifying vendor's products like hot water tanks and ventilation equipment. Very few of those are free software, but they probably could be and either they all need adding to the distro repositories or we need a more modular mechanism for installing developer/vendor-supplied apps. I believe autopackage is intended to fill that role but it hasn't been very popular so far (possibly because the internals are pretty ugly). I mention it because no-one else has so far - maybe something like that is worth more effort.

On the energy monitoring front there are plenty of things like mango, diyzoning, owfs and temploggerd whcih are basic apps for this stuff but none are in Debian yet (or weren't last time I looked). No doubt they will be at some point but until then anyone wanting to use them has an installation problem. I tried local building but found this to be very hard indeed for the two java-based apps there when targeting an arm box. Obviously that wasn't something the developers had tried.

I do agree with those who say that developers should not be doing packaging and system integration. They are not well-placed to understand the issue of less-common architectures, complex system interactions and really have better things to be doing with their time.

I am inclined to agree that a bit more modularisation of distro repositories would be a good thing. Ubuntu's model of 'core stuff', 'common stuff' and 'everything else' is a step in this direction. Debian's monlithic tools are becoming stretched, especially on small systems (where the package management overhead can amount to 40% of the rootfs storage requirement). I'm not sure there are 'millions' of applications, but there are many tens of thousands and dealing with that efficiently is a challenge.

One other thing which I don't think has been said loudly enough in this debate is that users really do value stability. The central repository model does provide a lot more of that than the 'install random stuff off the net' model. It really is worth something, and whilst they also value being able to install 'latest' of a few apps there is a real potential tradeoff there which needs to be managed somehow. The users I deal with are _much_ more interested in having a stable system than they are in the latest and greatest. They only upgrade apps when they find they need to for some significant feature. Making that easy and reliable _and discoverable_ for them is where we have much room for improvement.

I find that the Debian backports model works pretty well for this, and could be greatly extended to provide a much larger chunk of the new stuff users want. However it is not a particularly discoverable mechanism - not helped at all by the way many users are used to the Windows model of 'just click here to install this stuff'. There is a significant element of user education to get them to stop and think for a mo and see if what they want is already in the packaging system, or would be if they/it added a suitable repo. Most software websites don't help at all with this, as they encourage the Windows model and rarely mention the 'distro' model.

So, it's a thorny issue, and I agree there is room for improvement, but I also feel that the good part of what we have is very valuable, to everybody, including naive users on laptops, even if they don't really appreciate it. Make it easier to go outside that model by all means, when necessary, but try hard not to just make thing unreliable and/or insecure as a result.

Applications and bundled libraries

oblio — Wed, 31 Mar 2010 06:30:59 +0000

Oh come on!

You really think that 25.000 applications can replace millions? No they can't.

First of all, if you look really closely, those aren't 25.000 applications. They're 25.000 *packages* - including libraries (who cares about those?), documentation, meta packages, maybe even development versions of packages (sources).

Secondly, you're forgetting about statistics and evolution. More variation and competition means there's a greater chance of success. Only a small percentage of applications are any good.

Thirdly, for each niche there are many design decisions. Out of 10.000 crappy applications for a certain niche, you'll have 100 decent applications, 10 really good ones, and 3 great ones, each having a different design philosophy (therefore you can't replace great application A with B or C).

Should I write my own "custom" application for everything that doesn't fit that tight collection of 25.000 "applications"? (answer: no, on Windows you find a tool already made for 99% of regular desktop activities)

Applications and bundled libraries

cas — Sun, 28 Mar 2010 11:11:08 +0000

what you are describing here is not a system, it is a random, chaotic mess.

distributions are made by "big picture" systems people. they want the ENTIRE system to work smoothly as an integrated whole. i.e. they're mostly systems administrators rather than programmers (although there's a lot of crossover there's also a very obvious distinction between the two).

applications like firefox, chrome, etc are made by programmers. their focus is far narrower, all they really care about is their application - even at the expense of the larger system that it will be installed on.

this is not to say one kind of developer or the other is "better" - they're not, and BOTH are absolutely essential. but software works best when programmers and sysadmins work together, rather than try to work around each other.

Applications and bundled libraries

cas — Sun, 28 Mar 2010 10:53:05 +0000

what this describes is classic "DLL Hell".

If Mozilla and Chrome want to import that disastrous practice into linux, they're going to find that a lot of people say "thanks, but no thanks"(*)

firefox is good, but it's not so good or so irreplacable that it's worth tolerating that nightmare.

(*) quoted text passed through an extreme politeness translator.

Applications and bundled libraries

rqosa — Thu, 25 Mar 2010 21:38:20 +0000

> The LSB is not adequate to solve the problems to which bundling is perceived as a solution.

Why not? The whole point of the LSB is to have "libraries by default in the OS", and to have an unchanging ABI for those included libraries, just like they are with Windows or Mac OS X. (That is, for a single version of Windows or Mac OS X, at least, since the API/ABI has changed between releases.) An application compiled for the LSB can depend on those libraries without bundling them with the application, and any other libraries must be bundled with the application (which can be done by linking it statically, or can be done by putting the library in a directory under /opt and then setting RUNPATH or RPATH to that directory). This is essentially what developers of apps for Windows and Mac OS X must do already.

In short, your proposed solution for Linux to include "libraries by default in the OS" exists, and it's called the LSB.

Applications and bundled libraries

DarthCthulhu — Thu, 25 Mar 2010 16:19:35 +0000

I'm a bit late to this party, but I'll cheerlead a little anyway.

It's already possible (and easy!) for a user to get the benefits of both models. All you need is the right distribution. And that distro is GoboLinux.

Software installation is very, very easy on GoboLinux and you can have multiple versions of a software all running happily together without the need for every software bundling its own libraries. If some bit of software needs a specific library version, no problem! It will install that version which will still live happily with others.

How is it able to do this? Through a better directory layout, some symlink magic, and Recipes. Recipes are the means to install software; at its base, it's really just a URL with the software location and a list of prerequisites. The system will automatically go through and download/copy/compile all the needed libraries and so on before installing the main software. Recipes can install both source and precompiled binaries.

Currently, the distro is centrally-based, but there's no reason why that has to be the case. It's quite easy for individual developers to make their own Recipes for software installation and publish them. There's a local recipe store just for that, in fact.

It's also very easy to write a Recipe. In fact, there's a tool that does it mostly for you (albeit command line). Most of the time you only need to know the URL of the software you're interested in.

The main downside of the distro is that it's basically developed by two guys at present, the official release is positively ancient (you'll need to set aside a day or so to download and upgrade Recipes), and that, if things go wrong, it can take a little technical knowledge to fix (never try compiling glibc -- always go precompiled with that one). I've also, personally, never had the automatic kernel upgrade work correctly; it always requires manually moving the kernel modules to the proper place in the directory tree. Irritating, but fairly straightforward once you understand the way the new directory tree works.

GoboLinux really is the best of both worlds. Give it a try.

KSM

johill — Mon, 22 Mar 2010 21:06:31 +0000

Not really, it seems extremely unlikely that they actually get the code ordered the same way
and aligned at the same point within a page.

KSM

Bones0 — Mon, 22 Mar 2010 20:38:04 +0000

I wonder what implications page sharing via KSM has for the memory use discussion. As long as all the private libraries are built with the same compiler options, isn't there a good chance that the majority of the memory could be shared behind the scenes?

If the result of those KSM scans could be remembered between reboots, increased CPU usage should not be an issue. And if the file system supported shared chunks analogously to KSM's shared pages, disk space might not be a problem either.

Applications and bundled libraries

lkundrak — Mon, 22 Mar 2010 18:43:28 +0000

So, how about v8?

Applications and bundled libraries

lkundrak — Mon, 22 Mar 2010 18:39:18 +0000

I stopped reading your comment at "focused on getting something working well" point. Are you really sure avoiding distributions shipping the code qualifies as working well? When the software can't even be installed via standard methods?

Applications and bundled libraries

Frej — Mon, 22 Mar 2010 16:06:55 +0000

Just for the record, the windows way is also a bit crappy from a user standpoint.
With OSX it's better. For many apps you just drag the 'icon' to wherever you want. This simplicity is for users the same as when we want everything to be a file. It can be expressed as 'everything is just an object' for users. Of course this can't cover all cases, but the simplicity is attractive.
Why isn't a program just a file?

In short this way normal users actually have a chance of
1) Locating the program after 'installing it' (Since they decided the location)
2) Uninstall is just dragging same file (program) to trash.

But i agree, if you want to manage your computer (i think it's fun...) linux is for you. But if you don't, package systems are pretty annoying.

Applications and bundled libraries

ikm — Sun, 21 Mar 2010 18:31:59 +0000

> packaging happen 'upstream'. "Make install" should not drop binaries onto /usr/local, it should produce a 'deb' or 'rpm' file. Software then should not be distributed through tarballs or central source code repositories, but through built packages.

Funnily enough, that's exactly what happens under Windows. When you release a piece of software for Windows, all you have to provide is an .exe installer. Why? Because that's what end users expect and because it's quite easy to do (you only have one single platform and packaging format). And everyone's happy! Yes, libraries get bundled, they waste space, have bugs, but this doesn't seem like a major issue for most programs.

In contrast, under Linux you just can't provide 1) debs for three flavors of Debian and two flavors of Ubuntu, 2) rpms for each of the existing RH-derivatives, 3) ebuilds for gentoo and gentoo-derived distros, 4) whatever else other formats are out there. This is just crazy! Thus the only reliable and easy form is to provide compilable source. And let all those zillions of distros do the rest.

So what plagues linux, in my opinion, is that hailed "diversity". It's just too diverse to provide an easy way to install a piece of software. Linux needs its own Microsoft to make something a standard at last.

Applications and bundled libraries

nye — Sun, 21 Mar 2010 00:13:03 +0000

>I'm sorry, but "stable distro" and "latest versions of A, B, C" just don't jibe.

That is because in your mindset every application is inextricably part of The System. That isn't the way anyone thinks outside of the Linux ecosystem, and it's frustratingly difficult for one side to understand the other.

The average user wants to continue with the same stable system (with the appropriate fixes if they're towards the higher end of the average), but with the option of whatever software versions they choose. A Windows user doesn't expect that updating Firefox may require them to reinstall every other application on their system to support a complex web of interdependencies - the idea would be beyond ludicrous. This highly desirable goal is currently achieved by bundling libraries - perhaps it always will be.

It doesn't have to be that way, but the [overly IMO] rapid pace of Linux distribution releases means that having separate system and applications package trees would rapidly lead to massive combinatorial explosion.

Applications and bundled libraries

cmccabe — Sat, 20 Mar 2010 23:10:22 +0000

> Good point and I agree. The current system is great for sysadmins. But we
> need a system that works for both users,devs and sysadmins. Currently it
> works for companies (they pay), where there are people dediated to support
> the system. You don't have that on a personal computer ;).

I use Fedora Core 12 and I turned on automatic updates.

That is "a system that works for me" and I didn't need to pay or hire anyone to support the system.

There's a lot of areas where the Linux desktop is behind Windows. But in the area of automatic updates, Linux is way ahead. This matters not only because you get nice features, but because updating regularly is an important part of securing your system.

Firefox and Chrome rolled their own update system because their main audience is Windows. There is no system-wide update on Windows, except for Microsoft's code. They could nicely strip out all their updater code on Linux, and cooperate with upstream, but it's more work than just doing things the same way on both platforms.

The bundled libraries issue is the same problem. On Windows, you have to bundle all your libraries with your app, because there's no dependency management framework. You can't really trust the DLLs in the Windows folder because someone else might have put a different version there. And you can't put your version there because you might break somebody else who needs to use the earlier version.

Anyway, web browsers are kind of special. They've almost grown into mini operating systems over the last decade. Unfortunately, most of the wheels they've reinvented were rounder the first time around. At least it's an open platform, by and large.

Applications and bundled libraries

man_ls — Sat, 20 Mar 2010 12:51:41 +0000

It may seem obvious, but I have to ask -- why not revive libjingle and push the patches upstream?

Test suites can help

jrn — Sat, 20 Mar 2010 01:00:52 +0000

Does your program have a test suite? Even basic tests can do a lot of good, both in making non-functional functionality more obvious and spurring people to fix it. You might be interested in Mike Hommeys recent work on Mozilla: before and after.

Applications and bundled libraries

dirtyepic — Fri, 19 Mar 2010 20:58:41 +0000

> Why do we even have a system packager?

You've answered your own question. Because every app developer thinks their app is so important that users need to be using the latest version the day it's released, with no thought to system stability or security. Who cares about libpng indeed.

How VMWare does it

gurulabs — Fri, 19 Mar 2010 20:57:42 +0000

I use VMware Workstation on Linux. They allow you to go either way. By default it uses system libs, but you can set environment variables to control behaviour.

For example:

VMWARE_USE_SHIPPED_GTK="force" /usr/bin/vmware

Applications and bundled libraries

halla — Fri, 19 Mar 2010 18:48:35 +0000

Yes, they are. Both Qt4 and GTK2 are part of LSB. See http://refspecs.linux-foundation.org/LSB_4.0.0/LSB-Desktop- generic/LSB-Desktop-generic.html.

Applications and bundled libraries

marcH — Fri, 19 Mar 2010 18:33:20 +0000

I like this.

Going even further, most people actually need only the same small fraction of these 25K packages. So it would be nice to make this centralized repository much more modular, because every apt-XXX invocation is F.... dead slow on any low-end machine.

Applications and bundled libraries

vonbrand — Fri, 19 Mar 2010 18:16:26 +0000

I'm sorry, but "stable distro" and "latest versions of A, B, C" just don't jibe.

Whenever I really needed installing non-official software like that, after much looking around I usually decided not to do it. And when I did, the "application A" for which I truly, really, no-other-option-works, had to get a later version it was something very localized (not exactly (pieces off) newest Gnome or KDE), and I installed that from source (and created a package for simple installation/update). The "dependency hell problems" mentioned mostly just weren't.

Where I did install a larger set of stuff was when we had Suns with Solaris, where many pieces were almost useless (like the infamous cc or its klunky sh, or its bloated beyond recognition version of X) . There the first step was what somebody called GNU > /usr/local (which did include X Windows, TeX and an assortment of other pieces). But it was still done carefully and as limited as reasonable.

Applications and bundled libraries

mikov — Fri, 19 Mar 2010 18:12:17 +0000

Isn't that exactly what I am saying? The LSB is not adequate to solve
the problems to which bundling is perceived as a solution. Neither is
distribution package management.

Applications and bundled libraries

dlang — Fri, 19 Mar 2010 18:02:28 +0000

if you look at the LSB, I believe that you will find that the GUI libraries are not part of it's specs.

Applications and bundled libraries

mikov — Fri, 19 Mar 2010 17:28:00 +0000

Unfortunately, the LSB libraries and the ones coming from package
management are not really standard, or we wouldn't be having this
discussion at all.

The GUI libraries are the biggest problems I still experience
horror trying to get vmware-server-console working on a new system.

But there are plenty of other things which come bundled with
Windows, like secure sockets (Firefox doesn't use them, but Chrome
does AFAIK) and so on.

The bottom line is that bundling for Linux will create larger and
clumsier applications than their Windows equivalents. I am not
advocating against it - it is a solution, but I think it is a
really really ugly solution. In this case it is better to work on a
proper solution than to settle for a horrible one.

What would be a "proper" solution? For example, a global repository
with libraries which can all exist side by side. For example, if
Firefox wants to bundle a library "libfoo", it will be called
"libfoo-firefox-e0b146e7-26e5-4c2c-90e0-bec9bac7218e.so.1.2" .
Other packages can use it, and decrease the duplication. There will
have to be extensive metadata, etc.

It is complex, but it is worthwhile and doable.

Applications and bundled libraries

jospoortvliet — Fri, 19 Mar 2010 17:06:58 +0000

Interesting. Yes, you understood what I meant, it's strange that you can't reproduce it. Maybe it does depend on the windowmanager used. Either way, I haven't used it in a while now - I did use it when reading lots of PDF's while doing research for a paper. I vividly recall how I disliked Acrobat, as it was far less nice to use for reading. I actually went home behind my own computer screen to read, rather than doing it at work (where I only had windows available).

But I guess everyone's habits are different, as are preferences ;-)

I just wanted to illustrate a very small yet nice feature Okular has which makes it (to me) nicer than Acrobat. It has more of those, of course ;-)

Applications and bundled libraries

tzafrir — Fri, 19 Mar 2010 16:12:08 +0000

You can consider whatever is in the LSB as "standard library".

Alternatively, consider any library that comes with the package management system as "standard library" :-)

Now, check what libraries Firefox bundles in its installation for Windows (those are not system libraries)

Applications and bundled libraries

dbruce — Fri, 19 Mar 2010 15:38:48 +0000

"It can work for a few hundred packages easily enough to do it the 'apt-get way'. It can scale upwards to several thousands. But to be on par with something like what Windows provides you have to scale to millions and you have a shitload of programs that nobody in the Debian project (or Fedora or Redhat or anybody else) will never be aware of, much less know enough to package and build them for end users.

There is just simply no chance in hell that a centrally managed, closed system like the current package management status quo can ever hope to scale and meet the diverse needs of the average population."

I don't buy that at all. The fact that there are "millions" of Windows programs available (assuming that is even true, which I doubt) just points out the inefficiency of closed-source development, where it is not possible to build on someone else's program without explicit licensing, fees, etc. The "millions" of programs are the problem with Windows, not an advantage of Windows. When I show people a Linux system, they invariably are more impressed by centralized package management than by any argument concerning freedom.

If you look at everything that is in Debian, it is hard to come up with a niche that isn't covered. Some of the OSS programs may lack important functionality, but in principle they could be improved to cover everything that their commercial counterparts provide.

I just don't see any need to provide "millions" of programs. If you have some need that isn't addressed by the ~25K packages in Debian, it probably requires a custom application anyway.

Static Library

mgedmin — Fri, 19 Mar 2010 15:32:44 +0000

Static linking has the same disadvantages as bundling own libraries, with
the addition that you can't easily see which apps bundle which libraries,
making e.g. security support that much harder.

Applications and bundled libraries

mgedmin — Fri, 19 Mar 2010 15:28:14 +0000

Linking statically would not be an improvement over bundling.

Applications and bundled libraries

jengelh — Fri, 19 Mar 2010 11:36:23 +0000

Symbol versioning would not work in this case, I think, because as you recompile $broken_program (which can easily happen in things like rawhide/factory), you are linking it against $new_symbol.

Applications and bundled libraries

smurf — Fri, 19 Mar 2010 11:30:48 +0000

Yeah, for major changes. For minor ones (like the aforementioned bug workarounds), symbol versioning is a much better choice.

Applications and bundled libraries

hummassa — Fri, 19 Mar 2010 10:57:30 +0000

About why it affects security patching: because then you have to patch it in two places, instead of
one.
About modified-sqlite: yes, it is a library. if it is required by chrome and supercedes regular-sqlite
(without any api/abi incompatibilities), it should be packaged as another, modified, version (and
SONAMEd accordingly); else, it should be packaged as another package altogether (and anyway, yes,
security patching must be done in each package, regular-sqlite and chrome-sqlite, but in the first
case, you can have only chrome-sqlite in memory even if another program wants to use sqlite)

Static Library

nikanth — Fri, 19 Mar 2010 06:05:29 +0000

Use static libraries and provide a single binary. Why do we need dynamic libraries, unless we want to provide multiple binaries for a package?

Static linking could achieve the goal without violating the Linux model.

Applications and bundled libraries

mikov — Fri, 19 Mar 2010 03:54:48 +0000

It seems to me nobody has commented on one huge difference between
Windows and Linux. Windows already caries a huge amount of libraries
by default in the OS. That may not be apparent, but all the thousands
of Win32 APIs, are in essence standard libraries available everywhere
(more or less). This includes GUI controls, etc.

There is no equivalent in Linux. The only somewhat standard library
is libc. A "bundled" Linux app would have to bundle everything under
the sun - X11 libraries, GUI toolkits, etc. This is crazy and in no
way comparable to Windows.

Applications and bundled libraries

djao — Thu, 18 Mar 2010 22:22:29 +0000

I have okular installed here (Fedora 12) and I cannot reproduce the behavior you describe.

One problem is that, most of the time, when I'm scrolling through a PDF, I want to read the pages from the top down (i.e. scroll forward through the file). However, in order to scroll forward by dragging the main page with the left mouse button, the mouse cursor itself actually has to move up in order for the page content to move down. So my mouse cursor never hits the bottom of the screen like you describe, unless I'm scrolling backwards, which happens very rarely. When I scroll forward, the cursor hits the top of the screen, and when it hits the top, it certainly doesn't automatically wrap the cursor to the bottom.

Since I cannot reproduce this behavior, I have to make certain assumptions about what you mean. Assuming you meant that the mouse cursor wraps from top to bottom, I can see how it would be a worthwhile option, but I would never use it myself. Most of my pdf reading occurs on a laptop, with a touchpad, in which case dragging the page is worse than useless -- it requires holding down a button as well as moving a finger along the touchpad, whereas the scrollwheel is built in to the touchpad and only requires moving a finger along the touchpad, and thus involves strictly less work. The only time I use dragging is for fine (pixel-level) scrolling control that cannot be achieved with the scroll wheel, but in such cases wraparound is unnecessary.

In addition to the lack of utility, my own opinion is that the bottom of the screen should be an absolute boundary to movement, not an invitation to wrap the cursor around to the top of the screen, no matter how worthy the justification may be. Moreover, if the PDF is displayed in a window, rather than full screen, then automatic cursor wraparound would be even more jarring, as it would jump from the top of the window to the bottom of the window rather than the top of the screen to the bottom of the screen.

Applications and bundled libraries

dlang — Thu, 18 Mar 2010 22:17:54 +0000

what I've seen is that when you try to drag outside the window, it gets interpreted as a scroll request and the window starts scrolling until you move the mouse back into the window area.

I don't know if this is implemented by the window manager or by the individual app.

Applications and bundled libraries in the Ideal World (LSB)

dlang — Thu, 18 Mar 2010 22:14:34 +0000

look at what LSB defines. it only defines a very small portion of the system.

if the application only uses the things that are defined by LSB, then it would work on either distro.

but if the application uses things that are outside the scope of the LSB, then it may not work.