this is sad [LWN.net]

this is sad

Posted Jun 18, 2016 18:51 UTC (Sat) by amacater (subscriber, #790)
In reply to: this is sad by josh
Parent article: Klumpp: A few words about the future of the Limba project

The value of Debian is that it cares about dependencies and dependency chains - and licences. Policy and developer practice helps reinforce this.

Ubuntu's Snap format and Flatpak and the others that seek to produce "all dependencies in one place" are fantastic if you want e.g. Firefox on a phone and that's all you want.

Anywhere else, they seem large, bloated and don't serve to produce a better ecosystem once you've got several of them around.

Fundamentally - you've got to understand what your program needs.

to post comments

this is sad

Posted Jun 19, 2016 2:58 UTC (Sun) by drag (guest, #31333) [Link] (4 responses)

If you look you'll see that they are not substantially larger then their rpm/deb equivalents if done correctly. Recently they re-did the Libreoffice snap that took the size down from ~1GiB down to 280-ish MiB or so. That is only 70-ish megs larger then equivalent rpm or deb files and even then those don't include Java, which is necessary to unlock all the Libreoffice features.

this is sad

Posted Jun 19, 2016 8:56 UTC (Sun) by lsl (subscriber, #86508) [Link]

As it is, the flatpak variants are also strictly less useful. With a standard LO installation, I can easily use its filters from a script to convert weird document formats to something more usable. I can just install a JRE to unlock the remaining features. Does the latter work with the Flatpak? What would I need to go through to achieve the former?

LibreOffice is already the kind of software that's a pretty good match for what Flatpak can do. The utility of many other programs will be much more severely restricted when decoupled from the system.

Interaction with the system is a desired feature of most software. These are not smartphone apps. It's also the hard part of any sandboxing effort. Annoying the user with thousands of file picker dialogs is not going to cut it except for the most trivial of programs.

this is sad

Posted Jun 19, 2016 8:59 UTC (Sun) by nim-nim (subscriber, #34454) [Link] (1 responses)

They're comparing pathological cases, apps which have been decades in writing, with lots of specific private code, and which have been forced to sanitize their requirements due to years of distro pressures.

That's pretty much the ideal case for flatpack or snap (I suspect just taking apache openoffice instead of libreoffice would be quite less rosy).

Even there they have ~ 30% of wastage.

That's the usual "trade dev convenience for everyone else's" and "cargo cult windows/apple/android to win market share¹" (the only change is who is being cargo culted, usually via themes of other superficial things)

It will probably end with the usual "dev does not understand why no one follows him".

I just hope they won't damage too much the platform in the meantime.

¹ Years of efforts poured in the Linux desktop, with lavish use of distro gifts and exceptions, have produced little in terms of market share. Meanwhile the people who focused on doing their own thing instead of aping the mistakes of others have actually found their way to people homes (in appliances, in owncloud, etc). With things like Office 365 and skype Microsoft is actually on the way to the network desktop that could have been done years ago Linux side if people had not focused on burying its *nix network heritage and copying every half-hassed network-ignorant Microsoft or Apple tech they could find out.

this is sad

Posted Jun 19, 2016 9:46 UTC (Sun) by halla (subscriber, #14185) [Link]

"trade dev convenience for everyone else's"

It has nothing to do with "dev convenenience" -- I would prefer not to have to make binaries, it's a timesink. And "everyone else's" only applies if by "everyone" you mean "everyone but end users".

this is sad

Posted Jun 19, 2016 21:03 UTC (Sun) by callegar (guest, #16148) [Link]

Doesn't this just prove that you can bundle all libraries in DEBs exactly as the snap packages do? This comparison is against the LibO provided DEBs, that bundle a lot of stuff. If you compare with the Debian provided DEBs, I think you get 280MB vs ~100MB, which is almost 200% larger.
Increasing size is not such a big issue for the disk (you can get a bigger one), it is an issue for your cache memory.

this is sad

Posted Jun 20, 2016 0:47 UTC (Mon) by zblaxell (subscriber, #26385) [Link] (13 responses)

Debian's APT tool is an awesome technical solution to an important problem that few others have attempted to solve, much less successfully solve as well as Debian has. apt-get scales up to distributing and installing 5-digit numbers of packages by 3-digit numbers of developers to 8-digit numbers of users, and Debian does an admirable job of ensuring continuous service from devices running their software. I've used it while wearing developer, administrator, and end-user hats for twenty years, and for what it does, it's far superior to anything else.

The problem is that APT isn't a solution to a problem anyone not participating actively in the Debian/GNU project expects to be solved. It's a workaround at best, and it's a failure (i.e. the opposite of success) when measured against several important performance standards.

APT needs minutes of CPU time on a PC-class machine to navigate through a carefully curated (i.e. expensive and error-prone) database of packages labelled according to their dependency relationships. A single-package install on a smartphone-class machine could easily drain a fully charged battery just trying to solve the dependency graph, let alone doing all the IO to perform a non-trivial dependency upgrade.

Assuming APT doesn't finish a package install request by saying "sucks to be you, I can't figure out how to install that," the result can still be disappointing. The configuration APT generates can be specific to an individual user or server instance, which has severe QA and support implications even when it doesn't fail partway through in some disastrous way (which is a thing that happens sometimes, and must be planned for).

In order to get any sort of reproducible behavior and mitigate the technical risks, one has to create chroots and use filesystem snapshots to replicate the various possible development and runtime configurations, back out of bad install/upgrade disasters, and sometimes even to run the applications as well. This would all be worthwhile to build a unique custom setup or development prototype, but in most cases the intended result is to just run a configuration similar to some well-known template shared by thousands of users--and a monolithic binary download is a much better way to get that in practice. Once you've crossed the line and implemented containers or VMs to resolve dependency issues, APT is only useful as a dumb package downloader.

APT edits the runtime in ways that carry elevated risks of catastrophic failures (e.g. bricked device) compared to its competitors based on mostly read-only (i.e. monolithic "system" image) or logically isolated components (i.e. chrooted containers). Often the last successful interaction an end-user has with the device is answering "yes" to the question "Do you want to install this upgrade?" and that's not a story anyone wants their end-users to tell from a first-person perspective.

APT relies on its maintainers to provide a detailed catalog of all the non-trivial dependencies between packages as they are found "in the wild." Everyone else considers the wildness to be the root cause of their problems, and tries to solve that instead. They integrate dependency packages into a single domesticated whole, simplifying the dependency graph down to a handful of (possibly very large) nodes. Ruthless decisions are made to control the size of the dependency graph: we will have feature X, feature Y in the next release, feature Z is deprecated and users will be forced to upgrade or opt out of accepting future releases, no inter-package dependencies are permitted without the permission of the vendor, etc. Vendors didn't invent these strategic restrictions because they're greedy or evil--they *discovered* them, because they're trying to build products that work well enough to sell.

Using apt-get requires specialist skill, and that skill isn't getting any cheaper over the decades. Today, machines range in price from $5 to $1500, and most users' data is worth not more than $100 (thanks to ransomware vendors for researching this for us!). APT easily absorbs $1600 in support costs when something goes wrong, making it more expensive to fix than to throw away. There's no guarantee that spending $1600 will fix APT either, so throwing it away is a more predictable way to spend money.

The one advantage APT used to have was the ability to do in-place live upgrades on a server with a high probability of success. Debian used to be good at that sort of thing, but they started failing in 2008 and haven't gotten better since. If every upgrade carries the risk of a service interruption, it's both cheaper and safer for a server operator to just start over with a new machine, validate all the old applications on a completely clean install, and cut services over from the old machines at some network layer above the individual machines. This is now SOP in the industry, and it means that even if Debian and APT did still support live upgrades well, nobody needs it any more.

APT could have a place among the set of users who will spend thousands of dollars worth of their own time to upgrade a $20 router into a $200 one. There are vendors who serve those users (Raspberry Pi is especially successful here), but even these vendors tend to build monolithic device-specific images on SD cards instead of integrating their changes into the base distribution.

this is sad

Posted Jun 20, 2016 2:02 UTC (Mon) by pizza (subscriber, #46) [Link] (6 responses)

> APT needs minutes of CPU time on a PC-class machine to navigate through a carefully curated (i.e. expensive and error-prone) database of packages labelled according to their dependency relationships. A single-package install on a smartphone-class machine could easily drain a fully charged battery just trying to solve the dependency graph, let alone doing all the IO to perform a non-trivial dependency upgrade.

You both vastly overstate apt's CPU requirements and understate modern smartphone processing and I/O capabilities.

And a random apt package's I/O requirements are most likely better than a random android app's requirements, especially when you consider that modern Android actually *compiles* the app upon installation, and nearly everything interesting going on under the hood (on Android) happens via bog-stock sqlite databases.

this is sad

Posted Jun 20, 2016 4:42 UTC (Mon) by zblaxell (subscriber, #26385) [Link] (5 responses)

> You both vastly overstate apt's CPU requirements

You're right, I fudged apt's CPU requirement, but not in the direction you're thinking. I'm also aware of current smartphone processing capabilities even though I didn't state them.

I've seen apt keep an 8-core 3.2GHz machine with 24GB of RAM busy overnight trying to solve a dependency problem. That's with just 22,000 packages on a machine many times more powerful than a current smartphone. Feeding a million apps to apt (assuming we ignore the insanity of downloading a million-app package database) would crush its tiny brain.

sqlite trickle-writes a few pages at a time, usually in response to external stimuli (i.e. it's rate-effectively rate-limited by the user). An apt-get upgrade is gigabytes of I/O all at once, and nothing works right until it's done.

this is sad

Posted Jun 20, 2016 12:28 UTC (Mon) by pizza (subscriber, #46) [Link] (4 responses)

> I've seen apt keep an 8-core 3.2GHz machine with 24GB of RAM busy overnight trying to solve a dependency problem.

That sounds more like there's something wrong with the dependency tree due to broken packages.

Granted, some dep solving algorithms are better than others (witness the significant improvement that dnf brought over yum) but even back in the yum days it still took on the order of a couple of minutes to do the depsolving of a major distro upgrade. (And I might add that apt was noticably faster than yum when doing the same sort of thing..)

Still, this isn't exactly an apples-to-apples comparison; your typical GNU/Linux installation has an order of magnitute more software components installed than a smartphone.

And, I might add, after even a minor point update of Android, the system has to "optimize apps" for the new OS (aka "recompile the bytecode") which takes my tablet about 30 minutes for under 100 "apps", leaving the tablet considerably warmer in the process. There's quite a bit of crap going on there.

this is sad

Posted Jun 20, 2016 17:18 UTC (Mon) by zblaxell (subscriber, #26385) [Link] (3 responses)

> Still, this isn't exactly an apples-to-apples comparison;

If you want to understand why users want apples instead of apt, you have to compare apt to apples.

> your typical GNU/Linux installation has an order of magnitute more software components installed than a smartphone.

The "order of magnitude more software components" is the problem statement. Apt is a symptom of this problem, not a solution. Complexity is not competitive.

If I pick a random desktop machine, it has 128 leaf-dependency packages installed, not even double the number found on a modern smartphone (certainly not an order of magnitude more). Each of these is effectively stand-alone with one or two big dependencies (e.g. ten of them are bits of KDE, one is all of LibreOffice, one is most of xorg, one is Chromium, ssh, rsync, etc). Like a modern smartphone, almost half of these packages turn out to be games.

Apt tracks 2484 separate package objects to support the top 128. If I pick a random package and try to upgrade it, apt informs me it breaks 56 other packages immediately, and those have dependencies too. I've done this dance many times before. There will be well over a thousand packages upgraded before it's done, and I'll have to intervene multiple times during the process as apt will pick a solution from 525600 candidates for upgrading this desktop that isn't the one that Debian-QA tested, so it doesn't work.

Most users (myself included) won't do such an upgrade, because the cost and risk just aren't competitive. It's easier, faster, and more reliable to just debootstrap in a chroot or a new root LV and rebuild the local site configuration manually (that's what tools like etckeeper and puppet are for, after all). This will create a new tree of 2484 moving parts that will, unless there is a radical change of direction in the way Debian manages dependencies, also be too expensive to maintain.

This mess is what failure to say "no" to complexity looks like.

this is sad

Posted Jun 20, 2016 17:48 UTC (Mon) by pizza (subscriber, #46) [Link] (2 responses)

Okay, so you're advocating for the "just replace the system" definition of an "upgrade".

That's all fine and dandy. Except that, in order to set up the system to begin with, someone is still going to have to deal with that complexity. And end-users are going to have to live with whatever choices were made for them.

this is sad

Posted Jun 20, 2016 18:05 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]

FWIW, there are several projects that already do transactions upgrades. For cloud instances, CoreOS is popular. There is also

https://fedoraproject.org/wiki/Workstation/AtomicWorkstation

Pushing binary deltas of images is easier. Layering packages into a underlying image based update systems is a bit more tricky but it is feasible to do.

this is sad

Posted Jun 20, 2016 19:25 UTC (Mon) by zblaxell (subscriber, #26385) [Link]

Not really advocating, just clarifying the motivation behind the simplified dependency model used in projects like flatpak, Limba, etc instead of apt, libsolv, etc. The optimal model is probably somewhere between those extremes: a mixture of aggregation to simplify the dependency graph for common cases, and isolation to allow individual applications and the host system to resolve dependencies without disrupting each other.

It's a safe bet that most users do want to enable someone else to make an apparently monolithic supportable package (or least an aggregation of components that behave as a single unit) out of thousands of mundane technical choices on their behalf. Those that want to make choices themselves self-select themselves away from the rest of us--they just build what they need from source without bothering anyone.

this is sad

Posted Jun 20, 2016 7:39 UTC (Mon) by niner (guest, #26151) [Link] (5 responses)

You start with the assumption that APT is the pinnacle of dependency managers and conclude that if APT has performance issues, the concept as a whole cannot work. Your conclusion is based on a flawed assumption and therefore wrong as well. I'd guess that openSUSE's libsolv is the current top of the line dependency solver which is the reason for why it's not only used by openSUSE's zypper but also by Fedora's dnf and probably others. Until you demonstrate that even libsolv fails in your scenario, your argument stands on a very weak base.

this is sad

Posted Jun 20, 2016 11:07 UTC (Mon) by nye (guest, #51576) [Link] (3 responses)

>Until you demonstrate that even libsolv fails in your scenario, your argument stands on a very weak base.

The problem has been proven NP-complete, so it doesn't matter how many times you cry "but that's only one example!".

Until you demonstrate that P=NP, your argument stands on a very weak base.

this is sad

Posted Jun 21, 2016 11:00 UTC (Tue) by niner (guest, #26151) [Link] (2 responses)

Theoretical limits are irrelevant when it does work well enough in practice. If the dependency solver performs well with millions of packages and comparatively small dependency trees, it doesn't matter if it would take years with a hundred times more candidates. IOW it doesn't need to solve the general problem, only the specific one you encounter in practice.

If it were different, people would not even have to bother trying to write something as widespread available as a syntax highlighter for highly dynamic programming languages like Perl or maybe even C++. Because to fully understand all parts of the source code, they'd have to solve the halting problem. So they don't. They do not pretend to handle completely arbitrary input programs but only those that matter in practice.

this is sad

Posted Jun 21, 2016 12:12 UTC (Tue) by nye (guest, #51576) [Link] (1 responses)

>If the dependency solver performs well with millions of packages and comparatively small dependency trees

That's a fairly major 'if'. It's already not too great with tens of thousands of packages[0]; I don't believe it can ever scale another three orders of magnitude (which is what would be required to be truly mainstream) without a change in approach.

If I understand correctly, package managers have already given up on finding 'optimal' solutions[1] and now use heuristic SAT solvers, but regardless, at some point distros have to let go of the idea that the installability of a given package is a function of *every other possible package*. Eventually to continue scaling there simply has to be some demarcation line that lets the solver prune the tree to something small enough to keep dependency resolution tractable. One way to do this is by separation of a core platform from leaf packages that don't (or rarely) have complex interdependencies, which coincidentally happens to describe how almost every end user actually wants a system to work.

[0]in the sense that both apt and aptitude can take a while to find a solution, especially on a less powerful machine, and aptitude will happily suggest a seemingly endless selection of terrible 'solutions'

[1] in the sense of calculating the minimal set of changes required to satisfy the request

this is sad

Posted Jun 21, 2016 12:22 UTC (Tue) by nye (guest, #51576) [Link]

>If I understand correctly, package managers have already given up on finding 'optimal' solutions[1] and now use heuristic SAT solvers

Just to expand briefly on this: I can't find a quick definitive answer in five minutes to whether libsolv, as used by zypper, is a heuristic solver or a pure solver, but there are unavoidable problems either way: if it's pure then it can't scale efficiently; if it's heuristic then it can't guarantee optimal results, and could potentially fail to find possible solutions.

Reducing the problem space by dividing it into smaller sets with fewer interdependencies is a solution to both.

this is sad

Posted Jun 20, 2016 12:13 UTC (Mon) by zblaxell (subscriber, #26385) [Link]

Even if you made the cost of solving dependency graphs zero, it wouldn't solve the other problems, only work around them.