Keen: The case against upstream packaging [LWN.net]

Arch maintainer Kyle Keen speaks out against direct delivery of software by upstream projects. "Maintainers' greatest power is the ability to outright say 'This is not good enough for our users' and consequently punish an ISV by either patching out the offensive part or in extreme cases removing the software from the repositories. ISVs know this and so don't act out. After 20 years of enforced good behavior this has lead to the idea of ISVs as 'the benevolent upstream developer.' This is why Linux doesn't have spyware, doesn't come with browser toolbars, doesn't bundle limited trials, doesn't nag you to purchase and doesn't pummel you with advertising."

to post comments

Keen: The case against upstream packaging

Posted Jun 15, 2016 20:58 UTC (Wed) by bmds (guest, #106429) [Link] (37 responses)

In the first paragraphs it didn't seem really clear what this article was really about: just snap or any way to distribute software that doesn't go through the repositories of a distribution first? It turns out, this article seems mainly aimed at the scenario of a wide adoption of snap and snap-like systems combined with loosely controlled app-stores. I don't really see its points correctly applying to the practice of upstream providing their own repos or packages made to play nice with specific versions of a distribution. For example, it seems to me perfectly reasonable for CFEngine to provide a Debian package for their LTS version because it couldn't get in before the freeze. Sometimes we want upstream to be able to get around the (hard to fix) bureaucracy of a distribution.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:23 UTC (Wed) by pizza (subscriber, #46) [Link] (36 responses)

> I don't really see its points correctly applying to the practice of upstream providing their own repos or packages made to play nice with specific versions of a distribution.

Except the entire point of snap/etc is that you don't have to play nice with any given distro version, nicely enabling "loosely controlled app-stores" with all of the benefits and pitfalls that entails.

People are advocating for snap/etc because "playing nice" is too hard/too much work/etc.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:28 UTC (Wed) by halla (subscriber, #14185) [Link] (34 responses)

"Play nice with any given distro"

Ubuntu 16.04 comes with Krita 2.9.7. That is absolutely ancient, it's about a thousand bug fixes and a hundred productivity-improving features behind current.

I am glad that it also has snap, so the users of this LTS release can get something up to date.

For the rest, in this discussion about the new ways of distributing applications to users -- people seem to forget that software comes in two kinds: the kind where you don't care about the version number, because any old version gets the work done just as fast, and the kind where a user needs the latest version because that means they're more productive.

For some people, bash needs to be the latest version; I'm not one of them. For some people, any old Krita is fine; but the people who are earning their income with Krita need the latest version.

And that's why I support Krita on Ubuntu 12.04 and CentOS 6.5.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:39 UTC (Wed) by pboddie (guest, #50784) [Link] (15 responses)

Can "backports" repositories help with getting newer releases to users of stable distributions? Genuine question!

Keen: The case against upstream packaging

Posted Jun 15, 2016 22:07 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Not really. You usually end up backporting all the dependent libraries for complex applications like Krita. At which point it's just easier to bundle up everything.

Keen: The case against upstream packaging

Posted Jun 16, 2016 6:36 UTC (Thu) by halla (subscriber, #14185) [Link] (8 responses)

In practice, no. Maintaining the Krita Lime repo for Ubuntu was already a major time sink -- partly because Launchpad is horrible, partly because it throws all the differences between different versions of Ubuntu in your face. Add to that OBS for OpenSUSE and a special repo for CentOS 6.x, and we haven't even started covering a decent chunk of distributions, but we do have lost a big chunk of development time.

One binary that runs on all Linux distributions, like AppImage, is a wonderful thing. I wouldn't have made a snap if Michael hadn't contributed it, and I can confirm that it's easy to use and that the Ubuntu appstore is easy to use: supporting snap takes me about five minutes per release, excluding waiting for the build. That's something I can afford. In contrast, we still haven't managed to setup a new version for Krita 3.0 in the Krita Lime ppa. I don't know about flakpak yet.

Keen: The case against upstream packaging

Posted Jun 16, 2016 11:43 UTC (Thu) by pizza (subscriber, #46) [Link] (4 responses)

> One binary that runs on all Linux distributions, like AppImage, is a wonderful thing.

Only by bundling pretty much the entire OS with it, basically not relying on anything being present but the kernel. Which only gets you so far, as there are important things that are relatively tightly coupled with the kernel. The elephant in the room being the graphics subsystem but this affects anything that needs to talk to hardware.

Oh, and printing. Which is such a complicated mess that I'm still amazed it works as well as it does.

Keen: The case against upstream packaging

Posted Jun 16, 2016 13:13 UTC (Thu) by halla (subscriber, #14185) [Link] (3 responses)

Nah, you're exaggerating. It's much less than that, as you can verify by mounting the image and checking it. And even if you weren't, I wouldn't give a damn. What gets bundled inside the image is only relevant for armchair pundits: for users the fact that this is possible is just a wonderful thing.

Keen: The case against upstream packaging

Posted Jun 16, 2016 15:31 UTC (Thu) by pizza (subscriber, #46) [Link] (2 responses)

So please tell me how I can use darktable, with its use of OpenCL, on CentOS6 with a snap-like package? Keep in mind that the kernel+DRM is too ancient to support the modern, bleeding-edge-by-necessity userspace libraries that would be bundled with the package.

Yes, CentOS6 is positively ancient but supporting exactly that sort of scenario is how the likes of Snap are being sold to us.

As for printing, I'm still stuck dealing with bug reports from people using a *prerelease* driver that Ubuntu bundled with 14.04 LTS, and won't update. Unless the snappy applications bundle more-up-to-date printer drivers. Although unless snap packages are allowed to claim arbitrary USB devices, I don't know how that would possibly be expected to work.

Keen: The case against upstream packaging

Posted Jun 16, 2016 16:54 UTC (Thu) by halla (subscriber, #14185) [Link]

"So please tell me how I can use darktable, with its use of OpenCL, on CentOS6 with a snap-like package?"

I don't know, ask a Darktable developer? I do Krita, and that's all I do -- and have done for the past ten years.

Keen: The case against upstream packaging

Posted Jun 20, 2016 12:04 UTC (Mon) by hkario (subscriber, #94864) [Link]

I'm using Steam on Arch Linux

believe me, you can't have a cake and it it too

I constantly have to override the shipped with Steam glibc and libGL to make it work (at all) on Arch

yes, for applications that don't need OpenCL, OpenGL or sound, it may be possible to just bundle the world and be done with it, but it's not a universal solution

Keen: The case against upstream packaging

Posted Jun 16, 2016 11:53 UTC (Thu) by ovitters (guest, #27950) [Link] (2 responses)

> supporting snap takes me about five minutes per release

How much work is AppImage? Initially, per release, etc?

Keen: The case against upstream packaging

Posted Jun 16, 2016 12:19 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

I think any system, once set up, should be in the minutes-per-release category (compile time notwithstanding).

The initial setup time is another matter.

Keen: The case against upstream packaging

Posted Jun 16, 2016 13:16 UTC (Thu) by halla (subscriber, #14185) [Link]

Setting it up was a bit hard, at least it took me a lot of time, but I was new to the whole thing. Starting the build and uploading the image takes about five minutes of my time, and I should find a volunteer to automate it for me, of course.

Keen: The case against upstream packaging

Posted Jun 16, 2016 12:18 UTC (Thu) by Wol (subscriber, #4433) [Link] (4 responses)

> Can "backports" repositories help with getting newer releases to users of stable distributions? Genuine question!

Which is no ******* use whatsoever when you're running the latest version of the distro!!!

That's why my distro of choice is now gentoo. Can't remember why, I wanted the latest version of lilypond for some bugfix or other. And running the then *current* version of SuSE, I was running a well-old version of lily.

Cheers,
Wol

Keen: The case against upstream packaging

Posted Jun 16, 2016 15:36 UTC (Thu) by pboddie (guest, #50784) [Link]

Maybe what I consider "backports" is different to everybody else's definition, but I seem to remember that the idea of such repositories was to deliver later stuff than what the normal repositories provide for any given distribution release. Whether it is the absolute "latest and greatest" is another matter.

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:30 UTC (Fri) by ceplm (subscriber, #41334) [Link] (2 responses)

Again, I don't get the logic. Instead of filing a bug with the packager of the given distribution (and possibly helping him/her to fix the bug), you switch distro or start to use bundled minidistros (aka Snap/Flatpak)? Doesn't make any sense to me.

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:38 UTC (Fri) by raven667 (subscriber, #5198) [Link] (1 responses)

If the distro maintainer already doesn't have time to keep up with upstream versions, how would filing a bug report create more time?

Keen: The case against upstream packaging

Posted Jun 17, 2016 18:32 UTC (Fri) by ceplm (subscriber, #41334) [Link]

Filling the bug and helping her?

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:27 UTC (Fri) by ceplm (subscriber, #41334) [Link] (17 responses)

Well, it is called a bug in the distro. And it should be fixed. Did you ask nicely maintainer of Krita in Ubuntu to upgrade?

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:41 UTC (Fri) by raven667 (subscriber, #5198) [Link] (16 responses)

The whole point of a distro, especially an LTS distro, is that they _don't_ upgrade versions of software, I'm not sure why you think calling that a bug would change this behavior, it goes against almost every policy that defines Debian or Ubuntu or SuSE or Fedora or Redhat, Firefox is one of very few exceptions to this policy in most distros.

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:57 UTC (Fri) by halla (subscriber, #14185) [Link] (13 responses)

And that's why those policies are misguided -- there is no reason to keep application software like Blender, Krita, Digikam, Shotwell, QtCreator and so on and so forth at a fixed version number. For the base system, yes, maybe, for applications that people actually earn their living with, no.

(Note: I have no clue about server stuff, I only work for and care about end users.)

Keen: The case against upstream packaging

Posted Jun 17, 2016 18:13 UTC (Fri) by raven667 (subscriber, #5198) [Link] (2 responses)

Right, which is why we are talking about Snap and Flatpak because talking about this with the distros hasn't changed their fundamental policies in the last 20 years, even talking about the system as a seperable Core that provides a standard API/SDK with Applications on top of that hasn't gone anywhere, with LSB pretty much flaming out by being unusably small.

Keen: The case against upstream packaging

Posted Jun 18, 2016 7:54 UTC (Sat) by ceplm (subscriber, #41334) [Link] (1 responses)

As usually with most discussions about new technology, there is even less understanding of the existing world than of the new technology itself.

> the distros hasn't changed their fundamental policies in the last 20 years

That just isn't true. Even RHEL (remember, supported for seven years at least) now regularly (in a year or two frequency) rebases some packages (including but not limited to whole Gnome), and LibreOffice and Firefox got regular updates. And of course, it doesn't matter for Fedora, because it lasts only for something over a year.

Keen: The case against upstream packaging

Posted Jun 18, 2016 15:29 UTC (Sat) by raven667 (subscriber, #5198) [Link]

Isn't that kind of the exception that proves the rule. The fact that you can list these packages as exceptions to the norm shows that the norm exists and highlights that every application isn't being supported the way that Firefox and LibreOffice are.

Keen: The case against upstream packaging

Posted Jun 18, 2016 7:55 UTC (Sat) by pabs (subscriber, #43278) [Link]

Those policies are why people use those suites of those distros. There are plenty of rolling release distros for folks who don't like those policies.

Keen: The case against upstream packaging

Posted Jun 18, 2016 22:00 UTC (Sat) by xtifr (guest, #143) [Link] (8 responses)

Actually, many people like having some stability in their apps, and would rather not have things constantly changing underneath them. The "stable" and LTS distros aren't just for server-side stuff. Not everyone needs or wants the latest-and-greatest when it comes to the stuff they use to earn their living. Heck, I know one SF writer who keeps multiple backups of his favorite DOS-based word processor, because he knows that if it dies, he'll never be able to replace it. The company that makes it went out of business in the nineties. (He was overjoyed when I told him about DOSbox, since one of his big worries is making sure his systems are able to keep running it.) That's an extreme case, but there are a lot of people who have a milder version of the same attitude.

Of course it's different for apps (especially open-source apps with their release-early approach) that still need key features. Then, it's reasonable for people to want to keep up with the latest-and-greatest. But one person's key feature is another person's unnecessary frill. So you can't easily divide apps up into will-want-updates and will-want-stability. It's a personal decision. So really, the best option is to have both. The stable/LTS distro versions for stability, and bundled packages for the latest-and-greatest. (Or "greatest", depending.) :)

Keen: The case against upstream packaging

Posted Jun 19, 2016 3:00 UTC (Sun) by raven667 (subscriber, #5198) [Link] (1 responses)

> That's an extreme case, but there are a lot of people who have a milder version of the same attitude.

I've found that kind of rigid attitude toward tools leads to extremely fragile workflows, as you describe, that become critically disrupted when the world inevitably changes around you. I personally use new tools from time to time and try to stay flexible, for example I just changed my main work desktop from Mac OS 10.11, which I've been using for 10 years, to Fedora 24 (beta) with GNOME3. The biggest change is that the hot-corners aren't where my muscle memory is used to them being and keyboard commands being CTRL-? instead of META-?. I kind of feel bad for people who are so set in their ways that they are completely lost when encountering a new environment, to me that seems like a weakness or liability.

Keen: The case against upstream packaging

Posted Jun 19, 2016 12:15 UTC (Sun) by ceplm (subscriber, #41334) [Link]

Of course, you are a geeek. And geeks love to tinker (I do too, I still haven't settled on my $EDITOR after fifteen+ years using Linux). Try to talk with some “normals”, be them large enterprise customers of Red Hat (yes, I am an employee), who have to pay multimillions of USD to retrain employees, or ask my 78-year old mum. They are absolutely not interested in the latest and brightest, but they want something which works the same all the time. There is over a million of people downloading Never10 (https://www.grc.com/never10.htm) because want still the same, because there are people for whom their computer is just a tool for something else, not an object of tinkering.

Keen: The case against upstream packaging

Posted Jun 19, 2016 7:20 UTC (Sun) by halla (subscriber, #14185) [Link] (5 responses)

Yes, there are a few people like that. But it's a rare attitude, and if they're happy with what their distribution provides, fine, as long as they don't report bugs, ask for support or write pissy tweets about not being able to load the latest brush pack.

Then there are people who are using Lubuntu 10.04 or Windows XP or OSX 10.5 but are mailing me about providing Krita 3.0 for those platforms... I'm also not catering to them.

On the whole, outliers can go and care for themselves: that's not my job.

Keen: The case against upstream packaging

Posted Jun 19, 2016 17:54 UTC (Sun) by ceplm (subscriber, #41334) [Link] (4 responses)

Well, that's the discussion about facts, and we don't have any hard evidence for the either part. I think that there is in fact a huge majority of people like that, who prefer their system just work as it did always. However, as I said, neither of us has any hard evidence, so I would suggest to stop this thread right here.

Keen: The case against upstream packaging

Posted Jun 19, 2016 20:31 UTC (Sun) by halla (subscriber, #14185) [Link] (3 responses)

Well, sure, if you want that. But I adduce my experience as a maintainer of an application that's being used by hundreds of thousands of users, someone who is getting feedback daily (and going a bit crazy trying to answer all of them). Sure, that's no "hard evidence", since I can hardly find the time to keep up as it is, let alone prepare statistics in a rigorous, repeatable way for you to check.

The thing is, it's pretty rare for someone like me to give people in the Linux community the benefit of their experience: most maintainers of free software projects do not have the same kind of wide community engagement I have. Even a project like gcc, which has many more users doesn't get that -- gcc typically is something you don't want to update because, bleh, lots of work, and nothing that actually improves the experience of the people using the application its compiled with, perceptibly. And the maintainers who have even more direct experience than me, like Ton Roosendaal, don't bother with LWN.

You can dismiss all of my sententiousness because it doesn't fit your world-view... But my experience is a fact, and you ignore it at the peril of ignorance.

Keen: The case against upstream packaging

Posted Jun 20, 2016 11:49 UTC (Mon) by ceplm (subscriber, #41334) [Link] (2 responses)

I absolutely do not want to dismiss your experience, and I know that you are very much involved in the community of users of Krita. However, we (Red Hat) had an interesting experience with Mozilla Foundation. For years, they mostly ignored Firefox on Linux claiming there are almost no users of the desktop Linux (and thus Firefox). The problem was they were counting only downloads from the mozilla.org site (which is were majority of Windows and Mac users got their Firefox from anyway). Only when we persuaded them that huge majority of Firefox users just use packaged versions from distributions, they agreed that most likely they have more users on Linux than on Macs (for example) and started to take bugs filed by Linux users (or indirectly by the Linux packages maintainers) seriously.

So, I wonder, whether you have right understanding of your user base after all. I would think that most people who contact you and who build their Krita themselves are some kind of tinkerers, and it is possible that actually majority of users of Krita just downloaded packages from their distros, use them, and not having any problems with the program just never bother to contact you.

Keen: The case against upstream packaging

Posted Jun 20, 2016 13:22 UTC (Mon) by halla (subscriber, #14185) [Link] (1 responses)

Well, it's hardish to figure out what people run who install from distributions, and how often that happens. But I think that bugs in bugzilla can sort of indicate relative levels of popularity, with some skew. The most enthusiastic users are the ones who have been around longest will also be the most prolific bug reporters. Another skew is that on Linux, the crash dialog offers to send a bug report, which doesn't work on Windows. Still some stats I just gleaned seemed interesting to me:

There is a total of 5820 bug reports for Krita, and we got 1724 in the last year. Keep in mind that Krita dates back effective to 2000 or so...

Of these bug reports, 888 were for Linux, 596 for windows, 40 for OSX and 200 unspecified. On linux, we have 146 crash reports vs 69 on Windows.

The distribution breakdown is sort of crazy and supports my gut feeling that every distribution other than *buntu is a marginal thing for my user group, especially if you count Mint as Ubuntu, too:

(k)ubuntu: 153
compiled sources: 148
mint: 60
arch: 44
debian: 36
opensuse: 27
fedora: 10
gentoo: 10
redhat: 3

Without http://www.davidrevoy.com/article193/guide-building-krita... we wouldn't have had so many reports against compiled sources, for sure.

In the end, my claim isn't that we don't have users on Linux, but that the users on Linux want the latest version:

Here are the versions we got reports for in the past year:

git: 281
unspecified: 344
3.0: 326
2.9: 667
2.8: 81
2.7: 5
2.6: 4
2.5: 4
2.4: 8
2.3: 1
2.2: 1

Keeping in mind that 3.0 has been out for only a few weeks and 2.9.0 was released a bit more than a year ago (https://krita.org/en/item/krita-2-9-0-the-kickstarter-rel...) , I think that this bears out the idea that users want the latest version immediately.

(And now my build is done...)

Keen: The case against upstream packaging

Posted Jun 20, 2016 13:34 UTC (Mon) by halla (subscriber, #14185) [Link]

Oh, and just for Linux, the version breakdown is as follows:

unspecified: 178
git: 244
3.0: 98
2.9: 304
2.8: 43
2.7: 3
2.6: 3
2.5: 4
2.4: 8
2.3: 1
2.2: 1

Keen: The case against upstream packaging

Posted Jun 17, 2016 18:31 UTC (Fri) by ceplm (subscriber, #41334) [Link] (1 responses)

Well, the bug was probably necessary in previous non-LTS versions of Ubuntu/Debian. But you were talking about many years old version of Krita, right? And yes, the experience taught us that even in RHEL rebases are necessary for desktop applications, so I am not sure that Debian/stable strategy of never changing anything is the best one. OTOH, there are backports repos, where you can get more recent version, cannot you?

Keen: The case against upstream packaging

Posted Jun 20, 2016 15:43 UTC (Mon) by pboddie (guest, #50784) [Link]

You can also run newer stuff in a chroot. I do this quite a bit with a Debian unstable chroot made by a debootstrap wrapper I use. You can get relatively modern packaged versions of software (that people like to claim do not exist) that also get updates, although the chroot isn't exactly integrated with the rest of the system, but at least you're still getting distro-level support and not the "multi-gigabyte image tossed over the wall onto ephemeral file-sharing site" that appears to be the fashionable future of software deployment.

Keen: The case against upstream packaging

Posted Jun 16, 2016 18:00 UTC (Thu) by patrakov (subscriber, #97174) [Link]

> People are advocating for snap/etc because "playing nice" is too hard/too much work/etc.

Playing nice is in some cases impossible - e.g. because your software is actually illegal in USA, and you still want to reach EU users.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:19 UTC (Wed) by jospoortvliet (guest, #33164) [Link] (33 responses)

Reacting to the quoted part: right, because all those open source projects would love to put spy ware in, what do they care about users?

If the rest of the article is as ignorant it isn't worth reading.

I am no fan of upstream packaging - if the distros do a good job, better. If not (can be for many reasons including conflicting demands of users vs distribution rules, app or language limitations etc) upstream can feel forced to package. And, to beat the same drum again, it is high time for improvements to packaging. Hi, Snappy!

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:27 UTC (Wed) by pizza (subscriber, #46) [Link] (4 responses)

> Reacting to the quoted part: right, because all those open source projects would love to put spy ware in, what do they care about users?

Let's be honest here -- The point of Snappy isn't to make Open Source packaging better or simpler; it's to enable proprietary, binary-only stuff that can be "released once, run everywhere", and to build an "App store" model around that proprietary stuff.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:42 UTC (Wed) by sjj (guest, #2020) [Link] (3 responses)

An appstore is clearly in Canonical's strategy. I don't see this as a necessarily bad thing. They need revenue and so do many open source projects, which could sell their application in an appstore for nontechnical users and supporters. I don't oppose commercial software vendors on Linux either.

Would this erode the Free/OS community in the long run? It's a possibility.

Keen: The case against upstream packaging

Posted Jun 15, 2016 23:19 UTC (Wed) by johannbg (guest, #65743) [Link]

Agreed and appstore is now the norm for end users and upcoming generation however people wont be partaking in fragmented appstores in the linux ecosystem.

Having multiple app market targeting the same platform wont solve anything since it just results in people chasing multiple appstores instead of chasing multiple Linux distributions ( or upstreams having to produce multiple apps of the same application, in different format just for different appstores instead ).

If this is going to work from my pov then there needs to be one community driven and maintained app store which paid and free applications coexist nicely and the community should be focusing the discussion more in the direction of the implementation of the app market itself because the infrastructure for it will require humans to review the application available there on top of the tools which is a lesson to be learned from Google playstore where end users have been subjected to scams like "Virus Shield" and malware apps based on malware family's like Gunpoder, cloned applications etc. basically the community should be asking itself what can we learn from Apple,Amazon,Google,Microsoft,Samsung etc app stores. Discussing pros and cons in implementations and policy's surrounding those appstores and settle on a single app format/specification for upstream to use..

Keen: The case against upstream packaging

Posted Jun 15, 2016 23:49 UTC (Wed) by pizza (subscriber, #46) [Link] (1 responses)

I'm not opposed to the notion of appstores in principle, but in practice it will quickly devolve to an utter cesspool.

Sure, an "open source project" could "sell" their application, at least until someone else comes along, wraps it in some sort of adware, adds a letter to the name, and puts it up for "free". I've seen this happen often enough on android.

Sure, you can mitigate some of that by using humans for review and curation -- so who's going to pay them? Who's going to be the gatekeeper, trusted to do the right thing? Will any of that get kicked back to the folks doing the under-the-skirt work on libraries and other tools that aren't wrapped in a sexy "app" someone might pay for?

The only ones making money on appstores are the folks running the stores and the folks that use their apps to provide/upsell some sort of paid service. While there are outliers that do quite well, the rest are barely scraping by with adware that's really just prettied-up spyware, if not borderline malware.

> Would this erode the Free/OS community in the long run? It's a possibility.

Methinks the probability is well in excess of 0.5.

Yes, snappy and its ilk will make it easier for a handful of complex, end-user-focused F/OSS projects to provide more timely updates to end-users. But that's just a happy coincidence used to sell it to the more naive amongst us.

Keen: The case against upstream packaging

Posted Jun 16, 2016 3:33 UTC (Thu) by drag (guest, #31333) [Link]

It depends on whose 'appstore' you are using.

From the 'Day in the life of a Fedora packager' blog post they mention https://github.com/fedora-infra/anitya to track upstream progress and releases. A similar mechanism can be used for tracking and pulling built packages.

Distributions possibly work together to establish a mechanism which upstream progress is tracked. Packages are signed so you know who originated them. They check upstream or ISV or whatever, vet the licensing, and make sure they are rebuildable for people that need that sort of thing. Unit tests to confirm builds can be put into place to validate that the source releases match the binary release if such a thing is a problem. Licenses can be tracked, dependencies can be tracked and once everything checks out then the packages are made available via 'FLOSS APP STORE' or whatever.

Once the ISV is trusted and there are no major complaints against it's conduct or reliability then it will require little effort to maintain updates for dozens of distributions at the same time. If users start noticing issues then projects can be later be reprised, corrected, rejected or forked.

There definitely need to be something in place. I don't want to be accidentally installing Firefx Browser from Mozila by accident.

Keen: The case against upstream packaging

Posted Jun 16, 2016 1:36 UTC (Thu) by robmv (guest, #103765) [Link] (27 responses)

> because all those open source projects would love to put spy ware in, what do they care
> about users?

Not all, but what is easier, to compromise the personal laptop those upstream developers use to build the package and browse the network (any kind of website) or Fedora, Debian, etc. build infrastructure?

Keen: The case against upstream packaging

Posted Jun 16, 2016 21:27 UTC (Thu) by jospoortvliet (guest, #33164) [Link] (20 responses)

I dunno. Compromise ONE subsystem of ONE of hundred of Debian Developers and you can sneak your code in all of debian's software. You trust ALL OF THEM to keep their systems super secure?

I wouldn't trust EVERY upstream more but many of them, for sure.

Keen: The case against upstream packaging

Posted Jun 17, 2016 20:43 UTC (Fri) by ballombe (subscriber, #9523) [Link] (19 responses)

Except that there have been various instances of upstream software that contained malicious code, but I have never heard of a single instance of a distribution package containing malicious code that was not upstream.

Upstream is a far bigger target.

Keen: The case against upstream packaging

Posted Jun 18, 2016 13:42 UTC (Sat) by NAR (subscriber, #1313) [Link] (18 responses)

Well, remember the openssl modifications by the Debian packagers? Not malicious by intent, still didn't really help the users...

Keen: The case against upstream packaging

Posted Jun 19, 2016 16:33 UTC (Sun) by ballombe (subscriber, #9523) [Link] (17 responses)

This has nothing to do with what I was replying to.

And actually Openssl was the prime example why you need packagers. Upstream was completely linux-hostile and you could barely compile the thing without patching it let alone install it on Linux.
All distribution had to patch it and upstream was breaking the ABI in minor releases, so you would end up maintaining three releases.

And given the openssl codebase is a disaster waiting to happen, it is really unfair to blame the Debian packager for the error.

Keen: The case against upstream packaging

Posted Jun 19, 2016 19:54 UTC (Sun) by bronson (subscriber, #4806) [Link] (16 responses)

If you're trying to blame the OpenSSL project for the bug then please explain this: why did it only affect Debian?

(no need to claim LibreSSL is proof that everyone was affected... the bug was Debian-and-derivatives-only after all)

Keen: The case against upstream packaging

Posted Jun 20, 2016 7:59 UTC (Mon) by paulj (subscriber, #341) [Link] (15 responses)

They _were_ fixing a clear bug in the upstream code. Fixed it spectacularly badly, but still the upstream code was buggy...

Keen: The case against upstream packaging

Posted Jun 20, 2016 17:17 UTC (Mon) by bronson (subscriber, #4806) [Link] (14 responses)

By "clear bug" do you mean "irritating valgrind warning?"

As an aside, whenever anyone tries to use a comment to explain questionable code, I tell them: /* even if n != i, use the full array */

Keen: The case against upstream packaging

Posted Jun 20, 2016 17:52 UTC (Mon) by zlynx (guest, #2285) [Link] (13 responses)

Talking about the case of reading data from an uninitialized array?

"Irritating warning" quickly becomes "compiler removed all the code because of undefined behavior." so it is definitely something to fix.

You know a C compiler is free to assume that an unset local variable is any convenient value and rewrite the code to assume it was true or false or zero.

I've seen

void func f(void) {
   int i;
   while(i) {
        i = j[i]; /* and other stuff I don't recall */
   }
}

I've seen the compiler completely remove that while(i) loop in optimization.

Keen: The case against upstream packaging

Posted Jun 20, 2016 19:05 UTC (Mon) by bronson (subscriber, #4806) [Link] (12 responses)

Pure hogwash. No conforming compiler would ever remove that loop. Sure, if it's unbelievably smart it might terminate the loop early -- before it walks into the uninitialized data -- but that's no big deal.

And, this had nothing to do with local variables. Not sure what your point is there.

Keen: The case against upstream packaging

Posted Jun 20, 2016 19:59 UTC (Mon) by zlynx (guest, #2285) [Link] (7 responses)

Perhaps we were thinking about different bugs.

But here's your pure hogwash:

$ cat optimize-out-local-variable.c
#include <stdio.h>

unsigned char c[256] = "abcdefghijklmpoqrstuvwxyz";

int f(void) {
	int i;
	while(i) {
		i = c[i];
	}
	return i;
}

int main() {
	int v = f();
	printf("v is %d\n", v);
	return 0;
}

$ gcc -fwhole-program -O3 -S optimize-out-local-variable.c
$ cat optimize-out-local-variable.s
	.file	"optimize-out-local-variable.c"
	.section	.rodata.str1.1,"aMS",@progbits,1
.LC0:
	.string	"v is %d\n"
	.section	.text.startup,"ax",@progbits
	.p2align 4,,15
	.globl	main
	.type	main, @function
main:
.LFB12:
	.cfi_startproc
	subq	$8, %rsp
	.cfi_def_cfa_offset 16
	xorl	%esi, %esi
	movl	$.LC0, %edi
	xorl	%eax, %eax
	call	printf
	xorl	%eax, %eax
	addq	$8, %rsp
	.cfi_def_cfa_offset 8
	ret
	.cfi_endproc
.LFE12:
	.size	main, .-main
	.section	.rodata
	.align 32
	.type	c, @object
	.size	c, 256
c:
	.string	"abcdefghijklmpoqrstuvwxyz"
	.zero	230
	.ident	"GCC: (GNU) 6.1.1 20160510 (Red Hat 6.1.1-2)"
	.section	.note.GNU-stack,"",@progbits

Keen: The case against upstream packaging

Posted Jun 20, 2016 20:45 UTC (Mon) by bronson (subscriber, #4806) [Link] (6 responses)

You didn't respond to what I was calling "pure hogwash" (that the OpenSSL entropy-adding loop could ever be optimized away) and you're still talking about local variables. As I said before, uninitialized locals have absolutely nothing to do with this bug.

Are you trying to turn this thread into a discussion of the C standard...? Still confused.

Keen: The case against upstream packaging

Posted Jun 20, 2016 20:57 UTC (Mon) by zlynx (guest, #2285) [Link] (5 responses)

Okay ... please just forget it because we're obviously not communicating. I thought you meant something else originally and you aren't even reading what I write.

Keen: The case against upstream packaging

Posted Jun 20, 2016 22:20 UTC (Mon) by bronson (subscriber, #4806) [Link] (4 responses)

No, I read and tried to reply to everything you wrote.

I'm still trying to understand why you're bringing up uninitialized locals since, while you're 100% correct, that doesn't have anything to do with this bug. In the entropy-adding loop, all locals were properly initialized.

I'm also trying to understand why you think a compiler could optimize away the entropy-adding loop. C compilers can be strange, but omitting the entire loop just because it walked into uninitialized data at the end of the buffer would be crazy.

Little help here?

Keen: The case against upstream packaging

Posted Jun 21, 2016 8:16 UTC (Tue) by paulj (subscriber, #341) [Link] (3 responses)

It was still an uninitialised variable on the stack - a local of a calling function. And the OpenSSL RNG was still horrifically broken - its entropy was largely dependent on uninitialised stack variables + plus the PID (and the PID being mixed in masked the problem). There's nothing in the C standard that guarantees that stack variables:

- Will not be the exact same in run after run

- Will not be the exact same across different machines

- Will have meaningful amounts of entropy.

Even if such code happens to provide some entropy in practice on real machines, that's still fragile. If and when the compiler does IPO and figures out that the buffer argument is uninitialised, as zlynx points out, the compiler can assume the buffer is whatever constant it wants (e.g.).

It's easy to criticise the Debian maintainer, and he should probably have been a bit more careful. However, he did ask on the openssl-dev list and even got an answer from an @openssl.org person to say those lines should be removed. However, the OpenSSL code was inherently buggy and fragile.

Keen: The case against upstream packaging

Posted Jun 21, 2016 14:22 UTC (Tue) by bronson (subscriber, #4806) [Link] (2 responses)

I agree, code like that is flat out inexcusable. But any standards-compliant compiler will compile it correctly. Although it made OpenSSL inherently fragile, I don't think it's reasonable to call it a bug. It only confused humans, not computers. (if potentially confusing code is a bug, then it's time to panic about a lot more bugs in the kernel!)

I'm not quite sure what you mean when you say the OpenSSL juicer was largely dependent on stack variables + pid... Do you mean the broken one? They removed the loop entirely -- the broken was only dependent on the pid.

Before the change, of course, it was largely dependent on /dev/random. It only mixed in uninitialized data after it collected sufficient random data. No matter how smart the compiler is, it's still obligated to process the random data. After that, who cares if it mixes in any uninitialized data or not?

Keen: The case against upstream packaging

Posted Jun 21, 2016 14:37 UTC (Tue) by paulj (subscriber, #341) [Link]

Ah yes, sorry, the beginning of the buffer still had /dev/random data.

Letting the RNG go on over uninited data was still wrong though. At best, there's nothing you can assume about the contents of that data. At worst, it's under the control of an attacker - see DJBs arguments on why that's a bad idea. Also, he notes that if you must mix-in lower quality sources, you should mix in those /before/ the higher quality ones.

The underlying mixing function was fine, of course, and I agree it's unlikely a compiler would leave out the trawl over uninited memory. However, with the right amount of whole-programme optimisation, inlining and inter-procedure optimisation, it is still /possible/ that inlined versions could have the mix optimised out. And who knows, maybe one day tool chains will optimise /that/ aggressively by default.

The "fix" in Debian was definitely wrong. However, really, the deliberate passing in of uninited stack data was wrong too.

Keen: The case against upstream packaging

Posted Jun 22, 2016 14:02 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> It only confused humans, not computers.

Who do you think code is written for? While not a bug, it still might be something to fix.

Keen: The case against upstream packaging

Posted Jun 20, 2016 20:09 UTC (Mon) by madscientist (subscriber, #16861) [Link] (3 responses)

I don't agree with your aggressive stance on this. Clearly the behavior is undefined, according to the C standard, so the question of what a "conforming compiler" would do is not helpful: a compiler could do pretty much anything here and still be conforming. It wouldn't have to be particularly smart, either, IMO. If you write a loop "while(0) ..." then almost every (optimizing) compiler will throw out the loop so I see nothing magical or ultra-intelligent about doing the same here if the compiler "knows" that the value of i is 0. If the value of i is undefined then the compiler can just assume it's anything.

Keen: The case against upstream packaging

Posted Jun 20, 2016 23:25 UTC (Mon) by bronson (subscriber, #4806) [Link] (2 responses)

Except the value of i is always defined: it's the size of the buffer.

The issue was that the buffer had valid data at the beginning, and uninitialized data at the end. Instead of just looping over the good data, the OpenSSL devs thought "eh, more bits can't hurt" and just looped over everything.

While that's an extremely bad idea (like, horrifyingly punishingly bad), I think a conforming C90 compiler is still obligated to execute the entire loop. C99 allows more leeway to ignore indeterminate values but, near as I can tell, the compiler must still assume the uninitialized portion is still filled with any arbitrary sequence of 0-255.

There's an interesting discussion here on uninitialized characters, seems to support this: http://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_338.htm I haven't looked at C11, maybe this discussion changed things?

Yes, I regret saying "hogwash." I was just trying to place my stake in the sand: despite what the OP said, conforming compilers may not optimize this loop away. That way, if I'm wrong, somebody can slap me around with facts. :)

Keen: The case against upstream packaging

Posted Jun 21, 2016 13:09 UTC (Tue) by madscientist (subscriber, #16861) [Link] (1 responses)

I think you're missing the trees for the forest, to paraphrase a saying; the only interesting bits of the original example for the purposes of this discussion are:

void func f(void) {
   int i;     // here i is not initialized
   while(i) { // here i is read
      // whatever stuff
   }
}

The first time through the loop, i is uninitialized and since it could have register storage class, this is actually undefined behavior according to the C standard.

Keen: The case against upstream packaging

Posted Jun 21, 2016 14:00 UTC (Tue) by bronson (subscriber, #4806) [Link]

OH, I get it. I was too hasty in my reply. I wish I could go back and edit it to say:

> No conforming compiler would ever remove *OpenSSL's entropy-adding* loop.

Since I didn't specify my subject explicitly ("that loop"), people will think I'm talking about the most recent loop they saw (zlynx's pseusdocode), rather than the loop under discussion (OpenSSL's entropy-adding loop).

For the record, I agree 100% with everything you and zlynx said about local variables and optimization. I ignored it, though, because local variables nothing have to do with the OpenSSL bug. The OpenSSL loop would never be optimized away, and the valgrind warning is just a warning.

My apologies, thanks for persisting.

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:35 UTC (Fri) by ceplm (subscriber, #41334) [Link] (5 responses)

We are not talking about unaware upstream developers, but about those who intentionally do stupid things (e.g., bundling security libraries, if not directly including spyware) and Snap/Flatpak conveniently remove one level of third party who could control what they do. And if you think, that upstream developers never do anything stupid, you have never maintained packages for normal distribution.

Keen: The case against upstream packaging

Posted Jun 17, 2016 19:46 UTC (Fri) by drag (guest, #31333) [Link] (4 responses)

Going 'flatpak' or 'snappy' does not eliminate the possibility that a person could audit the packages or recreate them to validate the contents. Certainly I would consider it a significant bug if somebody can't recreate a package from source.

And the point of the above post isn't that app developers don't do stupid things.

The point is saying that if you are pulling signed packages from upstream you can know those packages have not been modified by verifying the signature.

If I pull from upstream I have to trust that the developer's workstation is secure, their build system is secure, and that they have exclusive control over their signing keys.

If I pull from distro-built packages then I have to trust the security of the developer's system, their keys, and then also the security of every developer and all systems and all keys that are used as part of the distribution build system.

This is one of the few advantages that Google App store for Android has over traditional Linux systems. The ISV/upstream people control the signing keys for the packages they ship, not Google. That is if I understand things correctly.

Keen: The case against upstream packaging

Posted Jun 17, 2016 20:35 UTC (Fri) by ceplm (subscriber, #41334) [Link] (3 responses)

Well, this goes certainly in both ways. How is every upstream developer’s system secured? Is it really better secured than the security of the build systems maintained by Fedora/Debian et al. I wonder. I know that the Fedora systems are fiendishly secured against any intrusion or inclusion of something nasty.

Keen: The case against upstream packaging

Posted Jun 17, 2016 23:58 UTC (Fri) by HelloWorld (guest, #56129) [Link] (2 responses)

> Well, this goes certainly in both ways. How is every upstream developer’s system secured?
You're relying on the upstream developer's system's security anyway. If it gets compromised, an attacker will be able to publish a manipulated source tarball signed with the key of the upstream developer, and the packager will presumably package that – after all, it has the correct signature.

Keen: The case against upstream packaging

Posted Jun 18, 2016 1:01 UTC (Sat) by andresfreund (subscriber, #69562) [Link] (1 responses)

It's a heck of a lot easier to detect that in source, than in binary format.

Keen: The case against upstream packaging

Posted Jun 23, 2016 13:39 UTC (Thu) by HelloWorld (guest, #56129) [Link]

We shouldn't delude ourselves into assuming somebody actually tries to detect malicious new “features” every time a new version of some free software package is released.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:41 UTC (Wed) by flussence (guest, #85566) [Link] (4 responses)

The other side of this is the maintainer can sometimes be evil too.

Debian poisoning the well and leaving Ubuntu users exposed to hundreds of unfixed ffmpeg CVEs for years springs to mind...

Keen: The case against upstream packaging

Posted Jun 16, 2016 1:15 UTC (Thu) by misc (guest, #73730) [Link]

Did Debian prevented Ubuntu from fixing the CVEs, or was no one interested into doing the work in the Ubuntu community ?

Keen: The case against upstream packaging

Posted Jun 16, 2016 3:25 UTC (Thu) by NightMonkey (subscriber, #23051) [Link] (2 responses)

Do you have a source for that story?

Keen: The case against upstream packaging

Posted Jun 16, 2016 12:34 UTC (Thu) by viiru (subscriber, #53129) [Link]

This seems like a very creative reading of the ffmpeg vs libav split in which the Debian ffmpeg package maintainer was on the libav side and simply followed that side after the split.

Keen: The case against upstream packaging

Posted Jun 16, 2016 17:15 UTC (Thu) by flussence (guest, #85566) [Link]

None that I'd care to spend hours dredging up, but here's some pointers off the top of my head: Google threw their resources at fixing ffmpeg because that's what they use, and libav security alerts for years-old CVEs have been a regular fixture on this site for almost as long as the fork's existed.

Keen: The case against upstream packaging

Posted Jun 15, 2016 21:53 UTC (Wed) by drag (guest, #31333) [Link] (18 responses)

> The promise: Phone app stores came out of nowhere and suddenly had millions of developers appear out of the woodwork overnight! If we make our distribution more like phone app stores then we get millions of application developers!

This is a strawman. Who believes this? Who thinks that Linux app store = millions of developers?

The problem with distributions is one of scalability.

Say you have package X. Package X is critical and required the combined efforts of 3 people working to maintain and audit it's packaging for any distribution. If you have a 12 different distributions working in isolation then you need no less then 36 people to maintain this package if you want it to reach end users.

Conversely if you have 'universal packages' then 3 people can maintain X and still have the same level of high quality packaging. That frees up the effort of of maintaining 10 packages with the same level of scrutiny.

Effectively you multiplied the efficiency of packaging software by 1100% by moving to a distributed model of packaging over individual centralized services.

If every piece of 'Good' software written for Linux was packaged and audited properly by every distribution then the argument against decentralized packaging approaches would be sound. There would be nothing to be gained by increasing efficiency. However there is a huge amount of software that is 'invisible' to users because distributions lack the manpower necessary to repackage every piece of software under the sun.

> Low-quality bugs reported to the distribution's tracker often becomes a good bug when the maintainer reports it upstream. Without distros the ISV would have to carry the full weight of support.

You don't need to have distros packaging software to have them run bug reports and work with developers to resolve issues. These are related, but not interdependent tasks.

If distributions want to take on the task of helping ISVs package then I doubt anybody would object to that either. Suse, for example, features some really nice build infrastructure.

Beyond that ISVs can join larger organizations like Apache or GNU to help take much of the administrative burden of running a successful open source project. Debian could, for example, provide services for ISVs and give them 'Debian social contract certification' and provide means for users to vet ISVs packages against Debian's standards. They can provide resources, tools, support, auditing, and do everything they currently do just without having to duplicate the work that every other distribution does.

> The promise: Sandboxing makes you immune to bad ISVs.

Strawman. I doubt anybody can say that containers or sandboxing is a 100% foolproof anymore then it can be claimed that distributions packaging their own software will protect users 100% from problems. They are both approaches that are used to mitigate potential issues.

> The promise: By cutting out the maintainer middle-man, security fixes can be pushed out to users faster.

This is less of a issue who is doing the packaging and more to do with how the packaging works. Security fixes have inherent risks associated with them. Security fixes break things and require a lot of effort and vetting before they can be shipped.

Current packaging technology as it is cannot deal with situations were bad security fixes need to be rolled back. Somethings like Nix address this, but quick and safe rollbacks of updates is not easily done with things like dnf or apt-get.

I don't know how well 'snappy' can deal with this, but I know that Flatpak with it's OStree runtimes can deal with this. If updates are atomic and rollbacks effortless then people can ship out emergency fixes and apply them much quicker then otherwise is possible.

> The promise: A universal package means that ISVs don't need be aware of every distro and its unique quirks.

> The reality: They don't need to and never have. Put out the sources with a makefile.

This is just false. Distribution differences filter up to both application developers AND end users. It is not a burden born entirely by distribution maintainers.

> The promise: App Stores are a re-spin of the traditional distribution repository. One central location, easy search, easy install, automatic updates. Its practically the same thing.

> Distributions provide extensive curation that you can't get without an army of volunteers.

Distributed packaging schemes promise to more efficiently use the time of the 'army of volunteers' so that packaging is done much more effectively with wider amount of software for a wider audience. Distributions individually packaging their own software is not necessary to have distributions audit and assist with the packaging of software. Nor does it eliminate their ability to give their 'badge of approval' to ISV packaging that they approve and support.

Debian, for example, can maintain their own 'app store' based on the software they have audited and comforms to the 'Debian Social Contract'. Just because they download binary packages instead of tarballs doesn't mean that they are powerless to advise users.

One thing beyond what is mentioned in the article is the benefit of upstream signing packages.

This reduces the 'surface of attack' for the open source community to the upstream project. If I have my packages signed by Fedora then I need to trust that Fedora is secure, that their build systems are secure, and that the maintainers are all honest, in addition to trusting that upstream is not only secure but are being honest. I have no reason to doubt Fedora, but they have had security problems in the past as had all major distributions. The chances that package maintainers can catch a clever back door slipped into a upstream project is extremely small. However if package is signed by upstream then that cuts out a huge amount of potential things that can go wrong.

Keen: The case against upstream packaging

Posted Jun 15, 2016 22:09 UTC (Wed) by HenrikH (subscriber, #31152) [Link] (5 responses)

>Current packaging technology as it is cannot deal with situations were bad security fixes need to be rolled back. Somethings like Nix address this, but quick and safe rollbacks of updates is not easily done with things like dnf or apt-get.

How is this not easily done? I see this happening from time to time with i.e Ubuntu kernel packages where they have to roll back some security patch that created unwanted side effects, i.e linux-image-2.6.12-ubuntu11 is released and then the patch is removed and linux-image.2.6.12-ubuntu12 is released. Works equally well with RPM where you simple increase the "%define release" in the spec file.

Keen: The case against upstream packaging

Posted Jun 16, 2016 1:53 UTC (Thu) by drag (guest, #31333) [Link] (4 responses)

> Ubuntu kernel packages where they have to roll back some security patch that created unwanted side effects,

Kernel packages are a exception. They are designed to have multiple versions installed in case one version is bad. This is why every time somebody mentions something about kernel updates they tell you to always make sure to have 2 or three versions installed before upgrading.

For a end user to 'roll back' you just reboot and select a older kernel at grub.

A typical update would be something like 'QT package update, recompiled packages depending on QT installed because of CVE foo'. Apt-get and friends don't have a easy way to back out of a change like that. There is a snapshotting plugin for yum or dnf, but I am not sure how well that actually works.

As a end user 'roll back' you have to find the updated packages, remove them, and then pin the older versions and install those. It's not a simple process. It's certain possible. But it's not fool proof and chances that something goes wrong is fairly high.

Even if the update is just one package it's no fun to uninstall openssl and reinstall a older one.

The stuff about distros providing a updated package or changing version numbers is nice, but it's not something you can depend on since it's out of your control (as a end user). If your production website goes down because of a update you are not going to wait around for the bug report to process in the distro and for them to release a new update to fix the old update. You are going to have to do a manual roll back instead of a installing a update patch to a security patch.

This is why the general approach is a 'wait and see' attitude to security updates and give it a few days before installing them, unless it is something of a very critical nature.

Compare this to the 'container' or 'bundled' approach were you can have, like the kernel, multiple versions of the same software available on the same machine at the same time. A 'roll back' consists of just shutting down the broken version and starting up the last known good one. It's much safer, faster, and more reliable. Thus less risk and less vetting needs to go into things before applying a critical update.

Keen: The case against upstream packaging

Posted Jun 16, 2016 10:41 UTC (Thu) by HenrikH (subscriber, #31152) [Link] (1 responses)

Ah ok, didn't understand that you where talking from an end-user point of view. Yes there I can see the problem, I think that the main problem is that DEB repositories can only hold one version at a time (at least that is what reprepro enforces). If more versions than just "current" could be privoded by the repositories then some form of rollback this package function could be added to apt-get.

Keen: The case against upstream packaging

Posted Jun 16, 2016 18:04 UTC (Thu) by drag (guest, #31333) [Link]

yeah, exactly. Traditional package management solutions are one-way. Things like NIX are designed to support rollbacks, but containers combined with good way to manage images can be completely 'transactional'.

Keen: The case against upstream packaging

Posted Jun 19, 2016 22:09 UTC (Sun) by lsl (subscriber, #86508) [Link] (1 responses)

> A typical update would be something like 'QT package update, recompiled packages depending on QT installed because of CVE foo'. Apt-get and friends don't have a easy way to back out of a change like that.

Sounds like exactly the kind of thing "{dnf,yum} history undo last" (s/last/$transaction_id/ if it was an older update) is intended to help with.

Keen: The case against upstream packaging

Posted Jun 20, 2016 3:39 UTC (Mon) by hummassa (guest, #307) [Link]

and apt-btrfs-snapper :D

Keen: The case against upstream packaging

Posted Jun 15, 2016 23:38 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

Who thinks that Linux app store = millions of developers?

It's also that if we look at the combined output of the millions of developers who contribute to the Apple app store, a large swathe of that output turns out to be stuff the Linux equivalent of which we neither need nor want.

Keen: The case against upstream packaging

Posted Jun 16, 2016 1:58 UTC (Thu) by drag (guest, #31333) [Link]

That is why you still need distributions to advise and curate.

Even though distributions are giving up control in that they no longer control what software is easily available for Linux it doesn't mean that are out of a job. People are still going to depend on their ability to advise, test, coordinate, and work with upstreams to do a good job. It's not like giving upstream the ability to make packages that is easily installed is all going to turn them into geniuses at integration and compatibility.

Keen: The case against upstream packaging

Posted Jun 16, 2016 8:55 UTC (Thu) by bustervill (guest, #85383) [Link] (5 responses)

> The problem with distributions is one of scalability.

I call BS on that.

The point in the article, which resonated the most with me, is the one about diversity.

Different distros have different views and values and that reflects on the way they package software. I personally can't stand the Debian's bureaucracy, especially when it comes to their "stable". But I understand that their organization with all the associated bureaucracy stem from certain values they uphold and there are people and organizations which benefit from those particular values.

So, if 12 distributions package a piece of software, you don't "fix" the problem by prepackaging it, universally, upstream. You still need 3 people to address each different way your software is prepared for use.

Scalability issues usually stem from upstream, not from the maintainers.
If you provide a sane build method with your software, the work and attention needed from a maintainer is significantly reduced. For example - most of the software from the GNOME project: - http://ftp.gnome.org/pub/GNOME/sources/ - builds reliably and in the same way.

From scalability point of view, an universal package would help only if you try to avoid the pain points of making your program buildable and auditable by others.

Keen: The case against upstream packaging

Posted Jun 16, 2016 18:31 UTC (Thu) by drag (guest, #31333) [Link] (4 responses)

Debian, Fedora, Ubuntu, et al All use the same software in the same manner. They use the same compilers, with similar compiler options, with the same kernels, and address the same concerns in similar manners. If you look at any high-level software they all do all the same thing. If you look at distribution releases that release about the same time they are more or less identical in the versions of software they ship.

If you copy and paste the firefox binary from Debian and run it in Fedora and it will probably 'just work'. May need some messing around with library symbolic links or LD_PATH or whatever, but you can make it work. In the past I've ran plenty of Ubuntu software in Fedora when it's not been available for Fedora. Sometimes you run into issues with compiler incompatibilities or ABI changes, especially with C++ stuff, but it's as much of a headache to deal with between releases of the same distro as it is between different distros. The point being is that the software itself is plenty compatible, but it's the packaging system that is complicating things.

For ISVs like Mozilla or Chrome or Steam... you'll find that they don't compile their software over and over and over baring major ABI incompatibilities. Instead they just compile one binary and then repackage it a over and over again.

That is not to say there are NOT differences between distributions, it is just that user-facing applications is rarely where those fundamental differences show up. Differences are more often the design of bureaucracies (is it a corporate model? democratic? meritocracy?), the goal of the distributions, release schedules, support schedules, ISV certifications, hardware certifications, support by major projects that are not yet (or are partially) incorporated into distributions, installers, network configurations, and so on and so forth.

For example if I want a open source alternative to Active Directory for managing large numbers of Linux systems then I will probably want FreeIPA. FreeIPA is best ran on CentOS, so that is what I'll use. If I want to do a bunch of Kubernetes stuff then I will look at Atomic or CoreOS. If I want to do a bunch of standalone docker or vagrant stuff then I will probably want to use Ubuntu 14.04 for now. If I want to look at using a OS for use in helping porting applications to docker containers then Ubuntu or Alpine is a good choice. If I work for somebody that contracts with the government then probably Redhat will be required. Oracle databases? Unbreakable Linux or RHEL. Large scale desktop deployment? Suse or Ubuntu is probably appropriate, etc etc.

But what about LibreOffice, KDE, Gnome, Emacs, Eclipse, or Steam? I want Gnome stuff to work identically across distributions; customizations are fine, but not at the expense of platform stability and compatibility. I want Eclipse to work the way that matches the documentation online. I want the versions that match the projects that I need to use. I can't depend on distribution time-based releases to coincide with the versions I may need to deploy to my users. I want the plugins developed by somebody on Ubuntu to work correctly for me on CentOS. I don't want nasty surprises or changes or deviations. In desktop applications I want them to work as intended by the desktop application developers. If I write a Python Flask web-application then I want it to work on any platform that it gets deployed on the same way that it worked when I was testing it. These sorts of things.

Distributions offer awesome levels of 'value-added'-ness, but people also need a higher degree of consistency in application behavior and deployment. I think they can achieve both with less effort.

Keen: The case against upstream packaging

Posted Jun 17, 2016 6:54 UTC (Fri) by jospoortvliet (guest, #33164) [Link]

Indeed, most of the differences between distributions are minor, nuisances, irrelevant, arbitrary or simply not there. Serious benefits and real differences are very rare. If half the distributions merged we'd not lose anything of technical value. Technical I say for a reason as tools and philosophy are valuable too in their way but a serious consolidation and more collaboration and less NIH would hugely benefit the Linux ecosystem. Instead, we get Snappy et al. Well, I don't expect the ideal situation of consolidation and collaboration so it will have to do...

Keen: The case against upstream packaging

Posted Jun 17, 2016 11:44 UTC (Fri) by bustervill (guest, #85383) [Link] (2 responses)

The difference between Ubuntu packages and the equivalent Debian ones is massive. You can spend a week or two of your time here:
https://patches.ubuntu.com/

The Debian and Arch (for examle) philosophies about patching packages differ significantly. Arch aims to provide vanilla upstream software and applies patches very rarely, while Debian are *much* more inclined to modify the software.

Debian will patch to unofficially backport a rare bug or to remove a feature, which may be useful, but has controversial licensing compatibility. They will apply custom patches to the code to solve a real problem, while upstream is still pondering on the best approach. Debian have (and historically had many more) custom, distro-specific management/configuration frameworks, which are aimed to be the official Debian way, so every relevant package is patched to include integration with the said framework. I find it annoying, but it does bring consistency to the administration of areas in which, traditionally, there's a fragmentation.

I particularly remember, years ago, before the patents for the bytecode hinting which freetype was including had expired, Debian was patching freetype so it wouldn't include (somewhere still) patented software. It wasn't just freetype. Even with custom-built, unpatched freetype all the fonts looked horrible. Debian were patching various parts of the graphics stack. Their take on Firefox - Iceweasel - was built with particular options disabled or removed.

Because of such decisions, I wouldn't use Debian on my personal desktop machine. But if you are a company, which business is based on Linux software, or just uses it on their servers, Debian would be an excellent choice and will give you ridiculous stability and piece of mind on multiple platforms - not just x86/_64.

Ubuntu are patching, modifying and tuning vanilla code *even* more. Often in ways orthogonal to Debian's.

Fedora aim to include the newest stuff in every release. This often breaks compatibility or makes you do old things in a new way. They (RedHat) even write their own software to nudge things to follow their views.

All these decisions DO affect the end user.

Even if you don't directly use a common library, the missing/added features can be (and often are) noticed.
The performance profiles of common programs are different.
The mechanisms they use to arrange their webs of trust and the guarantees they can give you, differ.
The condition on which they support the same sets of packages differ.
Different is the attention they give to old software.

I don't believe all upstream vendors, even with their infinite wisdom, can satisfy all the use cases people care about. Mainly because there are users that care about stuff, about which upstream developers do not. Think about that, the next time you see a flame war about GNOME or systemd.

Vendor-provided universal package is just another preference and is certainly not obsoleting the work done by maintainers.

Keen: The case against upstream packaging

Posted Jun 17, 2016 14:47 UTC (Fri) by drag (guest, #31333) [Link]

> The difference between Ubuntu packages and the equivalent Debian ones is massive. You can spend a week or two of your time here:

Well I can spend a couple minutes and realize that the majority of the patches are '-debian maintainer', '+ubuntu maintaner' and '+notes'. Ones that are more substantial are ones that involve shuffling files around, removing symbolic links copying things.

The real differences come down to changes necessary to implement things like Unity.

> The Debian and Arch (for examle) philosophies about patching packages differ significantly. Arch aims to provide vanilla upstream software and applies patches very rarely, while Debian are *much* more inclined to modify the software.

If Debian feels a compelling need to modify packages there is nothing in snappy or flatpak that will preclude this activity. It is not necessary for Debian to package everything under the sun to be able to do, say, Iceweasel.

> All these decisions DO affect the end user.

My point in the previous post is that, yes, the Distros provide 'value-added' that do affect the end user.

If you can isolate the design choices that distributions make so that they no longer make software incompatible between distributions then this will enable more innovation and free up resources.

If Debian does put a lot of work into something like Iceweasel, why is it necessary for a Fedora user to format their hard drive and install a different OS in order to benefit from it?

Keen: The case against upstream packaging

Posted Jun 17, 2016 17:40 UTC (Fri) by ceplm (subscriber, #41334) [Link]

> Fedora aim to include the newest stuff in every release. [...] They (RedHat) even write their own software to nudge things to follow their views.

Just a nit: this implies that Fedora is same as Red Hat. I know members of the Fedora community who may cause a substantial injury if you held such opinion. It is really not true, and it has not been true for a long long time.

Keen: The case against upstream packaging

Posted Jun 17, 2016 15:50 UTC (Fri) by valhalla (guest, #56634) [Link] (2 responses)

> Say you have package X. Package X is critical and required the combined efforts of 3 people working to maintain and audit it's packaging for any distribution. If you have a 12 different distributions working in isolation then you need no less then 36 people to maintain this package if you want it to reach end users.

Not really, as the original article said packagers don't really work in isolation, they talk with each other and share patches.

Also, a huge number of distributions are actually derivative of some other distribution and take most of their packages from those, so they don't need any maintainer for it.

In some cases distributions even share maintainers: this is of course easier with derivative ones (e.g. I know of a lot of Ubuntu maintainers who are also Debian maintainers, and then apply a few ubuntu-specific patches to their packages)

Anyway, yes, having maintainers does require a significant workload, but is one that is easy to parallelize and distribute between multiple people.

Keen: The case against upstream packaging

Posted Jun 17, 2016 16:38 UTC (Fri) by drag (guest, #31333) [Link] (1 responses)

> Not really, as the original article said packagers don't really work in isolation, they talk with each other and share patches.

It was a over simplification made to illustrate a point, which is that the driving force behind this isn't 'App Store = Millions of Developers', but that it's a more efficient way to deliver what distributions deliver today with potentially much better results and significantly higher amounts of user freedom.

It would be nice that someday it would be easily possible for a regular desktop Linux user to able to do things like upgrade Libreoffice without having to perform major surgery on the OS (upgrade and/or reinstall) in a way that follows the 'best practice' of their Linux distro and is supportable.

It would be nice to be able to provide support to end users troubleshooting something with their application or desktop environment, as well, without the first question being 'What linux distro you are running? Need more info.'

Keen: The case against upstream packaging

Posted Jun 17, 2016 16:52 UTC (Fri) by raven667 (subscriber, #5198) [Link]

> App Store = Millions of Developers

To be fair though this isn't an unreasonable argument, with good simple packaging tools its simply more scalable if developers can cut production ready releases themselves without needing middlemen in each possible distro to do it for them. There are just millions more developers than distro maintainers and you can just look at the good and bad things about how app stores are implemented in the Apple and Google ecosystems to see how this plays out, it's not a theoretically discussion.

Keen: The case against upstream packaging

Posted Jun 18, 2016 12:32 UTC (Sat) by pboddie (guest, #50784) [Link]

Who thinks that Linux app store = millions of developers?

Maybe this guy does.

Keen: The case against upstream packaging

Posted Jun 19, 2016 5:08 UTC (Sun) by zblaxell (subscriber, #26385) [Link]

> to outright say 'This is not good enough for our users' and consequently punish an ISV by either patching out the offensive part or in extreme cases removing the software from the repositories

It never occurred to me that ISVs could equate QA and code contributions with punishment. That explains a lot.

I'm upstream and left unsaid is developers usually make lousy packagers.

Posted Jun 19, 2016 5:08 UTC (Sun) by mrdocs (guest, #21409) [Link] (2 responses)

Full on disclaimer: I work for SUSE. I am also upstream for Scribus, which is a desktop app. I also have been a project manager and engineer for some significant software projects.

My experience in 15 years of open source development is most developers are horrible packagers:

From the "Well my special app needs X, we cannot do it any other way'" argument to a complete lack of system admin knowledge e.g. setting correct permissions or suid on binaries, I think I have seen most of the insane things developers do in packaging.

Moreover, for programmers, packaging becomes a programming exercise. Like redefining every rpm macro from the systems defaults, because the developer thinks he or she is more clever than an enterprise distro ;-)

From a technical perspective, this is largely solved by OBS: http://openbuildservice.org/ OBS can build for almost any distro and for any architecture from tiny ARM5 to S390x mainframes and everything in between. Not just rpm, but .deb and Arch files as well. Even Windows binaries by abusing rpm.

I'm upstream and left unsaid is developers usually make lousy packagers.

Posted Jun 19, 2016 9:40 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

> My experience in 15 years of open source development is most developers are horrible packagers
And from _my_ experience when anybody says: "People are really bad at using the tool X, because they always keep making those silly mistakes" then it means that tool X is invariably total crap.

Sometimes it's an unavoidable crap, of course.

> From a technical perspective, this is largely solved by OBS: http://openbuildservice.org/ OBS can build for almost any distro and for any architecture from tiny ARM5 to S390x mainframes and everything in between. Not just rpm, but .deb and Arch files as well. Even Windows binaries by abusing rpm.
I wish people would stop spouting that nonsense.

No, building packages is NOT a problem. It's now trivially solved by running the build inside containers (Docker, LXC, Amazon AMI, whatever). The problem is making sure that all the dependencies are correct and then doing QA for all the builds.

On Windows and Mac OS you generally need to do QA for 3 latest versions of the OS - and that easily covers the period of about 5 years. I don't even want to compute the number of Linux variations during this period.

I'm upstream and left unsaid is developers usually make lousy packagers.

Posted Jun 19, 2016 11:31 UTC (Sun) by pboddie (guest, #50784) [Link]

From a technical perspective, this is largely solved by OBS: http://openbuildservice.org/ OBS can build for almost any distro and for any architecture from tiny ARM5 to S390x mainframes and everything in between. Not just rpm, but .deb and Arch files as well. Even Windows binaries by abusing rpm.

OBS certainly does generate different package formats but that's only part of the story, because one can easily make something that is technically compliant with the package layout but isn't compliant with distribution policy. I don't know whether OBS has improved in the past couple of years, but generating Debian packages with it was a very awkward experience, completely unnatural with respect to the Debian packaging documentation, and it didn't even properly support more recent policy features in a technical sense (in that I seem to recall it supporting only older source patching approaches). It was also a lot slower than pbuilder to use.

But I agree with the other things: developers and administrators have different expertise; it is only sensible that they learn from (and complement) each other. The trend towards "containerizing" everything tries to get around this - the "delicate geniuses" have an entire container for their own special software to play in - but it just enlarges the problem domain of "container administration" within systems administration.

Keen: The case against upstream packaging

Posted Jun 21, 2016 17:33 UTC (Tue) by gnu_lorien (subscriber, #44036) [Link] (3 responses)

Long ago I used to maintain a puzzle game for Linux. One of the major reasons I stopped working on it was because of how hard and time consuming it was to package. At the time I think it was Autopackage that was my preferred way around trying to make and maintain a whole lot of distribution packages. There were a lot of similar complaints about Autopackage as I'm seeing now and essentially all of it fell on deaf ears because distributions couldn't support my users the way I needed to.

It's interesting to see this same debate play out so many years later now that we have far more examples of "app stores" and other packaging formats. Nobody seems to be working to solve both problems at once:
- I want the latest version of some things quickly and easily
- I want stable versions of some things and for them to never change until they're broken

Distributions make it super difficult to get new versions and have a massive lag time from reading an announcement on a blog to being able to actually try out the new software. Direct distribution through "app stores" makes it easy to get the new version, but impossible to revert if I don't like it.

I would rather see distributions focus on satisfying the needs of their users that have led to the creation of Snap and AppImage rather than complaining about ideological problems and technical problems that discourage users.

Keen: The case against upstream packaging

Posted Jun 21, 2016 21:21 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> Direct distribution through "app stores" makes it easy to get the new version, but impossible to revert if I don't like it.
That's not necessarily true. Snaps/Flatpaks are actually easier to revert than regular DEB/RPM packages.

Keen: The case against upstream packaging

Posted Jun 26, 2016 22:02 UTC (Sun) by Wol (subscriber, #4433) [Link] (1 responses)

Because LSB and co always got things backwards. There is absolutely no point in the distro telling the app what's there.

If an APP could pass an "I require this" to the distro package manager - apt/yum/yast/whatever - and apps were tested against a couple of big distros, yes it would be annoying for the small distros to have to keep in line, but apps could then know that any "correctly" configured distro (for a random definition of "correct" :-) would be able to run the app.

(And software could be fixed/upgraded underneath it, with the package manager knowing what the proprietary apps needed!)

Cheers,
Wol

Keen: The case against upstream packaging

Posted Jul 11, 2016 10:40 UTC (Mon) by Klavs (guest, #10563) [Link]

I'm not an appdeveloper - but I run servers for people who do - and they have to keep all google's libs - for each android version they want to support - and build against each and everyone - as I understand it.

So even with android app-stores -you have to do many builds and test against each.

The biggest benefit seems to be that google gives app developers something unified to build against.

Couldn't you save a lot of "duplication of libraries" - by building such a "build-lib" setup - where distro/versions's could be added if one wants to support more.

If that turned out to become standard - distro's might even converge on what "jumps" in versions they wanted to take - making the variety a bit less - and making it easier for app-develepers on Linux ?