LWN: Comments on "Bassi: Dev v Ops"

Bassi: Dev v Ops

nix — Fri, 01 Sep 2017 15:10:04 +0000

If were were starting from scratch, I'm sure it would be split into a few more pieces than it is. But we're not, and backward compatibility is paramount here.

Bassi: Dev v Ops

Wol — Mon, 21 Aug 2017 16:52:34 +0000

Oh for the WordPerfect approach.

"This is a document. It's a DATA file". "This is a macro file. It's possibly dangerous. Do you want to run it?".

It was easy to do stuff like Word. You could easily use a macro to load or create a document and edit it. But you could NOT put macros inside a document.

Cheers,
Wol

Bassi: Dev v Ops

fknuckles — Mon, 21 Aug 2017 07:52:42 +0000

Comments are sorted by date, which means you will see the ones that apply to the most recent version.

Developers and users don't need to see the 6-month old comments. Developers get notified of reviews in the app store. If it contains feedback (such as your app crashes on startup on my {phone}), then from then, it is just a matter of whether or not they ahve the relevant hardware to test, or they can build and ship an updated version of their app with extra instrumentation so that the next crash will offer an opportunity for the user(s) to upload more contextual log data etc.

The system doesn't look elegant but it is quite effective.

Bassi: Dev v Ops

fknuckles — Mon, 21 Aug 2017 07:48:27 +0000

Ha ha ha. It's easy to see you're one of those types who think social media users are all idiots.

This is a shining example of the disconnect in expectation that devs (in Linux especially) tend to have. For one thing, you can't expect your users to know how to give you a proper bug report, and you should welcome all feedback, even if it's not actionable. How many times have you been to a website that was full of broken links, not because it's not maintained, but because the developer themselves never exercises that part of the website? It's the same with apps. If you have no way for users to give you feedback, or you have such high hurdles (because you're a bit arrogant and mis-informed about how much time your users have to spend on you), then you will simply get no feedback , and your website/app will be broke for a very long time indeed, with the attendant decay-feedback-loop that it's associated with it. Make it easy to give you feedback, even it's just "hey, this page/feature is broken". It's better for you to receive more feedback than you can deal with than to receive none at all.

If an app I'm using crashes and I go tell you about it, if you expect me to surrender considerable amounts of time and effort to report this to you "The way you want it (tm)", then you're terribly mis-informed. On every platform other than {Insert Linux Distro here}, I can simply uninstall your app and try the next one that might do what I want.

Developers try to put in all sorts of mechanisms to get you to give them feedback before you leave a negative review (and I personally make an effort to use these), but the app-store review is the ultimate feedback. If your app sucks, or your recent update broke stuff, I go and edit my rating and give it fewer stars. When you fix it, and if I'm still using it, I eventually update my rating accordingly.

I actively avoid even testing apps that have fewer than 3 stars on average.

Bassi: Dev v Ops

fknuckles — Mon, 21 Aug 2017 07:27:00 +0000

This theoretical (albeit speed is a practical concern for some of course) hasn't proven to be a stumbling block in Windows and Mac.

People don't download the same copy of their 300MB app every day, and the updates to the app don't have to be 300MB too. If you need an app, you don't refuse to use it because it's going to take 2 hours to download in your 5mbps DSL, and you don't refuse to use it because you have a 200GB monthly data cap. It's not app downloads that eat your data, it's youtube and netflix.

Bassi: Dev v Ops

Cyberax — Mon, 21 Aug 2017 05:18:06 +0000

Yep. The versioning hell and synchronization issues. glibc has a lot of problems because of the split libpthread, so many of them that musl libc utterly abandoned this practice.

Bassi: Dev v Ops

mathstuf — Mon, 21 Aug 2017 02:27:23 +0000

Is there a reason that glibc couldn't be split out into different shared libraries and the must-be-common bits under their own sonames?

Bassi: Dev v Ops

nix — Sun, 20 Aug 2017 21:18:38 +0000

> For the glibc, I think it could work too: glibc 2 and 3 could be installed in parallel, and higher-level libraries/components/apps are ported to glibc 3 layer by layer, from bottom to top. But it requires that every higher-level library is also parallel-installable, and that they bump their major version when porting to glibc 3. But it would probably take many years to port all the Linux userland libraries to glibc 3, we would realize that some low-level stuff are not that well maintained and that nobody works on those modules anymore.

That doesn't work because glibc provides shared data structures that are used by many consumers at once (a major example being the heap, and malloc()). All glibcs in a process's address space must therefore agree on the format of all such shared data structures, locking rules, etc. In practice this means that you can only have one soname of glibc linked into a process's address space at once (thus, a single copy of the library, which necessarily agrees with itself about all that stuff). This therefore means that soname bumps for widely-used libraries of this nature cannot be done incrementally, and are generally agonizing as a result.

Bassi: Dev v Ops

mcortese — Sun, 20 Aug 2017 09:17:41 +0000

is not a mystery, though, why it’s a dying model.

Well, it might not be a mystery, but I haven't found any explanation in the paper. Actually, I haven't even found any evidence that the current model is dying. Only that some other platforms use a different model. Is there a trend to switch away from a distro-based approach? The paper seems to assume so, but I fail to see the evidence.

Bassi: Dev v Ops

DOT — Sat, 19 Aug 2017 22:04:18 +0000

"If I use LibreOffice only to edit a few files I create, how can an attacker exploit any vulnerability in it? How many LibreOffice documents have you received that contained an exploit?"

I receive them via email daily. Granted, most of them are caught by my spam filter. But what if one makes it through and I get fooled into opening it?

Bassi: Dev v Ops

flussence — Sat, 19 Aug 2017 15:29:45 +0000

>This may be the most extraordinarily arrogant thing I've ever seen on lwn.

When it's framed as an isolated statement without context, it certainly does come across that way. Don't do that?

Bassi: Dev v Ops

NAR — Fri, 18 Aug 2017 14:00:55 +0000

If I use LibreOffice only to edit a few files I create, how can an attacker exploit any vulnerability in it? How many LibreOffice documents have you received that contained an exploit?

By the way, distributions only protect from known attacks. If you are really really really afraid of malicious code in cloned github repositories, you should use a throwaway virtual machine to open any files you've received anyway. Security is always a compromise. On one hand there's the extremely limited amount of software (and software combinations!) the distributions provide along with forced upgrades - on the other hand there's the perceived security and QA they provide.

well said

ssmith32 — Thu, 17 Aug 2017 21:10:25 +0000

While I mostly agree that the distro model is preferable, keep in mind that the "few percent" of apps where it makes sense to have developers bundle stuff directly to users are often the "critical" apps to the user.. for varying definitions of critical. So it's nice their building other options.

In other words, I agree with the general direction distros seem to be taking - keeping package management at the core, but also giving the option of something like flatpak for when it counts.

Bassi: Dev v Ops

jschrod — Thu, 17 Aug 2017 17:18:39 +0000

Since unpatched office suites are a major entrance door for exploits, and since you might use vim on arbitrary code you cloned from github, triggering actions in maybe-faulty plugins -- you should really evaluate your attitude.

I.e., I cannot follow your argument.

Well, maybe I really don't care about vim. But I do care that my Emacs is up-to-date cf. security issues. ;-)

Bassi: Dev v Ops

nybble41 — Thu, 17 Aug 2017 16:53:57 +0000

> xterm and gitk are probably not likely

Both xterm and gitk are expected to handle untrusted data. I know that xterm has had security issues in the past—"cat" the wrong "plain text" file with embedded control codes and it could overwrite arbitrary files owned by the user running the terminal (e.g. "\e]46;/path/to/file\a\e[?46h", which would set the log file name and start logging... if it weren't generally disabled at compile-time).[1] Note that this applies to the output of any command which copies from an untrusted file to the terminal without a filter, not just "cat".

The input for gitk can consist of arbitrary git repositories cloned from who-knows-where; the scope for an infection (though not the attack surface) is akin to a web browser.

[1] https://unix.stackexchange.com/a/15210

Bassi: Dev v Ops

pizza — Thu, 17 Aug 2017 15:24:32 +0000

The answer is: Any application (or library) that might accept or be passed input from an untrusted source.

xterm and gitk are probably not likely, but VLC, LibreOffice, browsers, email clients, and so forth? abso-effin-lutely!

> Probably less than dying from a brick falling on my head for a building and yet I don't wear hard hat when I go out to the street.

That's because there are laws about putting up protected pedestrian paths next to buildings that are likely to shed bricks.

Bassi: Dev v Ops

NAR — Thu, 17 Aug 2017 15:21:57 +0000

they realize it can't be rebuilt from sources anymore

It's 2017, not 1997 anymore. Most software are used by non-developers. They do not want to rebuild the software. They probably don't even know (or care) about the concept of building software.

the build happened on some developer's personal laptop that suffered a coffee-related death long ago

Bundling libraries does not exclude sane software development practices like using source control, automated builds, etc.

running your builds and test suites on all the kinds of hardware that people care about

I seriously doubt distributions can do this. Wasn't there an article here lately that Fedora can't be installed on a Mac, because they don't have the hardware? There may be a few packages where the distributions can do meaningful work, but in most cases they can only check that the software can overcome the obstacles the distributions themselves introduce.

Bassi: Dev v Ops

NAR — Thu, 17 Aug 2017 15:14:01 +0000

you must track all security issues of all dependecies that you ship

The thing is: for many (if not most) applications in many (if not most) installations it doesn't matter. Do I care if there's a "security" bug in xterm? In Gitk? In Vim? In VLC? In LibreOffice? Absolutely not. What are the chances that I actually get hacked because of a vulnerability in these? Probably less than dying from a brick falling on my head for a building and yet I don't wear hard hat when I go out to the street.

I want my browser as up to date ad possible, but I absolutely don't want to upgrade e.g. LibreOffice or Vim because the browser is updated - and this is where the distributions fail.

Bassi: Dev v Ops

jschrod — Thu, 17 Aug 2017 14:03:40 +0000

> Is it too much to ask to ship my own copies of the things my application depends on, easily?

As long as you are fully responsible for the security of your whole application - no.

That means that you must track all security issues of all dependecies that you ship, and that you provide timely application updates in case of any security updates in any of these dependencies.

I don't care about "efficiency". For me, your argument that this is not important is a straw man.

The case against bundling of dependencies is that almost none of the proponents explains how they'll handle security updates - or if they do so, tracking security issues of dependencies are not mentioned at all. See, I don't trust application developers that they do this tracking - but I trust distribution maintainers that each of them tracks the package they are responsible for.

Bassi: Dev v Ops

Cyberax — Thu, 17 Aug 2017 04:42:38 +0000

Developers can integrate with third-party advertisers or payment processors, this is not a problem. The biggest problem is that there's a HUGE entry barrier.

And I actually agree that it makes total sense for system-level software and the basic desktop environment, after all I don't really want to watch ads in "ls" command output.

The problem is that there's NO reliable way to distribute software in "classic" desktop Linux other than through distros.

Bassi: Dev v Ops

zblaxell — Thu, 17 Aug 2017 03:09:48 +0000

> > The main reason why there is no fart app in Debian is that so far nobody seems to have bothered to package and upload one.
> Duh. And that's the point of the article.

Well, that, and Debian doesn't collect and redistribute money from fart users (or fart advertisers) to attract fart developers and to fund fart reviewers and distribution infrastructure. Only a privileged few bother facing (justifiable) opposition from those who would not see their donations to the project squandered on digital flatulence.

Also, apt would explode given the number of packages in a non-trivial commercial app store (it barely keeps up with the growth of Debian now).

Both are solvable problems, if anyone cared enough. Some startup could just roll out an app store on top of straight Debian (maybe giving each app developer a unique URL and injecting files in /etc/apt/sources.list.d), and distribute all the fart apps they can find users to fetch.

Bassi: Dev v Ops

zblaxell — Thu, 17 Aug 2017 02:40:44 +0000

> True however quantitative differences between a "full distro" and "a little distro" could effect a crucial *qualitative* improvement

Indeed. A small team of competent developers can easily maintain a forked distro optimized for the needs of a single user. This is so easy that many individual users do it without realizing it (everyone who installs or customizes a few third-party packages effectively does this).

Packing an application binary with its dependencies is expanding slightly on this, to build a forked distro for users of a single application (possibly by picking a distro and copying bits of it into a package that can then be unpacked on another distro under different paths--i.e. the sort of thing LD_LIBRARY_PATH is designed for). This is much easier for the app developer than supporting either a full distro or integrating an application deeply into several different distros.

The security trade-off is non-trivial without a robust sandboxing mechanism in place, probably combined with routine eviction of non-updated apps from package repos/apps stores. We can learn some things from Apple here: keeping apps up to date is the app developer's problem, all the store has to do is define some technical standards and say "no" effectively (or "no, your app is built with an ancient version of libCVEfarm.so, and we won't distribute it any more until you upload a fixed version.").

Bassi: Dev v Ops

Cyberax — Wed, 16 Aug 2017 23:24:48 +0000

Right now distros are the only way to get the basic system working. The added value is that you don't have to care about thousands of small dependencies of a typical desktop.

However, this is a very slim advantage. CoreOS is becoming quite popular exactly because it removes much of the regular distro stuff.

Bassi: Dev v Ops

lsl — Wed, 16 Aug 2017 21:15:42 +0000

> In the Erlang world it is common that github repositories are specified in the rebar.config files, so the build process downloads the dependencies - of course, it's susceptible to bitrot. Bundling solves this issue.

Only if you simultaneously archive the set of source files and some records about the environment you used for that particular build. You know, like distributions do for every single package they build, if only to be able to point at the source corresponding to a particular binary package.

If you don't do that, bundling just the object code actually makes the problem worse. People might start to rely on the program just to get hit by the shit hammer months down the road when they realize it can't be rebuilt from sources anymore and no one really knows what it was built from anyway because the build happened on some developer's personal laptop that suffered a coffee-related death long ago.

If you have the resources to pull these things off (and others, like running your builds and test suites on all the kinds of hardware that people care about), then good for you. Many upstreams don't, despite producing useful software. That's where distributions tend to add a lot of value.

Bassi: Dev v Ops

niner — Wed, 16 Aug 2017 17:37:22 +0000

But that would be value added, which doesn't fit with NAR's comment that "Distributions don't add value nowadays".

Bassi: Dev v Ops

Cyberax — Wed, 16 Aug 2017 17:22:37 +0000

Because there are no other reasonable ways to get a basic system working?

Bassi: Dev v Ops

niner — Wed, 16 Aug 2017 12:25:55 +0000

If distributions add no value, why have I never seen a Linux user using anything but a distribution?

Bassi: Dev v Ops

NAR — Wed, 16 Aug 2017 11:14:12 +0000

I disagree very much. The upstream developer wants to create software for its users first and foremost, not some 3rd party entity. It should be easy for the end user to install and run the created software. "Here's the tarball, it's up to you to get those 30+ dependencies to be able to actually run it" is less than satisfactory - it's an insult to the user. The fact is that distributions have resources to package maybe about 0.1% of software out there. Distributions don't add value nowadays, they just keep this fantasy alive that "we can take care of your dependencies" - they can't.

autotools takes care of most of this for free

And what about Java? Python? JavaScript? Ruby? Does autotools find the dependencies for these languages? What if a dependency needs newer C compiler than the one included on the distribution (it happened to me)? In the Erlang world it is common that github repositories are specified in the rebar.config files, so the build process downloads the dependencies - of course, it's susceptible to bitrot. Bundling solves this issue.

Bassi: Dev v Ops

pizza — Wed, 16 Aug 2017 10:52:42 +0000

> However, the real job of the upstream software developer is to make their software easy to build by others.
> This may be the most extraordinarily arrogant thing I've ever seen on lwn.

Oh, come now. This barely registers on the scale.

While I personally disagree that "making their software easier to build by others" is the _most_ important thing an upstream developer can do, it's fairly high up there, because everyone benefits from more frictionless builds, including (and especially) the original developer, because it enables all manner of side denefits.

It's in everyone's interest -- upstream, downstream, and indirectly, end-users too, that the software be easily built. This reduces the burden on everyone -- upstream doesn't have to put as much work into building it themselves, downstream can re-build it effortlessly, and the end-user benefits as releases/builds can be made more frequent without undue burden.

A great example of this is Libreoffice (and its predecessor, go-OOo). They put a lot of effort into making the original OpenOffice codebase easier to build, which in turn allowed them to vastly speed up development cycles to the point where they generate full nightly (regression tested!) builds for every platform they support. Meanwhile, the complex original build process left ApacheOpenOffice unable to perform complete builds for several months, and even after they finally got their security fix release out a year after the code change was made, the underlying situation is no better, greatly raising the bar for would-be contributors.

Given the choice, why would anyone chose to contribute to AOO over LO, given the latter's vastly superior developer experience? And as an end-user, can anyone say with a straight face that AOO's approach has led to any sort of benefit?

So yes, you're correct in that an upstream developer's primary job is to make software that scratches their own itch, and F/OSS or not, they owe nothing more to anyone (I've said something similar here many times, including earlier in this thread). But making something easy for others to build (and thus, contribute to) greatly improves the developer experience, which surely improves the end-user experience if only because it leads to better software, delivered more rapidly. That is hardly an arrogant position to take.

> There is no way outside of the most remarkably toxic and self-centred mental state, that it could be reasonably be said to be to please distribution maintainers. And distribution maintainers who can't grasp that will, I guess, continue to be angry and baffled by the way they become less and less relevant.

Nine times out of ten, the stuff distribution maintainers ask for will, in the longer run, disproportionally reduce the support burden on the upstream author. Or are you seriously saying that shouldering all the stuff that distribution maintainers currently do is somehow less work?

Bassi: Dev v Ops

anselm — Wed, 16 Aug 2017 08:40:17 +0000

There are upstream developers with a “my way or the highway” attitude (DJB comes to mind) and there are upstream developers who go out of their way to make the life of distribution maintainers easier. Guess whose packages are generally better supported in distributions, and thereby get wider exposure. Hence, if you want your software to be widely used, accommodating distribution maintainers is probably not the worst strategy.

Bassi: Dev v Ops

rodgerd — Wed, 16 Aug 2017 05:16:16 +0000

> However, the real job of the upstream software developer is to make their software easy to build by others.

This may be the most extraordinarily arrogant thing I've ever seen on lwn.

The job of an upstream software developer could reasonably be said to make software that pleases them. It could reasonably be said to be to make software that pleases the users of that software.

There is no way outside of the most remarkably toxic and self-centred mental state, that it could be reasonably be said to be to please distribution maintainers. And distribution maintainers who can't grasp that will, I guess, continue to be angry and baffled by the way they become less and less relevant.

Bassi: Dev v Ops

lsl — Wed, 16 Aug 2017 05:06:58 +0000

Then link against glibc dynamically, It's stable anyway. You can still statically link all the other libraries you need (especially the huge ugly C++ things that break ABI every other week).

Bassi: Dev v Ops

zblaxell — Wed, 16 Aug 2017 03:39:54 +0000

My hosting provider is a for-profit entity which definitely wants you to download the 300MB version.

There are two costs being traded here: the cost of regression testing in the 270MB of dependencies multiplied by the number of supported (distribution, release) tuples, and the cost of pushing a third of a gigabyte to each user on a new install to reduce or eliminate the tuple-count cost factor. One of those costs is orders of magnitude larger than the other, and also costs things that aren't money, like end-user satisfaction, vendor reputation, maybe even consequential damages (not everyone gets to clickwrap those away).

If we have a million users, the bandwidth cost is $0.02 per user. Can we spend this money elsewhere to save money? To use system library dependencies, we need to hire people to verify the app on every Linux distribution (assuming these all have differences in build toolchain, source patches, enabled services, etc). For a distro with 1000 users of our app, that works out to a marginal QA budget of $18.00, not even one hour of one QA's salary, to solve the set of problems occurring on that distro that could have been solved by simply *not* making the download 90% smaller. No puny human QA is going to reliably solve GTK dependencies in an hour, so the most realistic alternatives to making the download 10x larger are to say "join the other 999,000 users using the distro we developed it for" or "your business isn't important to us."

Bundling has 99 problems, but bandwidth cost ain't one.

Bassi: Dev v Ops

rahulsundaram — Tue, 15 Aug 2017 22:06:20 +0000

> It's stupid, these days, to not use cmake

I doubt that perspective is anywhere near universal. Lots and lots of major projects are moving to meson these days, mostly from autotools and not cmake but we won't have a single leader in this space anytime soon.

Bassi: Dev v Ops

vasvir — Tue, 15 Aug 2017 19:38:21 +0000

I totally buy it.

Here is an entertaining thought.

If the convention followed by Linux kernel was followed by library authors Debian could drop the stable/testing/unstable branches and adopt a rolling release strategy.

After all the kernel does not have stable/development branches it is by all intent and purposes a rolling thing.

Yes I know there are older kernel support but it isn't a dev thing like it was 1.2/1.3 or 2.0/2.1. I think that with version 2.4 kernel adopted a rolling release strategy.

just a thought...

Bassi: Dev v Ops

halla — Tue, 15 Aug 2017 19:14:31 +0000

Autotools is not ideal. Autotools is outdated: pretty much everything relevant uses cmake these days, which is miles better, if only because, if well-used, it gives you a full list of missing dependencies. It's stupid, these days, to not use cmake. Don't use autotools, don't use meson, don't use scons, don't use whatever: just hold your nose, if needed, and learn to use cmake. Don't reinvent, work with it.

Anyway, of course, my project is easy to build for distributions; but distributions don't release often enough, backport repos are only a sticking plaster, especially when talking about applications for non-technical people. But there's something that's hardly been discussed in this thread: if get a bug for distribution X, I have to first figure out which exact versions of dependencies it has, which flags it thinks it needs to build with, what other Qt QImageIO plugins are present (like deepin's) and so on: in fact, it's come to a point I cannot give support for Krita on certain distributions, like gentoo or arch.

It's also come to a point where I hesitate to recommend to people who want to use Krita that they'd give Linux a try. I sometimes feel we are trying with fewer people than ever, to keep more different setups/configurations/distributions/desktops than ever spinning in the air.

Bassi: Dev v Ops

jdulaney — Tue, 15 Aug 2017 18:49:42 +0000

This and so much this.

Bassi: Dev v Ops

jdulaney — Tue, 15 Aug 2017 18:47:51 +0000

I might could help you get an rpm together.

Bassi: Dev v Ops

davexunit — Tue, 15 Aug 2017 18:20:09 +0000

There is a common misunderstanding that it's the upstream software developer's job to make builds for every distro. However, the real job of the upstream software developer is to make their software easy to build by others. If it's easy to build, it's easy for a distro dev to distribute your software. Making it easy build means following some best practices like using a well understood build system (autotools ideally), not bundling your dependencies (or making it *very* easy to use non-bundled versions), not making assumptions about the target system (autotools takes care of most of this for free), etc.

Bassi: Dev v Ops

davecb — Tue, 15 Aug 2017 16:31:00 +0000

Back in the Sun days, "good practices" described things we were sorta sure were good, and we couldn't see a downside to.

Conversely, "best practices" was a phrase that really meant "if you don't do this, you're an <expletive deleted/>, and when it breaks, we'll look real hard for a way to prove you voided your warranty"