Distribution-friendly projects - Part 1

[Editor's note: This article, which looks at the interactions of software projects and distribution providers, will be presented in three parts.]

Introduction

In today's world most users of Linux don't build their system from scratch by downloading the sources of the applications and libraries they need and building them by hand. Most users will use one or more distributions (the ones that best suit their needs), and they'll stick with the packages provided by the distribution for as long as they can.

Power users may know how to get the software they want and build it so it runs, but the average user won't go around looking for software that is not readily available to them. The job of a distribution is, of course, to provide as much software as its users will need, sometimes changing the software so that it suits the needs of its users better.

The distribution's developers, the so-called downstream developers, have different responsibilities compared to the original software developers, the upstream developers. The former are responsible directly to their users, while the latter are usually more focused on implementing their software correctly for their own standards (which means for instance implementing a protocol exactly as described by the standard, or supporting a file format exactly as it should be).

Most of the time, these two objectives are compatible with one another, and users face an interface that hides the details of the implementation. Sometimes though there are user requests that upstream developers won't acknowledge, for instance: to parse a file that was written improperly by a commonly-used tool (maybe a proprietary tool that does not support free software). In these cases, some distributions tend to edit the source, creating a modified version for that particular distribution, with a different behaviour, interface, or what not.

It's because of cases like this, especially in the last few years, that there have been many arguments between original developers and distributions, which sometimes involved legal threats, forks or removal of software from distributions' repositories. It's not fun to watch these arguments going by, and sometimes it's all because of differences in opinion between the developers, or in how their experiences have affected their views.

Starting with the idea that everybody wants to have the software they wrote used, this article will try to explain what distributors want and why they ask the original developers to cooperate toward that goal. People who worked both as an upstream developer and as a downstream maintainer usually know what is being done with their code in a distribution and why. For people who have only seen one side, understanding of the needs or the reasons of the other side might be a very difficult task.

Technical and philosophical needs

The majority of the points where upstream and downstream have different views can be divided into technical and philosophical points. On the technical side, distributors need to make the software build on their system, without lots of workarounds, and it should follow the same behaviour as other software in their setup. On the philosophical side, they have needs relating to user requests and expectations. Users expect some consistency in how software looks and behaves on their system. Often, both of these kind of matters relate to the policy (written or unwritten) of that distributor.

While one might actually expect a philosophical debate between developers on formats and how to implement a protocol, it's difficult to understand how so many arguments are caused by different technical requests. Unfortunately even the technical needs are often different between upstream projects and distributions. The only way to accommodate both is to provide choices, something that more times than not is considered bad by the upstream developers, who do not want the complication of too many choices.

I sincerely doubt there will ever be a time when all the upstream developers and the downstream maintainers will be on the same page, but it is possible to at least try to understand what the other side wants, and see if it's possible to cover their needs, without regressing. Even if that means increasing the complexity a bit. It is true that most of today's tools, in every area, are more sophisticated and complex than their equivalent years ago (tens of years for computer tools, hundreds of years for other areas).

[This ends part 1 of this article. Part 2 will look at the technical needs of distributions and the upstream developers. Finally, part 3 will cover the philosophical concerns and present some conclusions. Stay tuned for part 2, which should air in two weeks.]

---

(Log in to post comments)

I actually merged legal-related issues together with philosophical issues. They are
philosophical 
after all, just more "country-wide philosophy" rather than project philosophy ;) What is (or
might not 
be) legal in one country can easily be legal in another. But just wait till Part 3 and you'll
read it all :)

Thanks a lot for starting this article series, I await the next part.

Personally, I'm opposed to the upstream vs. downstream picture painted here. For me, upstream
projects are shared spaces where parties interested in a particular project convene to develop
it to address their (potentially disparate needs), i.e. so-called downstream developers should
actually engage in upstream development to get their needs met (such as features, or the
flexibility to make the software act and look in the ways required by the distribution). The
changes affected upstream then subsequently trickle downstream.

Not doing it this way, i.e. supposing that upstream provides a certain raw kind of software
and last mile development (including, frequently, the addition of entire new features) should
happen in the distributions, usually leads to severe quality control problems. The downstream
developers may not be or usually are not familiar enough with the codebases they work on to
affect significant changes without causing regressions or defects, and by doing the work
downstream rather than in the shared upstream space, they actively evade peer-review from
those who are in the know. 

With the kernel in particular, we've seen that problem grow to unholy dimensions (as distro
kernels often were significant forks in the 2.4 era), but thankfully mostly addressed it by
now (except for the embedded space). In the userspace arena, it's still pretty bad, however,
with distributions grabbing and applying undone patches from left and right to differenciate
their offerings from their competitors, with consequently little interest to share or push
upstream. The result is that userspace applications shipped by distributions can often times
be in considerably worse shape than the original upstream releases.

Now, you're Gentoo guy, and I know that Gentoo generally has a relatively sane policy when it
comes to the extent of the deviations from upstream that it will ship, and that it will
generally try to push upstream whatever it comes up with first. Unfortunately, that's not true
for everybody in the market.

Kernel 2.4 (Or actually: kernels before the new 2.6 development model: around 2.6.5?) were
examples of packages that required large ammounts of patches. Their release cycles were very
long. The latest stable kernel simply didn't support latest hardware. A distribution had to
backport many drivers just to support the latest hardware. And if you backport drivers, you
eventually have to backport some other useful features.

Indeed the fix for that was an upstream change of the development model.


Pushing upstream is the right thing to do (in the long run) for the downstream package
maintainer, as maintaining a difference means more work for new upstream versions. And people
are lazy.

> Kernel 2.4 (Or actually: kernels before the new 2.6 development 
> model: around 2.6.5?) were examples of packages that required 
> large ammounts of patches.

Upstream definitely has to do its part, yes. If the upstream project doesn't understand itself
as a shared space for a variety of stakeholders, there's little a distribution can do - except
perhaps fork, and make a new, proper upstream to go to.

Well I don't know anyone who straddles the upstream/downstream fence more then Diego. But he
really is the exception. This article talks about how they have some fundamental differences.
You may not like the upstream and downstream split, but it exists.

With Amarok, sometimes I end up feeling like downstream is more concerned with packaging
standards and guidelines then quality control. "If it compiles and starts, ship it" kind of
attitude. This is mostly found in non-profit poweruser distros. Examples include a couple
years ago Debian removing (or marking it as "suggested") the Ruby dependency on Amarok,
despite it being required for Lyrics (and now scoring as well). Recently some other distro I
hadn't heard of put Amarok scripts into a different package that wasn't default, breaking
these features. So a user came in to #amarok and blamed Amarok for not handling this issue
better (an issue created by the distro). (Gentoo has a 'ruby' use flag, but I've never heard
of such complaints from Gentoo users, I guess because USE flags are more obvious then random
extra packages.)

However probably the most annoying thing (commercial or not) downstream has done was hire a
Indian programmer to work on media devices for Amarok 1.3 when Amarok 1.4 had totally
re-worked media devices and had been released for a while. But the programmer had apparently
been directed to just use the Amarok that they had packaged already. It was a waste of their
money and a waste of a potential development resources for us. This is kind of a separate
"suits not understanding the most basic parts of collaborative development", which is odd
since it's not so different from in-house development. I had a similar issue recently with
someone adding some nice features for Amarok 1.4 when we have it in a feature freeze; they
didn't communicate with us until after they had done much of the coding.