Debian proposes dropping most architectures

[Posted March 14, 2005 by corbet]

From:		Steve Langasek <vorlon-AT-debian.org>
To:		debian-devel-announce-AT-lists.debian.org
Subject:		Bits (Nybbles?) from the Vancouver release team meeting
Date:		Sun, 13 Mar 2005 20:45:09 -0800
Archive-link:		Article, Thread

Hello all,

As promised earlier on -project[0], after the release team/ftpmaster
team meeting-of-minds last weekend, we have some news to share with the
rest of the project.

First, the news for sarge.  As mentioned in the last release team
update[1], deploying the testing-security queues has been held up
pending some infrastructure enhancements, without which
ftp-master.debian.org cannot handle the load of the added wanna-build
queues for testing-security.  This week, Andreas Barth and Ryan Murray
have been applying the finishing touches to allow the needed upgrade of
ftp-master.debian.org's openssh to a version that supports connection
caching, which is needed before the current ftp-master host will scale
to handle the addition of testing-security queues.  Once this happens,
the testing-security configuration should itself be completed for all
architectures in quick succession, with the result that testing-security
and testing-proposed-updates will be fully operational in the space of
two weeks.

This means that we are now (at long last) in the final stretch for
sarge's release, and we will be freezing soon once we know that d-i RC3
is golden.  While this does not yet give us a timeline with fixed dates
for the freeze and the release, this is noteworthy enough in itself that
we wanted to share the news immediately.

This also means that it's time again to ask maintainers to cut back on
uploads of new upstream versions of software, *particularly* of
libraries.  This hasn't been mentioned in recent release team updates,
since it's not very realistic to insist on no new upstream versions for
six months without a complete freeze; but as we get close to the freeze
it's particularly important to limit churn of library packages, since
they tend to delay packages that depend on them -- as has bitten us
several times recently.  If you believe a library needs updating for 
sarge, please talk to the release team (debian-release@lists.debian.org)
before uploading.


The much larger consequence of this meeting, however, has been the
crafting of a prospective release plan for etch.  The release team and
the ftpmasters are mutually agreed that it is not sustainable to
continue making coordinated releases for as many architectures as sarge
currently contains, let alone for as many new proposed architectures as
are waiting in the wings.  The reality is that keeping eleven
architectures in a releasable state has been a major source of work for
the release team, the d-i team, and the kernel team over the past year;
not to mention the time spent by the DSA/buildd admins and the security
team.  It's also not clear how much benefit there is from doing stable
releases for all of these architectures, because they aren't necessarily
useful to the communities surrounding those ports.

Therefore, we're planning on not releasing most of the minor architectures
starting with etch.  They will be released with sarge, with all that
implies (including security support until sarge is archived), but they
would no longer be included in testing.

This is a very large step, and while we've discussed it fairly thoroughly
and think we've got most of the bugs worked out, we'd appreciate hearing
any comments you might have.

This change has superseded the previous SCC (second-class citizen
architecture) plan that had already been proposed to reduce the amount of
data Debian mirrors are required to carry; prior to the release of sarge,
the ftpmasters plan to bring scc.debian.org on-line and begin making
non-release-candidate architectures available from scc.debian.org for
unstable.

Note that this plan makes no changes to the set of supported release
architectures for sarge, but will take effect for testing and unstable
immediately after sarge's release with the result that testing will
contain a greatly reduced set of architectures, according to the
following objective criteria:

- it must first be part of (or at the very least, meet the criteria for)
  scc.debian.org (see below)

- the release architecture must be publicly available to buy new

- the release architecture must have N+1 buildds where N is the number
  required to keep up with the volume of uploaded packages

- the value of N above must not be > 2

- the release architecture must have successfully compiled 98% of the
  archive's source (excluding architecture-specific packages)

- the release architecture must have a working, tested installer

- the Security Team must be willing to provide long-term support for
  the architecture

- the Debian System Administrators (DSA) must be willing to support
  debian.org machine(s) of that architecture

- the Release Team can veto the architecture's inclusion if they have
  overwhelming concerns regarding the architecture's impact on the
  release quality or the release cycle length

- there must be a developer-accessible debian.org machine for the
  architecture.

We project that applying these rules for etch will reduce the set of
candidate architectures from 11 to approximately 4 (i386, powerpc, ia64
and amd64 -- which will be added after sarge's release when mirror space
is freed up by moving the other architectures to scc.debian.org).
This will drastically reduce the architecture coordination required in
testing, giving us a more limber release process and (it is hoped) a
much shorter release cycle on the order of 12-18 months.

Architectures that are no longer being considered for stable releases
are not going to be left out in the cold.  The SCC infrastructure is
intended as a long-term option for these other architectures, and the
ftpmasters also intend to provide porter teams with the option of
releasing periodic (or not-so-periodic) per-architecture snapshots of
unstable.

Also, since the original purpose of the SCC proposal was to reduce the size
of the archive that mirrors had to carry, the list of release candidate
architectures will be further split, with only the most popular ones
distributed via ftp.debian.org itself.  The criterion for being distributed
from ftp.debian.org (and its mirrors) is roughly:

- there must be a sufficient user base to justify inclusion on all
  mirrors, defined as 10% of downloads over a sampled set of mirrors

To be eligible for inclusion in the archive at all, even in the
(unstable-only) SCC archive, ftpmasters have specified the following
architecture requirements:

- the architecture must be freely usable (i.e., without NDAs)

- the architecture must be able to run a buildd 24/7 sustained
  (without crashing)

- the architecture must have an actual, running, working buildd

- the port must include basic unix functionality, e.g resolving
  DNS names and firewalling

- binary packages must be built from the unmodified Debian source
  (required, among other reasons, for license compliance)

- binaries for the proposed architecture must have been built and signed
  by official Debian developers

- the architecture must have successfully compiled 50% of the archive's
  source (excluding architecture-specific packages)

- 5 developers who will use or work on the port must send in
  signed requests for its addition

- the port must demonstrate that they have at least 50 users

These objective requirements would be applied both to the other eight
architectures being released with sarge that are not currently regarded
as candidates for release with etch, and also to any porter requests
asking for new architectures to be added to the archive.

This plan has been signed off on by:

  Steve Langasek (Release Manager)
  Colin Watson (Release Manager)
  Andreas Barth (Release Assistant)
  Joey Hess (Release Assistant)
  Frank Lichtenheld (Release Assistant)

  James Troup (ftpmaster)
  Ryan Murray (ftpmaster)
  Anthony Towns (ftpmaster)

The following people in Debian leadership roles have also expressed their
support:

  Andreas Schuldei (DPL candidate)
  Angus Lees (DPL candidate)
  Branden Robinson (DPL candidate)
  Jonathan Walther (DPL candidate)

Again, if you've got any questions, comments or concerns, please raise them
on -devel so we can take them into account before putting this plan into
effect.


Further plans for etch
----------------------

After the release of sarge, the release team will stop tracking the
day-to-day status of RC bugs in testing for a while.  NMUs for broken
packages will be minimal; instead, our RC bug handling will mostly be
limited to the usual aggressive removal of broken packages from testing.
This will also be a key time for the QA team to focus on deeper quality
issues and methods for a change, instead of on individual
release-critical bugs.

Meanwhile, much of the release team's energy will be focused on
coordinating the many major changes that are sure to hit the archive
shortly after sarge's release.  We already know of a number of major
package changes coming up: gcc 4.0, python 2.4, xorg-x11 6.8.2, apt 0.6.
If you are planning any other transitions that will affect a lot of
packages, please let us know in advance.  We will need to complete the
larger transitions as fast as possible, to get testing back into a
nearly releasable state quickly again after the release.

It's also not too early to begin staging complex transitions for etch in
experimental, particularly those transitions that will take a while to
debug.

Post-sarge, we also know that with the new social contract, all
documentation needs to adhere to the DFSG.  The biggest impact of this
change is that documentation that is licensed only under the current
version of the GNU Free Documentation License (GFDL) needs to be made
available under a license that meets the requirements of the DFSG, or
else be moved to non-free.  Some maintainers have already opted to move
their GFDL documentation to non-free for sarge, but the vast remainder
will need to be dealt with soon after sarge's release to keep us on
track for etch.

Otherwise, it's business as usual for the sarge release.  Please bear in
mind previous release team updates [1][2] when preparing package uploads
to unstable; and if you have any questions, please ask the release team
before uploading.

Cheers,
-- 
Steve Langasek                                     [vorlon@debian.org]
Debian Release Team

[0] http://lists.debian.org/debian-project/2005/03/msg00015.html
[1] http://lists.debian.org/debian-devel-announce/2005/02/msg...
[2] http://lists.debian.org/debian-devel-announce/2004/11/msg...

(Log in to post comments)

Debian to drop most architectures ?

Posted Mar 14, 2005 20:09 UTC (Mon) by ballombe (subscriber, #9523) [Link]

This is not an official statement by the Debian project.

This is a prospective release plan for etch by the release team and the ftpmasters team.

To be an official statement, it would need to be endorsed by the DPL or voted by the Debian developers. None of that has happened.

It is far too soon to see whether this plan will be carried out, and until sarge is released the point is moot, imho.

Misleading title. . .

Posted Mar 14, 2005 20:42 UTC (Mon) by topher (guest, #2223) [Link]

This has a very misleading title.

Debian is not planning to, or even considering to, "drop" most architectures. What they're likely going to do is stop trying to produce a synchronized stable release for all architectures. Debian currently supports 11 architectures (for Sarge), and the fact is, it's too much. The time and effort required to get all of the architectures into a releasable state at the same time is one of the primary factors that keeps Debian releases as spread out as they are.

By selecting a handful of primary architectures (x86, PowerPC, IA64, and AMD64 (personally I think they should add Arm to it, also, and if needed (I don't know how many primaries they want at a maximum) move IA64 to secondary)), they greatly reduce this burden. Post-Sarge, they will do official stable releases for those architectures, in much the same way they do now. The secondary architectures will be allowed their own release schedule as determined by their port maintainers. No more holding up the entire stable release because (for example) the m68k architecture is having an issue.

Personally, as a Debian user, I think this is a good thing. I also think it will help move Debian towards a more regular (and attainable) release schedule.

Misleading title. . .

Posted Mar 15, 2005 0:25 UTC (Tue) by phython (subscriber, #3278) [Link]

Actually ARM is one of the architectures causing some build problems. So moving ARM down from a top tier arch is probably a good thing. Also, ARM needs a full binary break after sarge is released to get TLS and new tools

Misleading title. . .

Posted Mar 17, 2005 13:52 UTC (Thu) by wookey (subscriber, #5501) [Link]

It's true that there will be massive ABI breakage for etch to default to soft floating point (as there are only about two ARM CPUs with hardware floating point ever made so having this as the inefficient default for everything is pretty dumb). And whilst we are breaking everything there are a few other changes that are deemed to be a good idea.

Also although ARM is very widely used and lot of people base their work on debian's stuff or otherwise find it a useful resource (it's a big part of our business for example), the fraction of direct downloads from debian ftp mirrors is tiny: much less than 1%. So putting it on scc.debian.org makes sense to me. If embedded Debian ever reaches critical mass and millions of people start downloading debian-arm for their routers, then things might change and we could get bumped back up to mainline port status.

Debian to drop most architectures

Posted Mar 14, 2005 20:52 UTC (Mon) by alspnost (guest, #2763) [Link]

Well, tentative or not, if this goes ahead, it's a good thing. Let's leave the "support every architecture" thing to NetBSD, and stick to targeting the architectures that represent the overwhelming majority of the userbase.

I wonder whether Gentoo should do the same as well. I mean, Gentoo is a great, increasingly popular, but small Linux flavour, and you have to wonder how many people out there actually use it on MIPS, HPPA etc. If focusing on the "big 3" or "big 4" actually helped to use the resources better, and to make releases easier and quicker to achieve, then I'm all for it.

We've reached a situation where supporting the niches brings too much of a cost to the masses. Doubtless some people will see this as a great shame, but in the end, practicality is going to have to win out.

Debian to drop most architectures

Posted Mar 14, 2005 21:12 UTC (Mon) by ballombe (subscriber, #9523) [Link]

There is so many distribution for the 'masses', what benefit is there to have one more ? One the other hand, distributions for the less used architectures are scarse and Debian is a great resource here.

Debian to drop most architectures

Posted Mar 15, 2005 4:17 UTC (Tue) by yodermk (guest, #3803) [Link]

I guess the difference between Gentoo and Debian is that Gentoo is not "held up" for years while waiting for something to get fixed on an obscure architecture. The "stability flag" of each package (stable, testing, unstable) is separate for each architecture supported by Gentoo. So a package can be stable on x86 and testing on AMD64, or testing on x86 and unstable on IA64.

When a package is deemed "ready" for any given arch, it is moved to stable for that arch. As long as Gentoo has maintainers for any given architecture, especially if they're not likely to want to maintain more popular archs, it should be retained.

Gentoo's archs

Posted Mar 15, 2005 10:10 UTC (Tue) by Duncan (guest, #6647) [Link]

Very good points about the Gentoo archs. There's another to be made in
this context, however, that being that even the various main Gentoo archs
aren't "release locked" to each other. That is, each arch can release on
its own schedule. In practice, Gentoo x86 sets the main release schedule
and the other archs normally target the same schedule, but there's nothing
saying it /has/ to be that way. The individual archs can release as they
see fit -- it's only the benefit of the larger PR draw of the coordinated
releases that keeps most of the others (by choice) to the same schedule.

An example is my own arch, amd64. Where Gentoo main, that is x86, went
from four quarterly releases in 2004, 2004.0-2004.3, to a semi-annual
release cycle for 2005, 2005.0 and 2005.1, amd64 had originally planned to
stay with the four release schedule. As it so happens, a release-stopping
hiccup with 2005.0 (on amd64) has it delayed to only now getting close to
formal release (I believe there was a similar one for x86, but haven't
tracked it as closely so don't know its current status, and with a 6-month
window, it's still safely within window and will be for some time yet),
now, in mid-March, which means they'll likely only get three release
snapshots out this year, but anyway...

Of course, the other even BIGGER difference is the emphasis placed on
releases. With Gentoo, "releases" are fairly stable and well tested
snapshots of the living release tree taken at a specific time, with any
changes necessary as discovered during testing, with binary packages
available and designed to be convenient starting points for a new install.
Once a Gentoo installation is accomplished, the system is designed for
more or less constant updating as the packages become available and marked
either stable (arch) or unstable (~arch), as chosen by the local
installation admin. Other than occasional profile updates where somewhat
larger changes may be instituted (a switch from gcc-3.3 to 3.4, as the
default, or from a default 2.4 to 2.6 kernel and headers), if said changes
are considered disruptive on a normal install and therefore likely profile
masked, the system is designed to be kept up to date on a more or less
weekly basis, daily being possible for those that want it. Even profile
changes are generally less hectic than release upgrades on most binary
targeted distributions, where almost the entire system changes at once.
With Gentoo, once installed, there's really little reason to worry about
releases at all, because the updates are done on a routine basis, not as
part of a massive release update.

That is of course one of the reasons I like the Gentoo system.

Duncan

Gentoo's archs

Posted Mar 15, 2005 15:53 UTC (Tue) by jdv (subscriber, #712) [Link]

There's another to be made in this context, however, that being that even the various main Gentoo archs aren't "release locked" to each other. That is, each arch can release on its own schedule.

Imho, that is more or less what is now being proposed for debian. Well, except for the four or so most important archs, which remain locked.

Gentoo's archs

Posted Mar 15, 2005 18:14 UTC (Tue) by dlang (guest, #313) [Link]

check again, the secondary archs will get 'snapshot' releases, not full releases with security support.

there's a huge difference between the two.

Gentoo's archs

Posted Mar 15, 2005 21:25 UTC (Tue) by jdv (subscriber, #712) [Link]

Right, I misread -- I assumed that those snapshot releases would be more like real releases, but it would seem that is not true. Hopefully, a seperate community effort could pick up the pieces, though, and make a real stable distribution out of it, security support and all.

Article tweaked

Posted Mar 14, 2005 21:16 UTC (Mon) by corbet (editor, #1) [Link]

Just for the record: I've changed the title of the article and (slightly) the text to better reflect the fact that it refers to a proposal, not an adopted policy. My apologies for the first version, which wasn't so clear on that.