Building the whole Debian archive with GCC 4.1: a summary

Fwd:
Date: Sat, 25 Mar 2006 21:54:54 +0000
From: Martin Michlmayr <tbm-AT-cyrius.com>
To: debian-devel-AT-lists.debian.org
Cc: debian-gcc-AT-lists.debian.org
Subject: Building the whole Debian archive with GCC 4.1: a summary
Message-ID: <20060325215453.GA23620-AT-deprecation.cyrius.com>
Mail-Followup-To: debian-devel-AT-lists.debian.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.11+cvs20060126

Over the last 2.5 weeks I have built the complete Debian archive
on a quad-core MIPS machine donated by Broadcom using the recently
released version 4.1 of GCC.  In parallel, I have done the same
on an EM64T box donated to Debian by Intel and hosted by Stephen
Frost.

The purpose of this exercise was three-fold:
 - Find out about compiler problems in GCC 4.1 itself as well
   as in packages that may fail with the new version *before*
   GCC 4.1 is uploaded to unstable.  GCC, in particular G++,
   is becoming stricter regarding adherence to standards and
   packages may fail to build with 4.1 due to invalid code
   that was accepted previously.
 - Find out about MIPS specific problems in GCC 4.1 and to answer
   Matthias Klose's question [1] as to which platforms can move to
   GCC 4.1 as the default compiler once it is uploaded to unstable.
 - Find MIPS specific assembler warnings and create a list of all
   users of xgot (a MIPS specific toolchain problem).

Executive summary
-----------------

GCC 4.1 itself appears to be very stable, both on MIPS and AMD64.
There are, however, a large number of packages using code (especially
C++) which GCC 4.1 treats as errors.  Fortunately, most of them are
trivial to fix.  By compiling about 6200 packages, over 500 new
bugs have been discovered and submitted, 280 of which are specific
to the increased strictness of GCC 4.1.  Patches for 2/3 of those
GCC 4.1 specific bugs have been submitted.

Based on my findings, Ben Hutchings has prepared a summary of the most
common C++ syntax errors (that weren't treated as such before G++ 4.1):
http://womble.decadentplace.org.uk/c++/syntax-errors.html


Methodology
-----------

I generated a list of package that are "Architecture: any" or "mips",
sorted by upload (old packages first).  I then started compiling these
packages, and after the mirror pulse I would update my packages list
again (excluding packages which had no new version).  What I explicitly
did *not* do was to exclude packages which have known build failures
because I wanted to see if they might have GCC 4.1 issues too.  Another
thing I did on MIPS which the official build machines would not do
is to compile as "mips64" rather than "mips" (using a 64-bit kerneel but
32-bit userland, uname -m shows "mips64"; this can be changed by using the
linux32 program).  The aim of this was to identify mips64 specific
problems.  I compiled every packages that failed with "mips64" using
"mips" too, though.

In total, 6192 individual packages were compiled on MIPS, with 6761
compilations (because of new versions uploaded to the archive during
those two weeks).  A listing is available from [2] and all build logs
from MIPS from [3].  On AMD64, the number of individual packages
compiled was 5862.  This number is lower than the one for MIPS because
I started with MIPS first and then ignored packages on AMD64 with known
build errors.


Detailed summary of bugs found
------------------------------

While I've tried to keep count of the different errors, some of the
numbers are slightly off, partly because you do make some errors when
keeping track of so many bugs and partly because the classification
below is quite arbitrary and I slightly changed it over time.
(Normally, you'd go back and classify each bug again, but I didn't do
that because this was not a scientific study.)

1. New bugs I have filed in the BTS

 - gcc/g++ 4.1 strictness: 277 (see [4] for a list]
 - failures due to the new version of make: 4
 - old or missing build-dependencies: 50
 - host type cannot be recognized:
    - config.* out of date: 26
    - other method (mips64): 7
    - other method (amd64): 1
 - architecture specific bugs:
    - mips: 9
    - amd64: 7
 - GCC 4.1 compiler bugs: 6
 - packages that could (but don't) support mips: 5
 - non-i386 brokenness: 4
 - 64-bit brokenness: 2
 - a cast loses precision: 2
 - not using PIC: 1
 - .orig.tar.gz missing from archive: 1
 - other/generic: 50

2. The build is "successful" but there is a bug if you look closely

I tried to look at successful build logs but with over 6000 packages
I could obviously not do so in great detail.  Therefore, I'm sure there
are other bugs in "successful" builds I missed.

 - "Architecture" is "any" but should be "all": 45
 - package contains nothing useful (no binary, no headers, etc): 1
 - package contains no binary on !i386: 1
 - build doesn't show what commands are run (wishlist bug): 2
 - test suite not run (wrong command): 1

3. Bugs which I saw but which have already been reported

 - gcc/g++ 4.1 strictness: 2
 - gcc/g++ 3.4/4.0 strictness: 17
 - failures due to the new version of make: 6
 - old or missing build-dependencies: 66
 - host type cannot be recognized:
    - config.* out of date: 42
 - architecture specific bugs:
    - mips: 2
    - amd64: didn't count but there some
 - non-i386 brokenness: 27
 - 64-bit brokenness: 3
 - a cast loses precision: 2
 - not using PIC: 2
 - the "Architecture" is "any" but should be "all": 2
 - other/generic: 40

4. Most often seen warning

 - dh_*: Compatibility levels before 4 are deprecated.


Most common programming errors
------------------------------

Basically, it all boils down to broken C++ code.  There were a few
bugs in C code, but the majority was in C++.   The most common
errors I found (and some *approximate* numbers) are:

 - extra qualification: about 187 bugs
 - reliance on friend injection: 26 bugs
 - wrong escape characters (e.g. "\."; most commonly seen in regular
   expressions): 6 bugs
 - iterator problems (such as assigning 0 or NULL to an iterator): 3
 - template specialisation in wrong namespace
 - template reliance on a function declared later
 - use of template's base class members, unqualified, where the base class
   is dependent
 - use of "assert" without #include <cassert>: 5 bugs
 - dereferencing type-punned pointer will break strict-aliasing rules
   together (a warning) combined with -Werror: 6 bugs

If you intend to touch C++ code, please take some time to read
the following pages:
 - http://womble.decadentplace.org.uk/c++/syntax-errors.html
 - http://womble.decadentplace.org.uk/c++/template-faq.html
 - http://gcc.gnu.org/gcc-4.1/changes.html - a detailed list of changes
   in GCC 4.1.


Compiler bugs in GCC 4.1
------------------------

 - gcc ICE: #356231 (gnubik_2.2-5)
   http://gcc.gnu.org/PR26672 - fixed
 - g++ ICE: #356569 (3ddesktop_0.2.9-5.1)
   http://gcc.gnu.org/PR26757 - test case needs further reduction
 - gcc bug: #356574 (address of register requested)
   http://gcc.gnu.org/PR26004 - fixed
 - g++ bug: #356548 (cannot bind packed field)
   http://gcc.gnu.org/PR26670 - not yet assigned but related to known bugs
 - g++ bug: #356435 (no constructor)  [package bug: #356245]
   http://gcc.gnu.org/PR26755 - waiting
   This is a binary incompatibility!  4.0 has a different behaviour to
   3.3 and 4.1, which means that linking 4.1 code against 4.0 libraries
   may fail.  4.0 and 4.1 were supposed to be binary compatible so I'm not
   sure what will be done about this.
 - gcc bug: #356896 (miscompiles code, leading to segfault in program)
   http://gcc.gnu.org/PR26763 - confirmed

 - gcc or autoconf bug: #358644 (change in behaviour on mips regarding
   -fstack-protector)
   It's not clear right now whether this behaviour change was correct and
   autoconf should be changed or whether GCC should revert that change.
   Fortunately, this change only breaks 2 packages out of ~6200.


Status of 4.1 readiness
-----------------------

(These numbers are slightly off but give a good overview)

    * 248 Outstanding
    * 12 Forwarded
    * 10 Pending Upload
    * 3 Fixed in NMU
    * 12 Resolved

    * 186 Patch Available
    * 1 Confirmed
    * 38 Unclassified


Further work and unresolved issues
----------------------------------

These are issues I haven't had time to investigate yet:

 - look for packages that use a specific version of GCC (hard-coded) and
   see if they work with GCC 4.1.
 - a number of Java programs fail to build because jni.h cannot be found.
 - many gnustep packages didn't get built
 - many package that require kaffe didn't get built


MIPS specific comments
----------------------

1. Assembler warnings

20 packages had assembler warnings on MIPS.  A list will be sent to
debian-mips so Thiemo Seufer (or others) can investigate.

2. Packages that couldn't be built because of missing packages (depwait)

 - ghc-cvs/ghc6: 33 [see #274942]
 - libopenafs-dev: 3  [not ported to mips yet]
 - linux source/headers 2.6.15-1: 6 [2.6 is in the archive for mips now]
 - libdb4.2-java: 1 (openoffice)
 - gjdoc: 7
 - zeroc-ice/zeroc-icee-translators: 3
 - hmake: 1
 - libopenvrml5-dev: 1
 - libavifile-0.7-dev: 2
 - fp-compiler: 2
 - eclipse: 1
 - thunderbird-dev: 1
 - libgtkada2-dev: 1
 - mono/cil: 16

3. Packages not relevant to mips (notforus)

 - depends on syslinux: 1
 - mig: 1 (hurd)
 - low-level stuff: 1
 - cmucl: 1
 - mondo: 1
 - pbbuttonsd-dev (gtkpbbuttons): 1 [powerpc specific]
 - svga: 1
 - firebird2: 2

4. Packages which I've now ported to mips

 - numactl: add mips syscall definitions
 - liblinux-inotify2-perl: add mips syscall definitions
 - zeroc-icee-translators: add some simple #ifdefs for __mips__
 - zeroc-ice: add some simple #ifdefs for __mips__

5. Unclarified issue

7 packages fail to build because MIPS doesn't have a generic
cpu-feature-overrides.h header.  We have to check whether those packages
should be fixed (userspace should not depend on kernel headers anyway)
or whether the kernel can provide a generic cpu-feature-overrides.h.


Musings
-------

Obviously, compiling the archive from source leads to many important
insights.  What I found out over the last two weeks is just how much
work it actually is.  We should try to create better infrastructure
to make manual compilation of the whole archive easier.  It seems
that Roland Stigge has done some work on this already with his DARTS
project [5] but I have yet to take a look at it.


Acknowledgements
----------------

 - Broadcom for supporting our MIPS port through the donation of hardware
   to the project and developers.  Without Broadcom, our little and big
   endian MIPS ports would have a hard time meeting the release criteria
   regarding autobuilders.
 - Google for supporting my PhD, thereby allowing me to spend two weeks
   compiling the archive with GCC 4.1 and sorting out failures.
 - Intel for supporting some PhD work which led to this experiment.
 - Ben Hutchings for explaining many of the C++ bugs I found to me.
   I've learned more about C++ in these two weeks than I ever wanted to
   know. ;-)  Ben also submitted a number of patches for tricky bugs I
   couldn't fix.
 - Intel for donating an EM64T machine to Debian.
 - Thiemo Seufer for fixing all the tricky MIPS problems in Debian.
 - The GCC project for creating a great compiler, and in particular
   Andrew Pinski for doing lots of bug triaging work.
 - Roger Leigh for quickly implementing all my sbuild feature requests.
 - Everyone who compiles the archive from scratch from time to time.
   Kudos to you.  I'm now aware how much work it is!
 - Everyone who has fixed the bugs I filed already. :-)


References
----------

[1] http://lists.debian.org/debian-gcc/2006/02/msg00173.html
[2] http://people.debian.org/~tbm/logs/gcc-4.1/PACKAGES
[3] http://people.debian.org/~tbm/logs/gcc-4.1/
[4]
http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=submit...
[5] http://darts.alioth.debian.org/

-- 
Martin Michlmayr
http://www.cyrius.com/

Date: Sat, 25 Mar 2006 21:56:19 +0000
From: Martin Michlmayr <tbm-AT-cyrius.com>
To: gcc-AT-gcc.gnu.org
Subject: Building the whole Debian archive with GCC 4.1: a summary
Message-ID: <20060325215619.GA23744-AT-deprecation.cyrius.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.11+cvs20060126

Over the last 2.5 weeks I have built the complete Debian archive
on a quad-core MIPS machine donated by Broadcom using the recently
released version 4.1 of GCC.  In parallel, I have done the same
on an EM64T box donated to Debian by Intel.

The purpose of this exercise was three-fold:
 - Find out about compiler problems in GCC 4.1 itself as well
   as in packages that may fail with the new version *before*
   GCC 4.1 is uploaded to Debian.
 - Find out about MIPS specific problems in GCC 4.1 and to answer
   the question of Matthias Klose, Debian's GCC maintainer, [1]
   as to which platforms can move to GCC 4.1 as the default compiler.
 - Find MIPS specific assembler warnings and create a list of all
   users of xgot (a MIPS specific toolchain problem).

I thought I'd summarize my findings for the GCC developers so you know
what kind of broken code is out there in the wild.  Given that Debian
is a fairly large distribution, we probably have a better coverage of
the "typical free software" project (i.e. not just the successful
projects everyone talks about) than most other distros.


Executive summary
-----------------

GCC 4.1 itself appears to be very stable, both on MIPS and AMD64.
There are, however, a large number of packages using code (especially
C++) which GCC 4.1 treats as errors.  Fortunately, most of them are
trivial to fix.  By compiling about 6200 software packages (see [2]
for the list), over 500 new bugs have been discovered and submitted:
280 of which are specific to the increased strictness of GCC 4.1.
Patches for 2/3 of those GCC 4.1 specific bugs have been submitted.


Detailed summary of problems
----------------------------

Basically, it all boils down to broken C++ code.  There were a few
bugs in C code, but the majority was in C++.   The most common
errors I found (and some *approximate* numbers) are:

 - extra qualification: about 187 bugs
 - reliance on friend injection: 26 bugs
 - wrong escape characters (e.g. "\."; most commonly seen in regular
   expressions): 6 bugs
 - iterator problems (such as assigning 0 or NULL to an iterator): 3
 - template specialisation in wrong namespace
 - template reliance on a function declared later
 - use of template's base class members, unqualified, where the base class
   is dependent
 - use of "assert" without #include <cassert>: 5 bugs
 - dereferencing type-punned pointer will break strict-aliasing rules
   together with -Werror: 6 bugs
 - the use of "mips" as a variable name when GCC on mips defines it
   as a constant (arg!) and you end up with: const 1 = -1: 1 bug

A list of all GCC 4.1 specific bugs in packages can be found at [3].
The single biggest syntax error found is the use of extra qualification,
which fortunately are trivial to fix.


Compiler bugs in GCC 4.1
------------------------

I've found a number of GCC bugs in this process (the # bug numbers refer
to bugs in the Debian bug tracker, please use http://bugs.debian.org/nnn):

 - gcc ICE: #356231 (when building gnubik_2.2-5)
   http://gcc.gnu.org/PR26672 - fixed

 - g++ ICE: #356569 (when building 3ddesktop_0.2.9-5.1)
   http://gcc.gnu.org/PR26757 - test case needs further reduction

 - gcc bug: #356574 (address of register requested)
   http://gcc.gnu.org/PR26004 - fixed

 - g++ bug: #356548 (cannot bind packed field)
   http://gcc.gnu.org/PR26670 - not yet assigned but related to known bugs

 - g++ bug: #356435 (no constructor)  [package bug: #356245]
   http://gcc.gnu.org/PR26755 - waiting
   This is a binary incompatibility!  4.0 has a different behaviour to
   3.3 and 4.1, which means that linking 4.1 code against 4.0 libraries
   may fail.  4.0 and 4.1 were supposed to be binary compatible so I'm not
   sure what will be done about this.

 - gcc bug: #356896 (miscompiles code, leading to segfault in program)
   http://gcc.gnu.org/PR26763 - confirmed

 - gcc or autoconf bug: #358644 (change in behaviour on mips regarding
   -fstack-protector)
   In the past, the use of -fstack-protector on MIPS would result in
   an error.  Now it succeeds (with a warning) but then fails to link.
   autoconf expects it to fail, however.

   It's not clear right now whether this behaviour change was correct and
   autoconf should be changed or whether GCC should revert that change.
   Fortunately, this change only breaks 2 packages out of ~6200.

   Maybe other people can look at http://bugs.debian.org/358644 and
   comment.


Musing from the point of view of a distro
-----------------------------------------

I think GCC is a really solid tool and I was surprised not to see more
problems related specifically to MIPS given that it's not one of the
mainstream platforms.  As someone involved in a major Linux distribution,
I'm not terribly happy about the way errors are handled.  I'm specifically
talking about the extra qualification error here, which to the best of
my knowledge was never treated as a warning: -pedantic in 4.0 would cause
an error and now 4.1 causes an error by default.

While I appreciate G++ becoming more standards compliant, I think you
should do it in steps: issue a warning in one release (unconditionally,
i.e. without the need for -pedantic) and *then* in the next release make
it an error.  This gives developers a change to fix their bugs before
they become errors.  Granted the extra qualification bugs are trivial
to fix, but as a major distro we now have almost 200 packages that will
fail because of this error and this slows down our adoption of GCC 4.1.

I'd like to propose that you explicitly make it part of your process to
generate a warning in one release of GCC and only in the next one to
change it to an error.  Please give developer some advance warning.  As
you can see from my examples above, there is *lots* of broken code out
there.


Further work
------------

Given that I have fast MIPS hardware (thanks to Broadcom) I will try to
test GCC snapshots on MIPS from time to time and report problems to you.
If someone needs access to a fast MIPS machine for some testing, I may
be able to arrange something on a temporary basis.  Please contact me
privately if this is the case.


Acknowledgements
----------------

 - Google for supporting my PhD, thereby allowing me to spend two weeks
   compiling the archive with GCC 4.1 and sorting out failures.
 - Intel for supporting some PhD work which led to this experiment.
 - Broadcom for supporting Debian's MIPS port through the donation of
   hardware to the project and individual developers.
 - Ben Hutchings for explaining many of the C++ bugs I found to me.
   I've learned more about C++ in these two weeks than I ever wanted to
   know. ;-)  Ben also submitted a number of patches for tricky bugs I
   couldn't fix.
 - Intel for donating an EM64T machine to Debian.
 - The GCC project for creating a great compiler, and in particular
   Andrew Pinski for doing lots of bug triaging work.


References
----------

[1] http://lists.debian.org/debian-gcc/2006/02/msg00173.html
[2] http://people.debian.org/~tbm/logs/gcc-4.1/PACKAGES
[3]
http://bugs.debian.org/cgi-bin/pkgreport.cgi?which=submit...

-- 
Martin Michlmayr
http://www.cyrius.com/