Debian "sid" users beware of the dpkg 1.16.2 upload

Hi,

I'll be uploading dpkg 1.16.2 targeting unstable, by the end of
this weekend or beginning of next week the latest (after some final
polishing). Some pretty important points follow, the first section
in particular.


On-disk db damage
-----------------

The previous multiarch in-core db layout was bogus, resulting in a
possible inconsistent or broken on-disk db. If you are running any
dpkg derived from code that has never been in the main git repo (this
includes dpkg from the jenkins test builds [T], dpkg from experimental,
dpkg from Ubuntu, one of the personal pu/ branches, etc), any of the
following might affect you.

  [T] <http://jenkins.grml.org/job/dpkg-test-source/>

With those dpkg versions, when installing a M-A:same instance of
package P arch A, when a previous non-M-A:same version of package P
arch B was present in a config-files state, the installation would
incorrectly remove the control files on the infodb for the arch B
instance. You should check for any packages in the status db w/o
matching files under /var/lib/dpkg/info/.

Another effect of the bogus in-core db layout affected the disappearing
logic, so if you have been running any such dpkg versions, you should
also check in the logs that no package has been improperly disappeared,
although the installed files should still be present.

And finally (well, these are just some of the effects I've explicitly
looked for after revealing them through code review, there could be more
related to this, but I've not bothered after having reworked the in-core
db layout), upgrading from those versions to the new dpkg 1.16.2 might
cause the status db to get messsed up in some conditions. Before
upgrading, you should either downgrade to a non-multiarch enabled dpkg, or
make sure there's no package present (i.e. with status >= config-files)
with a mix of M-A:same and non-M-A:same instances. If there's such package
with two instances, the new dpkg will consider that a “cross-grade” and
as such replace one of them with the other, depending on the order they
are parsed, but leaving any control files untouched; if there's more than
two instances the new dpkg will outright refuse to parse such bogus and
inconsistent status db.

Do not file any bug report if you upgraded from a system with a damaged
db, w/o first reproducing it on a clean system.


New and changed interfaces
--------------------------

Several inadequate interfaces (relative to the unofficial dpkg versions)
have been changed during the past several months, some of those to make
it possible to reliably cross-grade packages (including dpkg itself, even
when the world view of what's native changes between dpkg and frontend).

The upload to experimental had unfinished command-line interfaces, so
if you have relied on those or on the ones from other unofficial dpkg
versions, you will need to adapt to the final ones. Any package name
output from dpkg tools will be accepted by those as valid input during
the same transaction. For input you should always be able to arch-qualify
package names (as in “dpkg:amd64”), for singleton packages (those that
only have one installed instance present) you can simply use the package
name, for co-installed instances you always need to arch-qualify them.
Whenever dpkg tools take a package name, only one specific instance is
expected (as in “pkgname:arch” for multiple instances, “pkgname” or
“pkgname:arch” for singleton instances), if the command takes a pattern
then a simple package name will match any of its present instances (the
equivalent of “pkgname:*”), other accepted patterns are “*:arch”.
Current apt in unstable might not correctly arch-qualify packages when
needed and dpkg could fail due to that, apt from experimental should
work fine.

Any program or script parsing dpkg tools output should be adapted for
the possibility of arch-qualified package names or for multiple instance
output from a single package name.

The new dpkg-query virtual field (for internally generated values), that
allows package instances to be referred unambiguously (will arch-qualify
them when needed), has been renamed to not stomp on the valid field
namespace from binary packages, the new name for the field is
“binary:Package” (in line with substvars naming conventions).

Also as (another) side effect of the new in-core db layout, spurious
virtual packages do not get printed by dpkg-query any longer, so you
do not need to care for them any more.

The output from --print-foreign-architectures has changed from being
space separated to new-line separated, and as such old apt versions will
only “see” the first foreign architecture. apt from experimental should
work fine though.

Registering allowed architectures has changed from a command line option
to two commands to add and remove them to/from a new architecture db.
Those are --add-architecture and --remove-architecture. When cross-grading
dpkg, there's nothing needed to be done, as with this it will be able to
tell automatically what are the new switched native and foreign
architectures.

If you need to know if the current dpkg supports multiarch, for example
to check if it will accept arch-qualified input, there's a new
«dpkg --assert-multi-arch» command (please see the man page).

«dpkg --audit» now performs some checks related to the invalid
architectures.

dpkg-checkbuilddeps has grown a new -a option to specify the
architecture.


Maintainer scripts
------------------

Something that maintainers might have not considered when switching
packages to be multiarch aware is that maintainer scripts calling
dpkg tools might need to specify the precise package names to get the
correct results for the desired package instance(s). Please, check
your maintainer scripts. (All dpkg-divert calls should be fine as they
are.)

In line with the above, the DPKG_MAINTSCRIPT_ARCH environment variable is
set by dpkg for the maintainer script, to match the architecture of its
binary package. You might rely on it to be always present as it was
already introduced [M] in squeeze's dpkg with multiarch in mind.

  [M] <http://lists.debian.org/debian-devel-announce/2010/03/msg...>


On-disk db layout
-----------------

This is nothing new, but just as a reminder, the internal on-disk db
layout has changed, and although programs are *not* expected to be
messing around with it, there's some (exceptional) times when they might
need access to that data, if that's the case, then «dpkg-query -L»,
«dpkg-query -f '${Conffiles}' -W» or «dpkg-query --control-path»
(interface introduced long time ago [M] precisely for this reason) are
the “correct” things to use, do not assume any given on-disk db layout
(see dpkg-query man page for more details).


Cross-grading
-------------

Some weeks ago, after having set the foundation in place during the
past months, I implemented support for cross-grading in dpkg (and as
a side effect fixed all in-core db issues), so after installing the
needed dependencies you can cross-grade dpkg itself, and any other
package desired. Take into account though, that apt does *not* support
cross-grading, it also has problems with virtual package dependencies
across architectures (for example with a foreign mawk), and AIUI if it
does allow such action it will perform it by removing first the package
to be cross-graded, so you should not use apt for that, as it might
render your system unusable if it removes essential packages or
required shared libraries.

A normal session to perform a dpkg cross-grade would go like this:

  # dpkg --print-architecture
  i386
  # dpkg --add-architecture amd64
  # dpkg --print-foreign-architectures
  amd64
  # apt-get update
  # apt-get download gcc-4.6-base:amd64 libgcc1:amd64 libc6:amd64 \
            libselinux1:amd64 zlib1g:amd64 libbz2-1.0:amd64 dpkg:amd64
  # dpkg -i gcc-4.6-base*.deb libgcc*.deb libc*.deb libselinux*.deb \
            libbz2*.deb zlib*.deb dpkg*.deb
  # dpkg --print-architecture
  amd64
  # dpkg --print-foreign-architectures
  i386

Note though, that not all packages might be properly marked as Multi-Arch
same, foreign or allowed, so a full system cross-grade might not yet be
possible.


regards,
guillem