LWN: Comments on "What's new in APT 3.0"

Dark mode?

cbushey — Sun, 27 Apr 2025 02:44:17 +0000

Nice one. Had to answer.

Dark mode?

surinameclubcard — Thu, 24 Apr 2025 18:57:39 +0000

Had to ask it.

GnuPG strength comparison ed448 vs ed25519 fixed (since 2024-11-25)

dkg — Mon, 21 Apr 2025 23:12:14 +0000

The article already mentions the concerns that GnuPG is abandoning the OpenPGP standard. To my mind, that's a pretty serious quality issue (no scare quotes needed), because as soon as there's a major repository signed with modern OpenPGP signatures (i.e. using the definitions in RFC 9580), then depending on gpgv means signature validation failure. sqv (and at least two other OpenPGP implementations ready for debian trixie) already support the newer OpenPGP standards.

For the signature verification that apt is doing, gpgv also has a particularly unwieldy interface -- you can't just supply it with a signing keyring and a signed repository manifest (either InRelease on its own, or Release + Release.gpg) and check the return code to see whether the signature is valid. Instead, you have to ask gpgv for the "status file descriptor" (via --status-fd) and build a state machine to parse and reason about its output.

If you (or anyone) is interested in other quality issues that GnuPG upstream is declining to fix, you might also want to look through the fixes proposed in the FreePG project. I also encourage people with a C programming background to look at the codebase, which suffers from a lack of testing compared to the size of the interface it presents, and carries significant unresolved technical debt.

(disclaimer: i'm a co-chair of the OpenPGP working group in the IETF; one of the debian maintainers for GnuPG itself; a contributor to the FreePG project; the author of a wrapper around gpgv to make it conform to standardized signature verification semantics; a long-time contributor to GnuPG; and a sad observer of that project's apparently deliberate slide toward irrelevance for the free software ecosystem)

Why reproducibility

SLi — Mon, 21 Apr 2025 14:50:00 +0000

Because you want reproducible but changeable. If you change one aspect, you want that aspect, to change, and nothing else to change randomly. You cannot get this with either filesystem images (you cannot get the result of "what if I had instead installed this version at the install time") nor with so called "functional reproducibility".

Lockfiles?

MrWim — Sat, 19 Apr 2025 15:23:56 +0000

I prefer apt lock files because they behave more like source code than docker images. We use them for building docker images, and ostree rootfses.

They’re small, so you can store them in git. They’re descriptive, so git diff is meaningful. This has been really helpful in the past. An adb upgrade in Ubuntu broke something[^1] for our customers after we deployed a new image. We were able to track it down really quickly because the git diff said what the old version was, and what the new one is.

[^1]: can’t remember what now

...but it is not going to be installed

niner — Sat, 19 Apr 2025 09:03:39 +0000

You make a compelling case for why it would not make sense, yet interactive conflict solving is exactly what zypper does and what I use quite often on my main machine running openSUSE. It tells you there's a conflict, gives you the list of possible solutions, you press the number and it continues happily.

Lockfiles?

ras — Fri, 18 Apr 2025 23:35:59 +0000

If you want that level of reproducibility, why not just use images, local repositories, or docker files?

To me, apt's main job it to select a set of package versions that work together, and have the latest patches. Even that is damned hard, both in the computation sense (it's NP complete) and writing code that produces the correct result without driving the user made with "this is ambiguous, should I do this?" prompts.

I wrote my own Debian resolver to get a more limited sort of reproducibility. Debian's version ranges, dependency types like conflicts and requires, pins, virtual package names that nominate a collection of possible packages make the problem so difficult it almost did my head in. I wanted a way that took an existing system and moved it to a known state, where "known state" means I give it a list of packages of I wanted installed (and consequently their dependencies), point it to a static upstream repository, and regardless of the state of the existing system always end up with the same result.

It sounds so simple, nonetheless apt can't do it because where you end up depends on the starting state of the system. The (semi-obvious) precondition is to insist the user provide a set of packages the final system that can be resolved unambiguously. From that you end up with a list of packages, and only those packages that must be present on the system. But calculating that list from just the package names is a nightmare, particularly when combined with the requirement you give the user a actionable error message on by their list was ambiguous. I was ironing out corner cases I stumbled across literally years later. Despite that, I was very happy with it in the end. All my systems we guaranteed to be in the same state even though I just nominated package names without versions.

Back to lockfiles - the more I think about it, the less sense it makes. As I said, the whole point of apt is you give it package names, it chooses the versions. The problem that programmers are trying to solve with lockfiles - ensure those versions work together, is what apt is doing for you. If are prepared to do that manually just maintain an image that has exactly the versions you want.

...but it is not going to be installed

NYKevin — Fri, 18 Apr 2025 20:23:34 +0000

In a simple case like this, "resolving" the conflict is just a matter of deleting one of the command line arguments, so it's minimally useful. In the general case, "resolving" the conflict might be arbitrarily complicated because you started dependency solving under the system's original configuration, the resolution would (presumably) entail adding or removing one or more packages, and then that might invalidate some or all of the solving you've already done, so now you have to backtrack all the way to the selected packages and start over with the new configuration. Then you might encounter another conflict, and need to prompt the user for another resolution. Eventually, you might ask the user a question to which they do not immediately know the correct answer, so now they want to pull up a man page (or examine some config file, or whatever), and have to switch to a different terminal (on pain of losing all their progress so far if apt exits).

At that point, it's worth asking whether interactive resolution would add much value as compared to dumping the user back to the shell and letting them iteratively resolve the conflict one step at a time. The (modern) shell is already designed for executing similar commands repeatedly, so what exactly is the user gaining from an interactive procedure?

...but it is not going to be installed

niner — Fri, 18 Apr 2025 18:21:45 +0000

Does it not ask the user how to resolve the conflict?

Lockfiles?

pabs — Fri, 18 Apr 2025 06:51:58 +0000

You should be able to yes, unless there are reproducibility bugs in the packages you are installing, or if the Debian snapshot ever goes offline or sees corruption.

Lockfiles?

pabs — Fri, 18 Apr 2025 06:50:43 +0000

See the reproducible installs comment nearby in the thread.

Automatic pager invocation

interalia — Fri, 18 Apr 2025 04:17:19 +0000

Ah thanks, I did not know about that convention!

Lockfiles?

amarao — Thu, 17 Apr 2025 13:34:09 +0000

Thank you for information.

And a big thank you for your work. And for diffoscope, it's amazing.

I did it without kas, in github runner environment... I almost did it, but I found, that with 12% chance one file in the image get's 666 permissions (including py files, which is kinda dangerous), and only if image is build in CI, not locally. Now it's solved by 'retry' for CI run, but I wonder what is real reason for such dangerous bit flip.

Anyway, I got the taste for reproducibility for images and start to spread the gospel everywhere I could. After having reproducibility, non-reproducible setups looks... unhygienic.

Lockfiles?

fmoessbauer — Thu, 17 Apr 2025 13:27:04 +0000

Bit by bit reproducible images and docker containers are possible, but still not trivial to achieve. The biggest remaining problems are reproducible installing as well as metadata (in case of containers). But it is doable, as shown in the https://github.com/siemens/kas project, where we build bit-identical containers for both x86_64 and arm64 in the Github CI. Users can just fork the project and reproduce the same container.

I also worked together with the team behind snapshot.d.o to make that service scalable and usable by CI systems. Now the service is behind a CDN, reasonably fast and also can be cached locally with a squid cache, if needed. Still, a lot of manual plumbing is needed to make (and keep!) things reproducible - and a lot of time spent in the diffoscope tool.

Lockfiles?

MrWim — Thu, 17 Apr 2025 13:02:52 +0000

We’ve been using lockfiles for building images with apt for the past 7 years. We don’t achieve bit-for-bit, but rather functional reproducibility:

https://github.com/stb-tester/apt2ostree?tab=readme-ov-fi...

Previously discussed here: https://lwn.net/Articles/821367/

I’m a big fan of lockfiles and reproducibility - the lack of which makes things so much harder to manage and reason about. I think it would be great if Debian were to embrace such an approach. Under that model apt would be responsible for creating a lockfile and downloading packages, and then dpkg would then be responsible for transitioning the running system to match the given lockfile.

I don’t have the will to pursue such a change in Debian myself though.

Automatic pager invocation

aragilar — Thu, 17 Apr 2025 13:01:54 +0000

This is presumably for the more human friendly interface `apt` rather than `apt-get` which apparently won't change so it can be scripted around. You could keep calling `apt-get` and not get the pager.

Lockfiles?

juliank — Thu, 17 Apr 2025 11:59:06 +0000

You can, you must mmdebstrap from a snapshot. All you need for reproducibilty is record the snapshot id.

Lockfiles?

juliank — Thu, 17 Apr 2025 11:57:19 +0000

For reproducible images, you obviously need to record the timestamp when you built the image and then request the archive snapshot for that timestamp, i.e. just set APT::Snapshot "timestamp"; or use snapshot urls directly. It's up to the archives to support the use of snapshots.

Debian images should be fully reproducible at this point, forever, well as long as the snapshots remain.

Obviously if you point your image build at a moving target instead of a snapshot, you don't get reproducible images. This is a feature.

The ability to request a specific, older version of a package is an anti-feature. We often see people who try to build repositories with

foo (= 1) Depends foo-dep (= 1)
foo (= 2) Depends foo-dep (= 2)

and then try to install foo=1 and have it fail. That's intentional and an important feature to avoid exponential number of combinations, and people from using old unmaintained versions of a package.

Lockfiles?

gioele — Thu, 17 Apr 2025 11:50:41 +0000

> The problem with reproducibility in Debian is that you have disappearing packages in the archive. Even if you pin it, a month later only newer version is available.

If you want reproducibility you must use the versioned archives provided by https://snapshot.debian.org/ instead of the main archive.

> The second problem is potential double upload, when the same version was replaced by a different binary artifact.

That cannot happen. It has happened only a few times in the past decades due to weird edge-cases (see <https://bugs.debian.org/1072205>), but now there is code in the infra to guard against it.

Lockfiles?

rweir — Thu, 17 Apr 2025 11:19:41 +0000

ah, you're totally right, sorry - I should have checked.

...but it is not going to be installed

IanKelling — Thu, 17 Apr 2025 10:25:43 +0000

This is excellent. Thank you for the great work.

Lockfiles?

jki — Thu, 17 Apr 2025 09:35:33 +0000

This changed, and people are using it at scale now, including via CI. Give it a try.

Lockfiles?

amarao — Thu, 17 Apr 2025 09:24:04 +0000

That's interesting.

Are they 'timeless'? Can I reproduce install made in 2023 in 2025? If so, this is great solution. If not, it's the same as my last attempt to have reproducible image. It works, but only until next fix to the perl (why do I have Perl in python-specific image????)

Lockfiles?

amarao — Thu, 17 Apr 2025 09:20:30 +0000

In my lifetime I saw a case or two, but it was many years ago and I can't be 100% sure, it was Debian, or 'some 3rd party deb repo'.

In the topic of better technology, the rules for Debian, whilst very honorable and important, are not the point. dpkg does not allow true (content) pinning, the technology itself have zero protection against reuploads (if apt catch reupload, there will be a warning, if there are two independent runs with different archive content, there will be zero indication of the problem).

As operator I absolutely love dpkg and apt, and it's amazing. But as developer with reproducibility goals, I have trouble to make reproducible images.

My best result is 'time-limited reproducible image' (which is a mockery for reproducibility). Until Debian pushes the next fix for some system package, my build can be reproducible. Even in this limited scope it has enormous benefits (like a developer can submit future image digest written in multiple places, together with proposed changes, and all CI need to do is to verify they match after building that image).

I don't know what proper solution should be. I love both 'security updates painless, automatic and for free', and 'reproducible forever' properties. I don't know if we can have both or not...

Lockfiles sounds like a great idea, but this is a big strain on snapshots (or mirrors, if snapshots become standardized).

Can they be 'merged' into archive in the way, which do not blow up size by x100?

Lockfiles?

porridge — Thu, 17 Apr 2025 09:09:36 +0000

I don't think double uploads are allowed in Debian proper, but I guess can happen when using unofficial repos by mistake or maliciously.

...but it is not going to be installed

juliank — Thu, 17 Apr 2025 08:32:59 +0000

Sorry this lost formatting and it had no context:

$ apt install postfix exim4
[...]
Unsatisfied dependencies:
 exim4 : Depends: exim4-base (< 4.98.1-1ubuntu2.1) but it is not going to be installed
         Depends: exim4-base (>= 4.98.1-1ubuntu2) but it is not going to be installed
         Depends: exim4-daemon-light (>= 4.98.1-1ubuntu2) but it is not going to be installed or
                  exim4-daemon-heavy (>= 4.98.1-1ubuntu2) but it is not going to be installed or
                  exim4-daemon-custom (>= 4.98.1-1ubuntu2) but it is not installable
Error: Unable to satisfy dependencies. Reached two conflicting decisions:
   1. exim4:amd64=4.98.1-1ubuntu2 is selected for install
   2. exim4:amd64=4.98.1-1ubuntu2 Depends exim4-daemon-light (>= 4.98.1-1ubuntu2) | exim4-daemon-heavy (>= 4.98.1-1ubuntu2) | exim4-daemon-custom (>= 4.98.1-1ubuntu2)
      but none of the choices are installable:
      - exim4-daemon-light:amd64=4.98.1-1ubuntu2 is not selected for install because:
        1. postfix:amd64 is selected for install
        2. postfix:amd64 Conflicts mail-transport-agent
           [selected postfix:amd64]
      - exim4-daemon-heavy:amd64=4.98.1-1ubuntu2 is not selected for install because:
        1. postfix:amd64 is selected for install as above
        2. postfix:amd64 Conflicts mail-transport-agent
           [selected postfix:amd64]

...but it is not going to be installed

juliank — Thu, 17 Apr 2025 08:31:09 +0000

The 3.0 solver will print you two conflicting chains from the root each.

Unsatisfied dependencies:
exim4 : Depends: exim4-base (< 4.98.1-1ubuntu2.1) but it is not going to be installed
Depends: exim4-base (>= 4.98.1-1ubuntu2) but it is not going to be installed
Depends: exim4-daemon-light (>= 4.98.1-1ubuntu2) but it is not going to be installed or
exim4-daemon-heavy (>= 4.98.1-1ubuntu2) but it is not going to be installed or
exim4-daemon-custom (>= 4.98.1-1ubuntu2) but it is not installable

becomes:

Error: Unable to satisfy dependencies. Reached two conflicting decisions:
1. exim4:amd64=4.98.1-1ubuntu2 is selected for install
2. exim4:amd64=4.98.1-1ubuntu2 Depends exim4-daemon-light (>= 4.98.1-1ubuntu2) | exim4-daemon-heavy (>= 4.98.1-1ubuntu2) | exim4-daemon-custom (>= 4.98.1-1ubuntu2)
but none of the choices are installable:
- exim4-daemon-light:amd64=4.98.1-1ubuntu2 is not selected for install because:
1. postfix:amd64 is selected for install
2. postfix:amd64 Conflicts mail-transport-agent
[selected postfix:amd64]
- exim4-daemon-heavy:amd64=4.98.1-1ubuntu2 is not selected for install because:
1. postfix:amd64 is selected for install as above
2. postfix:amd64 Conflicts mail-transport-agent
[selected postfix:amd64]

Automatic pager invocation

juliank — Thu, 17 Apr 2025 08:29:39 +0000

You can (un)set APT_PAGER, GIT_PAGER, SYSTEMD_PAGER, that all works (and they all respect PAGER if no specific variable is given)

...but it is not going to be installed

taladar — Thu, 17 Apr 2025 08:13:00 +0000

Will this new version of APT finally get rid of the annoying "but it is not going to be installed" output when some dependency of a dependency is not satisfied where you have to then manually try installing the named package to figure out the reason it is not going to be installed, potentially dozens of times until you hit the actual problem?

GnuPG strength comparison ed448 vs ed25519 fixed (since 2024-11-25)

ber — Thu, 17 Apr 2025 07:48:34 +0000

GnuPG 2.4.7 released on 2024-11-25 fixes the comparison (T7425).

It would be interesting to have more specific reports about the mentioned other "quality issues".

Automatic pager invocation

neilbrown — Thu, 17 Apr 2025 07:23:57 +0000

$TERM goes some way in that direction

Lockfiles?

Cyberax — Thu, 17 Apr 2025 06:56:23 +0000

It's not the same thing. I'm not aware of a way to build bit-for-bit identical _system_. Every time you do "apt-get install", you have a potential to end up with a slightly different system.

Automatic pager invocation

epa — Thu, 17 Apr 2025 06:49:31 +0000

There's a missing middle ground between a full tty (where you can use cursor movement commands) and sending output to a file. Another example is Emacs's shell-mode. If it didn't return true to isatty() then gzip might write binary data to stdout, and other programs wouldn't prompt interactively. So it has to claim to be a tty, but then some tools want to run a pager or do cursor movement.

There should be a similar function isinteractive() provided by the C library and kernel somehow. When you create a pty you can specify whether it should be a full terminal (returning true to both isatty() and isinteractive()) or just a kind of interactive thingy (isinteractive=true but isatty=false). I guess if any computer museums are using real printing terminals they could also benefit from this. You don't want to run less(1) on your Teletype, though I guess more(1) could work.

Lockfiles?

pabs — Thu, 17 Apr 2025 06:35:46 +0000

There is work on reproducible installs too, IIRC the live images are now reproducible from binary .deb files.

https://wiki.debian.org/ReproducibleInstalls

Lockfiles?

amarao — Thu, 17 Apr 2025 06:31:12 +0000

It work for Debian reproudicability. Now, try to build a reproducible image from them (debootstrap or docker). You would be very surprised how impossible it is.

Last time I tried, I got diffoscope angry on me because of perl security fix landing right in between rebuilds.

Lockfiles?

pabs — Thu, 17 Apr 2025 06:29:29 +0000

That would be the buildinfo files pioneered by the repro builds folks, they are working on reproducing binary packages based on them.

https://reproduce.debian.net/

Lockfiles?

pabs — Thu, 17 Apr 2025 06:28:33 +0000

https://reproduce.debian.net/

Lockfiles?

Cyberax — Thu, 17 Apr 2025 05:23:02 +0000

It runs on a single server, and won't scale if people start using it.

Automatic pager invocation

interalia — Thu, 17 Apr 2025 04:55:19 +0000

Often I don't mind auto-paging but other times it is annoying and I have to pipe to cat as dskoll mentioned in their comment. Maybe it'd be nice if there was another environment variable AUTOPAGER=true or something which programs could consult on whether to use such default paging behaviour. But then we might still hit a problem if we want auto-paging in some programs but not others.

Automatic pager invocation

dskoll — Thu, 17 Apr 2025 00:46:55 +0000

Yeah, I prefer manual control too, though something like apt ... whatever | cat should work too, because it probably only invokes the pager if standard output is a TTY.