Lockfiles?

Posted Apr 16, 2025 22:12 UTC (Wed) by amarao (guest, #87073)
In reply to: Lockfiles? by Cyberax
Parent article: What's new in APT 3.0

The problem with reproducibility in Debian is that you have disappearing packages in the archive. Even if you pin it, a month later only newer version is available. The second problem is potential double upload, when the same version was replaced by a different binary artifact.

Content addressing (like docker does with layers) would had solved this completely...

Lockfiles?

Posted Apr 16, 2025 22:46 UTC (Wed) by bluca (subscriber, #118303) [Link] (2 responses)

https://snapshot.debian.org

Lockfiles?

Posted Apr 17, 2025 5:23 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

It runs on a single server, and won't scale if people start using it.

Lockfiles?

Posted Apr 17, 2025 9:35 UTC (Thu) by jki (subscriber, #68176) [Link]

This changed, and people are using it at scale now, including via CI. Give it a try.

Lockfiles?

Posted Apr 17, 2025 6:28 UTC (Thu) by pabs (subscriber, #43278) [Link] (6 responses)

https://reproduce.debian.net/

Lockfiles?

Posted Apr 17, 2025 6:31 UTC (Thu) by amarao (guest, #87073) [Link] (5 responses)

It work for Debian reproudicability. Now, try to build a reproducible image from them (debootstrap or docker). You would be very surprised how impossible it is.

Last time I tried, I got diffoscope angry on me because of perl security fix landing right in between rebuilds.

Lockfiles?

Posted Apr 17, 2025 6:35 UTC (Thu) by pabs (subscriber, #43278) [Link] (2 responses)

There is work on reproducible installs too, IIRC the live images are now reproducible from binary .deb files.

https://wiki.debian.org/ReproducibleInstalls

Lockfiles?

Posted Apr 17, 2025 9:24 UTC (Thu) by amarao (guest, #87073) [Link] (1 responses)

That's interesting.

Are they 'timeless'? Can I reproduce install made in 2023 in 2025? If so, this is great solution. If not, it's the same as my last attempt to have reproducible image. It works, but only until next fix to the perl (why do I have Perl in python-specific image????)

Lockfiles?

Posted Apr 18, 2025 6:51 UTC (Fri) by pabs (subscriber, #43278) [Link]

You should be able to yes, unless there are reproducibility bugs in the packages you are installing, or if the Debian snapshot ever goes offline or sees corruption.

Lockfiles?

Posted Apr 17, 2025 13:27 UTC (Thu) by fmoessbauer (subscriber, #171204) [Link] (1 responses)

Bit by bit reproducible images and docker containers are possible, but still not trivial to achieve. The biggest remaining problems are reproducible installing as well as metadata (in case of containers). But it is doable, as shown in the https://github.com/siemens/kas project, where we build bit-identical containers for both x86_64 and arm64 in the Github CI. Users can just fork the project and reproduce the same container.

I also worked together with the team behind snapshot.d.o to make that service scalable and usable by CI systems. Now the service is behind a CDN, reasonably fast and also can be cached locally with a squid cache, if needed. Still, a lot of manual plumbing is needed to make (and keep!) things reproducible - and a lot of time spent in the diffoscope tool.

Lockfiles?

Posted Apr 17, 2025 13:34 UTC (Thu) by amarao (guest, #87073) [Link]

Thank you for information.

And a big thank you for your work. And for diffoscope, it's amazing.

I did it without kas, in github runner environment... I almost did it, but I found, that with 12% chance one file in the image get's 666 permissions (including py files, which is kinda dangerous), and only if image is build in CI, not locally. Now it's solved by 'retry' for CI run, but I wonder what is real reason for such dangerous bit flip.

Anyway, I got the taste for reproducibility for images and start to spread the gospel everywhere I could. After having reproducibility, non-reproducible setups looks... unhygienic.

Lockfiles?

Posted Apr 17, 2025 9:09 UTC (Thu) by porridge (subscriber, #15054) [Link] (2 responses)

I don't think double uploads are allowed in Debian proper, but I guess can happen when using unofficial repos by mistake or maliciously.

Lockfiles?

Posted Apr 17, 2025 9:20 UTC (Thu) by amarao (guest, #87073) [Link] (1 responses)

In my lifetime I saw a case or two, but it was many years ago and I can't be 100% sure, it was Debian, or 'some 3rd party deb repo'.

In the topic of better technology, the rules for Debian, whilst very honorable and important, are not the point. dpkg does not allow true (content) pinning, the technology itself have zero protection against reuploads (if apt catch reupload, there will be a warning, if there are two independent runs with different archive content, there will be zero indication of the problem).

As operator I absolutely love dpkg and apt, and it's amazing. But as developer with reproducibility goals, I have trouble to make reproducible images.

My best result is 'time-limited reproducible image' (which is a mockery for reproducibility). Until Debian pushes the next fix for some system package, my build can be reproducible. Even in this limited scope it has enormous benefits (like a developer can submit future image digest written in multiple places, together with proposed changes, and all CI need to do is to verify they match after building that image).

I don't know what proper solution should be. I love both 'security updates painless, automatic and for free', and 'reproducible forever' properties. I don't know if we can have both or not...

Lockfiles sounds like a great idea, but this is a big strain on snapshots (or mirrors, if snapshots become standardized).

Can they be 'merged' into archive in the way, which do not blow up size by x100?

Lockfiles?

Posted Apr 17, 2025 11:57 UTC (Thu) by juliank (guest, #45896) [Link]

For reproducible images, you obviously need to record the timestamp when you built the image and then request the archive snapshot for that timestamp, i.e. just set APT::Snapshot "timestamp"; or use snapshot urls directly. It's up to the archives to support the use of snapshots.

Debian images should be fully reproducible at this point, forever, well as long as the snapshots remain.

Obviously if you point your image build at a moving target instead of a snapshot, you don't get reproducible images. This is a feature.

The ability to request a specific, older version of a package is an anti-feature. We often see people who try to build repositories with

foo (= 1) Depends foo-dep (= 1)
foo (= 2) Depends foo-dep (= 2)

and then try to install foo=1 and have it fail. That's intentional and an important feature to avoid exponential number of combinations, and people from using old unmaintained versions of a package.

Lockfiles?

Posted Apr 17, 2025 11:50 UTC (Thu) by gioele (subscriber, #61675) [Link]

> The problem with reproducibility in Debian is that you have disappearing packages in the archive. Even if you pin it, a month later only newer version is available.

If you want reproducibility you must use the versioned archives provided by https://snapshot.debian.org/ instead of the main archive.

> The second problem is potential double upload, when the same version was replaced by a different binary artifact.

That cannot happen. It has happened only a few times in the past decades due to weird edge-cases (see <https://bugs.debian.org/1072205>), but now there is code in the infra to guard against it.