DNF5 delayed

Posted Aug 20, 2023 19:07 UTC (Sun) by Sesse (subscriber, #53779)
In reply to: DNF5 delayed by rsidd
Parent article: DNF5 delayed

> Debian recently celebrated 30 years, and dpkg and apt haven't undergone incompatible changes for, I think, at least 20 of those years?

But also not all that much new innovation; quality-of-life improvements (like apt-get → apt) and lots of bugfixes sure, but I can't recall anything really big since multiarch (2005) and possibly triggers (2007). dpkg is still using flat text files (one per package, read from disk on startup) for keeping track of installed files. apt doesn't have anything like libsolv AFAIK (I believe it's pretty much hand-rolled, and I still need to go to aptitude when apt can't figure out the conflicts). But it _is_ nice that it doesn't break under you, indeed.

DNF5 delayed

Posted Aug 21, 2023 13:53 UTC (Mon) by foom (subscriber, #14868) [Link] (9 responses)

> flat text files (one per package, read from disk on startup)

That seems to have been a fine choice, since apt and dpkg appear reasonably performant, whereas people are always complaining about how slow yum or dnf are?

DNF5 delayed

Posted Aug 21, 2023 14:25 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (2 responses)

Performance has never been a huge concern of mine as yum and dnf have always been way more *useful* in my book (though I did use `apt-rpm` way back in Fedora Core days, yum eventually improved enough to prefer it, probably around Fedora 8 or so). My main preferences for yum/dnf are around:

- dnf is a single tool that answers all my queries; I can never keep the set of apt tools for different queries straight (nevermind that some need more metadata downloaded…manually or not, I can't recall)
- `dnf install $file` works without hassle
- dnf tells me the versions that will be installed
- dnf won't "oh, there are no deps, but you obviously had no typos in your package name" auto-yes the "do you want to install this" prompt
- `dnf history undo` is so very nice
- dnf doesn't ask me questions in the middle of an installation process

DNF5 delayed

Posted Oct 9, 2023 8:42 UTC (Mon) by daenzer (subscriber, #7050) [Link] (1 responses)

> - dnf is a single tool that answers all my queries; I can never keep the set of apt tools for different queries straight (nevermind that some need more metadata downloaded…manually or not, I can't recall)

metadata for apt-file is now automatically downloaded along with other APT metadata.

Speaking of metadata, dnf automatically downloading it at random times can be annoying. dnf4 has --cacheonly to avoid that, dnf5 doesn't (yet?) though AFAICT.

> - `dnf install $file` works without hassle

This works with the new "apt" frontend (as opposed to the old "apt-get" one) as well.

> - dnf won't "oh, there are no deps, but you obviously had no typos in your package name" auto-yes the "do you want to install this" prompt

The flip side of this is is that if the package name has no typos, and the package doesn't require installing any dependencies, the separate confirmation step is superfluous.

> - dnf doesn't ask me questions in the middle of an installation process

Neither does APT, it's per-package configuration scripts. (Now with debconf, at least any such questions should happen back-to-back early on; in the old pre-debconf days, they could happen more or less anytime)

DNF5 delayed

Posted Oct 10, 2023 16:07 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

> Speaking of metadata, dnf automatically downloading it at random times can be annoying. dnf4 has --cacheonly to avoid that, dnf5 doesn't (yet?) though AFAICT.

Yes; `dnf -C info` is not uncommon in my history. Note that `dnf` does download file metadata on-demand, so you never fetch it unless using something like `dnf install */libfoo.so.*`. But it knew it needed it and updated it wanted instead of always being a separate call. It's good to know that there's at least one fewer command to care about now.

> This works with the new "apt" frontend (as opposed to the old "apt-get" one) as well.

I'll try and update my finger knowledge (`apt-get` is still in there as I think I needed it in the days before `apt` did everything and I haven't kept track of `apt` release announcements).

> The flip side of this is is that if the package name has no typos, and the package doesn't require installing any dependencies, the separate confirmation step is superfluous.

Eh, I prefer the consistent behavior more myself. There's `-y` for scripts and the like.

DNF5 delayed

Posted Aug 21, 2023 15:06 UTC (Mon) by Sesse (subscriber, #53779) [Link] (5 responses)

dpkg feels pretty slow to me, and it's only getting slower as a typical system gets more and more packages. I mean, even dpkg -i hello.deb (installing 277 kB of files) needs over 500 ms on a modern NVMe drive! On a HDD, we're talking about several seconds. full-upgrades can take many minutes just in unpacking packages, when the drive can sustain many gigabytes per second.

It may be that RPM is even slower, I don't know. But this is not fast by any reasonable standard.

DNF5 delayed

Posted Aug 21, 2023 17:44 UTC (Mon) by mbunkus (subscriber, #87248) [Link] (4 responses)

I think this might be more due to the fact that dpkg makes different tradeoffs than rpm wrt. file system security: it syncs the file system rather often. Here's a comparison of installing the same software I provide distro-specific packages for on Debian 12 & Fedora 38:

[0 root@8ea9c2baf151 …/mkvtoolnix] cat /etc/debian_version
bookworm/sid
[0 root@8ea9c2baf151 …/mkvtoolnix] strace -o ~/s.txt dpkg -i mkvtoolnix_79.0-0~ubuntu2304bunkus01_amd64.deb
Selecting previously unselected package mkvtoolnix.
(Reading database ... 86951 files and directories currently installed.)
Preparing to unpack mkvtoolnix_79.0-0~ubuntu2304bunkus01_amd64.deb ...
Unpacking mkvtoolnix (79.0-0~ubuntu2304bunkus01) ...
Setting up mkvtoolnix (79.0-0~ubuntu2304bunkus01) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.11.2-1) ...
[0 root@8ea9c2baf151 …/mkvtoolnix] grep -E 'fsync|sync_file_range|fdatasync|syncfs' ~/s.txt | wc -l
136

[0 fc38(64) root@149617e45639 ~…/x86_64] cat /etc/fedora-release
Fedora release 38 (Thirty Eight)
[0 fc38(64) root@149617e45639 …/x86_64] strace -o ~/s.txt rpm -Uhv mkvtoolnix-79.0-1.fedora38.x86_64.rpm
warning: mkvtoolnix-79.0-1.fedora38.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 10c052a6: NOKEY
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:mkvtoolnix-79.0-1.fedora38 ################################# [100%]
[0 fc38(64) root@149617e45639 …/x86_64] grep -E 'fsync|sync_file_range|fdatasync|syncfs' ~/s.txt | wc -l
5

The contents of both files aren't 100% comparable, but those numbers aren't even remotely comparable. Syncing hurts very much on HDDs, that's true.

DNF5 delayed

Posted Aug 21, 2023 17:48 UTC (Mon) by Sesse (subscriber, #53779) [Link] (3 responses)

dpkg syncs way too much compared to what you'd actually need, yes. It's entirely possible to fsync less and still be equally safe, so it's not like more fsyncs == safer.

dpkg is much faster under eatmydata, but still, reading the entire database into RAM (parsing text files line-by-line) is pretty unneeded.

DNF5 delayed

Posted Aug 21, 2023 17:54 UTC (Mon) by mbunkus (subscriber, #87248) [Link] (2 responses)

OK, pure conjecture here on my part garnished with some experience. If each invocation of dpkg reads the whole database, that should not take a lot of time safe for the first time — assuming reading is done with some proper chunking (meaning
only do a handful of big read calls, allowing for I/O speed). Unless the parsing algorithm itself is really bad, parsing several MB of data in-memory should be much faster than reading it from storage.

The next invocation should then get the whole database's data from the kernel's caches, shouldn't it? Sure, there are most likely more performant ways to store the data, or ways that would require fewer data to be read (and written, too), but does the database speed really matter that much compared to the FS syncs?

I'm talking about a system upgrade situation here, not about installing a single package.

Am I completely off base here?

DNF5 delayed

Posted Aug 21, 2023 18:11 UTC (Mon) by Sesse (subscriber, #53779) [Link] (1 responses)

> If each invocation of dpkg reads the whole database, that should not take a lot of time safe for the first time — assuming reading is done with some proper chunking (meaning only do a handful of big read calls, allowing for I/O speed).

How can you do a handful of big read calls to read thousands of files? There's one for each package installed.

> The next invocation should then get the whole database's data from the kernel's caches, shouldn't it?

Parsing 600000+ lines of text (example number from my laptop) takes real CPU time, even if the I/O is free or nearly so.

DNF5 delayed

Posted Aug 21, 2023 18:19 UTC (Mon) by mbunkus (subscriber, #87248) [Link]

> How can you do a handful of big read calls to read thousands of files? There's one for each package installed.

Ooooh I didn't know that. I thought it only reads the files directly in /var/lib/dpkg, not all the .list files, too. Good to know! I agree, that seems like a rather inefficient way to handle the information.

DNF5 delayed

Posted Aug 30, 2023 9:04 UTC (Wed) by jwilk (subscriber, #63328) [Link]

> multiarch (2005)

It was 2010 in APT; 2012 in dpkg.

> apt doesn't have anything like libsolv

APT has supported external solvers since 2011: https://packages.debian.org/unstable/apt-cudf