DNF5 delayed

Posted Aug 21, 2023 17:54 UTC (Mon) by mbunkus (subscriber, #87248)
In reply to: DNF5 delayed by Sesse
Parent article: DNF5 delayed

OK, pure conjecture here on my part garnished with some experience. If each invocation of dpkg reads the whole database, that should not take a lot of time safe for the first time — assuming reading is done with some proper chunking (meaning
only do a handful of big read calls, allowing for I/O speed). Unless the parsing algorithm itself is really bad, parsing several MB of data in-memory should be much faster than reading it from storage.

The next invocation should then get the whole database's data from the kernel's caches, shouldn't it? Sure, there are most likely more performant ways to store the data, or ways that would require fewer data to be read (and written, too), but does the database speed really matter that much compared to the FS syncs?

I'm talking about a system upgrade situation here, not about installing a single package.

Am I completely off base here?

DNF5 delayed

Posted Aug 21, 2023 18:11 UTC (Mon) by Sesse (subscriber, #53779) [Link] (1 responses)

> If each invocation of dpkg reads the whole database, that should not take a lot of time safe for the first time — assuming reading is done with some proper chunking (meaning only do a handful of big read calls, allowing for I/O speed).

How can you do a handful of big read calls to read thousands of files? There's one for each package installed.

> The next invocation should then get the whole database's data from the kernel's caches, shouldn't it?

Parsing 600000+ lines of text (example number from my laptop) takes real CPU time, even if the I/O is free or nearly so.

DNF5 delayed

Posted Aug 21, 2023 18:19 UTC (Mon) by mbunkus (subscriber, #87248) [Link]

> How can you do a handful of big read calls to read thousands of files? There's one for each package installed.

Ooooh I didn't know that. I thought it only reads the files directly in /var/lib/dpkg, not all the .list files, too. Good to know! I agree, that seems like a rather inefficient way to handle the information.