What goes into default Debian?

Posted Mar 20, 2021 8:45 UTC (Sat) by Sesse (subscriber, #53779)
In reply to: What goes into default Debian? by nix
Parent article: What goes into default Debian?

There's nothing preventing plocate from searching multiple databases, especially if you just want them to be searched serially. It's literally a for loop in the client—you probably want some sort of path rewriting in updatedb, but that could be done, too. (Then there's the question on whether you want to try to be maximally clever by pruning away the prefix or not. YMMV.) Due to io_uring, it should be reasonably performant to search databases on NFS, although I haven't tried.

You could probably even just make a shell script that calls plocate multiple times. The main reason I've never done it is that it's such a niche case nobody's ever asked for it—it requires a lot of admin intervention.

What goes into default Debian?

Posted Mar 23, 2021 20:12 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

Aha! If it's that easy, I might work on some patches. I do like the idea of plocate, but unless it works with a setup that allows N databases, one per NFS export point, it's kinda hard to make it do anything useful on a system where most filesystems you would like to run locate over are on NFS. (In my case, I usually run locate on the desktop and almost all the things I ever want it to find are on the server. Obviously the server's locate databases have to be built *on* the server: even with 10GbE I don't want updatedb throwing the stat data for thirty million files over the network every night :) and that means multiple databases if I want locate on the client to scan everything visible from the client, whether it's server-side or not).

What goes into default Debian?

Posted Mar 28, 2021 10:39 UTC (Sun) by Sesse (subscriber, #53779) [Link] (3 responses)

I pushed code for searching multiple databases to the git repository. Please give it a go.

What goes into default Debian?

Posted Mar 28, 2021 10:52 UTC (Sun) by zdzichu (subscriber, #17118) [Link]

That's why I love open source community!

What goes into default Debian?

Posted Apr 27, 2021 12:54 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

Ooh, excellent! I'll give this a try this weekend :) (yes, sometimes I do go months between catching up with LWN. Mea culpa etc.)

What goes into default Debian?

Posted Mar 27, 2022 15:50 UTC (Sun) by nix (subscriber, #2304) [Link]

Aaaand months and months after I said I'd try it, I finally did. This thing is fantastic :)

Before, with GNU findutils: 40 mins to build the locatedb, 10 mins if everything was in cache. Afterwards (hot cache figures only): 56 seconds. DBs about five times smaller. As for times:

% /usr/bin/time locate wombat
[...]
8.09user 0.08system 0:08.25elapsed 99%CPU (0avgtext+0avgdata 2084maxresident)k

% /usr/bin/time locate -r wombat
[...]
19.75user 0.40system 0:20.38elapsed 98%CPU (0avgtext+0avgdata 2196maxresident)k

% /usr/bin/time locate -r womb.t
[...]
24.98user 0.03system 0:25.08elapsed 99%CPU (0avgtext+0avgdata 2184maxresident)k

Afterwards:
% /usr/bin/time locate wombat
[...]
0.00user 0.00system 0:00.02elapsed 71%CPU (0avgtext+0avgdata 4140maxresident)k

% /usr/bin/time locate -r wombat
[...]
4.95user 0.10system 0:01.68elapsed 299%CPU (0avgtext+0avgdata 10952maxresident)k

% /usr/bin/time locate -r womb.t
[...]
5.15user 0.06system 0:01.72elapsed 302%CPU (0avgtext+0avgdata 11012maxresident)k

This is with a LOCATE_PATH with 19 databases in it, so I think we can safely say that the 20-fold increases in plocate time implied by this are... well... still pretty insignificant :)