What goes into default Debian?

“plocate LWN” takes 8 milliseconds.

I wrote plocate because mlocate's slowness was a real impediment to my (volunteer) sysadmin tasks. find is fine if you have a tiny system or a narrow search scope, but even on my laptop's SSD with a pretty small installation, it takes 2–3 seconds to run.

An analogy might be something like crond, but for file-system changes. It would maintain persistent state between boots, and could implement fs specific optimisations.

But as I say, I haven't really thought this through :-)

I once tried to make a build system with inferred dependency tracking using fanotify. On paper it's straightforward: fanotify gives a stream of events like "pid P opened file F for reading and G for writing", and from that we infer G depends on F. So we run the build and log all the file accesses and then process the log later and we have at least a rough idea of what file outputs use what other files as inputs. Easy.

Now try this with a build that has 4000 processes opening a million files on a dozen CPU cores. In nonblocking mode, the events can't be dequeued fast enough (there certainly isn't time to resolve filenames), and in blocking mode it dramatically slows down the build.

Same problem for incremental backups: the fanotify monitor plays along for a while, then a lot of files get updated at once and the monitor has to say "nope, can't do it, you'll have to do a full scan to find out what I missed."

updatedb doesn't use whitelists--it indexes everything not on a blacklist. Network filesystems are filtered with a blacklist that had to be updated when nfs4 came along, then again when smbfs^H^H^H^H^Hcifs came along, then again and again and again when a thousand fuse filesystems came along (the horror...semi-infinite generated file namespaces with expensive iterators popping up at random, vs. updatedb). If I start using a new filesystem tomorrow, I can have an updatedb-related disaster the following day.

updatedb would also index any directory under / that wasn't explicitly excluded by a blacklist. The whole point of using novel directories under / that no existing software knows about is that existing software will stay the hell out of them until explicitly directed there. Not updatedb! That thing goes looking for trouble, and as long as trouble isn't in a default list of a half-dozen excluded directories, it'll find it!

Most of the problems would have been trivially solved if updatedb used a whitelist of FHS directories (/etc, /bin, /lib, /lib64, /sbin, /srv, /usr, /var, /opt, /home if local, all with -xdev in find) and searched only those until told to do otherwise. A "normal user" will not store anything outside of $HOME anyway. Users who attach huge file stores to their machines can add the mount points to updatedb.conf, or use a standard (for Debian) path like /srv for the big file store. Users who use 'locate' every day can edit /etc/updatedb.conf.

My host configuration logs indicate locking didn't start happening in Debian until 2015 (give or take a year) with the introduction of updatedb.mlocate. Even then, the locking had an obvious bug until 2017. This was long after I had stopped caring what the locate package maintainer did any more--by 2002 I was already using /etc/apt/preferences to ensure the locate package could not be installed on production machines, and a few years later I stopped testing new versions.

I don't know what systemd or plocate does, but if it doesn't start with bind-mounting whitelisted filesystems in a private namespace and running updatedb chroot in that namespace, I don't need to see the rest of it. I've seen updatedb's blacklisting accidentally defeated by users and junior sysadmins and upstream software updates, and I've seen nothing to prevent this from happening again.

When we index files, we define a service profile, purchase or assign hardware from inventory, task that hardware with running the service, i.e. storing hundreds of millions of files, and providing an indexing service for them. We assign staff and robots to operate and monitor the hardware, periodically check that the hardware is healthy and services in the profile are all running correctly, check that the indexes are correct and up to date and indexing files in scope and not indexing files not in scope, and ensure there is enough storage for the index and enough free iops to update it with whatever frequency the service profile says we need. In other words, these indexers are _supervised_.

None of this happens when updatedb is installed by default. It's a production risk and wasted cost until you turn it off or take control of it. If you turn your back on it, it inevitably fills up /var without warning, and burns power and media lifetime even when it's working normally--and when it's not, it can take big servers all the way down.

As far as I can tell, among the standard Unix packages, this is a unique property of *locate? Off the top of my head I can't think of any other past or present default-installed service that potentially or actually does scheduled work proportional to the number of files your host can access.