A filesystem "change journal" and other topics [LWN.net]

A filesystem "change journal" and other topics

Posted Jun 14, 2018 6:04 UTC (Thu) by njs (subscriber, #40338)
In reply to: A filesystem "change journal" and other topics by nix
Parent article: A filesystem "change journal" and other topics

This is how the (proprietary) CrashPlan backup system works... except of course it can't use the yet-to-be-created change journal, so it has to register an inotify watch on every file in the system. Which is why all my systems have an override file in /etc/sysctl.d/ with some ridiculous setting like:

fs.inotify.max_user_watches = 4194304

It's absolutely worth it to get continuous, fine-grained backup snapshots, but clearly Linux is not supporting this as well as it could.

to post comments

A filesystem "change journal" and other topics

Posted Jun 19, 2018 19:33 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

I'm fairly sure that won't work if some of the changes are made by NFS clients. (The inotify watch gets fired on the client, not on the server, but it's on the server you'd want it for a backup system.)

Needless to say I won't be competing with CrashPlan because a) I won't have a wizzy GUI and b) I won't write something that slows the system to an utterly unusable crawl (or that's what the Windows version does, anyway, which I am forced to use on the work laptop I never turn on). Oh also CrashPlan actually exists. Code that doesn't exist has minimal effect on system performance!

A filesystem "change journal" and other topics

Posted Jun 20, 2018 14:27 UTC (Wed) by bfields (subscriber, #19510) [Link] (3 responses)

"I'm fairly sure that won't work if some of the changes are made by NFS clients. (The inotify watch gets fired on the client, not on the server, but it's on the server you'd want it for a backup system.)"

If you've observed this, please file a bug report against the NFS server.

Sure, a write that's still in the client's cache won't result in a notification on the server until it's flushed back to the server, but that will happen eventually (barring overwrites, a client crash before sync, etc.--but I think a backup application will be OK with those exceptions).

But any change that reaches the exported filesystem should definitely result in a notification.

Client-side notifications are much harder as they'd need some (very chatty) protocol to learn about changes made by anyone other than themselves.

A filesystem "change journal" and other topics

Posted Jul 8, 2018 15:09 UTC (Sun) by nix (subscriber, #2304) [Link] (2 responses)

> "I'm fairly sure that won't work if some of the changes are made by NFS clients. (The inotify watch gets fired on the client, not on the server, but it's on the server you'd want it for a backup system.)"
> If you've observed this, please file a bug report against the NFS server.

What, this works?! It didn't used to, but mind you that was many years ago. I assumed it still didn't work because I mentioned it here and someone -- perhaps Al Viro? -- asked astoundedly why I would ever expect changes on the client to fire watches on the server. I decided that inotify was therefore useless-by-design for more reasons than I thought it was, and never looked again.

If that's actually been considered a bug and fixed at some point in the last decade or so, then the only remaining problem is the scalability one, and for that one presumably just needs a better API. (Well, that and a way to get the right subset of change notifications back out to clients, but *that* is a problem that can obviously be solved entirely in userspace.)

A filesystem "change journal" and other topics

Posted Jul 9, 2018 13:46 UTC (Mon) by bfields (subscriber, #19510) [Link] (1 responses)

I'll admit I've never tested it!

But it should have worked since inotify was introduced. It's not really even up to knfsd, the notification hooks are mostly in lower level vfs code that knfsd calls. (And of course userspace servers like Ganesha couldn't avoid notifying even if they wanted to.)

Before inotify was introduced, there was dnotify, was was a poor fit since as I understand it only allowed directories. Sending the notification for a write to the parent directory is a little strange to start off with, for knfsd it's often impossible. Nevertheless looking through the history I see Neil Brown added code that attempts to do this on read and write in early 2004. Even on dnotify I expect notifications for other operations worked since it was introduced.

The harder problem again is notifying one client about changes that occurred on another. Yeah, in theory I guess you could do that with a separate protocol implemented in userspace. Sounds painful to me.

A filesystem "change journal" and other topics

Posted Jul 27, 2018 13:38 UTC (Fri) by nix (subscriber, #2304) [Link]

The harder problem again is notifying one client about changes that occurred on another. Yeah, in theory I guess you could do that with a separate protocol implemented in userspace. Sounds painful to me.

I think that's the only sensible approach. Whatever the client is, it would have to mirror the NFS server layout, with a dispatcher/broadcaster on the NFS server that inspected the NFS mount points and rebroadcast inotify notifications to the appropriate client: every client would then pretend to be something like gamin so that glib etc would Just Work with it.

This is much simpler than the alternative if the NFS server did not see changes done on clients: that would need every client to broadcast inotify requests it saw to the server, which then bounced them to every client, and that is getting seriously ugly, really.