Overlayfs issues and experiences

After some discussion with David, I think the file locking (and probably also the fanotify) semantics turn out to be pretty straightforward:

File locking really exists to ensure that changes to files are coordinated properly.

Union mounts and overlayfs both share the same basic design -- you have a r/o layer (or more than one) and a r/w layer. The assumption is that the r/o layer won't change out from under you, even if it's (for instance) a mounted NFS filesystem. The r/w layer is also assumed to not be shared between multiple hosts. Each host must have its own.

So, locking at the r/o layer is just not very interesting and may possibly be problematic. Consider two hosts setting a write lock on the same file on a NFS-hosted r/o layer. Pushing those lock requests to the server would unnecessarily serialize their access. The file is not going to change either way, so that's just unnecessary.

So, I think that we only want to worry about locking on the r/w layer. Furthermore since we have an assumption that the r/w layers are not shared, we only need to worry about locking on a single host. In the case of NFS or another remote fs, we don't really even need to send those lock requests outside of the client.

Unfortunately, there are some other problems that get in the way of fixing this properly with overlayfs currently. David is looking at ways to address those -- once that's fixed we should be able to make file locking work on overlayfs too.