Optional mandatory locking

Posted Dec 10, 2015 13:27 UTC (Thu) by cuboci (subscriber, #9641)
In reply to: Optional mandatory locking by philipstorry
Parent article: Optional mandatory locking

This is not about files I generate myself. The files I'm talking about are uploaded by customers. I have no control over that other than to notice a new file is there.

Optional mandatory locking

Posted Dec 10, 2015 15:59 UTC (Thu) by alankila (guest, #47141) [Link] (4 responses)

I'd say that your sftp/whatever server implementation should be able to generate an event when client is finished with uploading a file. This would generally be the case if you used a library that implements the protocol rather than e.g. separate unix process that just dumps stuff to filesystem.

Optional mandatory locking

Posted Dec 10, 2015 19:55 UTC (Thu) by cuboci (subscriber, #9641) [Link] (3 responses)

This is standard OpenSSH SFTP. What event is it able to generate once the upload is complete?

Optional mandatory locking

Posted Dec 10, 2015 20:19 UTC (Thu) by iabervon (subscriber, #722) [Link]

sftp-server logs transactions it performs on behalf of the client. I'm not sure if successful completion is what's at the INFO level or if that would be at a DEBUG level, but this would be a better trigger than any sort of locking, since sftp transfers can fail in the middle, and a locking-based method would either think it was done (and act on partial data) or think it was still going (and wait forever).

Optional mandatory locking

Posted Dec 10, 2015 22:42 UTC (Thu) by rotty (guest, #14630) [Link]

You could also use inotify, for example by incron to generate an event based on a file being open for writing being closed. There might be gotchas, but in principle, it should work (I've used it for auto-converting files uploaded via SMB).

Optional mandatory locking

Posted Dec 12, 2015 11:24 UTC (Sat) by alankila (guest, #47141) [Link]

Probably none, because you are using a system that just dumps stuff to unix filesystem, so you are stuck with something like inotify/dnotify or whatever it is called today. Ideally, you'd assemble your own SFTP daemon out of reusable components, rather than using processes solving parts of the problem and then being stuck trying to discover mechanisms by which they can interoperate.

Optional mandatory locking

Posted Dec 13, 2015 2:55 UTC (Sun) by giraffedata (guest, #1954) [Link] (1 responses)

It's worth noting that even if there were some file locking function that could let you block until the file isn't open for write, relying on that is still a hack in your situation, since there's no reason the program that generates the file, over which you have no control, couldn't open and close the file multiple times in the process.

Optional mandatory locking

Posted Dec 14, 2015 11:55 UTC (Mon) by cuboci (subscriber, #9641) [Link]

I'm aware of that. Problem is, right now I have no choice but to use standard components. Talk about being stuck between a rock and a hard place.