User: Password:
|
|
Subscribe / Log in / New account

New NFS to bring parallel storage to the masses

New NFS to bring parallel storage to the masses

Posted Jan 22, 2009 4:21 UTC (Thu) by jwb (guest, #15467)
Parent article: New NFS to bring parallel storage to the masses

Does a pNFS mount have the full POSIX semantics like a real local filesystem? Can you safely deliver mail on it? I've tested storage from vendors like Ibrix and Isilon and I've always found that they fail in simple scenarios, such as two clients that open the same file in O_APPEND mode. On local unix filesystems this works fine, and you get coherent results, but on most commercial cluster storage you get gibberish. The only place where I've successfully exercised the full POSIX feature set is Lustre, which works perfectly in my experience. I hope that pNFS takes after Lustre more than it takes after NFSv4.


(Log in to post comments)

New NFS to bring parallel storage to the masses

Posted Jan 22, 2009 12:50 UTC (Thu) by epa (subscriber, #39769) [Link]

Is there a test suite you can run to check a filesystem's POSIX compliance for cases like this?

New NFS to bring parallel storage to the masses

Posted Jan 22, 2009 16:18 UTC (Thu) by eli (guest, #11265) [Link]

I don't know if fsx specifically tests that case, but it may:
http://www.codemonkey.org.uk/projects/fsx/

New NFS to bring parallel storage to the masses

Posted Jan 22, 2009 16:21 UTC (Thu) by snitm (guest, #4031) [Link]

Can you elaborate on your "two clients that open the same file in O_APPEND mode" scenario? What are your expectations? What is each clients' write workload? Are they each just blasting N bytes into the same file without any higher-level application coordination? What are you saying Lustre gets right and Ibrix, Isilon, etc. get wrong?

New NFS to bring parallel storage to the masses

Posted Jan 22, 2009 16:56 UTC (Thu) by jwb (guest, #15467) [Link]

When a file is opened with O_APPEND, a call to write() causes a seek to the end of the file and a write, atomically. On a normal local filesystem, n-many clients can do this to the same file at once, and their writes will all be atomic. This also works on Lustre. It definitely does not work on ordinary NFS, and it also does not work on some of the other commercial distributed/cluster filesystems I have tested.

New NFS to bring parallel storage to the masses

Posted Jan 22, 2009 20:32 UTC (Thu) by felixfix (subscriber, #242) [Link]

There used to be a string attached to those atomic writes. If it was too many bytes, either by absolute limit (4096 bytes?) or crossed a page boundary, it was split into multiple atomic writes. But I haven't had need to worry about this for many years, so I may misremember details.

New NFS to bring parallel storage to the masses

Posted Jan 24, 2009 14:14 UTC (Sat) by xav (subscriber, #18536) [Link]

I don't think there has ever been a guarantee on write() to be atomic. What write() does is return the number of bytes it could store, that's all.

New NFS to bring parallel storage to the masses

Posted Jan 24, 2009 17:19 UTC (Sat) by jwb (guest, #15467) [Link]

"If set, then all write operations write the data at the end of the file, extending it, regardless of the current file position. This is the only reliable way to append to a file. In append mode, you are guaranteed that the data you write will always go to the current end of the file, regardless of other processes writing to the file. Conversely, if you simply set the file position to the end of file and write, then another process can extend the file after you set the file position but before you write, resulting in your data appearing someplace before the real end of file."

http://theory.uwinnipeg.ca/gnu/glibc/libc_144.html


Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds