NTFS-3G isn't optimized yet [LWN.net]

NTFS-3G isn't optimized yet

Posted Jun 19, 2007 0:26 UTC (Tue) by szaka (subscriber, #12740)
Parent article: ZFS on Linux: It's alive (LinuxWorld)

The article mentions NTFS-3G has decent and good performance. Please also note that we're still designing and implementing missing functionalities and didn't start the optimization work yet.

But we're aware of and identified already several things which could be improved in the kernel, FUSE and NTFS-3G that could increase efficiency dramatically (lower CPU usage and/or higher I/O). Some of these:

read/write data copies between the kernel and a fuseblk fs could be optimized away, only metadata calculations (and transaction based caching) is needed in user space.
kernel double caches read/write buffers (FUSE and blockdev sides, fuseblk fs should use direct io).
non-properly aligned writes are split in the kernel and the FUSE fs ends up doing twice as much work as it would be needed (e.g. EncFS is hit by this, halving its performance when used with NTFS-3G).
NTFS-3G doesn't have any of its own caches yet (directory, inode, etc).
find(1) could be speed up 5 folds in general by fully filling struct dirent in readdir.

And of course several things could be optimized here and there for different kind of corner cases and workloads (allocators, extent and transparent file compressors/decompressors, etc).

(Log in to post comments)

NTFS-3G isn't optimized yet

Posted Jun 19, 2007 6:59 UTC (Tue) by ncm (subscriber, #165) [Link]

This sort of posting is the second best thing about LWN. I would like to read an article by szaka exploring the above in greater depth. (Delaying NTFS-3G to write it would be more than justified.)

NTFS-3G isn't optimized yet

Posted Jun 19, 2007 12:39 UTC (Tue) by Lionel_Debroux (subscriber, #30014) [Link]

I second this, a detailed article would be a good thing. It's good that a number of bottlenecks have been identified over the months.

And, it's not true that the optimization work on NTFS-3G didn't start yet ;-)
"NTFS-3G isn't yet as optimized as it could be" is nearer from the truth, since there's still room for improvement, as szaka wrote above.

The new allocator introduced in March yields a leap on performance with large files, and those large files are also less fragmented than they used to be. Even before that, the performance on large files was improved over the versions: in July or August 2006, it was hard to copy a file weighing more than several hundreds of megabytes (depending on the fragmentation), but the stable 1.0 version, the last one before the improved allocator, was faster.

Keep up the good work :)

ZFS - horse of a different colour

Posted Jun 20, 2007 22:21 UTC (Wed) by qu1j0t3 (subscriber, #25786) [Link]

NTFS isn't in this race, as anyone who's looked into ZFS would understand. As csamuel points out, speed is only one of the minor selling points of ZFS (and of course, it's not going to win any sprints boxed up in FUSE).

http://www.sun.com/2004-0914/feature/
http://www.sun.com/bigadmin/features/articles/zfs_part1.s...

etc, etc.

ZFS - horse of a different colour

Posted Jun 21, 2007 0:33 UTC (Thu) by szaka (subscriber, #12740) [Link]

This is not about a race. In fact we are already working implicitely on Linux ZFS support "by" FUSE. One of the major driving forces behind NTFS-3G development is exactly that, to also help ZFS development.

Open source NTFS-3G with FUSE (Linux, FreeBSD), MacFUSE (OS X), ReFUSE (NetBSD) is for interoperability which is used by many, regularly. Most users don't care what's the technology, only if it works (reliably, usable, featureful enough). This includes performance, which would be typically the major theoretical complain if a user space driver were involved (as you also commented).

But the truth is, if it exists at all, far more complex.

The hybrid driver architecture involves overhead but it seems that there are many ways to minimize and offset this to a level where the dominant performance factor will be the design of the file system and the quality of the driver's implementation (used data structures, caches, algorithms, etc running on a specific, speedily changing hardware architecture).

I hope, that at some point NTFS-3G can help to dispel most of the performance related doubts and demonstrates that at least the block device based user space file system drivers can be a viable technical solution used with trust by millions. Only a couple of years work left then hopefully we will be a bit more clever :-)