| |||||||||||||
XFS: the filesystem of the future?XFS: the filesystem of the future?Posted Jan 21, 2012 11:55 UTC (Sat) by and (subscriber, #2883)Parent article: XFS: the filesystem of the future?
> For one or two threads, XFS is still slightly slower than ext4, but it
> scales linearly up to eight threads, while ext4 gets worse, and btrfs gets > a lot worse
I don't know which benchmark result motivated this statement, but the image on the right still shows ext4 being twice as fast as the new XFS, at least for one and two threads...
(Log in to post comments)
XFS: the filesystem of the future? Posted Jan 21, 2012 16:29 UTC (Sat) by hmh (subscriber, #3838) [Link]
The thing is, on any server worth of notice you will have 8 to 32 cores, and mkxfs will usually give you at least 4 aggregation groups per filesystem.
AFAIK, that means you'll be using xfs with at least 4 threads per filesystem on servers. Also, the lower IOPS matters a lot.
Desktops and laptops might well be best served by a different fs, which is fine. Use the best tool for the job, and the tools need not all be the same tool.
XFS: the filesystem of the future? Posted Jan 21, 2012 22:20 UTC (Sat) by csamuel (✭ supporter ✭, #2624) [Link] Maybe I'm misunderstanding, but I always thought you would want as many IOPS as possible from your storage system. Otherwise you're going to bottleneck horribly on some codes (yes badly written Java bioinfomatics codes with your 1 byte synchronous I/O's, I'm looking at you).
XFS: the filesystem of the future? Posted Jan 21, 2012 23:57 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
some software is limited by iops, other software is limited by throughput. you can's say that one is all that matters.
XFS: the filesystem of the future? Posted Jan 22, 2012 10:56 UTC (Sun) by ttonino (subscriber, #4073) [Link]
I think the 'IOPS' in the above graphs are the result of benchmarking. So from 2 to 4 threads, the ext4 benchmark result goes up a little, but the number of IO's hitting the disks explodes tenfold.
Which means that ext4 starts to produce inefficient I/O patterns with multiple threads, while XFS is better at combining the I/O's.
Compare to the CPU load while running a disk benchmark; you want that to be as low as possible, compared to throughput.
XFS: the filesystem of the future? Posted Jan 23, 2012 0:26 UTC (Mon) by dgc (subscriber, #6611) [Link]
Hi hmh,
> The thing is, on any server worth of notice you will have 8 to 32 cores,
The default for non-raid devices (single drives or hardware RAID that does not advertise it's configuration) is 4AGs below 4TB, and then scales at 1AG per TB. I was testing on a 17TB volume, so the default mkfs config gave 17 AGs because the virtio device doesn't pass on RAID alignment from the host.
I even pointed out in the talk some performance artifacts in the distribution plots that were a result of separate threads lock-stepping at times on AG resources, and that increasing the number of AGs solves the problem (and makes XFS even faster!) e.g. at 8 threads, XFS unlink is about 20% faster when I increase the number of AGs from 17 to 32 on teh same test rig.
If you have a workload that has a heavy concurrent metadata modification workload, then increasing the number of AGs might be a good thing. I tend to use 2x the number of CPU cores as a general rule of thumb for such workloads but the best tunings are highly depended on the workload so you should start just by using the defaults. :)
> Desktops and laptops might well be best served by a different fs, which is fine.
Which for me all run XFS.
Indeed, laptops and desktops are signficantly more powerful than you give them credit for. My desktop has 8 CPU threads, 500MB/s of IO bandwidth and can do 70,000 random 4k write IOPS and that cost less than $AU1500 when I bought it a couple of years ago. That's a serious amount of capability at very low cost, yet ext4 struggles to make use of all that capability. XFS, OTOH, is a perfect fit for such configurations, especially if you are running highly parallel applications (like kernel builds) all the time on your desktop....
This is one of the things I'm trying to make people aware of - that high performance (SSD) and large scale storage (4TB drives) are here right now and are affordable on your desktop and laptop. Filesystem concurrency and high throughput is not a supercomputer or high end server problem anymore - what you need from your desktop filesystem to use the maximum potential of your storage hardware is very different from what was needed 5 years ago....
> Use the best tool for the job, and the tools need not all be the same
I couldn't have said it any better myself. :)
Dave.
XFS: the filesystem of the future? Posted Feb 7, 2012 2:14 UTC (Tue) by jmalcolm (guest, #8876) [Link]
This is really one of my favourite things about Open Source. Much of the amazing technology that allows me to get the most out of my modest hardware was originally designed for very high-end workstation and data-center type use. As commodity hardware approaches the capabilities of yesterday's high end the benefits of the software trickle-down to the rest of us.
XFS was not designed for home media server or my development laptop but I sure enjoy it anyway.
XFS: the filesystem of the future? Posted Jan 22, 2012 12:59 UTC (Sun) by dgc (subscriber, #6611) [Link]
Hi and,
Scalability is not about peak single thread throughput - it's about how much of that per-cpu throughput is maintained as concurrency increases. Also, there were more results in the presentation than just those in the article above.
As I explained in the talk, these are simple workloads designed to demonstrate fundamental differences in behaviour between the filesystems. The absolute numbers don't matter - it's the trends in performance that are important. To summarise, XFS shows linear scaling in both throughput and IO patterns from 1 to 8 threads for file creation, traversal and removal. ext4 goes extremely non-linear at the IO level as the thread count increases and that's what causes throughput to suffer. BTRFS has a mix of CPU usage and IO scalability issues - sometimes it is CPU bound, other times it is IO bound like ext4 - and so goes non-linear for different reasons.
FWIW, I didn't run numbers for 16 threads for the presentation simply because I didn't have a week to wait for ext4 and BTRFS to create, traversal an unlink the 400 million files such a workload would have created. If they ran as fast as XFS (roughly 6 hours for the complete 16 thread workloads), then I would have presented them....
Dave.
XFS: the filesystem of the future? Posted Jan 22, 2012 13:30 UTC (Sun) by and (subscriber, #2883) [Link]
Hi Dave,
my comment was not meant as an attack on XFS (I think it's a really nice file system and I even used it for a while), I was just wondering about the IMHO glaring discrepancy between the article's text and the actual results shown.
On a different matter: does XFS still scale (almost) linearly for 16 or 32 threads? if yes, neither XFS nor the VFS layer are probably hitting the scalability wall.
to summarize how I understood you: you recommend using XFS for "big storage" systems, while -- for the time being -- the desktop use-case is still better served by ext4?
XFS: the filesystem of the future? Posted Jan 22, 2012 15:06 UTC (Sun) by dgc (subscriber, #6611) [Link]
> my comment was not meant as an attack on XFS
I didn't take it that way - I was trying to explain what it meant ;)
> (I think it's a really
There's always the problem of context. Jon has done a pretty good job of communicating my message, but it's hard to convey all the context that I put around those charts. IOWs, for explanations of the finer details of the charts you really should watch the video first - I do actually explain that ext4 is faster for 1-2 threads on most of the workloads I presented and I explain why that is the case, too...
That's in the talk, too, and I think Jon mentioned it in the article. ;)
For some raw numbers at 16 threads, XFS performance increases from about 100k file creates/s to about 130k file creates/s, but CPU usage doubles and most of it is wasted spinning on the contended VFS inode and dentry cache LRU locks. I have some patches that I'm working on to minimise that problem.
No, that's exactly the opposite of what I am saying. My point is that even for desktop use cases, XFS is now so close ext4 performance on single threaded metadata intensive workloads that it ext4 has lost the one historical advantage it held over XFS.
The example I use in the talk is untarring a kernel tarball - XFS used to take a minute, ext4 about 3s. That's one of the common "20x slower" workloads that people saw all the time. Now XFS will do that same untar in 4s. And the typical 50x slower workload was then doing a 'rm -rf' on that unpacked kernel tarball. XFS has gone from about a minute down to 3s, compared to 2s for ext4....
While these XFS numbers are still slightly slower from a "benchmark perspective", in practice most people will consider them both to be "instantaneous" because they now both complete faster than the time it takes to type your next command. IOWs, users will notice no practical difference between the performance of the filesystems for common desktop/workstation usage.
Let's now fast forward a few months: your desktop and server system filesystems will be BTRFS, whilst XFS is fast enough and scales well enough for everything else that BTRFS can't be used for. I can't see where ext4 fits into this picture because, AFAIC, XFS is now the better choice for just about every workload you wouldn't use BTRFS for....
So I'm not sure ext4 has a future - ext4 was always intended as a stop-gap measure until BTRFS is ready to take over as the default linux filesystem. This milestone is rapidly approaching, so now is a good time to look at the future of the other filesystems that are typically used on systems. Everyone knows what I think now, so I'm very interested in what users and developers think about what I've said and the questions I've posed. The upcoming LSF workshop could be very interesting. :)
Dave.
|
|||||||||||||