| |||||||||||||
XFS: the filesystem of the future?XFS: the filesystem of the future?Posted Jan 21, 2012 10:45 UTC (Sat) by drag (subscriber, #31333)In reply to: XFS: the filesystem of the future? by jmalcolm Parent article: XFS: the filesystem of the future?
dedupe is overrated.
From what I understand on Solaris is that in order to have a effecient De-dupe you need to be able to maintain a table of the 'deduped' items in RAM. That way when you need to access a file the filesystem knows where the actual bits are located without having to look them up. Something like that.
http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-...
Does not seem to be a significant advantage as in many situations 5GB is considerably more expensive then 1TB of disk space.
The amount of RAM required to keep ZFS happy can be staggering sometimes.
However the kick-ass things that more modern file systems bring are things like online compression, checksum'ng, raid-like features, easy subvolumes. That sort of thing is very nice to have from a administrative, integrity, and availability viewpoint...
(Log in to post comments)
XFS: the filesystem of the future? Posted Jan 21, 2012 18:13 UTC (Sat) by jmalcolm (guest, #8876) [Link]
Indeed. I have also read that deduplication on ZFS is 'broken' but I did not have a link to back that up. Still, technology has a way of improving such that 'resource usage' and maturity concerns of today become unimportant in 'the future'.
I agree that snapshotting/cloning are exciting features of systems like ZFS. The 'time-slider' that Sun added to Nautilus in OpenSolaris invoked quite a lot of jealousy in me. I also thought that Nexenta integrating ZFS into 'apt-get' with 'apt-clone' was simply brilliant. (Note: on Ubuntu, 'apt-clone' is something else)
As a developer, I sometimes do silly thing like build or install a bleeding edge version of an important library which I later regret. I would love to have a simple and seamless way to roll-back the clock or easily hit the save button just before I do something stupid. Version control is great for code repositories but it does not really help me when I mess up my filesystem or install a broken version on an IDE. Not that I do those kinds of things of course...
XFS: the filesystem of the future? Posted Jan 26, 2012 11:53 UTC (Thu) by jospoortvliet (subscriber, #33164) [Link]
Actually, openSUSE and SLE do this using btrfs. It's build into the zypper package manager and additionally has commandline (snapper) and GUI (in YaST) interfaces.
Based on btrfs, a timeslider in a gui filemanager would be possible too, I'm sure, either using btrfs directly or as GUI to snapper (but that'd be (open)SUSE specific unless other distro's pick up on snapper).
XFS: the filesystem of the future? Posted Feb 7, 2012 1:38 UTC (Tue) by jmalcolm (guest, #8876) [Link]
I did not know that about SUSE. Thanks.
XFS: the filesystem of the future? Posted Jan 21, 2012 23:10 UTC (Sat) by cmccabe (guest, #60281) [Link]
Yeah, I agree that dedupe is overrated, for most applications.
Also keep in mind that the more compressed and de-duped your data is, the more likely it is that you'll lose data when there's a hardware problem. Some filesystems, like HDFS, actually write out the data three times or more, which is a kind of anti-deduplication.
XFS: the filesystem of the future? Posted Jan 23, 2012 14:29 UTC (Mon) by jezuch (subscriber, #52988) [Link]
> Yeah, I agree that dedupe is overrated, for most applications.
On the other hand, cp --reflink is quite awesome.
> Also keep in mind that the more compressed and de-duped your data is, the more likely it is that you'll lose data when there's a hardware problem. Some filesystems, like HDFS, actually write out the data three times or more, which is a kind of anti-deduplication.
I guess that native RAID-ing in the filesystem is expected to offset this risk, in any "normal" situation at least.
XFS: the filesystem of the future? Posted Jan 23, 2012 18:48 UTC (Mon) by martinfick (subscriber, #4455) [Link]
It seems unfair to say that dedupe is overrated if you are only basing this on a single implementation (ZFS). There are many ways to dedupe which do not suffer from the same RAM problem (COW comes to mind), and I suspect that many more will be implemented in the future.
Also I suspect that you may not have considered that while RAM indeed is expensive compared to disks, if implemented properly deduping files will actually save RAM when a single file can be cached instead of many. Vserver unification, while not a full featured dedup, does allow for this RAM savings which can be huge in virtualised environments (and more).
XFS: the filesystem of the future? Posted Jan 24, 2012 21:42 UTC (Tue) by wazoox (subscriber, #69624) [Link]
Generally speaking, dedupe is trading CPU and RAM for storage space. It has also the serious drawback of making many sequential IOs random. It probably makes sense when your storage stack is horribly expensive, or when you really need to squeeze out some more bandwidth on a replicated system, etc. However given current hard drives prices (even with the current 50% price hike) and subsystem performance (any 500 bucks RAID card can do 1 GB/s), it's almost always a gain only for the vendor.
XFS: the filesystem of the future? Posted Jan 24, 2012 22:36 UTC (Tue) by khim (subscriber, #9252) [Link] On the other hand dedupe is pretty good fit for SSD. SSD is expensive (albeit less expensive then RAM) and seeks are not as important.
XFS: the filesystem of the future? Posted Jan 25, 2012 7:48 UTC (Wed) by wazoox (subscriber, #69624) [Link]
That's true, but so far dedupe is mostly touted for secondary-level storage, so SSDs are a bit of a stretch.
XFS: the filesystem of the future? Posted Jan 28, 2012 20:36 UTC (Sat) by robbe (guest, #16131) [Link]
> in many situations 5GB is considerably more expensive then 1TB of disk
> space.
How do you figure? This amount of memory sets me back for less than double the cost of disk space. But *only* if looking at ECC RAM versus cheap & big SATA storage. Go to SAS, as used in many servers, and the scale is more like 1:1.
And that's not even considering RAID, where net capacity is not 100%
Of course, in many environments, you need to get RAM and disks from your server vendor, and they mark up prices arbitrarily ... so your numbers can come out completely different.
|
|||||||||||||