| |||||||||||||
XFS: the filesystem of the future?XFS: the filesystem of the future?Posted Jan 20, 2012 20:40 UTC (Fri) by rfunk (subscriber, #4054)Parent article: XFS: the filesystem of the future?
"Reliability is not just a matter of not losing data - hopefully XFS is already good at that"
Heh, having lost an entire filesystem (and zeroed-out files on another filesystem) to XFS a while back, I find that wording grimly amusing. In my experience it really was quite good at losing data.
But seriously, XFS has long had a reputation for being more likely to eat your data than its Linux competition, at least in non-server-room use cases. Have they really fixed that? I mean, yes "hopefully" they have, but it would be good to know beyond just hope before we start switching to it based on speed.
(Log in to post comments)
XFS: the filesystem of the future? Posted Jan 20, 2012 20:55 UTC (Fri) by flashydave (subscriber, #29267) [Link]
The other issue that has bitten me numerous times is a kernel stack overflow when using xfs and NFS together which results in system lock up. On a given production system it occurs every 2 to 4 weeks - enough to be embarassing. There was a developers thread discussing this a while back but I have not heard of any fixes subsequently. This is the one issue that makes me cautious about using xfs in future. Can anyone bring me up to date?
XFS: the filesystem of the future? Posted Jan 21, 2012 20:02 UTC (Sat) by dmcguicken (guest, #57851) [Link]
!
Do you have any other details on this? I have a sneaking suspicion this is EXACTLY what I was seeing on a little VIA box of mine that I blamed on a faulty I/O daughterboard.
I was using a 1/2 TB external USB drive with XFS for my MythTV recordings, shared over the LAN via NFSv4... and I saw hard lockups every few weeks, pretty reliably.
XFS: the filesystem of the future? Posted Jan 23, 2012 10:21 UTC (Mon) by flashydave (subscriber, #29267) [Link]
XFS: the filesystem of the future? Posted Jan 22, 2012 12:04 UTC (Sun) by dgc (subscriber, #6611) [Link]
Hi Flashydave,
If you don't report problems like this to the developers, then we don't know you are having them. The reason the stack overflow fix has not been pushed is that it might have unexpected performance issues due to moving allocation work into workqueues where they have a full stack to work with. Seeing that there has only be a couple of reports of stack overflows on the list in the past 2 years, it doesn't appear to be a widely occurring problem. Hence the urgency for the fix does not appear to be that great and so fixing it can wait until we fully understand the implications of the proposed fix.
IOWs, the frequency or likelihood of occurence of a problem greatly influences decisions on whether to push fixes right now or wait for more testing. So, don't assume that we know how much your systems are affected by the problem even thought we might discussing a possible fix - report them xfs@oss.sgi.com so we are guaranteed to know about them and can take that into account.
As it is, I have been testing the fix for some time now so I'm now pretty confident it doesn't cause any regressions. I'm definitely considering re-prosing it again for the next merge cycle now that I have a lot more testing done on it...
Dave.
XFS: the filesystem of the future? Posted Jan 23, 2012 10:19 UTC (Mon) by flashydave (subscriber, #29267) [Link]
Sorry - I was under the impression from the threads I was reading it was a well documented issue and that it was not particularly helpful to simply say "me too".
XFS: the filesystem of the future? Posted Jan 30, 2012 0:11 UTC (Mon) by sbergman27 (guest, #10767) [Link]
"""
I was under the impression from the threads I was reading it was a well documented issue and that it was not particularly helpful to simply say "me too". """
Nevertheless, it was negligent of you not to jump through all the required hoops to register your bug report. Just because a hundred people before you had already reported it does *not* mean that the devs would necessarily have noted the problem then, or admit to it existing now. Your data loss was your fault. Because you did not bother to file a bug.
FS developers toil day and night, and the only payment they get is fame, glory, and money. Any data loss is the fault of the user.
XFS: the filesystem of the future? Posted Jan 30, 2012 10:59 UTC (Mon) by nix (subscriber, #2304) [Link] FS developers toil day and night, and the only payment they get is fame, glory, and money.What world do you live in, and how can I move there?
XFS: the filesystem of the future? Posted Feb 6, 2012 3:16 UTC (Mon) by chip (subscriber, #8258) [Link]
I gave up on XFS when it started leaving me files of NULs after crashes. I understand they fixed that problem, but until iops are my limiting factor why should I leave ext4?
XFS: the filesystem of the future? Posted Feb 7, 2012 16:26 UTC (Tue) by phoenix (guest, #73532) [Link]
Which version of the kernel, XFS, and NFS?
We use XFS and NFS on ~50 servers, serving 1-2 TB of RAID/LVM to 100-400 diskless stations each, without running into any kernel stack overflows or other lockups.
64-bit Debian 5.0.
If this is something new (since kernel 2.6.32), we'd like to know. :)
XFS: the filesystem of the future? Posted Jan 20, 2012 22:14 UTC (Fri) by ricwheeler (subscriber, #4980) [Link]
I think that you need to put some analysis out there.
Did you have bad storage? Misconfigured write cache? Did you run xfs_repair? Did you (or a vendor) do any analysis of the errors?
In my experience, XFS has been quite reliable and is the most common file system used in many storage appliances.
XFS: the filesystem of the future? Posted Jan 28, 2012 23:12 UTC (Sat) by sbergman27 (guest, #10767) [Link]
"Did you have bad storage? Misconfigured write cache?"
Yes. There must be some way to blame this on the user or his hardware.
What do you mean by "misconfigured write cache"? What special thing is it that you are supposed to do with the drive's write cache to keep XFS from eating your data?
XFS: the filesystem of the future? Posted Jan 29, 2012 0:47 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
if you have a drive that is caching data on ram on the drive and then writing it to the drive later, you will loose data in a power failure.
many drives have been found to lie to the OS abut when the data is saved (they report it being saved when it's only in the cache, not when it's actually saved to the platter)
if you have a drive like this, then you will loose data, no matter what filesystem you use, and even if you use a high-end raid controller.
XFS: the filesystem of the future? Posted Jan 29, 2012 1:36 UTC (Sun) by sbergman27 (guest, #10767) [Link]
"if you have a drive like this, then you will loose data, no matter what filesystem you use, and even if you use a high-end raid controller."
Especially if the filesystem itself is already being cavalier with the data, by holding it in the page/buffer caches as long as it can, and playing russian roulette with "features" like delayed allocation. All in the name of good benchmark numbers.
Drive caches are a small factor in comparison. We didn't really even used to worry, or even thing about them. Now, they seem to be the preferred scapegoat for Linux filesystem devs when data loss occurs. I *know* how reliable things were before we had barriers and FUA. And I know what I'm seeing now. With all due respect, I'm just not buying this explanation.
XFS: the filesystem of the future? Posted Jan 30, 2012 17:35 UTC (Mon) by Otus (guest, #67685) [Link]
> Drive caches are a small factor in comparison. We didn't really even used to worry, or even thing about them.
They used to be small. They've grown approximately at the same speed as HDD sizes, which is significantly faster than throughput, not to mention seek time.
A consumer HDD from c. 2000 might have a 2 MB cache and 40 MB/s throughput, so a full cache empties from sequential data in 50 ms best case. Current 2-3 TB drives have a 64 MB cache and 100-150 MB/s throughput, so a full cache takes around 500 ms minimum to empty.
For non-sequential data it's much worse.
XFS: the filesystem of the future? Posted Feb 2, 2012 16:57 UTC (Thu) by jd (guest, #26381) [Link]
Cache would not be a problem if:
(a) it was battery-backed, and
Battery-backed doesn't have to mean the whole drive has to remain powered-up, it just has to mean the DRAM gets enough juice to keep refreshing until regular power is restored *if* there is any unwritten content in it. In other words, if everything is flushed to disk then you don't need to keep the drive's RAM powered. If drive manufacturers were *really* clever, then only those blocks of RAM with unflushed content would need to remain powered.
It's hard to get a frame of reference, as most devices with RAM and a modern LIon battery also have a power-hungry CPU and an even hungrier RF system to feed. Here, you only need to keep selective RAM chips powered, no processing is required. I have absolutely no idea what kind of leakage of charge good batteries suffer, but it is probably small. Just keeping DRAM alive doesn't take a vast amount of power. This solution should be adequate to handle even Katrina-length power outages. Beyond that, disk corruption is unlikely to be your major concern.
XFS: the filesystem of the future? Posted Feb 5, 2012 19:45 UTC (Sun) by rilder (subscriber, #59804) [Link]
If you need data integrity (like database commits) you will need to enforce it from application with fsync etc or have a SSD which provides such guarantees.
Speaking of write cache, the directive is to disable it on disk if you have another battery backed write cache sitting behind, or speaking of laptops you can have them since they are battery backed, no one says to disable cache completely.
You can start reading about it here -- http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_problem_w...
Speaking of caching in page cache, it is just to provide better I/O locality as mentioned in talk, it is flushed periodically, if it is flushed as and when required you will end up with seek nightmare with disk.
XFS: the filesystem of the future? Posted Feb 6, 2012 2:52 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
> If you need data integrity (like database commits) you will need to enforce it from application with fsync etc or have a SSD which provides such guarantees.
having a SSD or battery backed cache does not replace doing fsyncs. If you don't do the fsync you don't know that the data is being written from the OS cache to the disk subsystem.
Shared pain Posted Jan 20, 2012 22:16 UTC (Fri) by gwolf (subscriber, #14632) [Link]
I was also an XFS enthusiast a while back (~5 years). In my experience, XFS does not lose data under normal use, but a hardware crash (power outage, etc.) leaves many zeroed files. What I read back then is that the filesystem was sure to be coherent (the structure would never be compromised), but the data itself was left for later, whenever there was time to flush the caches.
Shared pain Posted Jan 20, 2012 22:59 UTC (Fri) by zlynx (subscriber, #2285) [Link]
EXT4 acts the same way, creating files with zeros when run in writeback journaling mode. I run mine this way, although I do make sure that auto_da_alloc is turned on so that data is flushed when doing file replacement via rename.
I'd much rather have the performance.
Shared pain Posted Jan 21, 2012 12:45 UTC (Sat) by dany (guest, #18902) [Link]
> I'd much rather have the performance.
Its ok that you would, but would also your employer/customers like better performance over reliability? There is reason, that default ext4 mount mode in RHEL is ordered.
Shared pain Posted Jan 21, 2012 17:15 UTC (Sat) by ricwheeler (subscriber, #4980) [Link]
You don't need to give up reliability for performance in either ext4 or xfs.
Eric and I are both with the Red Hat file system team (as is Dave Chinner) and we would not be supporting XFS if it was not solid and reliable as well as high performance.
What you do need, as Eric mentioned, is to keep your box properly configured and to have applications that do the right things to persist data.
Jeff Moyer (also a Red Hat file system person) wrote up a nice article for LWN a few months back on best practices for data integrity.
Shared pain - article link Posted Jan 22, 2012 21:01 UTC (Sun) by ndye (subscriber, #9947) [Link] Jeff Moyer (also a Red Hat file system person) wrote up a nice article for LWN a few months back on best practices for data integrity. This article, I presume?
Shared pain Posted Jan 28, 2012 23:16 UTC (Sat) by sbergman27 (guest, #10767) [Link]
So you can either use Ext4 mounted with the nodellalloc option, or Ext3 mounted data=ordered, and sleep well at night. Or... you can use XFS and commission a code audit for every piece of important software you run, specifically checking for proper fsync usage. Cross your fingers, hoping the auditors didn't miss anything, and try to sleep well at night.
Shared pain Posted Jan 29, 2012 0:44 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
you may sleep well at night, but it will be the sleep of someone who has been fooled about the reliability of their data.
skipping fsync is not safe on any filesystem that's not mounted -sync
this is true for every OS
Shared pain Posted Jan 29, 2012 1:53 UTC (Sun) by sbergman27 (guest, #10767) [Link]
And again, you are glossing over the matter of the relative likelihood of data loss with various filesystems. It suits your purposes to turn it into a black and white issue. "Ext3 can lose your data too!", you cry.
Well, Both my mattress and J.P. Morgan *could* lose my money. So putting my money either place represents equal risk. If I understand you correctly, you are saying that Ext3 mounted data=ordered, Ext4 mounted with the defaults, and XFS mounted with the defaults, all represent equal risk to our data because any one of them *could* conceivably lose our data.
Again, I'm not buying it.
Shared pain Posted Jan 30, 2012 21:49 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
replying to multiple comments in one reply
I am not saying that the risk is equal, I am disputing the statement that ext3 is rock solid and won't loose your data without you needing to do anything.
Ext3 is one of the worst possible filesystems to use if you really care about your data not getting lost (and therefor implement the fsync dance to make sure you don't loose data), because it's fsync performance is so horrid.
The applications are not keeping the data in buffers 'as long as they can', they are keeping the data in buffers for long enough to be able to optimize disk activity.
The first is foolishly risking data for no benefit, the second is taking a risk for a direct benefit. These are very different things. Ext3 also keeps data in buffers and delays writing it out in the name of performance. Every filesystem available on every OS does so by default, the may differ in how long they will buffer the data, and what order they write things out, but they all buffer the data unless you explicitly mount the filesystem with the sync option to tell it not to.
You say that people will always pick reliability over performance, but time and time again this is shown to not be the case. As pointed out by another poster, MySQL grew almost entirely on it's "performance over reliability" approach, only after it was huge did they start pushing reliability. The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided.
This extends beyond the computing field. There are no fields where the objective is to eliminate every risk, no matter what the cost. It is always a matter of balancing the risk with the costs of eliminating them, or with the benefits of accepting the risk.
That's the problem... Posted Jan 30, 2012 22:54 UTC (Mon) by khim (subscriber, #9252) [Link] In theory, theory and practice are the same. In practice, they are not. Ext3 is one of the worst possible filesystems to use if you really care about your data not getting lost (and therefor implement the fsync dance to make sure you don't loose data), because it's fsync performance is so horrid. Right, but that's the problem: most developers don't care about these things (most just don't think about the problem at all, others just hope it all will work... somehow). Most users do. Thus we have a strange fact: in theory ext3 is the worst possible FS from "lost data" POV, in practice it's one of the best.
That's the problem... Posted Jan 30, 2012 23:16 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
trust me, users trying to run high performance software that implements data safety (databases, mail servers, etc) care about this problem as well.
For other developers, the fact that fsync performance is so horrible on the default filesystem for many distros has trained a generation of programmers to NOT use fsync (because it kills performance in ways that users complain about)
That's the problem... Posted Feb 2, 2012 3:45 UTC (Thu) by tconnors (guest, #60528) [Link]
> For other developers, the fact that fsync performance is so horrible on the default filesystem for many distros has trained a generation of programmers to NOT use fsync (because it kills performance in ways that users complain about)
Then there's the fact that fsync will spin up your disks if you were trying to keep them spun down (to the point where on laptops, I try to use 30 minute journal commit times, and manually invoke sync when I absolutely want something committed). I don't want or need an absolute gaurantee that the new file has hit the disk consistent with metadata. I want an absolute guarantee that /either/ the new file or the old file is there consistent with the relevant metadata. ext3 did this. It's damn obvious what rename() means - there should be no need for every developer to go through all code in existance and change semantics of code that used to work well *in practice*. XFS loses files everytime power fails *in practice*. If I need to compare to backup *everytime* power fails, then I might as well be writing all my data to volatile RAM and do away with spinning rust all together, because that's all that XFS is good for.
Another pathological (but instructive) case... Posted Jan 30, 2012 23:03 UTC (Mon) by khim (subscriber, #9252) [Link] Similar story happens with USB sticks: most users believe FAT (on Windows) is super-safe, NTFS (on Windows) is horrible - and Linux is awful no matter what. Why? Delayed write defaults. FAT on Windows is tuned to flush everything on ANY close(2) call. NTFS works awfully slow in this mode thus it uses more aggressive caching. And on Linux caching is always on. And users just snatch USB stick the very millisecond program window is closed (well... most do... the cautious ones wait one or two seconds). They feel it's their unalienable right. In these circumstances suddenly the oldest and the most awful filesystem of them all becomes the clear winner! Exactly because "I care about my data" does not automatically imply "I'll do what I'm told to do to keep it".
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 4, 2012 13:08 UTC (Sat) by Wol (guest, #4433) [Link]
Don't get me started ... :-)
I work with Pick (a noSQL db), and a LOT of the reliability problems that ACID is meant to fix, just *can't* *happen* in Pick.
Well, they can if the database was badly designed, but you can get similar pain in relational databases too...
Relational is a lovely mathematical design - I would use it to design a database without a second thought - but I would then convert that design to NF2 (non-first-normal-form) and implement it in Pick. Because 90% of ACID's benefits would then be redundant, and the database would be so much faster too.
You've heard my war-story of a Pentium90/Pick combo outperforming an Oracle/twinXeon800, I'm sure ...
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 4, 2012 13:27 UTC (Sat) by gioele (subscriber, #61675) [Link]
> I work with Pick (a noSQL db), and a LOT of the reliability problems that ACID is meant to fix, just *can't* *happen* in Pick.
This is getting off-topic, but could you explain which kind of reliability problems meant to be fixed by ACID DBs are cannot happen in Pick and why?
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 7, 2012 23:20 UTC (Tue) by Wol (guest, #4433) [Link]
Okay, let's do a data analysis. In Pick, you would do an EAR.
Look for what I call "real world primary keys" - an invoice has a number, a person has a name. Now we've got a primary key, we work out all the attributes that belong to that key. We can now do a relational analysis on those attributes. (forget that real-world primary keys aren't always unique and you might have to create a GUID etc.)
With an invoice, in relational you'll end up with a bunch of rows spread across several tables for each invoice. IN PRACTICE, ACID is mostly used to make sure all those rows pass successfully through the database from application to disk and back again.
In Pick, however, you then coalesce all those tables (2-dimensional arrays) together into one n-dimensional Pick FILE. And you coalesce all those rows together into one Pick RECORD. With the result that there is no need for the database to make sure the transaction is atomic. All the data is held as a single atom in the application, and is passed through the database to and from disk as a single atom.
That's also why Pick smokes relational for speed - access any attribute of an object, and all attributes get pulled into cache. Try doing that for a complex object with relational !!! :-) (Plus Pick is self-optimising, and when optimised it takes, on average, just over one disk seek per primary key to find the data it's looking for on disk!)
The problem I see with relational is it is MATHS and, to reference Einstein, therefore has no basis in reality. Pick is based on solid engineering, and when "helped" with relational theory really flies. Relational practice actually FORBIDS a lot of powerful optimising techniques.
And if designed properly, a Pick database is normalised therefore it can look like a relational database, only superfast. I always compare Pick and relational to C and Pascal. Pick and C give you all the rope you need to seriously shoot yourself in the foot. Relational and Pascal have so many safety catches, it's damn hard to actually do any real work.
(And because foreign keys are attributes of a primary key, you can also trap errors in the application. For example, the client's key is a mandatory element of an invoice so it belongs in the invoice. Okay, it's the app's job to make sure the record isn't filed without a client, whereas in relational you can leave it to the DB, but in Pick it's easy enough to add a business layer between the app and the DB that does this sort of thing.)
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 7, 2012 23:26 UTC (Tue) by Wol (guest, #4433) [Link]
Just to add, look at it this way ...
In relational, attributes that are tied together can be spread (indeed, for a complex object MUST be spread) across multiple tables and rows. Column X in table Y is meaningless without column A in table B.
In Pick, that data would share the same primary key, and would be stored in one FILE, in one RECORD. Delete the primary key and both cells vanish together. Create the primary key, and both cells appear waiting to be filled.
As far as Pick is concerned, a RECORD is a single atom to be passed through from disk to app and vice versa. From what I can make out, in relational you can't even guarantee a row is a single atom!
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 0:45 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
this has nothing to do with the ACID guarantees. the ACID guarantees have to do when you start modifying the datastore, specifically what happens if the modification doesn't complete (including that the system crashes in the middle of an update)
ACID is the sequence of modifying the file on disk so that you have either the new data or the old data at all times, and if the application says that the transaction is done, there's no way for it to disappear.
what you are talking about with Pick is a way to bundle related things together so that it's easier to be consistent. That doesn't mean that writing your record will happen in an atomic manner on the filesystem (if the record is large enough, this won't be an atomic action)
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 14:52 UTC (Wed) by Wol (guest, #4433) [Link]
So ACID is there to cope with the fact that, what the DB sees as a single transaction, the operating system and filestore doesn't, so it sits between the database and OS, and makes sure that the multiple success/fails returned by the OS are returned to the database as a single success or fail.
AND THAT IS MY POINT. In Pick, 90% of the time, this is unnecessary and wasteful, because what is a single transaction as seen by the DB is also a single transaction as seen by the OS and disk subsystem!
I agree with you, if the record is too big, it won't go onto the file store in one piece, but the point is that with a relational DB you can pretty much guarantee that, in practice, a transaction will never go onto the filestore in one piece. So ACID is needed. But in Pick it's unusual for it NOT to go on the filesystem in one piece. So most of the time ACID is an unnecessary complexity.
I'm ignoring file system failures like XFS/ext4 zeroing out your table in a crash :-) because I don't see how ACID can protect against the OS trashing your data :-)
What you need to do is realise that ACID sits between the database and disk. As you say, it guarantees that the database in memory is (a) consistent, and (b) accurately represented on disk. And because, *in* *the* *real* *world*, pretty much any change in a relational database requires multiple changes in multiple places on disk, ACID is a necessity.
But in the real world, most changes in a Pick database only involve a *single* change in a *single* place on disk to ensure consistency. So a "write successful" from the OS is all that's needed to provide a "good enough" implementation of ACID. (And if the OS lies to your ACID layer, you're sunk even if you've got ACID. See all the other posts in this thread about disks lying to the OS!)
(This has other side effects. Yes, a Pick database can get into an inconsistent state. But that inconsistent state MIRRORS REALITY. A Pick database can lose the connection between a person and his house. Or a car and its owner. But in reality a person can lose their house. A car can lose its owner. It's far too easy to assume in a relational database that everyone has a home, and next thing you can't put some poor vagrant in your database when you need to ...)
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 22:17 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
you don't seem to understand that writes to the filesystem are not atomic in just about every case, let alone dealing with the rest of ACID
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 9, 2012 0:17 UTC (Thu) by Wol (guest, #4433) [Link]
Writes to the file system where? At the db/OS interface? At the OS/disk interface?
Because if it's at the OS/disk interface, what the heck is ACID doing in the database? It can't provide ANY guarantees, because it's too remote from the action.
And if it's at the db/OS interface, well as far as Pick is concerned, most transactions are near-enough atomic that the overhead isn't worth the cost (that was my comment about "90% of the time").
Your relational bias is clouding your thinking (although Pick might be clouding mine :-) But just because relational cannot do atomic transactions to disk doesn't mean Pick can't. As far as Pick is concerned, that transaction is atomic right up to the point that the OS code actually puts the data onto the disk. And if the OS screws that up, ACID isn't going to save you ...
Think of a "begin transaction" / "end transaction" pair. It's almost impossible for that transaction to truly be atomic in a relational database - you will invariably need to update multiple rows. In Pick, it's more than possible for that transaction to be truly atomic at the point where the db hands it over to the OS. ACID enforces atomicity between the OS and the db. Pick doesn't need it.
What guarantees does ACID provide over and above data consistency? Because a well-designed Pick app guarantees "if it's there it's consistent". And if the OS screws up and corrupts it, neither Pick nor ACID will save you.
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 9, 2012 0:54 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
ACID has nothing to do with relational algebra
ACID is a feature that SQL databases have had, but you don't need to abandon SQL to abandon ACID and you don't need to have SQL to have ACID
Berkeley DB is ACID, but not SQL, MySQL was SQL but not ACID with the default table types for many years.
ACID involves the database application doing a lot of stuff to provide the ACID guarantees to users by using the features of the OS and hardware. If the OS/hardware lies to the database application about when something is actually completed then the database cannot provide ACID guarantees.
It appears that you have an odd interpretation about what ACID means, so reviewing
Atomicity
A transaction is either completely implemented or not implemented at all. For changes to a single record this is relatively easy to do, but if a transaction involves changing multiple records (subtract $10 from account A and add $10 to account B) it's not as simple as atomically writing one record. Remember that even a single write() call in C is not guaranteed to be atomic (it's not even guaranteed to succeed fully, you may be able to write part of it and not other parts)
Consistency
this says that at any point in time the database will be consistent, by whatever rules the database chooses to enforce. Berkeley DB has very trivial consistency checks, the records must all be complete. Many SQL databases have far more complex consistency requirements (foreign keys, triggers, etc)
Isolation
This says that one transaction can affect another transaction happening at the same time
Durability
This says that once a transaction is reported to succeed then nothing, including a system crash at that instant (but excluding something writing over the file on disk) will cause the transaction to be lost
What you are describing about Pick makes me thing that it has very loose consistency and isolation requirements, but to get Atomicity and Durability the database needs to be very careful about how it writes changes.
It cannot overwrite an existing record (because the write may not complete), and it must issue appropriate system calls (fsync and similar) to the OS, and watch for the appropriate results, to know when the data has actually been written to disk and will not change.
It's getting this last part done that really differentiates similar database engines from each other. There are many approaches to doing this and they all have their performance trade-offs. If you are willing to risk your data by relaxing these requirements a database becomes trivial to implement and is faster by several orders of magnitude.
note how the only SQL concept that is involved here is the concept of a transaction in changing the data.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 9, 2012 20:22 UTC (Thu) by Wol (guest, #4433) [Link]
Yup. I am being far looser in my requirements for ACID for Pick, but the reason is that Pick is far more ACID by accident than relational.
Atomic: as I said, a relational transaction in relational will pretty much inevitably be split across multiple, often many, tables. In Pick, all dependant attributes (excluding foreign-key links) will be updated as a single transaction right down to the file-system layer. So, as an example, if I have separate FILEs for people and buildings, it's possible I'll corrupt "where someone lives" if I update the person and fail to create the building, but I won't have inconsistent person or building data.
Consistency: IF designed properly, a Pick database should be consistent within entities. All data associated with an individual "real world primary key". Relations between entities could get corrupted, but that *should* be solved with good programming practice - in my example above, "lives at" is an attribute of person, so you update building then person.
Isolation: I don't quite understand that, so I won't comment.
Durability: Well, when I tried to write a Pick engine, my first reaction to actually writing FILEs to disk was "copy on write seems pretty easy...". And there comes a point where you have to take the OS on trust.
So I think my premise still stands - a LOT of the requirement for ACID is actually *caused* by the rigorous separation demanded by relational between the application and the database. By allowing the application to know about (and work with) the underlying database structure you can get all the advantages of relational's rigorous analysis, all the advantages of a strong ACID setup, and all the advantages of noSQL's speed. But it depends on having decent programmers (cue my previous comment about Pick and C giving you all the rope you need ...)
And one of the reasons I wanted to write that Pick db engine was so I could put in - as *optional* components - loads of stuff that enforced relational constraints to try and reign in the less-competent programmers! I want a Modula-2 sort of Pick, that by default protects you from yourself, but where the protections can be turned off.
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 9, 2012 20:36 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
atomic, your scheme won't work if you need to make changes to two records (the ever popular "subtract $10 from account A, add $10 to account B" example)
consistency, what if part of your updates get to disk and other parts don't? what if the OS (or drive) re-orders your updates so that the write to the record for person happens before the write to building?
As far as durability goes, if you don't tell the OS to flush it's buffers (which is what fsync does), then in a crash you have no idea what may have made it to disk and what didn't.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 10, 2012 16:17 UTC (Fri) by Wol (guest, #4433) [Link]
The ever popular "subtract $10, add $10" ...
Well, if you define the transaction as an entity, then it gets written to its own FILE. If the system crashes then you get a discrepancy that will show up in an audit. It makes sense to define it as an entity - it has its own "primary key" ie "time X at teller Y". Okay, you'll argue that I have to run an integrity check after a crash (true) while you don't, but I can probably integrity-check the entire database in the time it takes you to scan one big table :-)
Consistency? Journalling a transaction? Easily done.
And yes, your point about flushing buffers is good, but that really should be the OS's problem, not the app (database) sitting on top. Yes I know, I used the word *should* ...
Look at it from an economic standpoint :-) If my database (on equivalent hardware) is ten times faster than yours, and I can run an integrity check after a crash without impinging on my users, and I can guarantee to repair my database in hours, which is the economic choice?
Marketing 101 - proudly announce your weaknesses as a strength. The chances of a crash occuring at the "wrong moment" and corrupting your database are much higher with SQL, because any given task will typically require between 10s and 100s more transactions between the db and OS than Pick. So SQL needs ACID. With Pick, the chances of a crash happening at the wrong moment and corrupting data are much, much lower. So expensive strong ACID actually has a prohibitive cost. Especially if you can get 90% of the benefits for 10% of the effort.
I'm not saying ACID isn't a good thing. It's just that the cost/benefit equation for Pick says strong ACID isn't worth it - because the benefits are just SO much less. (Like query optimisers. Pick doesn't have an optimiser because it's pretty much a dead cert the optimser will save less than it costs!)
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 10, 2012 18:43 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
so that means that you don't have any value anywhere in your database that says "this is the amount of money in account A", instead you have to search all transactions by all tellers to find out how much money is in account A
that doesn't sound like a performance win to me.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 11, 2012 2:30 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]
Well, git works exactly the same way. Is it fast enough for you?
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 11, 2012 5:48 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
what gives you reasonable performance for a version control system with a few updates per minute is nowhere close to being reasonable for something that measures it's transaction rate in thousands per second.
besides, git tends to keep the most recent version of a file uncompressed, it's only when the files are combined into packs that things need to be reconstructed, and even there git only lets the chains get so long.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 11, 2012 13:44 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]
git/svn/... use store intermediate versions of the source code, so that applying all patches becomes O(log N) instead of O(N). But that's just an optimization.
NoSQL systems work in a similar way - they can store the 'tip' of the data, so that they don't have to reapply all the patches all the time. However, the latest data view can be rebuilt if required.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 12, 2012 15:57 UTC (Sun) by nix (subscriber, #2304) [Link]
Actually, even the most recent stuff is compressed. It just might not be deltified in terms of other blobs (which is what you meant, I know).
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 12, 2012 18:29 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
yes, everything stored in git is compressed, but it only gets deltafied when it gets packed.
and it's frequently faster to read a compressed file and uncompress it than it is to read the uncompressed equivalent (especially for highly compressible text like code or logs), I've done benchmarks on this within the last year or so
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 12, 2012 13:38 UTC (Sun) by Wol (guest, #4433) [Link]
Okay, it would need a little bit of coding, but I'd do the following ...
Each month, when you run end-of-month statements, you save that info. When you upate an account you keep a running total.
If the system crashes you then do "set corruptaccout = true where last-month plus transactions-this-month does not equal running balance". At which point you can do a brute force integrity check on those accunts.
(If I've got a 3rd state of that flag, undefined, I can even bring my database back on line immediately I've run a "set corruptaccount to undefined" command!)
And in Pick, that query will FLY! If I've got a massive terabyte database that's crashed, it's quite likely going to take a couple of hours to reboot the OS (I just rebooted our server at work - 15-20 mins to come up including disk checks etc). What's another hour running an integrity check on the data? And I can bring my database back on line immediately that query (and others like it) have completed. Tough luck on the customer who's account has been locked ... but 99% of my customers can have normal service resume quickly.
Thing is, I now *know* after a crash that my data is safe, I'm not trusting the database company and the hardware. And if my system is so much faster than yours, once the system is back I can clear the backlog faster than you can. Plus, even if ACID saves your data, I've got so much less data in flight and at risk.
But this seems to be mirroring the other debate :-) the moan about "fsync and rename" was that fsync was guaranteeing (at major cost) far more than necessary. The programmer wanted consistency, but the only way he could get it was to use fsync, which charged a high price for durability. If I really need ACID I can use BEGIN/END TRANSACTION in Pick. But 99% of the time I don't need it, and can get 90% of its benefits with 10% of its cost, just by being careful about how I program. At the end of the day, Pick gives me moderate ACID pretty much by default. Why should I have to pay the (high) price for strong ACID when 90% of the time, it is of no benefit whatsoever? (And how many SQL programmers actually use BEGIN/END TRANSACTION, even when they should?)
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 14:08 UTC (Wed) by nix (subscriber, #2304) [Link] From what I can make out, in relational you can't even guarantee a row is a single atom!Well, the relational algebra does not discuss storage at all, and does not stipulate where relations might reside on permanent storage (nor *which* might: you could perfectly well store join results permanently for all it cares). But in practice, in SQL... just try INSERTing half a row. You can't. Atomicity at the row level is guaranteed. I hate SQL, but at least it does this right.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 15:00 UTC (Wed) by Wol (guest, #4433) [Link] Well, the relational algebra does not discuss storage at all, and does not stipulate where relations might reside on permanent storage Which is exactly my beef with relational databases. C&D FORBID you from telling the database where relations should be stored for efficiency. But in REALITY it is highly probable that, if you access one attributed associated with my primary key, that you will want to access others. But it's a complete gamble retrieving the same attribute associated with other primary keys. Because Pick guarantees (by accident, admittedly) that all attributes are stored in the same atom as the primary key they describe, all those attributes you are statistically most likely to want are exactly the attributes that coincidentally get retrieved together.
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 15:05 UTC (Wed) by Wol (guest, #4433) [Link]
Atomicity at the row level IN THE DATABASE is guaranteed, yes.
What I meant was it's not guaranteed at the physical level in the datastore. Two cells in the same row could be stored in completely different "buckets" in the database, for example the data is stored in an index with a pointer from the row. I know that probably doesn't happen but if the guy who designed the database engine thinks it's more efficient there's nothing stopping him.
So even if you the database programmer *think* an operation should be atomic right down to the disk, there's no guarantee.
Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 8, 2012 21:53 UTC (Wed) by nix (subscriber, #2304) [Link]
It happens quite a lot, increasingly often now that databases are lifting the horrible restrictions many of them had on the total amount of data stored per row (Oracle and MySQL had limits low enough that you could hit them in real systems quite easily).
If it matters that data is written to the disk atomically, you have already lost, because *nothing* is written to the disk atomically, not least because you invariably have to update metadata, and secondly because no disk will guarantee what happens to partial writes in the case of power failure. So, as long as you have to keep a journal or a writeahead log to deal with that, why not allow arbitrarily large amounts of data to appear to be written atomically? Hence, transactions.
It is true that programs that truly use transactions are relatively rare: in one of my least proud moments I accidentally changed the rollback operation in one fairly major financial system to do a commit and it was a year before anyone noticed. However, when you *do* have code that uses transactions, the effect on the code complexity and volume can be dramatic. As a completely random example, I've written backtracking searchers that relied on rollback in about 200 lines before, because I could rely on the database's transaction system to do nearly all the work for me.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 9, 2012 20:30 UTC (Thu) by Wol (guest, #4433) [Link] It happens quite a lot, increasingly often now that databases are lifting the horrible restrictions many of them had on the total amount of data stored per row (Oracle and MySQL had limits low enough that you could hit them in real systems quite easily). Sorry, I have to laugh here. It's taken Pick quite a while to get rid of the 32K limit, but that limit does date from the age of the dinosaur when computers typically came with 4K of core ... And no limit on the size of individual items, or the number of items in a FILE. Cheers,
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 9, 2012 20:38 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
if a single item is larger than the track size of a drive, it is physically impossible for the write to be atomic. You don't need to get this large to run in to problems though, any write larger than a block runs the possibility of being split across different tracks (or in a RAID setup, across different drives). If you don't tell the filesystem that you care about this, the filesystem will write these blocks in whatever order is most efficient for it.
The entire noSQL family of servers is based on relaxing the reliability constraints of the classic ACID protections that SQL databases provided. Posted Feb 10, 2012 16:25 UTC (Fri) by Wol (guest, #4433) [Link]
:-)
Look at the comment you're replying to :-) In early Pick systems I believe it was possible for a single item to be larger than available memory ...
Okay, it laid the original systems wide open to serious problems if something went wrong, but as far as users were concerned Pick systems didn't have disk. It was just "permanent memory". And Pick was designed to "store all its data in ram and treat the disk as a huge virtual memory". I believe they usually got round any problem by flushing changes from core to disk as fast as possible, so in a crash they could just restore state from disk.
Cheers,
Shared pain Posted Feb 2, 2012 1:06 UTC (Thu) by darrint (guest, #673) [Link]
"Well, Both my mattress and J.P. Morgan *could* lose my money. So putting my money either place represents equal risk."
I'm not sure how that metaphor is supposed to work. A few years ago when the U.S. government passed new credit reforms they were time delayed by several months at the request of the targeted banks, supposedly to give them time to carefully update their big and very important computer systems. The reality was the banks used those several months to burn and pillage the assets of people like me. I was in debt, my own stupidty of course, and I probably lost over a thousand dollars in fees alone due to a few financial institutions playing shenanigans with the ragged edges of my account terms.
At the time I thought wistfully of how much more secure my money would be if I could just collect my pay in cash and drive it to my house.
Shared pain Posted Feb 2, 2012 14:08 UTC (Thu) by rwmj (subscriber, #5474) [Link]
You were in debt ...
Does your house loan you money? What does this negative money look like that you keep under your mattress?
Shared pain Posted Jan 29, 2012 2:02 UTC (Sun) by sbergman27 (guest, #10767) [Link]
"I'd much rather have the performance."
Why? And have you actually measured the "performance" that you are sacrificing reliability for? I'd certainly want to quantify what I was getting in return for the reduced reliability. In testing I've done, I couldn't tell the difference between data=ordered and data=writeback with ext4. Or for that matter, between default and nodelalloc.
Even for a single user desktop, I just don't see that the trade-off is a win. YMMV, I suppose. But I would encourage you to run some objective tests.
Shared pain Posted Feb 2, 2012 2:44 UTC (Thu) by tconnors (guest, #60528) [Link]
> I'd much rather have the performance.
Really? Why bother writing your data to disk at all then? RAM is *super* fast! Me, when I call rename(), I damn well expect either the previous file or the current version has hit the platter and is consistent with metadata. ext3 does this, and ext4 does this with some tweaks. rename() has been implied by programmers for generations to mean an atomic barrier.
XFS has never done this. That is why I don't use XFS. Because they have a damn stubborn following that insists that the perfectly reasonable semantic of close();rename(); is "wrong wrong wrongity wrong, burn you evil data hater!"
Shared pain Posted Feb 2, 2012 22:13 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
no, rename is an atomic barrier from the point of view of your software on the running machine.
however if the filesystem does not get unmounted cleanly, all guarantees are off. This has always been the case in Unix.
Shared pain Posted Feb 2, 2012 23:27 UTC (Thu) by khim (subscriber, #9252) [Link] Unix is dead, sorry. On Linux you have filesystems which was was unreliable against crashes at all (ext, ext2, etc) or which guarantee atomicity across reboots (ext3, ext4, btrfs). Oh, and there are XFS, too - looks like it's developers finally understood that filesystems exist to support applications, not the other way around (even if XFS fanbois didn't) P.S. Yes, ext4 and btrfs also had the problem under discussion. But they were quickly fixed.
Shared pain Posted Feb 2, 2012 22:50 UTC (Thu) by dgc (subscriber, #6611) [Link]
> rename() has been implied by programmers for generations to mean an
> atomic barrier
Not true. rename is atomic, but it is not a barrier and never has implied that one exists. rename() has been around for 3 times longer than ext3, so I don't really see how ext3 behaviour can possibly be what generations of programmers expect to see....
Indeed, ext3 has unique rename behaviour as a side effect of data=ordered mode - it flushes the data before flushing the metadata, and so appears to give rename "barrier" semantics. It's the exception, not the rule.
> XFS has never done this. That is why I don't use XFS.
Using data=writeback mode on ext3 makes it behave just like XFS. So ext3 is just as bad as XFS - you shouldn't use ext3 either! :P
> they have a damn stubborn following that insists that the perfectly
That's a bit harsh.
There's many good reasons for not doing this - lots of applications don't need or want barrier semantics to rename, or are cross platform and can't rely on implementation specific behaviours for data safety. e.g. rsync() is a heavy user of rename, but adding barrier semantics to the way it uses rename would slow it down substantially. Further, rsync doesn't need barrier semantics to guarantee that data has been copied and safely overwritten - it's written to be safe with current rename behaviour because it is both operating system and filesystem independent.
There have also been good arguments put forward for making this change, such as from Val Aurora (who I also quoted in my talk):
http://lwn.net/Articles/351422/
However, no-one has ever followed up on such discussions with patches to the VFS to make this a standard behaviour that you can rely on all linux filesystems to support. I'm certainly not opposed to such changes if the consensus is that this is what we should be doing - I might argue to maintain the status quo (e.g. because rsync performance is extremely important for backups on large filesystems) but that doesn't mean I don't see or understand the benefits of such a change.
Indeed, adding a new rename syscall with the desired semantics rather than changing the existing one is a compromise everyone would agree with. Perhaps you could do write patches to propose this seeing as you seem to care about such things?
Dave.
Shared pain Posted Feb 2, 2012 23:41 UTC (Thu) by khim (subscriber, #9252) [Link] rename is atomic, but it is not a barrier and never has implied that one exists. rename() has been around for 3 times longer than ext3, so I don't really see how ext3 behaviour can possibly be what generations of programmers expect to see.... Easy: most currently active programmers never seen a Unix with journalling FS and without the ability to safely use rename across reboots. Actually they very much insist on such ability - and looks like XFS developers try to provide the capability. But it's not clear if you can trust them: clearly they value POSIX compatibility and benchmarks more then needs of real users (who need working applications, after all, filesystem needs are just minor implementation detail for them). Indeed, ext3 has unique rename behaviour as a side effect of data=ordered mode - it flushes the data before flushing the metadata, and so appears to give rename "barrier" semantics. It's the exception, not the rule. When "exception" happens in 90% cases it becomes a new rule - it's as simple as that. However, no-one has ever followed up on such discussions with patches to the VFS to make this a standard behaviour that you can rely on all linux filesystems to support. That's because we already have a solution: don't use XFS and you are golden. OSes exist to support application - as you've succinctly shown above with rsync example. The only problem: I'm not all that concerned with rsync speed. I need mundane things: fast compilation (solved with gobs of RAM and SSD, filesystem is minor issue after that), reliable work with bunch of desktop applications (which don't issue fsync(2) before rename(2), obviously). Since I already have a solution I don't see why I should push the patches. If you want to advocate xfs - then you must fix it's problems, I'm happy with ext3/ext4 (which may contain bugs but which at least don't try to play "all your apps are broken, you should just fix them" card).
Shared pain Posted Feb 3, 2012 4:42 UTC (Fri) by raven667 (subscriber, #5198) [Link]
> rename() has been around for 3 times longer than ext3, so I don't really see how ext3 behaviour can possibly be what generations of programmers expect to see
I'm going to go out on a limb and say that there are more people who are familiar with expected ext3 behavior than the entire number of people who have run UNIX so I do think that ext3-like behavior is what programmers in general expect these days.
Shared pain Posted Feb 3, 2012 5:01 UTC (Fri) by neilbrown (subscriber, #359) [Link]
This doesn't change the fact that the ext3 behaviour is a mistake, was not designed, was never universal and so should not be seen as a desirable standard.
Yes, there is room for improvement - there always is. Copying a mistake because it has some good features is not a wise move.
As Dave said - if there is a problem, let's fix it properly.
(and yes, my beard is gray (or heading that way)).
Shared pain Posted Feb 3, 2012 5:16 UTC (Fri) by raven667 (subscriber, #5198) [Link]
Even though the behavior was created by accident doesn't mean that it's not a good idea and that it's become a de-facto standard expectation. Why else would there have been so much noise with ext4?
Shared pain Posted Feb 3, 2012 5:25 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
so you are saying that because ext3 gives better behavior for people who don't code carefully, it's behavior is the gold standard, even though there is still room for data loss and the same ext3 mistake that gave you the better reliability if you are careless also gives you horrid performance if you try to be careful and make sure your data is really safe.
if you could get the advantages without the drawbacks, of course it would be nice, but the same flaw in the ext3 logic that gives you one also gives you the other.
Shared pain Posted Feb 3, 2012 5:49 UTC (Fri) by raven667 (subscriber, #5198) [Link]
> ext3 gives better behavior for people who don't code carefully, it's behavior is the gold standard
Its not even about coding carefully, doing the "correct" thing is not even possible in many of the use cases which are protected by the default ext3 behavior such as atomically updating a file from a program which is not in C such as a shell script. I learned, along with many admins, to use the atomic rename behavior to implement "safe" updates which may have been a misunderstanding at the time but can now be considered the new requirement.
At the time this issue was discovered with ext4 there was a frank exchange of ideas and the realization that the expected rename behavior is beneficial to overall reliability and we should make it work properly. I'd be interested in seeing this kind of thing handled at the VFS layer so that the behavior is consistant across all filesystems, that sounds like a great idea.
Shared pain Posted Feb 6, 2012 23:33 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]
the rename behavior was only 'usually safe' without fsyncs (like from scripts), and you could always have a script call 'sync' (a sledgehammer to swat a fly yes, but in the case of ext3, it generated the same disk I/O that a fsync on an individual file would)
yes the we can look at changing the standard, but the way to do that is to talk about changing the standard, not insist that the behavior of one filesystem is the only 'correct' way to do things and that all filesystem developers don't care about your data.
Shared pain Posted Feb 7, 2012 23:47 UTC (Tue) by Wol (guest, #4433) [Link]
I think this argument has long been hashed out, but the point is the unwanted behaviour is pathological.
And IT IS LOGICALLY IMPOSSIBLE if the computer actually does what the programmer asked it to. THAT is the problem - the computer ends up in an "impossible" state.
And if it is logically impossible to end up there, at least in the programmer's mind, it is also logically impossible to make allowances for it and fix the system!
The state, as per the program's world view, is
If the computer crashes in the middle of this we "magically" end up in state (d) old file is full of zeroes.
How do you program to fix a state that it is not logically possible to get to? In such a way as the program is actually guaranteed to work properly and portably?
Cheers,
Shared pain Posted Feb 7, 2012 23:53 UTC (Tue) by neilbrown (subscriber, #359) [Link]
Seems like the programmer has an incorrect model of the world.
Writing to a file has never made the data safe in the event of a crash. fsync is needed for that.
If the programmer did not issue 'fsync' but still expected the data to be safe after a crash, then the programmer made a programming error. It really is that simple.
Incorrectly written programs often produce pathological behaviour - it shouldn't surprise you.
Shared pain Posted Feb 8, 2012 3:41 UTC (Wed) by mjg59 (subscriber, #23239) [Link]
I appreciate that ordering has never been guaranteed by POSIX, but let's limit it to the actual argument rather than an obvious straw man. The desired behaviour was never for a rename to guarantee that the new data had hit disk. The desired behaviour was for it to be guaranteed that *either* the old data or the new data be present. fsync provides guarantees above and beyond that which weren't required in this particular use case. It's unhelpful to simply tell application developers that they should always fsync when we've just spent the best part of a decade using a filesystem that crippled any application that did.
Shared pain Posted Feb 8, 2012 4:08 UTC (Wed) by neilbrown (subscriber, #359) [Link]
> we've just spent the best part of a decade using a filesystem that crippled any application that did.
That's the heart of the matter to me.... but now XFS - a filesystem that didn't cripple correct applicatons - is getting a hard time because it doesn't follow the lead of a filesystem that did.
And yes, I know, technical excellence doesn't determine market success, and even the best contender must adapt or die when facing with an ill-informed market. So maybe XFS should adopt the extX model for rename even though it hurts performance in some cases - because if it doesn't people might choose not to use it - and who wants to be the best filesystem that nobody uses (though XFS is a long way from that fate).
So I'm just being a lone voice trying to teach the history and show people that the feature they like so much was originally a mistake and the programs that use it are actually incorrect (or at least not-portable) and maybe there are hidden costs in the thing they keep asking for..
I don't expect to be particularly successful, but that is no justification for being silent.
Shared pain Posted Feb 8, 2012 12:38 UTC (Wed) by mjg59 (subscriber, #23239) [Link]
Arguing "The specification allows us to do this" isn't something that convinces the people who consume your code. Arguing "Our design makes it difficult" is more convincing, but implies that your design stage ignored your users. "We made this tradeoff for these reasons" is something that people understand, but isn't something I've seen clearly articulated in most of these discussions. It just usually ends up with strawman arguments about well how did you expect this stuff to end up on disk when you didn't fsync, which just makes people feel like you don't even care about pretending to understand what they're actually asking for.
(Abstract you throughout)
Shared pain Posted Feb 8, 2012 13:24 UTC (Wed) by nye (guest, #51576) [Link]
>That's the heart of the matter to me.
Then you have misunderstood the nature of the problem.
The problem is that there are cases when atomicity is required but durability is not so important. With ext3 (et al.) it is possible to get one without the other, but with XFS (et al.) atomicity can only be gained as a side-effect of durability, which is more expensive.
Thus, ext3 provides a feature which XFS does not - one which filesystem developers, as a rule, don't seem to care about, but application developers, as a rule, do. The characterisation of anyone who actually cares for that feature as 'ill-informed' is grating, even offensive to many.
General addendum, not targeted at you specifically: falling back to the observation that XFS's behaviour is POSIX-compliant is pointless because - though true - it is vacuous. In fact POSIX doesn't specify anything in the case of power loss or system crashes, hence it would be perfectly legal for a POSIX-compliant filesystem to fill your hard drive with pictures of LOLcats.
Shared pain Posted Feb 8, 2012 22:29 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
and with ext3 it's not possible to get durability without a huge performance impact
with any filesystem you have atomic renames IF THE SYSTEM DOESN'T CRASH before the data is written out, that's what the POSIX standard provides.
ext3 gains it's 'atomic renames' as a side effect of a bug, it can't figure out what data belongs to what, so if it's trying to make sure something gets written out it must write out ALL pending data, no matter what the data is part of. That made it so that if you are journaling the rename, all the writes prior to that had to get written out first (making the rename 'safe'), but the side effect is that all other pending writes, anywhere in the filesystem also had to be written out, and that could cause 10's of seconds of delay.
for the casual user, you argue that this is "good enough", but for anyone who actually wants durability, not merely atomicity in the face of a crash has serious problems.
ext4 has a different enough design that they can order the rename after the write of the contents of THAT ONE file, so they can provide some added safety at relatively little cost
you also need to be aware that without the durability, you can still have corrupted files in ext3 after a crash, all it takes is any application that modifies a file in place, including just appending to the end of the file
Shared pain Posted Feb 8, 2012 19:48 UTC (Wed) by Wol (guest, #4433) [Link]
Lets just say that governments (and businesses) have wasted billions throwing away applications where the application met the spec but in practice was unfit for purpose.
And a filesystem that throws away user data IS unfit for purpose. After all, what was the point of journalling? To improve boot times after a crash and get the system back into production quicker. If you need to do data integrity check on top of your filesystem check you've just made your reboot times far WORSE - a day or two would not be atypical after a crash!
Cheers,
Shared pain Posted Feb 8, 2012 20:51 UTC (Wed) by raven667 (subscriber, #5198) [Link]
The hyperbole is getting a little out of control. Journaled filesystems have traditionally only journaled the metadata so any file data in-flight at the time of a crash would be lost and corruption would be the result. Pre-journaling any filesystem with a write cache would be susceptible to losing in-flight data and corrupting metadata leading to long fsck times after crash to repair the damage. All filesystems lose data in those circumstances, that doesn't mean that all filesystems are unfit for any purpose or that computers are fundamentally unfit for any purpose. The current state of the art is to be safer with regular data writes, even to the point of checksumming everything, that's nice but the world didn't end when this wasn't the case.
Shared pain Posted Feb 8, 2012 15:13 UTC (Wed) by Wol (guest, #4433) [Link]
"fsync is needed for that"
And what is the poor programmer to do if he doesn't have access to fsync?
Or what are the poor lusers supposed to do as their system grinds to a halt with all the disk io as programs hang waiting for the disk?
Following the spec is not an end in itself. Getting the work done is the end. And if the spec HINDERS people getting the work done, then it's the spec that needs to change, not the people.
THAT is why Linux is so successful. Linus is an engineer. He understands that. "DO NOT UNDER ANY CIRCUMSTANCES WHATSOEVER break userspace" is the mantra he lives by. And filesystems eating your data while making everything *appear* okay is one of the most appalling breaches of faith by the computer that it could commit!
Cheers,
Shared pain Posted Feb 9, 2012 1:26 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
> And what is the poor programmer to do if he doesn't have access to fsync?
use a language that gives them access to data integrity tools like fsync.
for shell scripts, either write a fsync wrapper, or use the sync command (which does exactly the same as fsync on ext3)
> Or what are the poor lusers supposed to do as their system grinds to a halt with all the disk io as programs hang waiting for the disk?
use a better filesystem that doesn't have such horrible performance problems with applications that try and be careful about their data.
> Following the spec is not an end in itself.
True, but what you are asking for is for the spec to be changed, no matter how much it harms people who do follow the spec (application programmers and users who care about durability)
There is no filesystem that you can choose to use that will not loose data if the system crashes. If you are expecting something different, you need to change your expectation.
Shared pain Posted Feb 9, 2012 7:39 UTC (Thu) by khim (subscriber, #9252) [Link] Somehow you've forgotten about the most sane alternative:Remove XFS from all the computers and use sane filesystems (extX, btrfs when it'll be more stable) exclusively. In a battle between applications and filesystems applications win 10 times out of 10 because without applications filesystems are pointless (and applications are pointless without the user's data). The whole discussion just highlights that XFS is categorically, absolutely, totally unsuitable for the use as general-purpose FS. And when you don't care about data integrity then ext4 without journalling is actually faster (see Google datacenters, for example). True, but what you are asking for is for the spec to be changed, no matter how much it harms people who do follow the spec Yes. application programmers and users who care about durability Applications don't follow the spec. When they do they are punished and fixed. Thus users who care about durability need to use filesystems which work correctly given the existing applications. Is it fair? No. It's classic vicious cycle. But said sycle is fact of the life. Ignore it at your peril. I, for one, have a strict policy to never use XFS and to don't even consider bugs which can not be reproduced with other filesystems. Exactly because XFS developers think specs trump reality for some reason. There is no filesystem that you can choose to use that will not loose data if the system crashes. If you are expecting something different, you need to change your expectation. That's irrelevant. True, the loss of data in the case of system crash is unavoidable. I don't care if the window I've opened right before crash in Firefox is reopened or not. I understand that spinning rust is slow and can lose such info. But if the windows which were opened hour before that are lost because XFS replaced save state file with zeros then such filesystem is useless in practice. Long time ago XFS was prone to such data loss even if fsync was used and data was "saved" to disk days before crash. After a lot of work looks like XFS developers fixed this issue, but now they are stuck with the next step: atomic rename. It should be implemented for the FS to be suitable for real-world applications. There are even some hints that XFS have implemented it, but as long as XFS developer will exhibit this "specs are important, real applications don't" pathological thinking it's way too dangerous to even try to use XFS.
Shared pain Posted Feb 9, 2012 9:12 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
if you use applications that follow the specs (for example, just about every database, or mailserver), then XFS/ext4/btrfs/etc are very reliable.
what you seem to be saying is that these classes of programs should be forced to use filesystems that give them huge performance penalties to accommodate other programs that are more careless, so that those careless programs loose less data (not no data loss, just less)
Shared pain Posted Feb 9, 2012 9:19 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
by the way, I've done benchmarks on applications that do the proper fsync dance needed for the data to actually be safe (durable, not just atomic filesystem renames that may or may not get written to disk), and even on an otherwise idle system ext3 was at least 2x slower, and if you have other disk activity going on at the same time, the problem only goes up (if you hae another process writing large amounts of data, the performance difference for your critical app can easily be 40x slower on ext3)
Shared pain Posted Feb 9, 2012 17:37 UTC (Thu) by khim (subscriber, #9252) [Link] Exactly. This is part of the very simple proof sequence. Fact 1: any application which calls fsync is very slow in ext3. You've just observed it. Yes, it's not fair to XFS. No, I don't think being fair is guaranteed in real world.
Shared pain Posted Feb 9, 2012 19:26 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
sorry, on my systems I'm not willing to tolerate a 50x slowdown just to make badly written apps be a little less likely to be confused after a power outage.
and I think that advocating that you have the right to make this choice for everyone else is going _way_ too far.
when I have applications that loose config data after a problem happens (which isn't always a system crash, apps that have this sort of problem usually have it after the application crashes as well), my solution is backups of the config (idealy into something efficient like git), not crippling the rest of the system to band-aid the bad app.
Shared pain Posted Feb 9, 2012 20:44 UTC (Thu) by Wol (guest, #4433) [Link]
And what is "badly written" about an app that expects the computer to do what was asked of it?
I know changing things around for the sake of it doesn't matter when everything goes right, but if I tell the computer "do this, *then* that, *followed* by the other", well, if I told an employee to do it and they did things in the wrong order and screwed things up as a *direct* *result* of messing with the order, they'd get the sack.
The only reason we're in this mess, is because the computer is NOT doing what the programmer asked. It thinks it knows better. And it screws up as a result.
And the fix isn't that hard - just make sure you flush the data before the metadata (or journal the data too), which is pretty much (a) sensible, and (b) what every user would want if they knew enough to care.
Cheers,
Shared pain Posted Feb 9, 2012 20:52 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
it is badly written because you did not tell the computer that you wanted to make sure that the data was written to the drive in a particular order.
If the system does not crash, the view of the filesystem presented to the user is absolutely consistent, and the rename is atomic.
The problem is that there are a lot of 'odd' situations that you can have where data is written to a file while it is being renamed that make it non-trivial to "do the right thing" because the system is having to guess at what the "right thing" is for this situation.
try running a system with every filesystem mounted with the sync option, that will force the computer to do exactly what the application programmers told it to do, writing all data exactly when they tell it to, even if this means writing the same disk sector hundreds of times as small writes happen. The result will be un-usable.
so you don't _really_ want the computer doing exactly what the programmer tells it to, you only want it to do so some of the time, not the rest of the time.
Shared pain Posted Feb 9, 2012 21:13 UTC (Thu) by khim (subscriber, #9252) [Link] so you don't _really_ want the computer doing exactly what the programmer tells it to, you only want it to do so some of the time, not the rest of the time. Sure. YMMV as I've already noted. Good filesystem for USB sticks must flush on close(2) call. Good general purpose filesystem must guarantee rename(2) atomicity in the face of system crash. You can use whatever you want for your own system - it's you choice. But when question is about replacement of extX… it's other thing entirely. To recommend filesystem which likes to eat user's data is simply irresponsible.
Shared pain Posted Feb 14, 2012 16:16 UTC (Tue) by nye (guest, #51576) [Link]
>when I have applications that loose config data after a problem happens (which isn't always a system crash, apps that have this sort of problem usually have it after the application crashes as well)
That can't possibly be the case. You must be talking about applications which do something like truncate+rewrite, which is entirely orthogonal to the discussion (and is pretty clearly a bug).
I suspect you haven't understood the issue at hand.
Shared pain Posted Feb 9, 2012 17:25 UTC (Thu) by khim (subscriber, #9252) [Link] What you seem to be saying is that these classes of programs should be forced to use filesystems that give them huge performance penalties to accommodate other programs that are more careless, so that those careless programs loose less data In a word: yes. not no data loss, just less Always and forever. No matter what filesystem you are using you data is toast in a case of RAID failure or lightning strike. This means that we always talk about probabilities. This leads us to detailed explanation of the aforementioned phenomenon: in most cases you can not afford dedicated partitions for your database or mailserver and is this world filesystem without suitable reliability guarantees (like atomic rename in a crash case without fsync) is pointless. When your system grows it becomes good idea to dedicate server just to be a mailserver or just to be a database server. But the window of opportunity is quite small because when you go beyond handful of servers you need to develop plans which will keep your business alive in a face of hard crash (HDD failure, etc). And if you've designed your system for such a case then all these journalling efforts in a filesystem are just a useless overhead (see Google which switched from ext2 to ext4 without journal). I'm not saying XFS is always useles. No, there are exist cases where you can use it effectively. But these are rare cases thus XFS will always be undertested. And this, in turn, usually means you should stick with extX/btrfs.
Shared pain Posted Feb 3, 2012 10:19 UTC (Fri) by khim (subscriber, #9252) [Link] Yes, there is room for improvement - there always is. Copying a mistake because it has some good features is not a wise move. This depends on your goal, actually. If your goal is something theoretically sound, then no, it's not a wise move. If your goal is creation of something which will actually be used by real users then it's the only possible move. (and yes, my beard is gray (or heading that way)). My beard is not yet gray, but I was around long enough to see where the guys who did "wise move" ended. I must admit that they create really TRULY nice exhibits in Computer History Museum. Meanwhile creations of "unwise" guys are used for real work. If your implementation unintentionally introduced some property and people started depending on it - it's the end of story: you are doomed to support said property forever. If you want to keep these people, obviously. If your goal is just to create something nice for the sake of art or science, then situation is different, of course. This is basic fact of life and it's truly sad to see that so many Linux developers (especially the desktop guys) don't understand that.
XFS: the filesystem of the future? Posted Jan 20, 2012 22:39 UTC (Fri) by sandeen (guest, #42852) [Link]
Regarding data loss on a crash;
#1, if your application follows proper data integrity practices (fsync et al) and your storage is properly configured, you will not lose data on a crash on xfs. Losing buffered data on a crash is expected with any filesystem, though there are differences in that behavior between various filesystems. Ext3 tended to push data out more regularly, which had its pros and cons, but people came to expect their data to be safe(er) even if it wasn't explicitly flushed. But nothing is really safe until it's explicitly synced.
#2, the un-synced "null files" behavior has been fixed since this commit from 2007 ... http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-...
XFS: the filesystem of the future? Posted Jan 20, 2012 22:46 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
and the ext3 behavior was linked to the pathologically bad fsync behavior that ext3 had where a fsync could take 10's of seconds to complete.
this combination of behavior has probably done more to harm data reliability (by encouraging bad programming practices and discouraging good programming practices) than anything else I can think of.
XFS: the filesystem of the future? Posted Jan 28, 2012 3:36 UTC (Sat) by sbergman27 (guest, #10767) [Link]
"#1, if your application follows proper data integrity practices (fsync et al)"
And if it doesn't happen to? How many filesystem users do code audits of all the programs their organizations runs, specifically looking for good practice regarding fsync? It's much better to just use a reliable filesystem.
"and your storage is properly configured, you will not lose data on a crash on xfs."
That's reasuring. I note that your recommended finger-pointing re: data loss, is now distributed between user and vendor. But it's definitely not XFS's fault!
"Losing buffered data on a crash is expected with any filesystem,"
No. Losing data on a crash is not "expected with any filesystem". (I've never had a customer quiz me as to whether lost data was "buffered" or not.) EXT3 was rock solid before they ruined it around 2.6.31 or so in the name of performance. A manual adjustment to data=journal should still restore it to its former glory. Though I fear that this new "performance over reliability" mindset in the Linux FS world might be eroding it in other ways, too.
"Ext3 tended to push data out more regularly"
EXT3 tended to keep your data rock solid safe.
"which had its pros and cons"
If I sent out a survey to my customers asking their opinion on reliability vs performance on filesystems, I already know what they would say. Reliability over performance: 100%. Performance over reliability: 0%.
It seems to me that, these days, Linux FS devs either live in ivory towers of performance benchmarks, or in California prisons. But none seem to live anywhere near me. We need more Stephen Tweedies.
-Steve Bergman
XFS: the filesystem of the future? Posted Jan 28, 2012 5:53 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]
ext3 has always had the possibility of loosing data in a unclean shutdown if you didn't do the fsync dance.
the window of loss was smaller, and most people don't actually have unclean shutdowns that frequently, so I believe you when you say you never experienced it.
but that doesn't mean that ext3 was 'rock solid' in the face of poorly written applications. There are plenty of people who lost data on ext3.
XFS: the filesystem of the future? Posted Jan 28, 2012 16:37 UTC (Sat) by sbergman27 (guest, #10767) [Link]
The window was *far* smaller. However, the question is not did anyone ever suffer any data loss. It's how many people and how much data in comparison to other filesystems. I know of no other filesystem as resilient at its default settings as was the pre2.6.31 (or so) ext3. I also never noticed any performance problems. Certainly nothing that my customers cared about.
And of course, we can still mount ext3 explicitly data=ordered.
Oh for the days when Linux filesystem devs cared more about reliability than benchmarks... no matter whose fault the data loss might be. Things have changed since the Tweedie days.
Today, I use ext4 with nodelalloc. I tried ext4 at its defaults. But the first time power was lost (and the UPS failed) we had to rebuild a bunch of C/ISAM files. Never in our long history with ext3 did we *ever* have anything like that happen. (And my customers are bad about letting their UPS's go.) Personally, I think nodelalloc should be default, with delalloc available as a mount option.
Current Linux filesystem devs just seem reckless to me.
XFS: the filesystem of the future? Posted Jan 29, 2012 0:53 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
the current filesystem devs are the same people as they were in the 'good old days'
go talk to a good database admin (or especially a database developer), they will tell you that all filesystems have always had these problems, and they can probably tell you horror stories about ext3 and it's "worst in the industry" fsync performance (it's been documented for a fsync to take 10s of seconds on ext3 vs ms for other filesystems under the same workload)
you can disable all caching by mounting a filesystem with the sync option, but the performance is going to be _so_ horrible (unless you have drives that lie about when the data is safe) that you will end up changing back.
XFS: the filesystem of the future? Posted Jan 29, 2012 3:09 UTC (Sun) by sbergman27 (guest, #10767) [Link]
"the current filesystem devs are the same people as they were in the 'good old days'"
No. In the Ext world, I hardly ever hear the name "Stephen Tweedie" anymore. And he's the main developer I trusted. A very careful, conservative, and patient devloper. Took forever to get Ext3 out, Which annoyed me at the time. But I've learned to appreciate his "good things come to those who wait" philosophy. Today, it's Ted T'so at the forefront of Ext. A *very* different person.
"go talk to a good database admin (or especially a database developer), they will tell you that all filesystems have always had these problems"
And there you go again. Casting it as "black and white". If Ext3 ever lost a byte of data on someone's machine in Kenya, then by the gods, it's just as bad as the current crop of linux data sieve filesystems.
Hey, if you can keep doing the black and white thing, I can compensate by adjusting the contrast knob a bit. ;-) Current linux filesystems may not be, exactly, data sieves. But they are a far cry from the halcyon days of Ext3 pre 2.6.31.
XFS: the filesystem of the future? Posted Jan 29, 2012 6:45 UTC (Sun) by raven667 (subscriber, #5198) [Link] If I sent out a survey to my customers asking their opinion on reliability vs performance on filesystems, I already know what they would say. Reliability over performance: 100%. Performance over reliability: 0%. I don't believe that is true. If you ask them then sure, they will pick reliability but in actual operations they will performance first most of the time. Witness the popularity of MySQL/myisam which makes that exact trade off, if reliability was the most important thing everyone would use PostgreSQL and no one would have even heard of MySQL. What people actually do and what they say are often diametrically opposed.
XFS: the filesystem of the future? Posted Jan 29, 2012 20:51 UTC (Sun) by sbergman27 (guest, #10767) [Link]
"I don't believe that is true. If you ask them then sure, they will pick reliability but in actual operations they will performance first most of the time."
No, they would not. None have any complaints about our filesystem performance, either on my CentOS 4 machines using Ext3 mounted data=ordered, or on my later servers with Ext4 mounted nodelalloc. Benchmarking of Ext4 with and without nodelalloc always come out pretty much a wash. I've never understood what the fuss was about. Delayed allocation just doesn't improve performance noticeably. Nor did Ext3 with data=ordered in any of the server scenarios I have been involved with. And I see no appreciable difference regarding fragmentation rates, either.
If you've done your own benchmarks with your own server workloads which disagree with mine, I would be interested in hearing about them.
But regarding delayed allocation in ext4 and xfs, and the "wonders" of making data=writeback the default for ext3, I must observe that the Emperor has no clothes.
XFS: the filesystem of the future? Posted Jan 21, 2012 10:13 UTC (Sat) by jengelh (subscriber, #33263) [Link]
>having lost an entire filesystem (and zeroed-out files on another filesystem) to *blip*. But seriously, *blip* has long had a reputation for being more likely to eat your data
Yes yes, the usual complaints. "omg X suxx, and Y is so much better." Until Y ate your data, then they are all "omg Y suxx, Z is so much better."
XFS: the filesystem of the future? Posted Jan 29, 2012 3:13 UTC (Sun) by sbergman27 (guest, #10767) [Link]
'"omg X suxx, and Y is so much better." Until Y ate your data, then they are all "omg Y suxx, Z is so much better."'
I've never waivered from my stance that Ext3 was the most reliable filesystem ever. Not in 11 years.
XFS: the filesystem of the future? Posted Jan 22, 2012 11:52 UTC (Sun) by pkolloch (subscriber, #21709) [Link]
Well, I had that zero-ing out of data in a small scale with ext4. Since it has happened in the context of system freezes, something else might be to blame, though.
XFS: the filesystem of the future? Posted Feb 8, 2012 15:25 UTC (Wed) by Wol (guest, #4433) [Link]
No. ext4 and xfs both had the same bug. So basically anything that stopped disk i/o and forced a reboot could cost you data.
Basically what happened was
(a) user writes file
What should, of course, happen next is that the file data is flushed to disk, but because it hasn't been journalled and the system has crashed, it's not there to be written.
And that's why the rename gave you a file full of zeros, because the new file header overwrote the old one, but the new file contents hadn't been flushed to disk.
The fix, with "mode=journal" or whatever, basically blocks the metadata from updating until after the data has updated, by whatever means seems most suitable.
Cheers,
XFS: the filesystem of the future? Posted Jan 23, 2012 15:36 UTC (Mon) by cruff (subscriber, #7201) [Link]
I successfully ran 100 TB of XFS file systems under Irix for years without any data loss as the disk cache for the NCAR Mass Store. The performance was great with the middle of the road RAID hardware that we could afford. Of course, that wasn't Linux, so YMMV. :-)
XFS: the filesystem of the future? Posted Jan 23, 2012 15:52 UTC (Mon) by rfunk (subscriber, #4054) [Link]
Yeah, I used XFS on Irix for a while before it came to Linux, which is why I used to trust it.
One issue, however, was that XFS was designed with server room assumptions. So when I used it in a Linux desktop context, with a less-reliable power supply, bad things happened. One of those bad things was losing an entire filesystem once; another was the zeroed files that people have mentioned.
Of course this was probably around 2005, so I wouldn't be surprised if it's improved since then, but I'm also not going to assume that it has.
XFS: the filesystem of the future? Posted Jan 23, 2012 16:06 UTC (Mon) by wazoox (subscriber, #69624) [Link]
I've been using XFS for 15 years, and currently manage 250 systems in the 10 - 200 TB range; I've lost files once over a known XFS bug, and though I hear here and there about "XFS zeroing files" I never encountered the problem.
I'm afraid it's in the same ballpark as people who have 3 hard drives and say "OMG this brand sucks because one of my drives failed". I have several thousands spinning hard drives under my guard, and I think I have quite better statistics on what works and what doesn't; the same goes for filesystems, how many filesystems do you manage? If it's less than a couple of hundreds, the fact that you encountered a particular problem with a particular filesystem once doesn't carry much significance.
XFS: the filesystem of the future? Posted Jan 23, 2012 16:25 UTC (Mon) by rfunk (subscriber, #4054) [Link]
I said I was talking about non-server contexts. XFS was designed for server rooms, where the power never goes out. It sounds like that's where you're using it.
XFS: the filesystem of the future? Posted Jan 23, 2012 18:40 UTC (Mon) by wazoox (subscriber, #69624) [Link]
Well, I use it on all of my desktop systems too, and my work desktop has an uptime of 176 days. I don't reboot often :)
XFS: the filesystem of the future? Posted Jan 29, 2012 3:47 UTC (Sun) by sbergman27 (guest, #10767) [Link]
"Well, I use it on all of my desktop systems too, and my work desktop has an uptime of 176 days. I don't reboot often :)"
I hope you don't connect to the Internet with that security-hole ridden kernel you're running. You should reboot after kernel updates.
XFS: the filesystem of the future? Posted Jan 30, 2012 8:03 UTC (Mon) by youareretarded (guest, #82640) [Link]
are you retarded? nobody gets exploited by missing a kernel update? it sounds like you don't know shit about the kernel if you think someone is going to be remotely exploited through a kernel bug introduced in the last 180 days. it's always the software, never the kernel.
XFS: the filesystem of the future? Posted Jan 23, 2012 21:46 UTC (Mon) by dgc (subscriber, #6611) [Link]
Actually, XFS was designed for storage subsystems that didn't lie to it, not "server storage". You can say exactly the same for ext3, ext4, btrfs, etc.
"Consumer storage" violates the write ordering guarantees that these filesystems require to have journal recovery work because they have volatile write caches. That's why we have write barriers and use them by default on these filesystems these days. XFS was the first filesystem to enable them by default, another reason it was always slower on metadata intensive workloads than ext3/4.
"server storage" doesn't violate write ordering in an effort to improve performance, so XFS has always worked fine and performed well on that class of storage.
Dave.
XFS: the filesystem of the future? Posted Jan 31, 2012 17:36 UTC (Tue) by Cato (subscriber, #7643) [Link]
Do you have evidence that all "consumer storage" devices violate write ordering guarantees, or simply don't flush pending writes on request?
As far as I can tell, some consumer drives do lie about when writes have been flushed, and write back caching is the default anyway.
Some relevant links:
http://serverfault.com/questions/15404/sata-disks-that-ha...
http://brad.livejournal.com/2116715.html - disk testing tool
XFS: the filesystem of the future? Posted Jan 31, 2012 19:30 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
I don't think anyone is meaning to say that all consumer storage devices are broken, but I also don't think that there's much dispute that some are.
XFS: the filesystem of the future? Posted Feb 2, 2012 21:19 UTC (Thu) by dgc (subscriber, #6611) [Link]
> Do you have evidence that all "consumer storage" devices violate write
> ordering guarantees,
Any device with a volatile write cache tells the OS that the write IO has been completed before it actually is written to stable storage. IO completion is supposed to mean "the IO is complete" and any device with a volatile write cache is actually lying - the write is not yet on stable storage, so is "lying" about the completion status of the IO to the OS. Pretty much all consumer devices ship with a volatile write cache enabled by default for performance reasons.
Barriers and cache flushes were introduced to provide a mechanism that allowed filesystems to force such drives to order writes the way the filesystem wants correctly. The original barrier mechanism was "cache flush, write, cache flush" and could make the drive slower than not caching in the first place depending on the workload. More recently we just use the FUA mechanism if the drive supports that, and that has neglible performance overhead.
> As far as I can tell, some consumer drives do lie about when writes
If drives lie about cache flush or FUA completion on volatile writeback caches, then that's a bug in the disk firmware.
FWIW, the difference with server storage (SAS drives) is that most ship with the volatile write cache turned off by default. They don't need it for performance because the SCSI/SAS protocol is much more efficient than SATA and so in most cases a write cache isn't necessary. You can turn it on, but you don't need to to reach full disk performance....
Indeed, it's not just filesystems that don't like volatile write caches. if you turn on volatile write caching on disks behind a RAID controller, the disk will now violate the write ordering guarantees that the RAID controller relies on to maintain data safety (exactly the same as for filesystems). You still lose data or corrupt filesystems on power loss in this case, even though the OS and RAID controller are behaving correctly.
Dave.
XFS: the filesystem of the future? Posted Feb 3, 2012 4:34 UTC (Fri) by raven667 (subscriber, #5198) [Link]
> They don't need it for performance because the SCSI/SAS protocol is much more efficient than SATA and so in most cases a write cache isn't necessary.
I think you are correct on every other point but I dont think this is right. SATA is pretty much the SCSI protocol as is SAS, they are only slightly incompatible for marketing rather than technical reasons. The big performance difference historically between consumer (IDE) and enterprise (SCSI) drives was tagged command queuing which is now very common in SATA drives as well although it wasn't so common in early SATA implementations. A tagged command queue allows the drive to implement an elevator which is a big win against a naive implementation without one.
XFS: the filesystem of the future? Posted Feb 3, 2012 12:35 UTC (Fri) by Jonno (subscriber, #49613) [Link]
Actually, SATA (Serial ATA) uses a slightly extended ATA command set over a serial bus, while SAS (Serial Attached Scsi) uses the SCSI command set over the same serial bus.
The SCSI command set is generally considered "better" than the ATA command set, though the difference isn't quite as large as the grand parent suggests. Write caches are still beneficial for SCSI (including SAS) performance, but the difference is not quite as large with SCSI as with ATA. That, as well as the fact that the average enterprise customer are more concerned about reliability than the average home user, are the reason that most SAS drives have write cache disabled by default, while most SATA drives have write cache enabled by default.
XFS: the filesystem of the future? Posted Feb 3, 2012 17:30 UTC (Fri) by raven667 (subscriber, #5198) [Link]
The "extended ATA" command set that's used on SATA devices not operating in Legacy IDE mode is the SCSI command set. This goes all the way back to ATAPI which is the SCSI command set encapsulated with the IDE bus protocol. Drives and controllers are capable of speaking either SATA-II or SAS protocols without any cost difference AFAICT but don't, for largely marketing reasons rather than engineering ones. As I was saying before, having a command queue on the drive allows for the drive to have an IO elevator which is _the_ big performance win, details about how commands are named and whatnot is not really an important factor.
scsi misinformation Posted Feb 3, 2012 14:08 UTC (Fri) by quanstro (guest, #77996) [Link]
the argument here seems to be circular. sas is faster
because it's sas.
sata and sas send the same data in the same size fises/frames
could you please explain the mechanism where by sas is going to
scsi misinformation Posted Feb 3, 2012 19:00 UTC (Fri) by raven667 (subscriber, #5198) [Link]
> the argument here seems to be circular. sas is faster because it's sas
That's the power of branding, replacing rational thought with mental shortcuts which put things in "good" or "bad" boxes.
XFS: the filesystem of the future? Posted Feb 4, 2012 13:00 UTC (Sat) by zomonto (guest, #82108) [Link]
> The original barrier mechanism was "cache flush, write, cache flush" and could make the drive slower than not caching in the first place depending on the workload. More recently we just use the FUA mechanism if the drive supports that, and that has neglible performance overhead.
No, libata always disables FUA by default. You can enabled it with a
XFS: the filesystem of the future? Posted Feb 8, 2012 13:09 UTC (Wed) by yungchin (guest, #72949) [Link]
> Pretty much all consumer devices ship with a volatile write cache enabled by default for performance reasons.
Dave, I was wondering, given the optimisations you discussed in the talk, where there's now lots of merging and reordering going on before sending things to the i/o scheduler, do you still expect much performance improvements from these hardware caches? (Or - that's of course the hidden question here - should we from now on happily disable them, at least for most use cases?) Thanks.
XFS: the filesystem of the future? Posted Feb 1, 2012 11:33 UTC (Wed) by Cato (subscriber, #7643) [Link]
Maybe I didn't quite get your point - would be good to understand exactly what write ordering guarantees are provided by server storage but not consumer storage. Is this just that a consumer hard drive's write cache will reorder writes without respecting the kernel's write barriers?
XFS: the filesystem of the future? Posted Feb 1, 2012 19:28 UTC (Wed) by dlang (✭ supporter ✭, #313) [Link]
some consumer drives lie about when the data has actually been written to disk (making write barriers ineffective), in those cases the OS will send more writes to the drive and the drive will go ahead and re-order them with the other writes that are in it's buffer.
XFS: the filesystem of the future? Posted Feb 1, 2012 20:13 UTC (Wed) by raven667 (subscriber, #5198) [Link]
And as someone else pointed out drives these days seem to have up to 64MB write buffers so that could be a lot of corruption if that data goes missing in-flight when the OS was told that it was permanently committed.
XFS: the filesystem of the future? Posted Feb 1, 2012 22:20 UTC (Wed) by magila (subscriber, #49627) [Link] While there's been a lot of talk about consumer devices "lying" in this thread, they really don't behave any differently from server drives with the same cache settings. Both consumer and server drives provide an option to turn on write caching. Both will lie about writes completing when write caching is enabled. Both will only signal command completion when data has gone to disk if write caching is disabled[1]. The only real difference is the default, most (but not all) enterprise drives ship with write cache disabled while all consumer drive ship with it enabled. Both give the option of changing the setting to whatever the user pleases.[1] I vaguely remember hearing a story several years ago that a handful of ATA drive models where not respecting write cache settings. This was an isolated incident. Newer drives can reasonably be assumed to handle write caching correctly.
XFS: the filesystem of the future? Posted Feb 2, 2012 1:39 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
when I talk about a drive lying, I'm not talking about normal write caching, I'm talking about it either not respecting write cache settings, or lying about data integrity commands that are supposed to work even in the face of write caching (cache flush commands for example)
most consumer drives don't have these problems, but a few have been found to have them.
unfortunately you cannot just assume that newer dries will not have the problem. On the database mailing lists you see a couple drive models every year where someone runs across the problem yet again.
XFS: the filesystem of the future? Posted Feb 2, 2012 2:40 UTC (Thu) by magila (subscriber, #49627) [Link]
"On the database mailing lists you see a couple drive models every year where someone runs across the problem yet again."
I'd be rather surprised if that were the case. The code that handles cache flushing isn't something which usually changes between models. If a manufacturer's firmware had a bug in that area I'd expect to see it across the board, not just randomly poping up periodically on different SKUs.
XFS: the filesystem of the future? Posted Feb 2, 2012 13:11 UTC (Thu) by cladisch (✭ supporter ✭, #50193) [Link]
> The code that handles cache flushing isn't something which usually changes between models. If a manufacturer's firmware had a bug …
You won't get any manufacturer to admit it, but this is not a bug, it's a feature (to get higher benchmark numbers).
XFS: the filesystem of the future? Posted Feb 2, 2012 17:38 UTC (Thu) by magila (subscriber, #49627) [Link] You might not believe it, but I can say based on first hand experience that hard drive manufacturers take data integrity very seriously. None of them would risk losing customer data just to gain extra performance. The potential backlash from data loss would be far worse than scoring lower on a benchmark.Plus the people running benchmarks, especially for tier 1 OEMs, aren't stupid. Lying about cache flushes is pretty easy to detect so the likelihood of getting away with it is pretty low right from the start. Pissing off OEMs is another thing hard drive manufactures would never, ever take risks with.
|
|||||||||||||