LWN: Comments on "That massive filesystem thread"

per-inode noatime

pjm — Wed, 10 Jun 2009 10:06:04 +0000

One can more or less achieve that by applying chattr +A to the whole filesystem and then chattr -A on specific files. Note that chattr +A on a directory will be inherited by any files created in that directory (excluding moving an existing file into that directory).

fsync() and disk flushes

nix — Mon, 27 Apr 2009 10:43:21 +0000

You don't need a UPS. A battery-backed disk controller is just as good
(and perhaps better because the battery failing doesn't take your machine
down if the power is otherwise OK, while the UPS failing *does*).

fsync() and disk flushes

bersl2 — Mon, 27 Apr 2009 06:24:18 +0000

It's hard to believe there are disk drives out there (not counting an occasional broken one) that write trash over random areas as they power down. Disk drives I have seen have a special circuit to disconnect and park the head the moment voltage begins to drop. It has to park the head because you can't let the head land on good recording surface, and it has to cut off the write current because otherwise it's dragging a writing head all the way across the disk, pretty much guaranteeing the disk will never come back. I believe it's a simple circuit that doesn't involve any controller intelligence.

There is a related failure mode where the drive's client loses power and in its death throes ends up instructing the drive to trash itself while the drive still has enough power to operate normally. I've heard that's not unusual, and it's the best argument I know for a UPS that powers a system long enough for it to shut down cleanly.

One of these happened to me. $DEITY as my witness, I will never run an important system without an UPS again.

Bonus: The drive was a Maxtor. Serves me right.
Double bonus: That still wasn't traumatic enough to compel me to make backups.

Put hardware in between RAM and Disk features to work in between these

hozelda — Fri, 17 Apr 2009 00:05:15 +0000

Is there a reason why solid state drives aren't use on PCs as a cache level ahead of the main disk drives? These can even be made replaceable (it's just a cache anyway). Wouldn't these improve the performance/correctness trade-offs enough for many use cases?

The end of LWN comment dialog?

jschrod — Wed, 08 Apr 2009 00:05:25 +0000

Well, I just decided to give you feedback, from someone who is subscribed to LWN quite a bit longer than you and who did not participate in this topic after you took over all its discussion threads: You showed that LWN really needs a KILL file feature where one can put a poster in it; you, in particular. Others have succinctly explained why, no need to repeat this.

But your self-rightousness doesn't allow to understand this, obviously. Luckily, there are still some discussion threads where you don't try to take over. I hope the likes of you will remain few on LWN in the future, this is not Slashdot, after all.

The end of LWN comment dialog?

jospoortvliet — Sun, 05 Apr 2009 17:11:39 +0000

Maybe. Probably for this one response. However, as I pointed out, there are wider repercussions to be expected from such behavior, and it is worth to show, as a community, that we disapprove of such ways of communicating. Even if that is done in a rather unfriendly manner.

On re-reading the thread, I think you are right in that ajross was more impolite than bojan, which often leads to a downward spiral and isn't helpful... bojan's post wasn't that far off from the normal tone on this site.

Anyway. This is went pretty far off-topic, and I think we mostly agree. For as far as we don't, we at least agree on that ;-)

The end of LWN comment dialog?

GreyWizard — Sun, 05 Apr 2009 16:27:22 +0000

It's better to change nothing than to make the situation worse.

The end of LWN comment dialog?

jospoortvliet — Sun, 05 Apr 2009 16:20:45 +0000

True, if you meant to point out that 'be polite YOU JERK' isn't very effective, I agree. I do however think that it's better than nothing. Nothing changes nothing.

The end of LWN comment dialog?

GreyWizard — Sun, 05 Apr 2009 15:42:52 +0000

Both of your arguments could reasonably be applied to a comment that says "please be polite" but both fail for "be polite you jerk." Someone who is accidentally rude is much more likely to respond to the "you jerk" part than the "be polite" part. The contradiction immediately destroys the credibility of the person posting because he or she is not willing to be held to the standard set for others.

A truly polite request for more courtesy might help but it's difficult to be sure because such things are quite rare. Giving in to the temptation to scold even just a little makes the comment worse than useless. Unless you are absolutely certain you can do it right it's better to focus on substantive issues and avoid appointing yourself a courtesy cop.

The end of LWN comment dialog?

jospoortvliet — Sun, 05 Apr 2009 12:43:36 +0000

I disagree on your argument that saying 'be polite you jerk' merely drags things down, for two reasons.

First of all, some people don't notice their behavior is unnecessarily impolite. Pointing it out can help them (if they are willing to be reasonable in the first place). Never pointing out somebodies failures will make them fail forever.

Second, it shows you care about being polite. If others show they care too, a culture of 'you should be polite' can be maintained. As you might have noticed from the differences between FOSS communities, culture is important and heavily influential. And it can be changed.

Some things to note:
- people DO care about what others think of them. No matter how much they scream 'no I don't', they do. It is our nature.
- people should know their arguments are not supported by being mean - it is the other way around.
- I agree that a 'be polite you yerk' might not always be the best way to correct someone. A personal mail can do more. However, it won't show up in public (unless an apology is made), thus it does not much to influence others who might think it is acceptable behavior because the guy got away with it. Of course, giving a good example is better than anything else.
- Of course discussing without end whether somebody was polite enough or not muddies the discussion and lowers the SNR.

The end of LWN comment dialog?

GreyWizard — Sun, 05 Apr 2009 03:34:43 +0000

Pardon me for saying so but you have not understood what I wrote. I did not urge anyone to "say whatever [their] brain produces" or anything equivalent. Nor did I issue a rallying cry against decency. Elevating the level of discourse would be a wonderful thing and if you've got an effective suggestion for how to do so I would be glad to read it.

But saying "be polite you jerk" merely drags things even further down into the muck.

The end of LWN comment dialog?

jospoortvliet — Sat, 04 Apr 2009 09:05:52 +0000

Hmmm. Just say whatever your brain produces, and if somebody has a problem
with what comes out, it's on their own plate.

Living in a country where that mode of thinking is the norm, I can tell
you it also has disadvantages... If only because the resulted hurt
feelings can muddy the discussion more than you might think. Besides, it
chases people away who would otherwise have contributed constructively -
it's not acceptable behavior in all cultures. Ever wondered why the FOSS
community is still predominantly western, despite many smart developers in
countries like India?

A little decency now and then doesn't hurt. I know people who, knowing how
blunt they can be, ask someone else to read certain emails before sending
them. After all, reality is that people DO have feelings.

That massive filesystem thread

dirtyepic — Sat, 04 Apr 2009 07:44:04 +0000

nevermind, you said cache didn't you.

That massive filesystem thread

dirtyepic — Sat, 04 Apr 2009 07:42:21 +0000

wouldn't mutt keep telling you there's new messages an hour after you've read them then?

fsync() and disk flushes

giraffedata — Sat, 04 Apr 2009 00:01:25 +0000

The answer is RAID and UPS, but not that way. The RAID goes over the UPS; e.g. a mirror of two disk drives, each with its own UPS.

Such redundancy also makes it possible to test the UPS regularly and avoid the problem of two dead batteries when the external power fails.

The UPS doesn't count if you don't test, measure, and/or replace the its battery regularly.

fsync() and disk flushes

giraffedata — Fri, 03 Apr 2009 23:49:58 +0000

It's hard to believe there are disk drives out there (not counting an occasional broken one) that write trash over random areas as they power down. Disk drives I have seen have a special circuit to disconnect and park the head the moment voltage begins to drop. It has to park the head because you can't let the head land on good recording surface, and it has to cut off the write current because otherwise it's dragging a writing head all the way across the disk, pretty much guaranteeing the disk will never come back. I believe it's a simple circuit that doesn't involve any controller intelligence.

There is a related failure mode where the drive's client loses power and in its death throes ends up instructing the drive to trash itself while the drive still has enough power to operate normally. I've heard that's not unusual, and it's the best argument I know for a UPS that powers a system long enough for it to shut down cleanly.

fsync() and disk flushes

nix — Fri, 03 Apr 2009 00:02:55 +0000

RAID doesn't really solve sudden-power-loss situations: in fact RAID-5 in
particular can make it much worse (turning small-range corruption into
apparent scattershot corruption).

A UPS, or battery-backing, is the answer (well, moves the failure point:
if it's a UPS, the UPS must fail before you lose: if it's battery-backed,
you often have to lose the battery first, then power, which is likely to
happen because you often have no idea the battery has failed until it's
too late).

In conclusion: we all suck, our data is doomed, the Second Law shall
triumph and Sod and Murphy shall dance above our mangled filesystems.

sticks & stones

bojan — Thu, 02 Apr 2009 23:56:35 +0000

Nothing to forgive. All is perfectly fine. I enjoy a robust discussion.

fsync() and disk flushes

xoddam — Thu, 02 Apr 2009 23:31:45 +0000

I haven't crashed many disks but my limited experience is similar. If you get to the point where data loss is caused by the hardware, it is likely to be trashing a whole lot more than the contents of the cache at shutdown. The solution to this problem is RAID; journals solve a different problem altogether.

fsync() and disk flushes

anton — Thu, 02 Apr 2009 23:28:33 +0000

And if you suddenly lose power, in his experience, the drive is actually much more likely to wipe out some arbitrary track of data from the disk than it is to have anything in the write cache and lose it.

While I have experienced drives that damage sectors or tracks on power loss, I consider these drives faulty; and with such drives the problem does not seem to be limited to drives that are trying to write something at the time. However, most drives don't wipe out arbitrary data in my experience.

But I have tested two drives with a test program for out-of-order writing, and found that they both wrote data several seconds out of order with a certain access sequence. If we don't see more frequent problems from this, that's probably because the disks don't optimize accesses as aggressively as some people imagine.

sticks & stones

xoddam — Thu, 02 Apr 2009 23:17:21 +0000

> In a post not so long ago, someone accused me of hiding behind Ted's authority

I plead guilty and I apologise. That was immediately after replying to someone else's post the gist of which was "Ted wrote ext2 and ext3 in the first place, he is therefore above criticism." It concluded with the words "Know your place", which got me riled.

[proverb: in the midst of great anger, never answer anyone's letter]

Your words were not so condescending but they had much the same emphasis: all ur filesystems are belong to POSIX (not users) 'cos POSIX is the law, and by the way Ted's interpretation is the only correct one because he's the primary implementor.

I hope you understand where I was coming from. Forgive me.

That massive filesystem thread

anton — Thu, 02 Apr 2009 23:16:34 +0000

The problem with what one might call the fsync() RANDOMLY_LOSE option is that it is something which must be used by everyman to avoid data loss, which if you get it wrong there is no sign unless you lose power at exactly the right time, and which nearly all programs you might clap eyes on other than Emacs have historically got wrong

s/other than/including/. However, I don't agree that this application behaviour is wrong; if the application wants to jump through hoops to get a little bit of extra safety on low-quality file systems, that's ok, but if it doesn't, that's also ok. It's up to the users to chose which applications they run and on which file system.

That massive filesystem thread

iabervon — Thu, 02 Apr 2009 20:23:21 +0000

Linus actually overstated git's use of fsync(). There are three relevant cases:

git writes to a brand new filename, and then updates an existing file to contain the new filename instead of an old filename. It will optionally do a fsync() between these two operations.
git writes all of the data from several existing files to a single new file, and then removes the existing files. It will always do a fsync() between these two operations.
git updates an existing file by writing to a temporary file and renaming over the existing file. It will never do a fsync() between these two operations.

That is, git relies on the assumption that a rename() is atomic with respect to the disk and dependent on all operations previously issued on the inode that is being renamed. It uses fsync() only to make sure that operations to different files happen in the order that it wants.

Now, obviously, if you want to be really sure to keep some data, write it once and never replace it at all. That'll do a good job of protecting against everything, including bugs where you do something like "open(), fsync(), close(), rename()" but forget or mess up the "write()". Obviously, this isn't an option for a lot of situations, however, but it's what git does for the most important data.

fsync() and disk flushes

iabervon — Thu, 02 Apr 2009 19:15:59 +0000

Linus pointed out that, in his experience, the only way you can possibly lose data that has gotten to a drive's write cache is to suddenly lose power. And if you suddenly lose power, in his experience, the drive is actually much more likely to wipe out some arbitrary track of data from the disk than it is to have anything in the write cache and lose it. That is, if you want to be really certain that you don't lose data, you need to have a couple seconds of power for your disk drive after its input goes idle, or you might lose some data completely regardless of anything the application that writes it could possibly do.

Rename undo

butlerm — Thu, 02 Apr 2009 18:13:21 +0000

You are right, if you want guaranteed preservation of in-order semantics
there are no alternatives other than journalling the data or delaying all
journal commits until the corresponding data has been written. Both
options are available (e.g. data=journal, and data=ordered), and both have
serious performance problems. Of course, if that is really what you need,
than the price is worth paying.

"data=writeback" is the current alternative which doesn't make any pretense
to the preserving in-order semantics of data and meta-data after a crash.
You get a snapshot of your meta-data at a certain point of time, but the
data may be trashed.

Rename undo is a much less severe compromise to in-order semantics after a
crash. It is not point in time recovery, it is consistent version recovery.
That can have some unexpected side effects, but none remotely as severe as
losing the data completely.

In the case you mention, if you write a new version, rename it over the old
one, change the security permissions on the replacement, and then the
system crashes, you are not going to get the new (unwritten) data, the new
inode, and the new permissions, you are going to get the old inode, the old
data, and the old permissions. The permissions go with the inode (and the
data), not the directory entry. That is what you want. The old inode (and
the old data) has to be kept around until the data for the new inode is
completely on disk. Otherwise you cannot undo the rename replacement after
the fact.

Totally superfluous.

rvfh — Thu, 02 Apr 2009 14:30:38 +0000

> The application developer who has the choice of using these options doesn't know whether the data is expendable or not
How then are they expected to know when to flush?

And anyway, do we really not know which files are important and which are not?
Examples:
* pid file, browser cache: don't care
* conf file, document, code: care
* database file: care a lot

But I do thank you for challenging this idea ;-) Please feel free to give counter-examples and -arguments.

Totally superfluous.

xoddam — Thu, 02 Apr 2009 14:14:49 +0000

The application developer who has the choice of using these options doesn't know whether the data is expendable or not, and the end-user doesn't care about the implementation details.

Things go strangely pear-shaped when the most irrelevant, trivial data (eg. GNOME configs when we're only using GNOME because it's a default someone else chose) goes missing or gets corrupted.

I most definitely don't care if GNOME forgets where I put a window or two. But I do care if it fails to start.

What we end-users want (I wear a developer hat much of the time but I'm *always* a user) is not to be annoyed by the things we don't care about. O_EXPENDABLE and its ilk are an invitation for corner-cases to bite end-users. End-users don't deserve such treatment.

Rename undo

xoddam — Thu, 02 Apr 2009 14:01:34 +0000

> If one wants high performance rename replacements, rename undo is much more practical.

I'm intrigued, but not satisfied. Telling the journal that a metadata change is 'committed' means that the post-crash-recovery state will reflect the change (journal replay).

Surely the only satisfactory way to commit data before committing the metadata change is to delay *all* journal commits in-order until after the relevant file data is written in place, or to journal the data itself.

For performance reasons it's probably much saner not to journal most data, especially for random access within large files, but I'm thinking that if it makes sense to allocate-on-commit to preserve the in-order semantics of atomic rename, it might also make good sense to special-case data journalling for newly-written (created or truncated) files when they are renamed (perhaps only for small files, and allocate-on-commit larger ones as users will likey expect a delay).

Having the ability to unwind a specific kind of metadata change seems very confusing. I fear that winding back a rename could well result in a violation of expected in-order semantics w.r.t. metadata after crash recovery. Or might it be possible to wind back an entire 'transaction', all other metadata changes since the rename included?

That massive filesystem thread

butlerm — Thu, 02 Apr 2009 06:40:34 +0000

'softupdates' is proof of the complexity of keeping two fully generalized
versions of the meta data around at all times - not only that but
converting back and forth between the two versions on demand.

You can't really "queue" a rename, without doing something comparable to
what softupdates does, because the rename has to take immediate affect from
the application perspective. To do that, somewhere there must be a layer
to keep track of the difference between what the user visible meta data is,
and what the committed meta data is. If the differences are sufficiently
general, that is a major problem. If one wants high performance rename
replacements, rename undo is much more practical.

It would be practical to update atimes on a low priority basis, with the
caveat that a lot of memory may be consumed holding metadata blocks around
until the atime updates are complete. In addition, on a system under
sufficient load, moving I/O to a low priority thread doesn't really help
anyway.

Recursive linking

man_ls — Thu, 02 Apr 2009 06:21:51 +0000

Sorry, it was a stupid attempt from a foreigner at an April Fools' prank :D I was hoping that the recursive link would give it away, but maybe it was too plausible altogether.

Will try to do better next time :D)

That massive filesystem thread

davecb — Thu, 02 Apr 2009 00:37:20 +0000

How do filesystems other than ext3/4 do it?

Well, the Unix v6 filesystem implemented
in-order writes, as did 4.x BSD and the
other pre-journaled filesystems. POSIX allows
reordering to make coalesence easy, as
a lot of research was being done at that
time to get better performance.

A colleague at ICL (hi, Ian!) did his
masters at UofT on that, and found you
could get a performance improvement
and still preserve correctness by using
what I'd characterize as tsort(1), which
worked better than BSD/Solaris soft updates.

--dave

That massive filesystem thread

bojan — Wed, 01 Apr 2009 22:38:42 +0000

Just a few points, so please don't get offended. I apologise in advance to all sensitive LWN readers for any injury caused by this post.

> Why invent a new system call which cannot (by necessity) be honored by ext2, or ext4 without a journal?

Even if there was some kind of magical law that said that you could not order commits on the non-journaled file system this way, it can always be trivially implemented through - wait for it - fsync(), which has acceptable performance characteristics on such file systems.

> Everything is working now fine in ext3

Sure. Except fsync(), which locks the whole system for a few seconds. Hopefully, this will get fixed (or at least its effect reduced) as a result of the hoopla.

> Well, now that Ts'o's commit rights have been officially revoked I think that the whole discussion is moot.

Now you are really making a fool of yourself.

ext4 trees

corbet — Wed, 01 Apr 2009 21:46:53 +0000

I'm confused. The article said that Ted's trees had not been pulled yet. In fact, that happened today; a bunch of ext4 work went into the mainline, including a number of patches which increase robustness for applications which don't use fsync(). I dunno what you were trying to link to, but it didn't work. I've not seen anything about revocation of commit rights. (It's hard to "revoke commit rights" in a distributed system in any case; at worst you can refuse to pull from somebody else's repository.)

Maybe it's an April 1 post that went over my head?

That massive filesystem thread

man_ls — Wed, 01 Apr 2009 21:37:29 +0000

It is a worthless effort. Each filesystem must keep its house clean. Why invent a new system call which cannot (by necessity) be honored by ext2, or ext4 without a journal? Everything is working now fine in ext3, and if it doesn't work right in ext4 people will just look for a different filesystem.

After reading that Linus is not pulling from Mr Tso's trees made me suspect. Well, now that Ts'o's commit rights have been officially revoked I think that the whole discussion is moot. I wonder if the next ext4 head maintainer will learn from this painful experience and just do the right thing.

That massive filesystem thread

bojan — Wed, 01 Apr 2009 20:55:12 +0000

I will just answer this one comment, so that nobody gets "offended".

Quite the opposite. I'm all for fixing bugs and giving application programmers the _right_ tools for the job. If some Linux developers took a second to lift their noses out of the specifics of Linux and actually looked around, this could be fixed for _everyone_, not just for some Linux specific file systems. That is my point, in case you didn't get it by now.

That massive filesystem thread

Steve_Baker — Wed, 01 Apr 2009 19:41:09 +0000

Perhaps instead they could just invert the meaning of the 'A' file
attribute when a filesystem has been mounted with the noatime or relatime
option to force strict atime updates for files so marked. That way you
can mount your filesystem(s) noatime and only put the A attribute on your
mailboxes and you're done.

I/O barriers and LVM

quotemstr — Wed, 01 Apr 2009 19:08:51 +0000

No, they don't do this. LVM's lack of barrier support has resulted in my not using LVM in some circumstances. If kernel.org kernels grow LVM barrier support and are otherwise stable, it might be worth using them until distribution kernels are updated to include the functionality.

I/O barriers and LVM

sbergman27 — Wed, 01 Apr 2009 19:03:54 +0000

"""
This change alone is making me consider using a kernel.org kernel, something I haven't done in years.
"""

Are you saying that distro kernels already do this? If so, I'm not suggesting that you are wrong. Just interested in more info.

That massive filesystem thread

quotemstr — Wed, 01 Apr 2009 18:09:36 +0000

By your logic, we should never fix bugs. Remember the 25 year old readdir bug? Don't you agree it was good to fix that? What if a program, somewhere, depended on that behavior? In reality, programs use rename for atomic replacement. POSIX doesn't say anything about guarantees after a hard system crash, and it's just disingenuous to think that by punishing application authors by giving them as little robustness as possible, you're doing them some kind of portability favor.

Yup. It's the beginning of the end.

foom — Wed, 01 Apr 2009 16:10:07 +0000

This is starting to get very repetitive...all these points have been made already at least once in one
of the other article's threads. I'd like to suggest that it might be in everyone's interest to move on to
more useful pass-times than rehashing the same arguments over and over again every time there's
an update on the subject.