LWN: Comments on "The many faces of fsck"

The many faces of fsck

rhkramer — Wed, 22 Jul 2009 15:28:27 +0000

Echoing others, this is a great article--thanks! (I say that even though
I haven't read the entire article yet.)

Nevertheless, I do want to make one comment (somebody can correct me if I
really am wrong):

For a long time it's been my understanding that certain "fixes" by fsck
(and similar fixes in the Windows world, like in chkdsk or whatever "they"
call it) result in the deletion of data.

I think that is alluded to in the somewhat cryptic statement: "Repair here
means, as it does in the rest of the article, returning the file system to
a usable consistent state, rather than to some platonic ideal of what the
file system would have been without the corruption."

I personally feel that it would be much better to be less cryptic, even
for (and maybe especially so for) Linux newbies.

(I do recognize that some of the events that prompt the use of fsck (like
an improper shutdown (possibly due to a power loss event), can cause data
loss regardless of what happens during a(n) fsck.)

hard links to directories

rbfair — Sun, 21 Oct 2007 17:52:41 +0000

Apparently Apple has implemented directory hard links in their hfs+ 
filesystem, and it is the underpinning for the new 'Time Machine' backup 
in Mac OS-X leopard, where each incremental backup of an unchanged subtree 
is a single directory hard link. Seems like a very reasonable usage - though
Time Machine presumably will not work on NAS boxes lacking directory
hard link support (like Linux file systems.)

[Ref:
http://www.appleinsider.com/articles/07/10/12/road_to_mac...

hard links to directories

sflintham — Sat, 15 Sep 2007 14:59:11 +0000

For what it's worth, and assuming I haven't misunderstood, Acorn's RISC OS had/has the concept of thing-which-is-a-file-and-directory. There were programs that made things like zip files behave as if they were a directory, and IIRC when another program asked the OS 'is this a file or a directory', it returned a bitmap where (say) bit 1 signalled 'this is a file' and 2 'this is a directory', and both would be set. Naturally this slightly broke programs that were written before they added this feature which just checked for a 1 or 2 value, but I don't remember it causing too many problems.

Not trying to argue a case for or against making the change in Linux here, just thought I'd drop this in...

hard links to directories

nix — Fri, 14 Sep 2007 19:34:45 +0000

. and .. appear as hard links to users of the VFS. That doesn't mean they
have any physical existence, especially not at filesystem roots.

The VFS and layers below manage . and .. entirely on their own: users just
call mkdir(2), rmdir(2), or rename(2) on the directory and the kernel does
all the rest.

The file-as-directory idea, while beguiling (especially for structured
file formats) has *many* more problems, not least that whether specific
programs were willing to treat a file as a directory or vice versa, or
both, would depend on currently-irrelevant details of how they were coded.
It is currently safe for a program to stat() something and then switch on
its d_type; but what d_type does a file-and-directory return? If it's new,
nothing much will support it unless you hack the S_ISREG() and S_ISDIR()
macros to return true for the new type too (and that's an ABI change: new
glibc major version!): if it's nailed to S_IFREG, very little will treat
it as a directory, and vice versa if it's nailed to S_IFDIR.

I wish it could be implemented, but unless we're willing to throw away our
existing software, I suspect we'll be stuck with kludges that expose both
as different faked-up-but-aliased inodes somehow, one a file, the other a
directory.

hard links to directories

Duncan — Fri, 14 Sep 2007 11:07:38 +0000

Thanks. Useful information. I hadn't thought out the implications of ..
to multiple named hard-linked dirs myself, so I'm glad you pointed it out.
It also explains the notes on directory unlink that I'd read (the
root-only thing, and that the exception was directory specific) but never
really understood. I understood that the subject went deeper than I
understood; now I'm beginning to see just a few of the ways it does so.

But just to be sure, now, . and .. are indeed hard links within a
filesystem, on at many (but not necessarily all) filesystem
implementations? You specified that the entries in filesystem roots are
faked (as I was beginning to realize), but left not quite as nailed down
as I'd like that they are indeed hard links within the filesystem, even if
they are exception links that require kernel calls to create/destroy.

Anyway, the kernel calls thing explains somewhat why filesystems haven't
implemented this yet. That's an additional complication, and would
certainly be heavily scrutinized if anyone tried introducing a filesystem
of that nature into the kernel mainline... which pretty neatly ties into
exactly why reiser4 has had the problems it has had getting into mainline,
given the original file as directory idea and implementation thereof. Now
I understand the implications of /that/ debate a bit better too, and a bit
more about what the nature of the introduced races and the complaints
about them entailed.

Duncan

hard links to directories

nix — Fri, 14 Sep 2007 06:32:38 +0000

The . and .. directory entries in the roots of filesystems don't actually
exist, but are faked by the VFS. (There's no reason why the filesystem
couldn't generate all . and .. entries that way, and IIRC some do.)

Once upon a time . and .. really were real hardlinks,
and /usr/lib/{mk,rm}dir (IIRC) were setuid root programs that
created/removed the directory and manipulated these links. This turned out
to have unfixable races, so was replaced with an in-kernel call, but some
Unixes (notably Solaris) maintained the semantics that unlink() of a
directory as root was permitted (which caused some aggro on the git list
recently).

Multiple hardlinks to a directory are theoretically implementable, and
under Linux the biggest problem (`where does .. take us?') is solvable
using the dcache; but allowing it will confuse users to some extent.

(Linus said that he had no major objections to allowing directory
hardlinking --- the we don't corrupt filesystems kind --- about seven
years or so ago, but nothing happened after that that I know of.)

hard links to directories

Duncan — Fri, 14 Sep 2007 03:13:52 +0000

> So if a directory has a link count > 1, this must be an error.

What are all the (normally hidden) . (me) and .. (my parent) links if not
hard links?

Your assumption that additional links are not allowed doesn't necessarily
hold for all filesystem types, either. Looking it up, the GNU coreutils
info entry for ln has the following:

> On all existing implementations, you cannot
> make a hard link to a directory, and hard links
> cannot cross file system boundaries. (These
> restrictions are not mandated by POSIX, however.)

Thus multiple named directory links are apparently not only possible, but
not restricted by POSIX either, so an implementing filesystem could indeed
allow such things, it's simply that few (none known to the info entry
author quoted above at the time the statement was written) have done so.

Hmm... That leads to two questions.

1) What does creating a subdir do if it doesn't create a hardlink entry in
the form of .. back to the parent? Does the above only refer to directly
created named (not . and ..) links?

2) The no crossing filesystem boundaries is interesting. The kernel must
fix that up by creating a logical .. link in the root dir of a mounted
filesystem, since it self-evidently couldn't be a hard-link, given that
the filesystem could be arbitrarily mounted anywhere. Likewise, the
kernel must remove the .. entries from chroots.

So maybe those . and .. entries aren't hard links after all, which begs
the question (avoids the question, begs that it be asked and answered,
whatever [1], I'm a literal constructionist in this case) I asked at the
beginning, what /are/ they, then? However, that's out of my league.
Perhaps Val can trivially answer.

[1] http://www.onelook.com/?w=beg+the+question

Duncan

hard links to directories

gw666 — Thu, 13 Sep 2007 08:19:25 +0000

UNIX file systems have the wonderful quality of allowing more than one hard link to an inode (which can be file or directory).

You cannot hard link a directory:

$ ln foo/ bar ln: `foo/': hard link not allowed for directory

So if a directory has a link count > 1, this must be an error.

First availability of fsck

addw — Wed, 12 Sep 2007 06:16:01 +0000

the first fsck was designed for the Fast File System

No, fsck was available long before then with the original Unix file system. For instance see: http://uglx.org/sys3

War story

cstanhop — Tue, 11 Sep 2007 06:07:13 +0000

I just want to say that this is a great article. I've enjoyed the others along this line, and I look forward to more articles like this one.

My fsck war story is one where the fsck program was of absolutely no help, and I got to play human fsck for a day.

This was back in '96 on a Caldera Linux distribution with ext2. I had just finished installing and configuring a linux box that acted as the file, email, web server, DNS, dial-up router, and proxy for the three person company I was part of. It was a big deal for us, and it had taken me a better part of a week to get everything running as I had planned. Everything was complete except (feel free to shudder knowingly) we hadn't received the tape drive system we intended to use as the backup for the system. However, I was primarily working as an EE at the time, and as a third of the work force, I needed to get back to my other work.

The system was running fine for a couple of days, but then one morning I came in and my boss told me it refused to boot. My heart sank as I went to the terminal and saw some error about the file system not being recognized. A week's work gone! I had a suspicion somebody had neglected to shut it down properly because they had forgot the root password, but under my baleful gaze oaths were sworn that this was not the case. Anyway pointing fingers wasn't going to get the thing to boot, and I should have figured out a way to back it up even without the tape drive.

So, I tried booting from a floppy and running fsck, but fsck wouldn't have anything to do with the drive. I briefly considered the possibility of rebuilding everything, but I was determined to not lose all that hard work. So, I read up all the information I could find about ext2, and I fired up ext2ed. Working with ext2ed, it appeared that a drive sector in the root inode had become corrupted. Reading that sector gave errors, but I could poke around in other sectors. So, based on what I had been reading about ext2 and knowing how I had laid out the drives, I had ext2ed rewrite the root inode. Once ext2ed wrote the root inode, the drive began to read the sector without errors. Yea me!

So, now I ran fsck. Success again! Absolutely no errors! But we were still spooked, so my boss had run out and bought equivalent geometry drives from some other manufacturer. I "mirrored" the disks with dd, then I booted from the originals just to see if worked. It did, so we swapped in the new drive and tried it, and it worked too. As far as I could tell, nothing had been lost and it was only that sector in the root inode that caused problems. We continued running with the new drives and kept another new drive as a mirrored backup for a while until we could get the tape backup working.

All I can say is "phew!" Luckily it only took me a better part of the day to recover from that potential data disaster. With a couple of earlier exception in the days of DOS, that's the closest I've had to get to my file system. I completely appreciate all the work people put into them so I can get on with the things I want to do. Even the slow fscks are appreciated (when they work). At least I learned the value of backups. Lots and lots of backups...

The Sound of fsck

BrucePerens — Tue, 11 Sep 2007 04:23:55 +0000

In the early 80's, Kirk McCusick pronounced it "fisk" while delivering his paper on the (then new) BSD FFS. I don't know how much closer to origin we can get.

Bruce

Always consistent

vaurora — Fri, 07 Sep 2007 00:48:32 +0000

Yeah, "wafl_check" indeed used to be "wack." I don't know the story behind the name change, but it seems likely that either the pronunciation was too confusing ("You want me to whack the file system? Are you crazy?"), or someone performed a humor-ectomy after NetApp got really successful.

Always consistent

jwb — Thu, 06 Sep 2007 20:34:47 +0000

NetApp's fsck is (or at least was, at one time) "wack" which is surely the easiest to pronounce.

Always consistent

rfunk — Thu, 06 Sep 2007 19:51:55 +0000

Yeah, the description of zdb reminded me of the early days of XFS on Irix,
when we were told that no fsck was necessary because it was always
consistent. Then later it was discovered that, yes, even journaling filesystems
can get corrupted, and xfs_check and xfs_repair were added. (Which I've
personally had to use on at least two separate occasions.)

Also, I must dispute the author's characterization of "xfs_check" as more
pronouncable than "fsck". fsck lacks that nasty underscore, so it's much easier
to tell someone to run fsck than xfs_check. "fsck" is also easier to say (however
you say it) and type.

The "yes" command existed before "-y"

hamjudo — Thu, 06 Sep 2007 14:04:05 +0000

There was no need to wear out the enter key.
Before fsck had -y, you could have used
yes | fsck /dev/rd02a

You should go back in time and tell the younger version of yourself that. It will save your fingers.

If you used fsck before the yes(1) command, then suggest to yourself that you write a yes(1) command, it is only a few lines longer than /bin/true.

The many faces of fsck

jwb — Thu, 06 Sep 2007 04:32:08 +0000

This is a great article. I'm glad you mentioned the mysterious WAFL fsck program and ZFS. I think people should approach claims of "always consistent" with extreme skepticism. The existence of WAFL's fsck is little-known because NetApp salesmen swear that it does not exist. The file system is always consistent, therefore you can't possibly need a fsck. That is, until you come in one morning and your filers are fscking themselves, and you call support, and support tells you "oh, that's the fsck, didn't you know about it?"

So when Sun claims that ZFS doesn't need a fsck, one tends toward disbelief. And given that they are 10 years behind NetApp in the file system design division, I bet they have less experience with all the weird modes of failure, too.