LWN: Comments on "The 2007 Linux Storage and File Systems Workshop"

fsck / xfs - versus ZFS

qu1j0t3 — 2007-04-17T14:06:11+00:00

It would be wrong to assume ZFS has the same failure modes as XFS. See, for instance: Bill Moore's blog.

Gee that's tough.

wmf — 2007-03-31T00:57:57+00:00

But ZFS doesn't use compare-by-hash; that's one of its advantages over previous work like Venti/
Fossil.

NVCache for journal

sweikart — 2007-03-25T02:20:38+00:00

> first, every write on the system would have to go through the
> flash, so longest life access pattern or not, it may still have
> problems.

Actually, for most of us journal users, only the metadata writes
go into the journal.

> ... is that the most efficient method to record an atime update
> in a journal?

You raise a good point. As a system administrator, I don't mount
with noatime, because atime is too useful. But, atime is not so
critical that I need it to be journaled; so, I'd love to mount
nojournaledatime.

-scott

NVCache for journal

dlang — 2007-03-25T01:48:01+00:00

useing flash for a journal would have some headaches (potentially fixable)

first, every write on the system would have to go through the flash, so longest life access pattern or not, it may still have problems.

second, when the write is complete the system needs to go back to the hjournal (flash) and mark it as being completed

third, the chunks of data going into the journal are of many different sizes (yes, in the end it all gets down to writing fixed size disk blocks, but is that the most efficant method to record an atime update in a journal? probably not)

good drivers and layouts that are flash aware may be abel to address these, if the API gives them sufficiant control (for example, since flash can be chagned from 0 to 1 without eraseing, make sure the 'entry commited flag is 1 when complete and you can just flip that bit, in theory)

DualFS

giraffedata — 2007-03-23T22:04:25+00:00

DualFS is a file system by Juan Piernas that separates data and meta data into separate file systems.

That's separate devices, not separate file systems. A single DualFS file system has both metadata and file data, and the innovation is that they are stored in such a way that access to one doesn't interfere with access to the other.

And it's worth noting that while ideally these devices would not share a head, the experiments reported are done with the devices being partitions of a single physical device and still show improvement.

The storage systems contract

giraffedata — 2007-03-23T21:56:49+00:00

Storage systems have a simple and important contract to keep: given user data they must save that data to disk without loss or corruption even in the face of system crashes.

s/to disk//

The fact that there's a disk in there, if there is one, is none of the other party's business. That's why clauses in that contract specifying what things such as "fsync" mean are so ambiguous.

The contract is actually quite complex in the area of what data is allowed to be lost in the face of a system crash.

Vectorized Read?

vmole — 2007-03-23T21:36:45+00:00

<p>Oh, duh, right. I knew that, at one time in the distant past. Sorry for the noise.

Vectorized Read?

giraffedata — 2007-03-23T21:34:26+00:00

... and you can read at specific offsets (which is what readv(2) gives you), isn't that being able to read specific blocks?

You need to be able to read not just specific offsets, but specific discontiguous offsets. readv() does not give you that. You have to read a contiguous area of the file (block device). You cannot make it read from 4K to 8K and 12K to 16K.

What readv() adds to read() is that you can read that contiguous file region into discontiguous memory, whereas with read() it has to go into a single contiguous range of memory addresses.

Vectorized Read?

zlynx — 2007-03-23T21:28:50+00:00

I am going on the man page for readv.

ssize_t readv(int fd, const struct iovec *vector, int count);
struct iovec {
void *iov_base; /* Starting address */
size_t iov_len; /* Number of bytes */
};

iov_base and iov_len apply to RAM addresses, not disk block or character addresses. The descriptor "fd" is read linearly for "count" iovec structures.

So, readv does *not* give you the ability to read at specific offsets. As I understand it from reading the documentation.

Vectorized Read?

vmole — 2007-03-23T20:22:08+00:00

Please explain further (I'm not doubting you, just trying to understand). If you open a block device (e.g. /dev/hda1), and you can read at specific offsets (which is what readv(2) gives you), isn't that being able to read specific blocks? I mean, I understand that it's not the underlying physical disk blocks, but isn't the mapping between physical disk blocks and the logical blocks that the filesystem sees all handled in hardware?

A very quick glance at the e2fsprogs source indeed seems to use open("/dev/hda1") and read/write.

Where's Mr. Tso when we need him?

Vectorized Read?

zlynx — 2007-03-23T20:05:49+00:00

readv and writev both use a vector of memory buffers, but they are not for writing a vector of disk blocks.

We probably need readiov/writeiov and readviov/writeviov or something like that.

I also had a crazy idea just now. What if they used device mapper to create a dm device with a linear view of every block fsck needed to read? Let readahead run on that.

How about readahead(2) or fadvise or posix_fadvise?

Vectorized Read?

vmole — 2007-03-23T16:55:23+00:00

Yes, that's what I meant by "RAW_IO on the partition" :-) But, again, if that would work, then readv(2) would seem to provide the required vectorized API, so why are the smart people saying they need a new API?

The 2007 Linux Storage and File Systems Workshop

nix — 2007-03-23T16:26:42+00:00

I can't see how you can eliminate opendir()/readdir(), but telldir() and
the wierdness around it could be zapped quite easily.

Outside of scripting language interfaces, the Linux kernel itself, and
glibc, I see hardly any uses of telldir() on a modern Linux box (Ruby,
Perl &c scripts aren't likely to depend on the detailed semantics of
telldir() anyway because non-POSIX systems don't implement them). The
Midnight Commander VFS implements a telldir operation but never appears to
call it...

... in fact, on my system here, I have no actual *uses* of telldir() at
all. Even strfry() is called more often.

I'd say telldir()'s semantics could be changed pretty easily. Something
like it might remain potentially useful, but its (extremely annoying to
implement) current semantics don't seem to matter to most real code.

NVCache for journal

xanni — 2007-03-23T07:41:14+00:00

This looks like it could potentially be a really big win for DualFS in particular!

Vectorized Read?

ldo — 2007-03-23T04:14:09+00:00

Because fsck needs block addressed access, and aio_read() is based on (fd, offset, count).

Hint:

fd = open("/dev/sda", ...)

NVCache for journal

sweikart — 2007-03-23T03:56:49+00:00

> Also, a file system can use the NVCache to store its journal ...

This seems like a great approach. If the NVCache could appear as
a block device that could be partitioned, filesystems that use jbd
(like ext3) could use it right away (a smaller NVCache partition for
read-mostly filesystems like /usr, a larger NVCache partition for
write-mostly filesystems like /var).

I assume journals are treated as ring buffers for writing, which
is the right access pattern for prolonging the life of flash.

-scott

The 2007 Linux Storage and File Systems Workshop

thedevil — 2007-03-22T18:48:30+00:00

>>Ted Ts'o suggested that someone should try to go through committee to get telldir/seekdir/readdir fixed or eliminated<<

So, if committee (POSIX I suppose) goes along with that, how are we going to scan directories?

Vectorized Read?

vmole — 2007-03-22T18:35:45+00:00

Because fsck needs block addressed access, and aio_read() is based on (fd, offset, count). Otherwise, you don't even need AIO; you'd just use readv(2). OTOH, it seems like readv() would be sufficient given the RAW_IO to the partition...so maybe someone else should answer this :-)

The 2007 Linux Storage and File Systems Workshop

jwb — 2007-03-22T17:08:02+00:00

What do you mean by Ext4 integration? Lustre already includes all Ext4 features and then some. In fact you might say that Ext4 is just rolling features into the mainline kernel that have long been used in Lustre. mballoc, delalloc, and extents have all been in Lustre for years.

What ext4 really needs is ...

jospoortvliet — 2007-03-22T15:16:46+00:00

A /me too from here. I hate file corruption, esp if it goes unnoticed... Having some redundancy in the filesystem (reiser4 promised to bring that, zfs as well) would be great as well, to allow users to recover bad files.

Vectorized Read?

jospoortvliet — 2007-03-22T15:02:28+00:00

Wouldn't this increase latency? And doesn't the kernel already do this, in a limited way?

The 2007 Linux Storage and File Systems Workshop

nix — 2007-03-22T11:56:05+00:00

Of course Coda itself was an enhancement of AFS (losing most of its scalability in the process, though)...

The 2007 Linux Storage and File Systems Workshop

nix — 2007-03-22T11:53:37+00:00

The page cache is aware of blocksizes differing from PAGE_SIZE, which provides a lot of what's needed, but that code is complex and delicate, and extending it to allow pieces of multiple files to co-exist in a single page-cache page is quite unlikely to be done (the memory savings are, after all, marginal, at half a page per file, and the complexity increase is significant).

fsck / xfs

wookey — 2007-03-22T11:47:46+00:00

I too have found the hard way that yanking the power on XFS (or just hitting reset at a bad time) is a very bad idea. All the files that had pending writes just end up as the correct length of zeros. When this is includes your package database, perl binaries and a load of other libs, this is quite bad.

The xfs_repair tool did do a pretty-good repair job (once I fixed it so it ran! http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=414079) but it did take about 5 hours to do it on a pair of 200GB mirrored drives. Then I got to re-install everything to fix the damage.

About 3 days faff in total. Fair dues though - there was no user-data loss and the system was recoverable, but I've never had this trouble with reiser3 on my laptop or ext3 on other boxes. So, yes, XFS is a really nice filesystem (live resizing, nice and fast) but I'd avoid it unless there is a UPS around.

The 2007 Linux Storage and File Systems Workshop

wingo — 2007-03-22T10:20:23+00:00

This would be a nice topic to revisit in an article -- networked filesystems. Samba for unix<->unix is something I'd like to know more about.

Gee that's tough.

snitm — 2007-03-22T05:55:29+00:00

LVM2 Snapshots are quite bad. For starters they are done at the block-level whereas ZFS provides file-level snapshots (aka redirect on write). LVM2 snapshots don't scale well either; seeing as each snapshot imposes a copy out penalty because there isn't a shared exception store (aka LVM snapshot LV) for all snapshots of an origin LV.

The 2007 Linux Storage and File Systems Workshop

drag — 2007-03-22T02:01:26+00:00

I don't take them too seriously. :)

But they aren't to far off. One thing worth noting is that Ext4 integration isn't mentioned anywere on them, but it's obvious that Ext4 is going to play a large part in it.

One thing about CFS, which I think is important to keep in mind, is that they are decendents of the failed Coda and then the Intermezzo projects. I don't know the exact relationships, but I think that they were developers in those projects.

The thing is is that they learned the hard way that distributed network file system protocols aren't a easy thing to make, even if you are good at it. It takes a lot of time and effort to get anything going and a long time of development to get to the point were you can actually release anything.

So it's not something that lends itself to the Linux-style development proccess of 'release early', 'release often'.

So they formed CFS to pursue the money nessicary to support themselves while they hacked on Lustre full time. The HPC market is the easiest and most profitable place to target for this sort of stuff, and they know that from Beowolf stuff that open source and distributed computing can lead to dramatic results.

Persistant allocation without zeroing :== bug

davecb — 2007-03-21T17:40:25+00:00

To be fair, I should comment that preallocation of disk blocks
for write is a **good** idea, and one which the Samba team,
amoung others, would welcome.
The "impedance mismatch" between FAT and NTFS on one
hand and Unix file systems on the other has caused lots of
problems wen Unix servers return an out of space indication
in circumstances where Windows clients think an error can't
happen (;-))
Of course, if the preallocated space isn't written to, I'd hope
the blocks would be zeroed on close, and previous to the close,
would not be avilable to be read. The latter introduces an impedance
mismatch with Unix files opened fro both write and read...

--dave

Persistant allocation without zeroing :== bug

davecb — 2007-03-21T17:19:48+00:00

Oh quite: that's what the GCOS programs was
intended to do. It just didn't succeed.

--dave

Gee that's tough.

bronson — 2007-03-21T16:06:04+00:00

For snapshotting, just run whatever filesystem you want on top of LVM. LVM is a little hard to get used to at first but it's definitely worth the effort. Here are some notes I took when setting it up on my systems: http://wiki.u32.net/LVM

I now put all nontrivial partitions on LVM. Works for me. I'll let others argue whether LVM snapshots are worse than ZFS, or if ZFS is a layering violation. :)

The 2007 Linux Storage and File Systems Workshop

k8to — 2007-03-21T15:53:00+00:00

You've mentioned this before in response to my discussing of issuses encountered trying to use NFS for similar purposes.

My response is still that sshfs is a great thing, and pretty useful for trivial tasks, or remote manipulation of low-bandiwidth, high-latency small set sof files. It's much easier to edit some remote configuration thing with a local tool via sshfs than most anything else.

But sshfs still can't handle a variety of normal file sharing activities reasonably. It fails entirely on mmap and large files make it choke because it hasn't got sufficient cache sophistication. Over a LAN you'll never get 10% of your throughput while maxing your CPUs on the ciphers. If the ssh link actually goes down (this happens), the whole thing gets very unhappy and it is impossible to recover.

Basically the only thing that makes sshfs "better" than the traditional lousy network filesystems we love to hate is that it has a well defined focus. It's a no-server-configuration filesystem for accessing small numbers of smallish files without high performance expectations. It is a remarkably pleasant tool when used inside its scope, but one of the reasons it is pleasant is it has a much narrower scope.

The 2007 Linux Storage and File Systems Workshop

saffroy — 2007-03-21T13:47:24+00:00

It's nice that you mention Lustre, actually I was kind of surprised that it would not be mentioned in this article. Lustre definitely has a great potential (great scalability, sequential I/O performance, client cache, excellent POSIX conformance), and I feel it could be a good general purpose global fs someday.

That is, if its creators (CFS) let it grow out of its niche HPC market: at the moment, I feel they're more concerned about implementing the features asked by their paying customers, which are big supercomputing centers. I'm certainly not blamining them for that, but for instance, they are more sensitive to large file throughput (tens of GB/s) than to file creation rates (Lustre is still damn slow here).

If the community or the customers push in the right direction, Lustre can become an excellent distributed fs for nearly everyone, but I feel it has yet to happen -- and I hope it will.

Oh, and don't take CFS roadmaps too seriously. ;-)

Gee that's tough.

mennucc1 — 2007-03-21T13:40:00+00:00

> Most of the features of ZFS are aviable on Linux right now.

but still I would like to have snapshots in EXT (maybe in 5 :-) ?)

The 2007 Linux Storage and File Systems Workshop

nix — 2007-03-21T12:25:46+00:00

Of course, you can't usefully store named pipes or devices on NFS-shared filesystems, either...

The 2007 Linux Storage and File Systems Workshop

drag — 2007-03-21T02:54:33+00:00

OpenAFS is, indeed, very nice.

But it's Windows support is crap. Not because OpenAFS is not cool, but because Window uses SMB or Microsoft's DFS to do it's thing and nobody else's. OpenAFS has to use a sort of SMB emulation were it deals with AFS stuff then translates that to something that the system can use.

But if your just dealing with Linux clients then that's not a problem.

Also the file and directory permission model is bizzare and isn't realy compatable with just standard Unix-style ACL (user/group/world read/write/execute) model. So people used to Linux permissions have to relearn how to deal with AFS permissions.

It's not posix, and it's not compatable with special file types like named pipes.

Also there is no real way to access your data unless your AFS server stuff is actually running. OpenAFS tends to incure a higher amount of knowledge and administration stuff isn't very easy to deal with.

Then it's large file performance is realy bad. It's just plain slow and the volumes are very limited in size.

What it's VERY good for is if you have a large distributed network.

Say you have a wireless network or a WAN-wide thing were you have a entire campus of computers to take care off. It handles disconnection very well, it's caching support is very good for semi-offline work (ie you can still edit a file even if you temporarially lost contact with the servers.

It's security stuff is nice. The volume management is very nice, snapshotting and mirroring stuff. It's safe to use over the internet and unencrypted wireless networks.

And as a special bonus it's /afs/ directory tree is very handy. It allows people to move volumes around, change servers, setup mirrors, and all sorts of stuff without having to have the clients know of any of these changes. Were as with NFS or SAMBA if you change out file servers or whatnot then the clients all have to be reconfigured to know the new locations and names of the servers and directories. With OpenAFS this is not nessicary.

But considuring the lack of posix support and poor large file performance as well as permission issues it's not realy a replacement for NFS. It's a alternative that is usefull in places were NFS is not.

And it's poor Windows support means that it's not usefull as a replacement for Samba.

But it's nice.

The OpenAFS points out a huge problem for Linux in general though. AFS is ancient. It's old old old. It's like X Windows/Athena/Kerberos ancient. Still, even with it's age, it's still MUCH more sophisticated then NFS or SMB network protocols. Nobody has realy produced anything better.

Lustre, maybe. It certainly has a lot of cool features and is fast. But I don't think that it has any security.

Supports lots of stuff. TCP networking, ininaband, and all sorts of other bizzare interconnects.

Supports ACLs, extended ACLs, extended attributes. Lots of high aviability and high performance features. Failover, extra redudancy. You can use it as root FS. It supports Quotas.

It doesn't require special patches and kernel recompiles for Linux client support.

The only thing that it lacks is robust security. They plan on supporting GSSAPI and Kerberos with the 1.8.0 release. This is due out by the end of this year according to their roadmap...
For Unix and Windows comaptability it supports SMB and NFS v2/v3/v4 export.

The 2007 Linux Storage and File Systems Workshop

saffroy — 2007-03-21T00:02:55+00:00

"There's no good solution today."

Of course there is one, it's called OpenAFS. Except installing and managing an AFS cell is not quite easy (although the Debian packager provides useful scripts to quickly start a simple server)...

But AFS has its share of good features:
- good Linux and Windows clients (and other *nixes as well)
- Kerberos authentication
- powerful ACLs
- excellent scalability: large sites (MIT, CMU...) run cells with thousands of clients (and several servers sharing the load of course)
- client-side caching using local storage
- the namespace is global, users can use the same paths on all clients
- data is organized in volumes (subtrees) that can be mounted anywhere in the global namespace, both by admins and users
- volumes can be snapshotted, and users can mount snapshots (no need for the admin to restore from tapes when user accidentally deletes a file and notices immediately)

...and more. :)

I've used OpenAFS on Linux for years, and never complained (it helps that I have a competent and knowledgeable admin).

Persistant allocation without zeroing :== bug

saffroy — 2007-03-20T23:46:59+00:00

The most common use I see for this feature is to have (more) predictable performance, and no allocation error (ENOSPC) when *writing* to a file (eg. for realtime apps that capture data). If you plan to capture large amounts of data, zeroing blocks is not very convenient...

The 2007 Linux Storage and File Systems Workshop

drag — 2007-03-20T23:25:22+00:00

I doubt it's usefull for large numbers of users, but for my personal stuff the fuse-based Sshfs is a superior replacement to NFS or Samba.

Faster, strong encryption, strong authentication aviable, trivially easy to setup. Robust.

The downside is that you can't use it for anything that requires special file typs, like named pipes. So ~/ is out. No booting from it. But for serving up large media files or sharing abritrary directories between computers it's great.

Persistant allocation without zeroing :== bug

k8to — 2007-03-20T22:28:19+00:00

Giving them the benefit of the doubt, perhaps they mean that the space is allocated, but the zeroing is done lazily. Ie. hand zeros to reading apps reading uninitalized space, or perhaps errors, or whatever. Ie. allocation doesn't necessarily mean you get to see what's there. Yes allowing apps to read uninitialized reclaimed disk space leads to data leaks.

The 2007 Linux Storage and File Systems Workshop

k8to — 2007-03-20T20:03:01+00:00

The piece I haven't seen so far is "here's how to configure samba to remove all the crap you don't care about if you haven't got windows". Granted, the Samba 4 hasn't shipped, but a document or setting along those lines would be useful *now*. Does it exist?

When I went looking I found a whole lot of complexity. I was really *not* interested in learning all the details of SMB and windows networking, I just wanted my Linux machines to share files.