LWN: Comments on "Btrfs at Facebook"

Btrfs at Facebook

flussence — Wed, 26 Aug 2020 11:45:44 +0000

Sounds like email, or at least the variants that don't stuff multi-gigabyte mailboxes into a *single* file...

Btrfs at Facebook

anselm — Fri, 21 Aug 2020 17:18:19 +0000

Things are maybe not so bad if – like Facebook – you have actual file system developers on your payroll.

Btrfs at Facebook

rep_movsd — Fri, 21 Aug 2020 16:31:55 +0000

Back in 2003-2004 I was on a small team that built an instant messenger client called Rediff Bol 2.0

Version 1.0 had been a huge success in India, but they wanted a snazzy UI like Yahoo.
Their message server used this same retarded technique of saving messages as files.... on a shared NFS mount.

I won't go into the horrors of such a design idea (which to be fair did have one thing going for it - simplicity) but it's insane to use a filesystem as a database.

Im sure FB devs had their reasons (maybe legacy), but fundamentally a crap idea. Filesystems are designed to be moderately efficient at various sizes of files.
I don't think any FS dev in their wildest nightmares would have considered a use case of zillions of files which are smaller than a floppy disk sector

Btrfs at Facebook

redneb — Tue, 11 Aug 2020 22:17:34 +0000

This is not true anymore, it was a bug that was fixed in 4.14 (11/2017). There are two commits about this and the pull request says: "degraded read-write mount is allowed if all the raid profile constraints are met, now based on more accurate check".

Btrfs at Facebook

anton — Sun, 12 Jul 2020 15:35:46 +0000

Basic striping and mirroring work well, and have done so for years

If only btrfs was user-friendly when a drive of a mirror pair fails; instead it gives you one shot at dealing with the problem, and on the next reboot becomes irreversably read-only; that was certainly the state of the FAQ when we installed a new server earlier this year. So we decided to try out ZFS instead, and its handling of a simulated disk failure and replacement was very smooth, hardly perceptible.

Btrfs at Facebook

khim — Thu, 09 Jul 2020 20:51:52 +0000

Useful automatic deduplication is work in progress.

You can run duperemove in background and everything would be nicely deduplicated but performance would be so awful then you just don't want to do that.

But if that's developer's system you can do that when developer is sleeping - then it works nicely.

Btrfs at Facebook

intelfx — Wed, 08 Jul 2020 10:22:18 +0000

> Then there is the issue of RAID support, another longstanding problem area for Btrfs. Basic striping and mirroring work well, and have done so for years, he said. RAID 5 and 6 have "lots of edge cases", though, and have been famously unreliable. These problems, too, are on his list, but solving them will require "lots of work" over the next year or two.

I'd be *really* pleased if this was to happen.

Btrfs at Facebook

jezuch — Tue, 07 Jul 2020 12:15:56 +0000

Git actually deduplicates even binary files - that's a convenient side effect of content-addressable storage. Btrfs is in theory able to share extents between identical copies of files, but I think you have to tell it these are copies - automatic deduplication is a work in progress I think?

Btrfs at Facebook

TMM — Tue, 07 Jul 2020 03:37:41 +0000

I don't work for facebook, but given that they said they used cheap flash drives I'm assuming that they are also using somewhat cheap raid controllers. The controller where I saw this was a highpoint rocketraid controller which did the same thing. I imagine that more cheaper (soft)raid controllers which are actually mostly just sata controllers do this to write their raid metadata.

Btrfs at Facebook

ncm — Sun, 05 Jul 2020 21:51:07 +0000

Or, indeed, just about any conventional database.

Btrfs at Facebook

ncm — Sun, 05 Jul 2020 21:02:01 +0000

Facebook seems to prefer SSD, for Reasons, but a TB SSD is pretty normal now. FB can certainly afford a thousand per developer, and something to plug them all into, what with large multiple $millions annual profit per employee. Not spending as much per head as could possibly be useful would be irresponsible even for a wildly less profitable enterprise. If they don't *all* have 3990Xes with 1TB RAM and 3 4k screens yet, heads should roll.

But as a practical matter, I gather from discussion with core devs (a couple of years ago) that it is part of FB's received engineering culture to copy dependencies, so there are, e.g., hundreds or thousands of copies of libz scattered throughout their repository, in many, many versions, and likewise now, I expect, libzstd. Btrfs would be able to share blocks for them when unpacked, but maybe not so much so as represented within git?

Imagine the engineering effort just to consolidate all those libz instances, identifying version, bug, and local patch dependencies, while every week more appear. Imagine the effort just to bring them all up to the current release: not a thing to make your CV shine.

Btrfs at Facebook

caliloo — Sun, 05 Jul 2020 19:02:19 +0000

Whatsapp was built with erlang, i wouldn’t be surprised this has something to do with their method of message storage....

Btrfs at Facebook

josefbacik — Sun, 05 Jul 2020 02:10:48 +0000

There’s an index log that compresses well. Mostly they’re in it for the snapshotting.

Btrfs at Facebook

gus3 — Sat, 04 Jul 2020 22:03:19 +0000

I just (an hour or so ago) bought a 4T USB drive for backups for less than $100. My old backup drive is showing some age, so it was time.

That's big enough for four 800GB containers mapped to virtual space, plus filesystem overhead. The only choke-points would be the USB3 connection and head seek time on the platters. Of course, SSD would eliminate the latter.

It isn't bleeding edge, or even leading edge anymore. It's COTS (commodity, off-the-shelf).

Btrfs at Facebook

pizza — Sat, 04 Jul 2020 12:04:27 +0000

> For a complex system, there will be different levels of design and taking into account low-level technical details for the high-level design would be wrong.

I'm not talking about "low level details", I'm talking about being aware of what is or isn't possible (or practical) to implement.

Outside of a handful of fields (mostly to do with advanced mathematics) those "designs" still need to be realized in the real world.

Btrfs at Facebook

scott@deltaex.com — Fri, 03 Jul 2020 22:52:04 +0000

Compressing ciphertext is effectively the same as cracking it.

Btrfs at Facebook

nivedita76 — Fri, 03 Jul 2020 22:49:10 +0000

I didn't understand it that way: they're not talking about compression issues when discussing the messages, but problems with storing them along with the metadata instead of in separate blocks. My guess is that if text messages are compressed at all, the messages are compressed, then encrypted and sent; and decrypted and decompressed on the recipient's phone. The larger ones for photos, videos, audio would already be in a compressed format anyway.

Page tables

corbet — Fri, 03 Jul 2020 16:58:42 +0000

The kernel was (and still is) built for a given page-table depth; the compiler then shorts out the code for the levels that aren't actually used.

Btrfs at Facebook

shemminger — Fri, 03 Jul 2020 16:51:27 +0000

800G is huge. All of Windows is a single git repo of 300G with 35M files.
Microsoft built Git Virtual Filesystem to deal with that, and avoid having to do massive downloads and checkouts.

Btrfs at Facebook

champtar — Fri, 03 Jul 2020 16:11:08 +0000

Encrypted content compress really poorly if at all, thus my interrogation, how can btrfs compression can help. See chatcannon for an explanation.

Btrfs at Facebook

scientes — Fri, 03 Jul 2020 14:29:27 +0000

> But Btrfs, too, has run into scalability problems with this workload. Messages are tiny, compressed files; they are small enough that the message text is usually stored with the file's metadata rather than in a separate data extent. That leads to filesystems with hundreds of gigabytes of metadata and high levels of fragmentation. These problems have been addressed, Bacik said, and it's "relatively smooth sailing" now. That said, there are still some issues left to be dealt with, and WhatsApp may not make the switch to Btrfs in the end.

This sounds like a perfect use-case of kyotocabinet. I have heard reports that it is rock solid with huge loads, and you also put kyototycoon on top for tcp access (including a memcached-compatible mode).

Btrfs at Facebook

Wol — Fri, 03 Jul 2020 14:26:08 +0000

In this case it sounds like "the computer is the database", along the lines of the AS/400 or Pick, would be a good implementation ... :-)

Cheers,
Wol

Page tables

Wol — Fri, 03 Jul 2020 14:23:57 +0000

Was the third level emulated in software for processors that didn't have it in hardware? I'm sure I remember something of the sort - that it was better to have an "ideal" design and emulate round practical short-comings.

Cheers,
Wol

Btrfs at Facebook

kevincox — Fri, 03 Jul 2020 13:59:42 +0000

It doesn't seem large to me. They have a lot of projects going on, and I don't know how much history this contains. I wouldn't be surprised if a number of assets such as images were checked in as well so that they can be built into the final site/whatever. Then I would guess there are 27 versions of jQuery and a couple of compiled binaries.

None of these things are huge but when you sum them up across a large company 800GB seems like they were really keeping themselves in check on the average.

Btrfs at Facebook

champtar — Fri, 03 Jul 2020 13:35:17 +0000

That makes more sense, thanks

Btrfs at Facebook

Karellen — Fri, 03 Jul 2020 13:20:42 +0000

These machines contain the entire source code for the web site; it is a struggle to get the whole thing to fit into the 800GB allotted for each and still leave room for some actual work to be done.

800GB‽‽

Am I just getting old, or is that actually somewhat on the large side?

Page tables

corbet — Fri, 03 Jul 2020 13:18:19 +0000

Digging through the code...v1.0 had two-level page tables internally. The third level was added once processors started supporting it. The same is true of the fourth and fifth levels.

Btrfs at Facebook

judas_iscariote — Fri, 03 Jul 2020 13:10:13 +0000

wut? whatsapp message leave the user device encrypted.. you need the recipient private key to decrypt it.

Btrfs at Facebook

cpitrat — Fri, 03 Jul 2020 13:02:55 +0000

I guess WoL has a point. The issue here is that we're saying "design" for different things. For a complex system, there will be different levels of design and taking into account low-level technical details for the high-level design would be wrong.

But in this case, we're talking about deciding how to store the individual messages. It's not even about design, it's a technical decision to store them as individual files (if that's really what is done, that's where I have a doubt).

Btrfs at Facebook

pizza — Fri, 03 Jul 2020 12:44:49 +0000

> Actually, not taking into account the limitations of the tools is GOOD design.

I'm not so sure about that.

Because ignoring reality is how we get designs are predicated on likes of spherical cows and other things that don't actually exist, and result in overly complex implementations with more exceptions than rules. If it can even be implemented at all.

Btrfs at Facebook

Wol — Fri, 03 Jul 2020 12:25:42 +0000

Actually, not taking into account the limitations of the tools is GOOD design.

Not taking into account the limitations when implementing the design is bad engineering.

I think Linus has commented on various occasions that when he's taken things like processor limitations into account for the basic design, the resulting implementation has been, shall we say, suboptimal.

If on the other hand the design *ignores* the limitations, and then the implementation works round the limitations, the end result is much better (plus, of course, as the tools improve the workarounds can be ripped out. If the workarounds are part of the design, then adapting to better tools is something that rarely gets done).

For example, I think the early processors only had a two-level page table. The best design turned out to be three-level, and when the early code using the two-level hardware was replaced by a three-level design that took advantage of the hardware for two of them, the result was a major improvement.

Cheers,
Wol

Btrfs at Facebook

ibukanov — Fri, 03 Jul 2020 10:11:01 +0000

Engineers at Facebook are not stupid. I suspect there is a missing piece of information about this particular case that justifies using the file system for this that overweights the harm of storing billions of tiny files.

Btrfs at Facebook

chatcannon — Fri, 03 Jul 2020 07:20:47 +0000

My guess: The message metadata (sender, recipient, timestamp, possibly a sequential ID, etc) are not encrypted. Most messages are short enough that the message metadata are larger than the cyphertext of the message content, and that all of this (message metadata plus message content) is small enough to fit in the filesystem metadata space rather than needing a block of its own.

Btrfs at Facebook

cpitrat — Fri, 03 Jul 2020 06:36:45 +0000

Well, not taking into account the limitations of the tools you use in your design is a bad design. If you plan to store billions of files of a few bytes and your rationale is "a good filesystem should handle this well, all filesystems are bad", this is not what I'd call good engineering ...

Btrfs at Facebook

champtar — Fri, 03 Jul 2020 01:03:55 +0000

Am I the only one understanding that WhatsApp message are stored unencrypted ? If they are encrypted I'm not sure how Btrfs compression can help ?

Btrfs at Facebook

Tov — Thu, 02 Jul 2020 21:50:13 +0000

"You mean each message is a separate file? It sounds like a poor design ..."

Not if you really think about it. A message is a single entity, which should be "filed" individually. However, metadata overhead and poor performance of older file systems have tought us to implement even more layers of indirection to paste over poor file system design...

Btrfs at Facebook

cpitrat — Thu, 02 Jul 2020 20:59:00 +0000

"Messages are tiny, compressed files; they are small enough that the message text is usually stored with the file's metadata rather than in a separate data extent."

You mean each message is a separate file? It sounds like a poor design ...

Btrfs at Facebook

gbalane — Thu, 02 Jul 2020 19:16:11 +0000

Thanks for the update. One question from the last update, do FB still uses readonly stores on Btrfs for serving the news feed ?

Btrfs at Facebook

Kamilion — Thu, 02 Jul 2020 19:02:28 +0000

> Experience has led him to assume that such things are Btrfs bugs, but this time it turned out that the RAID controller was writing some random data to the middle of the disk on every reboot. This problem had been happening for years, silently corrupting filesystems; Btrfs flagged it almost immediately.

Which RAID controller would this be? If it's an LSI, I'd really love to know, and I doubt Broadcom would care about LSI's reputation at this point... Hopefully it's not an adaptec...