|
|
Log in / Subscribe / Register

Insane number of files created per second

Insane number of files created per second

Posted Aug 25, 2016 18:42 UTC (Thu) by ott (guest, #110845)
In reply to: Insane number of files created per second by pr1268
Parent article: Btrfs and high-speed devices

Say for example you want to check-out a source repository that has millions of small files, file creation can be a serious bottleneck.


to post comments

Insane number of files created per second

Posted Aug 25, 2016 23:00 UTC (Thu) by gerdesj (subscriber, #5446) [Link] (2 responses)

I'm intrigued. Do you mean that checking out on a btrfs is slowed down as opposed to say ext4 or xfs?

I've just checked out Libre Office a few times and copied it around a bit. As a long time Gentoo user on a fair few systems, I'm quite familiar with how some pretty large software projects behave from source to binary.

LO is "only" about 82,000 files at 2.2GB. On my laptop with a reasonably modern Core i7 quad core + HT, with 16GB RAM and 1 x SSD + 1 x spinning disc, it takes a fair old while to compile and needs rather a lot of space. On previous laptops it used to be an overnight thing. Check out times are the least of my worries.

Back to your assertion, even if you are checking out over a 10GBs-1 connection I doubt that the fs is holding you back. What kind of projects involve millions of files? Also what sort of repo are you using?

Cheers
Jon

Insane number of files created per second

Posted Aug 26, 2016 3:25 UTC (Fri) by ott (guest, #110845) [Link]

Disclaimer: I work at Facebook but not in that team, so I don't know the specifics. However Google has released some numbers about their monorepo (http://cacm.acm.org/magazines/2016/7/204032-why-google-st...), and it's about 9M files.
It was just to say, it's not "outrageous" to have a directory with million of files, which could be checked out all at once. The FS performance definitely plays a major role there.

Insane number of files created per second

Posted Aug 26, 2016 15:55 UTC (Fri) by cwillu (guest, #67268) [Link]

Again though, this isn't _about_ file creation, it's about metadata ops; file creation is a convenient way to exercise it, while also being relevant to some usages.

Grepping a tree for the first time today? Atime updates. "make clean" in a large repository? combinations of mtime and deletions. "apt-get upgrade" with a bunch of pending updates? Oh, you better believe there's a shittonne (SI technical unit) of file creations, fsyncs, mv's and other metadata updates.

It ends up being a major but sometimes hidden determinant of how fast you can get shit done.


Copyright © 2026, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds