One billion files on Linux
One billion files on Linux
Posted Aug 19, 2010 15:20 UTC (Thu) by zzxtty (guest, #45175)Parent article: One billion files on Linux
If people are wondering about the validity of 1 billion files I can give an example, I work with MRI data. We do a lot of fMRI which generates lots of files (DICOM images). One file is generated per slice, with fMRI you continuously scan someone for an extended period of time, a single scan can generated 20,000 files. If you've got several MRI scanners and have been up and running for a few years the 1 billion file mark is not large, So far this year we have generated over 23 million files on one of our scanners.
However, I'm not sure I'd want to store them all on one file system, it would be a nightmare to restore from tape if anything did go horribly wrong. This is where data management comes in, I create a new partition for each scanner, each year. Currently we run all this on midrange hardware raid and format with zfs, it appears to cope. Would be nice to move it all to Linux =)