| |||||||||||||
Optimizing Linux with cheap flash drivesOptimizing Linux with cheap flash drivesPosted Feb 19, 2011 12:40 UTC (Sat) by willy (subscriber, #9762)In reply to: Optimizing Linux with cheap flash drives by bronson Parent article: Optimizing Linux with cheap flash drives
It's thought to be too hard to "release specs". The algorithms for handling any particular generation (and type) of Intel flash are substantially different from each other. That's just one manufacturer ... Linux would have a horrendous time trying to keep up with the dozens of flash manufacturers each releasing a new generation of flash every 18 months, possibly in several different flavours (1, 2, 3 and 4 bit per cell).
It's probably not even possible for Linux mainline to keep up with that frequency, let alone the enterprise distros or the embedded distros (I was recently asked "So what changed in the USB system between 2.6.10 and 2.6.37?"). And then there's the question about what to do for other OSes.
It's not just a question of suboptimal performance if you use the wrong algorithms for a given piece of flash; there are real problems of data loss and the flash wearing out. No flash manufacturer wants to be burdened with a massive in-warranty return because some random dude decided to change an '8' to a '16' in their OS that tens of millions of machines ended up running.
So yes, as Arnd says, the industry is looking to abstract away the difference between NAND chips and run the algorithms down on the NAND controller. I'm doing my best to help in the NVMHCI working group ... see
(I work for Intel, but these are just my opinions).
(Log in to post comments)
Optimizing Linux with cheap flash drives Posted Feb 19, 2011 19:06 UTC (Sat) by ewen (subscriber, #4772) [Link]
Perhaps the middle ground is to come up with some (de facto?) standardised way for manufacturers to categorise how their flash algorithms are optimised. (In addition to any minimum/maximum speed claims, etc.) "Optimised for video streaming", and "optimised for FAT32" being two raised by the article as relatively common, but there's a need for several more categories. At least that way, even without knowing the exact details of how, one could attempt to match the media purchased to the intended workload. Because at the moment it seems tricky as a purchaser to do that, outside perhaps video streaming and assuming everything else is optimised for the FAT file system on it at purchase.
Ewen
Optimizing Linux with cheap flash drives Posted Feb 19, 2011 21:03 UTC (Sat) by arnd (subscriber, #8866) [Link]
The NVMHCI concept (thanks for the Link!) makes a lot of sense at the high end where the drives can be smart enough to do a good job at providing high performance and low wear.
However, at the low end that I looked at, most drives get everything wrong to start with: there is too little RAM and processing power to do the reordering that would be needed for ideal NAND access patterns, the drives only do dynamic wear leveling, if any, so they break down more quickly than necessary.
What we'd instead need for these devices is indeed a way to be smarter in the host about what it's doing. The block discard a.k.a. trim logic is one example of this that sometimes works already, but is not really enough to work with dumb controllers. What I'd like to see is an abstraction on segment level, using commands like "open this segment for sequential writes", "garbage-collect this segment now", "report status of currently open segments", "how often has this segment been erased?".
Optimizing Linux with cheap flash drives Posted Feb 19, 2011 22:40 UTC (Sat) by willy (subscriber, #9762) [Link]
Yes, these are quite different devices ... I would estimate a factor of 100+ difference in price, and probably similar factors in terms of capacity, speed, power, etc, etc.
The API you're suggesting makes a ton of sense for the low end devices. I don't think there's a whelk's chance in a supernova of it coming to anything, though. You'd need the SD association to require it, and I can't see it being in the interest of any of their members. When the reaction to "hey, your cards suck for this other filesystem" is "your filesystem is wrong", I can't see them being enthusiastic about something this radical.
I do see that Intel are members. I'll try to find out internally if something like this could fly.
Optimizing Linux with cheap flash drives Posted Feb 20, 2011 3:25 UTC (Sun) by Oddscurity (guest, #46851) [Link]
So in summary, if I want to run an ext3 filesystem on a USB stick, I'm better off formatting the stick as FAT32 and then running the ext3 as a loop?
Or would that be the wrong conclusion?
Not that it's all I took away from this great article, but I'm wondering what I can do in the meantime to optimise my use with such devices.
Optimizing Linux with cheap flash drives Posted Feb 20, 2011 4:07 UTC (Sun) by ewen (subscriber, #4772) [Link]
Alas, no, running ext3 in a loop on FAT32 doesn't magically change your file system access patterns from ext3 access patterns to FAT access patterns. (Eg, in that scenario the FAT would hardly ever change since you allocate a huge file for the loop and then just write within it, versus native FAT32 with it changing with each file change, so the cheap flash drives optimisation for the 4MB holding the FAT would be wasted; and you'd still get random updates frequently in "unexpected" -- by the naive firmware -- locations.)
It appears if you want to run ext3 on a cheap flash drive, you pretty much have to assume that it's going to be slower than advertised (possibly MUCH slower, especially for write), and that there's a very real risk of wearing out some areas of the flash faster than might be expected. Probably okay for a mostly-read workload if you ensure that you turn off atime completely (or every read is also a write!), but not ideal for something with regular writes.
If it's an option for your use case, then sticking with the original FAT file system -- and using it natively -- is probably the least bad option. Certainly that's what I do with all my portable drives that see any kind of regular updates. (It also has the benefit that I don't have to worry about drivers for the file system on any system I might plug it into.)
Ewen
Optimizing Linux with cheap flash drives Posted Feb 20, 2011 14:11 UTC (Sun) by Oddscurity (guest, #46851) [Link]
Thanks for the comprehensive answer.
I may as well switch to just FAT32 for part of the use cases and the other ones are dominated by reads, so can stay on ext.
|
|||||||||||||