Optimizing Linux with cheap flash drives

Perhaps the middle ground is to come up with some (de facto?) standardised way for manufacturers to categorise how their flash algorithms are optimised. (In addition to any minimum/maximum speed claims, etc.) "Optimised for video streaming", and "optimised for FAT32" being two raised by the article as relatively common, but there's a need for several more categories. At least that way, even without knowing the exact details of how, one could attempt to match the media purchased to the intended workload. Because at the moment it seems tricky as a purchaser to do that, outside perhaps video streaming and assuming everything else is optimised for the FAT file system on it at purchase.

Ewen

The NVMHCI concept (thanks for the Link!) makes a lot of sense at the high end where the drives can be smart enough to do a good job at providing high performance and low wear.

However, at the low end that I looked at, most drives get everything wrong to start with: there is too little RAM and processing power to do the reordering that would be needed for ideal NAND access patterns, the drives only do dynamic wear leveling, if any, so they break down more quickly than necessary.
The way that the SD card association deals with the problem is to declare all file systems other than FAT32 (with 32KB clusters) unsupported.

What we'd instead need for these devices is indeed a way to be smarter in the host about what it's doing. The block discard a.k.a. trim logic is one example of this that sometimes works already, but is not really enough to work with dumb controllers. What I'd like to see is an abstraction on segment level, using commands like "open this segment for sequential writes", "garbage-collect this segment now", "report status of currently open segments", "how often has this segment been erased?".

Thanks for that link :-) I've noticed that industry group before, but didn't pay much attention. To me, it's been just another flash chip interface standard from the JEDEC stable - notably without Samsung :-) After your remark about the standard connectors, I've taken a better look...

Since 2006 or 2007, there have been several revisions of the ONFI interface standard: 1.0, 1.1, 2.0, 2.1, 2.2, 2.3, and recently 3.0. The most visible differences are in transfer rate.
The "NAND connector" spec from 2008 is a separate paper - not an integral part of the main standard document. The NAND Connector paper refers to ONFI 1.0 and 2.0 standards documents. But - have you ever seen some motherboard or controller board with an ONFI socket? I haven't. In the meantime, there's ONFI 3.0 - it postulates some changes to the set of electrical signals, for the sake of PCB simplification - but there's no update to the "NAND connector" paper. To me that would hint that the NAND connector is a dead end - a historical branch of evolution that has proved fruitless... Please correct me if I'm wrong there, as I'd love to be :-)

ONFI 3.0 does refer to an LGA-style socket (maybe two flavours thereof), apart from a couple of standard BGA footprints. Which would possibly allow for field-replaceable/upgradeable chip packages, similar to today's CPU's. Note that the 3.0 spec doesn't contain a single occurrence of the word "connector" :-)

As far as I'm concerned, for most practical purposes, ONFI remains a Flash chip-level interface standard. It seems ONFI is inside the current Intel SSD's - it's the interface between the flash chips and the multi-channel target-mode SATA Flash controller. The multiple channels are ONFI channels. The SATA Flash controller comprises the SSD's disk-like interface to the outside world, and does all the "Flash housekeeping" in a hidden way.

Note that there's an FAQ at the ONFI web site, claiming that "No, ONFI is not another card standard."

From a different angle, note that the ONFI electrical-level interface (set of signals, framing, traffic protocol) is different from the native busses you can typically see in today's computers, such as FSB/QPI/PCI-e/PCI/LPC/ISA/DDR123_RAM. ONFI is not "seamless" or "inherent" to today's PC's: you have nowhere to attach that bus to, such that you'd have the Flash memory e.g. linear-mapped into the host system's memory space - which doesn't look like a good idea anyway, considering the Flash capacities and the CPU cores' address bus width (no it's not a full 64 bits - it's more like 32, 36 or maybe slightly more with the Xeons). Getting a "NAND connector" slot in your PC is not just a matter of the bus and connector and some passive PCB routing to some existing chipset platform. You'd need a "bridge" or "bus interface", most likely from PCI-e to ONFI (less likely straight from the root complex / memory hub). For several practical purposes, the hypothetical PCI interface would likely use a MMIO window + paged access to the ONFI address space, or possibly SG-DMA for optimum performance. I could imagine a simple interface using a general-purpose "PCI slave bridge" with DMA capabilities, similar to those currently made by PLX Corp. - except that those cannot do DDR, the transfer rates are too low, the FIFO buffers are perhaps too small for a full NAND Flash page and the bridges can't do SG-DMA... The initiative would IMO have to come from chipset makers (read: Intel) who could integrate an ONFI port in the south bridge. I haven't found a single hint of any initiative in that vein. There are even no stand-alone chips implementing a dedicated PCI-to-ONFI "dumb bridge". Google reveals some "ONFI silicon IP cores" from a couple fabless silicon design companies - those could be used as the ONFI part of such a bridge, if some silicon maker should decide to go that way, or maybe some are "synthesizable" in a modern FPGA.

As for the basic idea, which is to "present raw NAND chips to the host system and let the host OS do the Flash housekeeping in software, with full knowledge of the gory details": clearly ONFI isn't going that way. And quite possibly, it's actually heading in precisely the opposite direction :-) There is a tendency to hide some of the gory details even at the chip interface level. On the ONFI Specs page you can find another "stand-alone paper" specifying "Block Abstracted NAND", as an enhancement to the basic ONFI 2.1 standards document. The paper is also referred back to by the ONFI 3.0 standard (where it lists BA NAND opcodes). Looks like an "optional LBA access mechanism to NAND Flash" (does this correlate with the moment SanDisk got a seat at the ONFI table, by any chance?) And in the ONFI 3.0 spec, you can find a chapter on "EZ NAND", which is to hide some of the gory details of ECC handling (at the chip interface level).

Ahh well...

Perhaps a Java class stored on the storage card. It could implement some well-defined interface type and its constructor could take some parameters for things like a hardware interface class, memory buffer, debug logger and a few other things.

It could run from userspace with the right interface class. Or from the kernel if someone wrote a simplified Java interpreter or maybe a module compiler.

I suppose instead of Java it could be written in whatever VM it is that ACPI uses. Kernels already have interpreters for that.

It could be fairly nifty.