It was wrong back then and it's wrong now

Posted Nov 8, 2008 3:15 UTC (Sat) by Ze (guest, #54182) [Link]

Yes - and I've certainly had problems with the first and I'm sure I'll have problems with the second. The only sane way to resolve this problem is to offer as low access as possible - but not lower. I don't think checksum calculation for blocks on HDD belongs to OS kernel (it can be calculated in HDD more or less for free, but general-purpose CPU will spend significant power doing it), but bad blocks handling certainly should belong to kernel - it have more resources to cope. Yes and what about the bandwidth requirements of sending the checksum over? or if someone wishes to use that space for an error correcting code or a more secure hash? Ultimately the time spent transferring files from the disk should be a small percentage compared to the time spent waiting for them. However there are always going to be trade offs with flexibility , speed and other things. What is the goal: great storage subsystem or great system? If the latter then all these things must be done at the system level (and may be offered via NFS/CIFS). OBS is quite bad idea but Linux will need some support for it anyway. And it's useful is some strange places (for example in KVM/VMWare/Xen). The whole point though is that NFS and CIFS are unsuitable for some uses. This is where Object based file systems come in , they are like a stripped down form of NFS/CIFS. Ideally it'd be nice to have a nice layer setup that allows people to see the layers they want and not have to put up with the layers they don't need. One downside I see for object based file systems is with versioning file systems , which use the layout of the block layer to make having multiple versions cheap in space. That's a trade off though that may be worth it to some and not to others.

Posted Nov 9, 2008 0:37 UTC (Sun) by giraffedata (subscriber, #1954) [Link]

For rotating storage it hides bad block remapping.

Bad block remapping? It hides all block mapping. In a truly low level disk interface, Linux would address the disk by cylinder/head/sector. Indeed, there are things Linux could do more effectively if it controlled the storage at that level. And that's nowhere near the lowest conceivable abstraction, either.

I don't think checksum calculation for blocks on HDD belongs to OS kernel (it can be calculated in HDD more or less for free, but general-purpose CPU will spend significant power doing it)

I see no reason for it to be a cheaper computation in the HDD than in the main computer. If there are special purpose processors in the HDD to do it cheaply, it's because that's where we've decided to do it; not vice versa.

the search is for the abstraction which best mediates between the needs of the computer and the needs of the storage.

I'd like to put it differently, because it's not what I can do for the computer and the storage, but what they can do for me. So: Which abstraction best leverages the abilities of the computer and those of the storage, to provide the most efficient storage service?

There was a time when the best dividing line was such that the main computer watched the bits stream off the head until it detected the start of a record, etc. That let us consolidate expensive CPUs. Today, we can squeeze more storage service out cheaper by moving a great deal of that function to the other end of the cable. More recent technological changes might mean it's most efficient for file layout to move out there too.

I can think of a few reasons to stick function inside the storage product and the storage box instead of the main computer products and box:

• The implementation must change more when the storage layer below it changes than when the application layer above it does.
• Multiple main computers share the storage box.
• It's expensive to move information over the cable.