LC-Asia: Facebook contemplates ARM servers

If all you need is a cluster on a chip those are largely solved problems. Existing cluster applications have the necessary management tools, and I/O devices can be either hardwired to specific cores or shared using something like PCI IOV. It might be convenient to have an internal high speed Ethernet interface for each core with an embedded switching fabric, for example, although something designed for the purpose would probably be more efficient for on chip communication.

A long term solution to the cache coherency problem would indeed substantially complicate kernel design. For example, if the memory overhead of each core is a problem, what you really want is a unified physical address space, explicit sharing of read only code and data, and no sharing or highly restricted sharing of mutable code and data. There is no reason why the kernels running on each core can't trust each other, they just can't scalably share mutable data on an instruction by instruction basis.

And if you really want a single system image, a much more radical redesign would be required. In the long run, that is probably inevitable, because cache concurrency doesn't scale - not across per core caches at any rate.