LC-Asia: Facebook contemplates ARM servers [LWN.net]

LC-Asia: Facebook contemplates ARM servers

Posted Mar 14, 2013 19:27 UTC (Thu) by butlerm (subscriber, #13312)
In reply to: LC-Asia: Facebook contemplates ARM servers by robert_s
Parent article: LC-Asia: Facebook contemplates ARM servers

Why bother with cache coherency? Just make a chip with sixteen independent processors. Add a high speed internal interconnect and you would have a very nice cluster on a chip. Each processor could be associated with a fixed amount of RAM from the external DIMMs, with no instruction level access to other processor's RAM and no cache concurrency issues other than what you might have for DMA.

to post comments

LC-Asia: Facebook contemplates ARM servers

Posted Mar 14, 2013 22:12 UTC (Thu) by bronson (subscriber, #4806) [Link] (4 responses)

So each CPU gets 1/16 of the memory bus? You'd need an absurd amount of cache to make that competitive with current desktop offerings.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 0:47 UTC (Fri) by smoogen (subscriber, #97) [Link]

I would expect the answer would be to make 16 memory busses. Since at best an ARM would see a 4GB dimm each.. you could still come off cost wise :).

[And no this is not meant to be serious. But I filed a patent on it anyway.]

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 1:17 UTC (Fri) by butlerm (subscriber, #13312) [Link] (2 responses)

Each core on an eight core processor gets 1/8 of the memory bus already. If you have non-SMP cores, what is the difference? The most obvious one is if each core is less powerful you better make sure that your application can use all of them. If you are memory bandwidth constrained more cores aren't going to help, but they aren't going to hurt very much either.

An active market for processors with up to sixteen x86 processor cores certainly suggests that devices with sixteen or even thirty two smaller, somewhat lower clocked cores isn't out of line, provided you are dealing with applications that work well on a horizontally scaled, non-SMP basis. Not workstations, servers.

Cache coherency doesn't scale. That is why companies like Facebook and Google use racks of thousands of relatively lightweight servers in the first place, instead of hundred processor NUMA setups. Going to non-SMP cores on the same silicon substrate is the next logical step in that evolution. For any sufficiently large scale application, SMP is a crutch, and a power hungry, expensive one at that.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 1:43 UTC (Fri) by dlang (guest, #313) [Link] (1 responses)

you can't just connect processors to the memory bus, you need to setup some form of arbitration between them.

you can create systems with lots of cores, but it's not easy, it's not cheap, and it's not low power.

There's a lot more to making a non SMP system than just throwing cores on a memory bus.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 5:45 UTC (Fri) by butlerm (subscriber, #13312) [Link]

It's certainly wouldn't be trivial, but as long as the power required per core doesn't go up, you would come out far ahead of any kind of cache coherent design. Large scale applications can't rely on cache coherency, and so they don't. At some point it is just a waste of silicon.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 9:11 UTC (Fri) by epa (subscriber, #39769) [Link] (2 responses)

Or... each processor can see all the address space, but only write to 1/16 of it. So if you are processor zero you have full, cached access to the lower one gigabyte of memory, and read-only access to the rest. That would simplify cache coherency a great deal.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 16:39 UTC (Fri) by robert_s (subscriber, #42402) [Link] (1 responses)

> That would simplify cache coherency a great deal.

But probably complicate OS design massively.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 17, 2013 23:42 UTC (Sun) by butlerm (subscriber, #13312) [Link]

I don't think you need to go quite as far as saying you can't have shared writable memory at all, the issue is that hardware can't maintain cache coherency across core specific caches on a scalable or efficient basis. So what you need is for any caching of such memory to be carefully arbitrated in software, e.g. on an acquire and release basis, and for the use of shared mutable data structures to be minimized in favor of things that you can partition across cores on a relatively long term basis.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 11:48 UTC (Fri) by etienne (guest, #25256) [Link] (1 responses)

Possible problems:
- How to start/stop tasks on other processor? Use ssh?
- How to migrate tasks from overloaded processors to under-used processors?
- How to share page cache? Have kernel and all libraries in each address space?
- How to share input/output? Network, serial lines/keyboards, screens, hard disks?
You may decide to have one processor controlling the other 15, with special access to their memory, but usually that processor will be the weakest link.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 17, 2013 20:47 UTC (Sun) by butlerm (subscriber, #13312) [Link]

If all you need is a cluster on a chip those are largely solved problems. Existing cluster applications have the necessary management tools, and I/O devices can be either hardwired to specific cores or shared using something like PCI IOV. It might be convenient to have an internal high speed Ethernet interface for each core with an embedded switching fabric, for example, although something designed for the purpose would probably be more efficient for on chip communication.

A long term solution to the cache coherency problem would indeed substantially complicate kernel design. For example, if the memory overhead of each core is a problem, what you really want is a unified physical address space, explicit sharing of read only code and data, and no sharing or highly restricted sharing of mutable code and data. There is no reason why the kernels running on each core can't trust each other, they just can't scalably share mutable data on an instruction by instruction basis.

And if you really want a single system image, a much more radical redesign would be required. In the long run, that is probably inevitable, because cache concurrency doesn't scale - not across per core caches at any rate.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 15, 2013 17:05 UTC (Fri) by robert_s (subscriber, #42402) [Link] (1 responses)

So you need 16 copies of the OS and common software in memory?

How much memory were you planning on putting on those external DIMMs?

If "a lot", how were you planning on cramming all those lanes into the single smallish SoC package?

There are of course many-many-core chips like those made by Tilera, but note these seem to use a rather exotic and, well, "different" memory architecture that can't just be cooked up by a SoC vendor "throwing on" 16 cores to a die.

LC-Asia: Facebook contemplates ARM servers

Posted Mar 21, 2013 19:29 UTC (Thu) by dfsmith (guest, #20302) [Link]

Well, let's say there's about 100MB of system software. With the price of DDR3 at around $6/GB (http://www.dramexchange.com/), that makes it of the order 60 cents per CPU unit.

Fixing the issue to reduce system software overhead would probably cost 4 person-years; or about $1M of burdened payroll.

That makes payback at around 2 million CPU units. Feasible, but probably not cost effective. And by the time the redesign is finished, DRAM prices will have dropped again.