User: Password:
|
|
Subscribe / Log in / New account

The future calculus of memory management

The future calculus of memory management

Posted Jan 27, 2012 20:33 UTC (Fri) by djm1021 (guest, #31130)
In reply to: The future calculus of memory management by vlovich
Parent article: The future calculus of memory management

Thanks for the thorough cost analysis.

First, RAMster works fine over builtin 1GbE and provides improvement over spinning rust, so 10GbE is not required. True, it's probably not faster than a local SSD, but that misses an important point: To use a local SSD, you must permanently allocate it (or part of it) for swap. So if you have a thousand machines, EACH machine must have a local SSD (or part of it) dedicated for swap. By optimizing RAM utilization across machines, the local SSD is not necessary... though some larger ones on some of the thousand machines might be.

The key point is statistics: Max of sums is almost always less (and often MUCH less) than sum of max. This is especially true when the value you are measuring (in this case "working set") varies widely. Whether you are counting RAM or SSD or both, if you can "overflow" to a remote resource when the local resource is insufficient, the total resource needed (RAM and/or SSD) is less (and often MUCH less). Looking at it the other way, you can always overprovision a system, but at some point your ROI is diminishing. E.g. why don't you always max out RAM on all of your machines? Because its not cost-effective. So why do you want to overprovision EVERY machine with an SSD?

P.S. The same underlying technology can be used for RAMster as for VMs. See http://lwn.net/Articles/454795/
And for RAMster, see: http://marc.info/?l=linux-mm&m=132768187222840&w=2


(Log in to post comments)

The future calculus of memory management

Posted Jan 27, 2012 22:59 UTC (Fri) by vlovich (guest, #63271) [Link]

The point I was making was that 10GBe was too expensive (since 10GBe is about on performance-parity with mass storage). With 1 GBe, you're looking at a lot less performance than attached storage. So yes, if your servers are using a regular spinning disk, then this *MIGHT* help the performance a bit since latency may be theoretically better (but throughput will be much lower). However, as the cluster grows, the performance will degrade (latency will get worse since the switch has to process more + bandwidth will decrease since your switch likely can't actually do simultaneous n ports * 1gbe of throughput).

So the point I've been trying to understand is what kind of application would actually benefit from this?

spinning storage in each box, limited amount of RAM, lots of machines, hitting swap occasionally, extremely cost conscious, not performance conscious

So here's two possible alternatives that provide significantly better performance for not much extra cost:
* add more RAM to 1 or more boxes (depending on how much you have in your budget) & make sure heavy workloads only go to those machines.
* if adding more RAM is too expensive use an equivalently-priced SSDs instead of spinning disks - yes you have less storage per-machine, but use a cluster-FS for the data instead (my hunch is that a cluster-FS + SSD for swap will actually get you better performance than spinning disk + RAMster for swap).
* use a growable swap instead of a fixed-size swap? that'll get you more efficient usage of the storage space

Have you actually categorized the performance of RAMster vs spinning disk vs SSD? I think it's useful to see the cost/performance tradeoff in some scenarios. Furthermore, it's important to categorize the cost of RAM vs the cost of the system & what kind of performance you require from the application + what are the specs of the machines in the cluster.

Also, getting back to the central thesis of the article; if it's hitting swap you're looking at a huge performance hit anyways, so why require paying for the "RAM swap" you use when you already do it implicitly performance-wise (i.e. you are already encouraged to use less RAM to improve performance, so charging for the swap use doesn't do anything; if you could lower the RAM usage, you would already be encouraged to do so).

The future calculus of memory management

Posted Jan 27, 2012 23:46 UTC (Fri) by djm1021 (guest, #31130) [Link]

Good discussion! A few replies.

> what kind of application would actually benefit from this?

Here's what I'm ideally picturing (though I think RAMster works for other environments too):

Individual machines are getting smaller and more likely to share external storage rather than have local storage. Cloud providers that are extremely cost conscious but can provide performance if paid enough (just as web hosters today offer a virtual server vs physical server option, the latter being more expensive).

Picture a rack of "microblades" in a lights-out data center, no local disks, labor-intensive and/or downtime-intensive to remove and upgrade (or perhaps closed sheet-metal so not upgradeable at all), RAM maxed out (or not upgradeable), some kind of fabric or mid-plane connecting blades in the rack (but possibly only one or two GbE ports per blade).

> but throughput will be much lower

There isn't a high degree of locality in swap read/write so readahead doesn't help a whole lot. Also, with RAMster, transmitted pages are compressed so less data is moved across the wire.

> Have you actually categorized the performance of RAMster vs spinning disk vs SSD?

Yes, some, against disk not SSD. RAMster is just recently working well enough.

The future calculus of memory management

Posted Jan 28, 2012 1:32 UTC (Sat) by dlang (subscriber, #313) [Link]

> Individual machines are getting smaller and more likely to share external storage rather than have local storage.

I agree they are more likely to share external storage, but I disagree about them getting smaller.

with virtualization, the average machine size is getting significantly larger (while the resources allocated to an individual VM are significantly smaller than what an individual machine used to be)

there is a significant per-physical-machine overhead in terms of cabling, connectivity, management, and fragmentation of resources. In addition, the sweet spot of price/performance keep climbing.

as a result, the machines are getting larger.

for what you are saying to be true, a company would have to buy in to SAN without also buying in to VMs, and I think the number of companies making that particular combination of choices is rather small.


Copyright © 2018, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds