| |||||||||||||
Smarter write throttlingSmarter write throttlingPosted Aug 17, 2007 17:17 UTC (Fri) by giraffedata (subscriber, #1954)Parent article: Smarter write throttling
A lot of I/O through the LVM level could create large numbers of dirty pages destined for the physical device. Should things hit the dirty thresholds at the LVM level, however, the process could block before the physical drive starts writeback. In the worst case, the end result here is a hard deadlock of the system I don't see any deadlock in this description, but maybe I'm supposed to know something more about how LVM works. (It doesn't make sense to me, for example, that a physical drive starts writeback; I expect a physical drive to be the target of a writeback). But I've seen the pageout-based memory deadlock plenty of times in action when a process that has to move in order for memory to get clean (maybe it holds a lock for some dirty file) blocks on a memory allocation. At least at one time, these were possible in Linux in various ways, network-based block devices and filesystems being the worst offenders, and people addressed it with tuning policies that sought to make it unlikely that the amount of clean allocatable memory ever shrank to zero. "Throttling" usually refers to that sort of tuning -- a more or less arbitrary limit placed on how fast something can go. I hope that's not what this is about, because it is possible actually to fix deadlocks -- make them mathematically impossible while still making the full resources of the system available to be apportioned any way you like. It's not easy, but the techniques are well known (never wait for a Level N resource while holding a Level N+1 resource; have pools of memory at various levels). I hate throttling. Throttling is waiting for something artificial. In a well-designed system, you can wait for real resource and never suffer deadlock or unfairness.
(Log in to post comments)
Smarter write throttling Posted Aug 19, 2007 0:46 UTC (Sun) by i3839 (guest, #31386) [Link] Looking at it like a feedback system, the current negative feedback that limits writing at all is the amount of memory (or part of it thanks to tunables, but in the end it's memory that limits it). Practically this means you can have huge number of dirty pages outstanding, all waiting on one slow device.
Now combine the facts that you can't get rid of dirty memory quickly with that they probably aren't used anymore (rewrite doesn't happen that often I think), and you've just managed to throw away all free memory, slowing everything down. So basically you're using most of your memory for buffering writes, while the write speed doesn't increase significantly after a certain buffer size.
What is throttled here is the speed of dirtying pages, not the speed of writeout, to keep the number of dirty pages limited. (But although the dirtying is throttled, the average should in both cases be the write speed of the device, so perhaps "throttling" isn't quite the right word).
The deadlock mentioned is AFAIK the situation where the system is under heavy memory pressure and wants to free some memory, which it tries to do by writing out dirty pages, but doing that might require allocating some memory, which can't be done because that's what lacking, hence a deadlock. But I don't know if that's the mentioned LVM example, or if that's another case.
Smarter write throttling Posted Aug 19, 2007 1:48 UTC (Sun) by giraffedata (subscriber, #1954) [Link] What is throttled here is the speed of dirtying pages, Actually, nothing is throttled here. Rather, the amount of memory made available for writebehind buffering is restricted, so folks have to wait for it. Tightening resource availability slows down users of resources, like throttling would, but it's a fundamentally different concept. Resource restriction pushes back from the actual source of the problem (scarce memory), whereas throttling makes the requester of resource just ask for less, based on some higher level determination that the resource manager would give him too much. I think the article is really about smarter memory allocation for writebehind data, not smarter write throttling. (To understand the essence of throttling, I think it's good to look at the origin of the word. It means to choke, as in squeezing a tube (sometimes a person's throat) to reduce flow through it. The throttle of a classic gas engine prevents the fuel system from delivering fuel to the engine even though the engine is able to burn it).
|
|||||||||||||