The Grumpy Editor's video journey part 2: Video editors [LWN.net]

The Grumpy Editor's video journey part 2: Video editors

Posted Jan 23, 2008 18:03 UTC (Wed) by Cato (subscriber, #7643)
Parent article: The Grumpy Editor's video journey part 2: Video editors

Given the entertaining comments about the OOM killer kicking in with these video editors, it
would be useful to know some details such as amount of RAM in the test PC, size of video
images being edited at the time, and maybe whether some apps were more prone to being OOM
killed.

Also, isn't it possible to simply increase swap size (and turn off the relevant memory
overcommit sysctl) to ensure the OOM killer doesn't cause problems with video editing, as per
http://lwn.net/Articles/104179/ ?  If so, some pointers on this would be useful.  

Finally - are any of these apps multi-threaded to make use of multi-core CPUs?

(Log in to post comments)

OOM killer and such

Posted Jan 23, 2008 18:11 UTC (Wed) by corbet (editor, #1) [Link]

The test system has 3GB of memory - upgraded partway through the writing of the article in response to problems. It didn't help, though. The big offender is Cinelerra, which clearly has some memory leaks somewhere. I don't believe any of the other applications tested brought down the wrath of the OOM killer.

Increasing swap would only delay the problem a bit. Turning off overcommit would be a better answer - at least that should catch the problem before it gets completely out of control.

I have a multicore system, but didn't look at that aspect of performance. In general, I don't think these applications are multi-threaded in that way. One does get some benefit, though, in that the editor can run on one CPU while the X server gets the other.

OOM killer and such

Posted Jan 24, 2008 2:55 UTC (Thu) by Richard_J_Neill (subscriber, #23093) [Link]

On a 32-bit system, no single app can use more than 2GB of RAM (if I understand correctly). So
3GB won't help very much. This is a particular pain, because even when running a
memory-monitor on the taskbar, you can't easily see whether one particular app is about to run
out of addressable RAM. 

That's why my new desktop machine is 64-bit, so that firefox can address almost 8GB of real
RAM, and furthermore, leak into 16GB of swap, before it finally crashes!

OOM killer and such

Posted Jan 24, 2008 3:48 UTC (Thu) by corbet (editor, #1) [Link]

The virtual address space on a default-configured 32-bit system is 3GB. In any case, you don't bring in the OOM killer by running out of virtual space.

But, in any case, the system in question is a 64-bit machine, so all that is moot.

Single app vs. single malloc()

Posted Jan 25, 2008 22:54 UTC (Fri) by AnswerGuy (subscriber, #1256) [Link]

On 32-bit x86 Linux systems with a standard libc the largest chunk of contiguous memory
available to any process is ~2GB; there are some dribs and drabs (about another half-GB)
available after that.

This has to do with how libc aligns its ELF segments relative to the heap.

On PAE enabled systems one can have multiple different processes each with their own 2+GB of
allocated memory.

Interestingly I've heard that 32-bit processes running under a 64-bit kernel can get much
closer to 3GB contiguously allocated.  I guess the kernel is able to stay completely out of
the way in that case.

Unfortunately I haven't found any really good URLs that describe this in sufficient detail
(with sufficiently comprehensible diagrams) that I can really recommend them.  And my
comprehension is limited --- so I can't explain it better nor draw the diagrams that I'm
eventually hoping to find.

(It's unfortunate because I work at a company that makes EDA software --- which is notoriously
memory hungry.  So the question comes up far too often, despite the accelerating adoption of
64-bit systems).
JimD

OOM killer and such

Posted Jan 25, 2008 1:51 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

It's crazy to let a greedy application wake the OOM Killer, which is tantamount to a kernel crash. Use a virtual memory size rlimit (e.g. 'ulimit' shell command). Given how easy it is for a program to go wild with its memory demands, it has always surprised me that Linux defaults to unlimited memory per process and that few users change that.

I use rlimits obsessively, and I frequently have leaky programs die when they get too big. I never have to worry about them causing trouble for innocent processes.