CVE-2019-5736: runc container breakout [LWN.net]

CVE-2019-5736: runc container breakout

Posted Feb 12, 2019 19:03 UTC (Tue) by ibukanov (subscriber, #3942)
In reply to: CVE-2019-5736: runc container breakout by NYKevin
Parent article: CVE-2019-5736: runc container breakout

Looping as opposed to calling abort() is a useful option in a complex application. It allows to attach a debugger and investigate the live state.

Plus in the container world it is quite likely that realloc returns null not when the system is out of memory, but rather when the container hits its allocation limits. Administrator may rise the limits and let the application to continue and reach some stable state when it can be properly closed.

to post comments

CVE-2019-5736: runc container breakout

Posted Feb 12, 2019 19:28 UTC (Tue) by sorokin (guest, #88478) [Link] (3 responses)

> Administrator may rise the limits and let the application to continue and reach some stable state when it can be properly closed.

Should the same logic be applied to, say, open() function? File is not there, but, perhaps, it will be soon. Administrator may attach to the application and see that a necessary file is missing and put it into the right place.

CVE-2019-5736: runc container breakout

Posted Feb 12, 2019 21:03 UTC (Tue) by ibukanov (subscriber, #3942) [Link] (2 responses)

Memory allocation is much more widespread then open calls. So most applications do not even try to bother with allocation errors and assume that new/alloc/realloc never fails as dealing with those is too painful. This is true both for manual memory management and GC languages like Java or Go. This leads to the question of what to do if an allocation does report an error and one cannot propagate the error to the caller. Then doing a loop is not particularly worse then calling abort().

CVE-2019-5736: runc container breakout

Posted Feb 14, 2019 0:38 UTC (Thu) by wahern (subscriber, #37304) [Link] (1 responses)

Which caller? The immediately preceding function or the caller that invoked the application? Being able to attach a debugger certainly has value, but in production environments, and particularly in cloud infrastructure environments, that's not the most important characteristic. What you want is for things to fail fast so that the system can adjust in a timely manner; and usually fail fast to the nearest caller (at least nearest by API or contract boundary) as the nearest caller will usually be in the best position to react optimally (and if not it can simply bubble up). It's *exactly* analogous to buffer bloat--by trying to be too helpful these hacks are blunting or dropping feedback pressure, with the macro result being crappy behavior that is extremely costly to remedy, if it can be remedied at all.

The distinction between being "truly" out of memory and only being out of memory because of policy seems contrived and irrelevant, particularly in the era of nested containers and VMs.

In the context of a language like Go or Perl or simple C utilities or similar languages, aborting on allocation failure (rather than trying to implement the runtime in a way that makes it recoverable) is excused as a reasonable cost-benefit tradeoff as these programs are typically executed circumscribed contexts (microservices, CGI scripts) where the possibility of meaningful recover by higher layers up the stack is retained. But locking up the application? Sure, being able to attach a debugger has some utility, just as being able to attach a debugger to a car's breaking system which decided to fail in an open state as a convenience for the engineers. But that's hardly the most reasonable or desirable behavior in the vast majority of contexts.

CVE-2019-5736: runc container breakout

Posted Feb 14, 2019 10:59 UTC (Thu) by ibukanov (subscriber, #3942) [Link]

The code in question is a part of a long-running application that has non-trivial persistent state, not a microservice or CGI script.

In Linux realistically when realloc returns NULL, it is due to limits imposed on the application, not because the system is out of memory. Linux happily over commits and when the system does run out of memory, a process will be killed at an arbitrary point by OOM killer. In a production system this should not happen as it is extremely hard to write an application code that does not corrupt the state if the app can be killed at any point.

But if an application hits a memory policy limit, then waiting for the limit to be lifted is a reasonable behavior. First if the limit is imposed not on a single thread, but many threads/processes, then it makes sense to wait until the memory hog finishes. Second, many applications still may leave things in an inconsistent state if any memory allocation can call abort.