Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Posted Feb 25, 2017 0:13 UTC (Sat) by josh (subscriber, #17465)In reply to: Cloudflare Reverse Proxies are Dumping Uninitialized Memory by atelszewski
Parent article: Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Yes, that's exactly what I'm suggesting, though whoever selected C for the project is also at fault. (That may or may not have been the programmer(s) who actually wrote the bug, who on a relative scale deserve far less of the blame.)
Let me say that more explicitly and emphatically: *C (and whoever selected C) is at fault*.
You want to get from point A to point B. You see people running a three-legged race from A to B, via an icy road, wearing one roller blade and one ice skate, while pair-juggling three chainsaws with exposed wires and no blade guards, powered by hypergolic rocket fuel and plutonium. You can't help but marvel at the skill required. When you observe one of the inevitable impromptu flaming-amputation-splenectomies, your first reaction shouldn't be to analyze the very last thing they did leading up to it and decide you can avoid making that mistake yourself. Nor should you interview the people who make it to the end and ask them for their detailed advice on safer techniques for maintaining a grip on the chainsaws while sliding through the turns. Nor should you seek out gloves that offer a better grip. *The biggest mistake is getting involved with that whole mess in the first place.*
> And also, that anything else than C is safe. Is it?
Many safer languages exist that make it hard or impossible to overrun a buffer or read previous memory contents, yes. They don't solve all possible problems, but they do solve this one.
Posted Feb 25, 2017 1:03 UTC (Sat)
by rgmoore (✭ supporter ✭, #75)
[Link]
If this doesn't wind up in next week's "quotes of the week", somebody is slipping.
Posted Feb 25, 2017 2:51 UTC (Sat)
by ibukanov (subscriber, #3942)
[Link] (12 responses)
One needs at least dependable types that turns code in a messy mathematical proofs even for simple things.
Posted Feb 25, 2017 3:59 UTC (Sat)
by josh (subscriber, #17465)
[Link] (10 responses)
This bug read off the end of a buffer into memory. That can't happen in memory-safe code.
The broken parsing could still have happened, but the data exposure wouldn't have.
That doesn't mean *other* bugs couldn't happen in memory-safe code; you can write bugs in any language. But this particular bug couldn't have happened in memory-safe code.
Posted Feb 25, 2017 12:09 UTC (Sat)
by job (guest, #670)
[Link] (9 responses)
Of course it can. If the buffer is not cleared between requests for example. Or if it is a shared buffer managed with offset pointers. It couldn't happen in exactly the same way, but there are countless similar potential screw ups possible.
You could interject that no one would implement anything like that, but that is ignoring reality. I've seen much worse. One could just as easily argue that surely no one deploys something handling sensitive data in 2017 without Valgrind (which on the face of it would seem to have caught this) yet here we are.
The poster child for managed languages is Java. No pointers so we're safe, right? Yet Tomcat has had nasty bugs with implications almost identical to this. A memory safe language may help but won't get rid of the problem. You simply have to be careful with production code, design things to fail safe where possible and use whatever tools at your disposal to discover problems early. (Cloudflare seems to have been running this for at least half a year until Google called them out on it.) It's much too easy to post facto declare your favorite language superior, but it has no basis in reality.
Posted Feb 25, 2017 12:56 UTC (Sat)
by pizza (subscriber, #46)
[Link]
I agree, and I've personally written "worse" in Java.
And it was absolutely necessary too, because when profiling our code to identify performance bottlenecks, I discovered that over half our server load was due to overhead in constructing messages using the "correct" JavaWay(tm). I roughly quadrupled the number of users each server could handle by using shared buffers with offset pointers because we were being absolutely murdered by the java allocator/gc overhead. Just by itself, that saved the company enough money to more than pay my salary the entire time I worked there.
(The next major Java revision added features that would have obviated the whole mess, but by then the dotcom crash was long over and I had very happily abandoned Java middleware in favor of bare metal C)
Posted Feb 25, 2017 20:02 UTC (Sat)
by roc (subscriber, #30627)
[Link] (7 responses)
Rust, for example, lets you create references to slices of arrays. Accesses to slices are bounds-checked, so you get more safety using those instead of buffer offsets. Of course it's possible for people to use buffer offsets when they should use slices, but slices are built-in and have nice syntax so they're the idiomatic approach.
A main reason people use manual object recycling in languages like Java is to reduce load on the garbage collector, reducing memory or throughput overhead depending on how it's tuned. With Rust this is a non-issue because you can stack-allocate in many cases, and for heap objects a good allocator like jemalloc uses free lists to give you most of the benefits of recycling.
Posted Feb 26, 2017 14:33 UTC (Sun)
by Jonno (subscriber, #49613)
[Link] (5 responses)
The problem with Java memory management isn't that it puts objects on the heap and not the stack, it is that it never deallocates stuff except during garbage collection. If most objects would be deallocated once their last pointer went out of scope, and the GC only had to deal with edge cases (such as unreachable cyclic data structures), you wouldn't need to do manual object reuse just to reduce the load on the GC.
Sure, a good allocator do reduce overhead a bit, but when 99% of your problem is fighting the GC, improving the last 1% doesn't do you much good...
Rust does all the "deallocate once out of scope" stuff perfectly. On the other hand it doesn't provide a GC at all, so if you create a cyclic data structure of (non-weak) reference counted pointers, the object will never be deallocated...
Posted Feb 26, 2017 15:35 UTC (Sun)
by mathstuf (subscriber, #69389)
[Link] (3 responses)
Posted Feb 26, 2017 21:45 UTC (Sun)
by roc (subscriber, #30627)
[Link] (1 responses)
You can create cycles using Rc and Arc refcounting; they don't call for additional review.
Posted Feb 27, 2017 3:01 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link]
Posted Feb 26, 2017 22:11 UTC (Sun)
by Jonno (subscriber, #49613)
[Link]
You can't do it with references, but you can do it with Rc and RefCell. For example, the following safe function would leak whatever argument you gave it:
fn leak<T>(val: T) {
Obviously using a RefCell<Weak<Foo<T>>> instead of the RefCell<Option<Rc<Foo<T>>>> avoids the leak:
fn no_leak<T>(val: T) {
> The typical solution I've seen so far is to use a pool of objects and then indexes into that pool for an adjacency list or other data layouts.
That was a common workaround used before the stabilization of Weak in Rust 1.4...
Posted Feb 26, 2017 21:44 UTC (Sun)
by roc (subscriber, #30627)
[Link]
That's one of the problems.
> If most objects would be deallocated once their last pointer went out of scope, and the GC only had to deal with edge cases (such as unreachable cyclic data structures), you wouldn't need to do manual object reuse just to reduce the load on the GC.
That would certainly help. Indeed, quite a lot of work has been done on such reference-counting-based GC for Java and other languages. See e.g. the paper "Down for the count? Getting reference counting back in the ring" by Blackburn et al. The problem for Java is that because the heap is shared between threads, object reference count updates have to be atomic operations, which makes them expensive, so to keep overhead down the compiler/runtime have to work really hard to avoid such updates. Swift does something similar (but without collecting cyclic garbage).
In Rust this is much less of a problem because you can avoid all reference-counting for single-owner objects; when an object isn't shared across threads you can use a cheap non-atomic refcount; and the lifetime+borrow system means you can often pass and return references to objects without altering refcounts, with static checking that everything's OK.
> On the other hand it doesn't provide a GC at all, so if you create a cyclic data structure of (non-weak) reference counted pointers, the object will never be deallocated...
Yes, it's like Swift in that respect --- except that as noted above, you're using refcounting a lot less in Rust.
Posted Feb 26, 2017 23:16 UTC (Sun)
by josh (subscriber, #17465)
[Link]
This comes down to "easy to use versus hard to misuse". ( https://ozlabs.org/~rusty/index.cgi/tech/2008-03-30.html and https://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html )
C falls far lower on that scale than many other languages, generally somewhere on the "easy to misuse" scale. That doesn't mean other languages hit the "It's impossible to get wrong." level for every kind of bug, but they usually hit some point on the "hard to misuse" scale.
Posted Feb 28, 2017 16:49 UTC (Tue)
by paulj (subscriber, #341)
[Link]
To reduce the risk of this kind of error, one must use engineering and programming techniques that transcend C or Java or Python (i.e. could apply to any). To tell one self "Well, it was just cause they used C" would be to miss that lesson.
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
struct Foo<T>(T, RefCell<Option<Rc<Foo<T>>>>);
let x = Rc::new(Foo(val, RefCell::new(None)));
*x.1.borrow_mut() = Some(x.clone());
}
struct Foo<T>(T, RefCell<Weak<Foo<T>>>);
let x = Rc::new(Foo(val, RefCell::new(Default::default())));
*x.1.borrow_mut() = Rc::downgrade(&x);
}
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Cloudflare Reverse Proxies are Dumping Uninitialized Memory