LWN: Comments on "Cloudflare Reverse Proxies are Dumping Uninitialized Memory"

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

thestinger — Thu, 02 Mar 2017 05:26:30 +0000

You can also switch a single subdomain like assets.yourdomain.com to using the Cloudflare proxy and then only reference those assets with subresource integrity hashes in the HTML. Hashes should also be in the path to the file like ?hash and then you can set Cache-Control max-age to a year and Cloudflare will cache them for a long time after the first request. It doesn't force you to pass all traffic through it, but for DDoS mitigation that's needed. One option is only turning it on for the HTML and other non-static-asset requests when under attack. That's leaving a lot of caching on the table though. Even dynamic requests can often be cached. You can make Cloudflare cache anything via Cache-Control + a cache everything page rule, the only limitation is that it clamps lower max-age to a 30 minute minimum with lower minimums requiring higher tier plans than free (no-cache, etc. work fine to disable it). Anyway since they don't charge based on bandwidth Cloudflare can save enormous amounts of money for people, which is a big part of why it's being used everywhere. Cloudflare needs a ton of spare capacity to absorb DDoS so they can give it away from free and those users get low priority if a DDoS occurs that they can't handle.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

paulj — Tue, 28 Feb 2017 16:49:19 +0000

+1 to this. This bug is about sharing buffers between different users, and not handling exceptional cases correctly. It could have happened with a "safe" language too.

To reduce the risk of this kind of error, one must use engineering and programming techniques that transcend C or Java or Python (i.e. could apply to any). To tell one self "Well, it was just cause they used C" would be to miss that lesson.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

HelloWorld — Tue, 28 Feb 2017 12:35:02 +0000

Again, libraries are not the issue here, the language is. The nice thing about an array or a pointer in C is that they're not just types, they're parameterized types. So if you have a declaration like T *t; or T t[42], where T is some type, *t will be an expression of type T, and you can't do that sort of thing for user defined types. So you either get something that isn't type-safe (and thus also not memory-safe) or a shitload of C preprocessor macros, which people generally agree are messy, because they behave inconsistently with the rest of the language.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

ssmith32 — Mon, 27 Feb 2017 20:57:24 +0000

They also provide a DNS service, which I don't believe would be affected by this. I'd stay away from any service that mitm any private data. Been looking at shifting my DNS to them, though.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

NightMonkey — Mon, 27 Feb 2017 20:15:00 +0000

Dumb SysAdmin here. It would seem in other languages this problem would be resolved by a "standard library" for C to deal with arrays. (Yah, not THE "C Standard Library", but some "standard by convention/consensus only".) Something where a programmer say "Ah, I hate dealing with array protections. Good thing HackerGSmith maintains the excellent Array library on GitHub so I don't have to."

Does such a thing exist? Or is this question daft?

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

mathstuf — Mon, 27 Feb 2017 03:01:59 +0000

You need to use RefCell as well (as shown in the other reply), but that also triggers extra review (from me at least) since Rust's compile guarantees are moved to runtime there. A test suite checking for proper dropping of a cyclic graph would be warranted.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

josh — Sun, 26 Feb 2017 23:16:04 +0000

Exactly.

This comes down to "easy to use versus hard to misuse". ( https://ozlabs.org/~rusty/index.cgi/tech/2008-03-30.html and https://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html )

C falls far lower on that scale than many other languages, generally somewhere on the "easy to misuse" scale. That doesn't mean other languages hit the "It's impossible to get wrong." level for every kind of bug, but they usually hit some point on the "hard to misuse" scale.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

Jonno — Sun, 26 Feb 2017 22:11:54 +0000

> Creating a cyclic data structure in Rust, AFAIK, requires unsafe code (which calls for additional review) because you need a mutable reference to write to and conflicts with the non-mutable reference you'd like to write with

You can't do it with references, but you can do it with Rc and RefCell. For example, the following safe function would leak whatever argument you gave it:

fn leak<T>(val: T) {
  struct Foo<T>(T, RefCell<Option<Rc<Foo<T>>>>);
  let x = Rc::new(Foo(val, RefCell::new(None)));
  *x.1.borrow_mut() = Some(x.clone());
}

Obviously using a RefCell<Weak<Foo<T>>> instead of the RefCell<Option<Rc<Foo<T>>>> avoids the leak:

fn no_leak<T>(val: T) {
  struct Foo<T>(T, RefCell<Weak<Foo<T>>>);
  let x = Rc::new(Foo(val, RefCell::new(Default::default())));
  *x.1.borrow_mut() = Rc::downgrade(&x);
}

> The typical solution I've seen so far is to use a pool of objects and then indexes into that pool for an adjacency list or other data layouts.

That was a common workaround used before the stabilization of Weak in Rust 1.4...

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sun, 26 Feb 2017 21:45:27 +0000

> Creating a cyclic data structure in Rust, AFAIK, requires unsafe code (which calls for additional review)

You can create cycles using Rc and Arc refcounting; they don't call for additional review.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sun, 26 Feb 2017 21:44:21 +0000

> The problem with Java memory management isn't that it puts objects on the heap and not the stack, it is that it never deallocates stuff except during garbage collection.

That's one of the problems.

> If most objects would be deallocated once their last pointer went out of scope, and the GC only had to deal with edge cases (such as unreachable cyclic data structures), you wouldn't need to do manual object reuse just to reduce the load on the GC.

That would certainly help. Indeed, quite a lot of work has been done on such reference-counting-based GC for Java and other languages. See e.g. the paper "Down for the count? Getting reference counting back in the ring" by Blackburn et al. The problem for Java is that because the heap is shared between threads, object reference count updates have to be atomic operations, which makes them expensive, so to keep overhead down the compiler/runtime have to work really hard to avoid such updates. Swift does something similar (but without collecting cyclic garbage).

In Rust this is much less of a problem because you can avoid all reference-counting for single-owner objects; when an object isn't shared across threads you can use a cheap non-atomic refcount; and the lifetime+borrow system means you can often pass and return references to objects without altering refcounts, with static checking that everything's OK.

> On the other hand it doesn't provide a GC at all, so if you create a cyclic data structure of (non-weak) reference counted pointers, the object will never be deallocated...

Yes, it's like Swift in that respect --- except that as noted above, you're using refcounting a lot less in Rust.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

mathstuf — Sun, 26 Feb 2017 15:35:35 +0000

Creating a cyclic data structure in Rust, AFAIK, requires unsafe code (which calls for additional review) because you need a mutable reference to write to and conflicts with the non-mutable reference you'd like to write with. The typical solution I've seen so far is to use a pool of objects and then indexes into that pool for an adjacency list or other data layouts. The indexes should be opaque types which disallow algebra and other problems. I imagine it would then need to have a "compact" method to fill in gaps in the pool when necessary.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

Jonno — Sun, 26 Feb 2017 14:33:56 +0000

> A main reason people use manual object recycling in languages like Java is to reduce load on the garbage collector, reducing memory or throughput overhead depending on how it's tuned. With Rust this is a non-issue because you can stack-allocate in many cases, and for heap objects a good allocator like jemalloc uses free lists to give you most of the benefits of recycling.

The problem with Java memory management isn't that it puts objects on the heap and not the stack, it is that it never deallocates stuff except during garbage collection. If most objects would be deallocated once their last pointer went out of scope, and the GC only had to deal with edge cases (such as unreachable cyclic data structures), you wouldn't need to do manual object reuse just to reduce the load on the GC.

Sure, a good allocator do reduce overhead a bit, but when 99% of your problem is fighting the GC, improving the last 1% doesn't do you much good...

Rust does all the "deallocate once out of scope" stuff perfectly. On the other hand it doesn't provide a GC at all, so if you create a cyclic data structure of (non-weak) reference counted pointers, the object will never be deallocated...

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sun, 26 Feb 2017 03:33:22 +0000

To be fair, some W3C groups (e.g. Webapps) have done useful work in the last decade.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sun, 26 Feb 2017 03:32:22 +0000

In this case, clearing buffers would have reduced the amount of data leaked by some fraction, but they'd still have leaked data in the live allocations.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sun, 26 Feb 2017 03:29:50 +0000

Oh yeah, and such 'wrappers' tend not to protect you against things like use-after-free which are a huge source of exploitable bugs. Those that do are even more intrusive and high overhead.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sun, 26 Feb 2017 03:28:32 +0000

Not all "memory-safe" languages are created equal, and Rust has some big advantages of GC-based languages here. See my comment above.

> The only fault of C is that people use arrays directly instead of having wrappers that checks out-of-range access.

* Using such wrappers in C is horrendously ugly. C++ at least is better.
* The compiler and standard libraries are not designed to work well with such wrappers. For example Rust's iterators let you traverse an array (or other data structure) without explicit indexing so that you don't manipulate offsets directly and hence don't need bounds checks.
* Since "wrappers" etc aren't built in, projects choose different ones and then you get conflicts combining code.
* Since "wrappers" aren't built in, people just aren't accustomed to using them. They'll choose the convenient built-in thing first.

In reality how many projects use such wrappers? I can't think of any. It must be a small fraction at best. There are good reasons for that. Even C++ "vector" doesn't do bounds checking (if you use operator[], the natural syntax)!

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

flussence — Sat, 25 Feb 2017 22:45:28 +0000

I'm not sure what's more sad here; the possibility that the W3C would have people like this as members, or the assumption that namedropping them would work in your favour when people outside the academic bubble know well that the WHATWG has done all the actual work for the past 15 years.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

dmaas — Sat, 25 Feb 2017 21:09:46 +0000

Our experience with DDoS attacks is that if you aren't using some kind of distributed, edge-filtering ingestion system like CloudFlare, you won't even be able to log in to your web server or load balancer to change any settings or try to set up blocking. Massive TCP attacks just overwhelm anything you can do when a single piece of hardware is receiving the traffic.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

Cyberax — Sat, 25 Feb 2017 20:29:48 +0000

That's what Cloudflare does. It's a load balancer for sites with some caching.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

roc — Sat, 25 Feb 2017 20:02:03 +0000

True and fair enough, but not all memory-safe languages are created equal.

Rust, for example, lets you create references to slices of arrays. Accesses to slices are bounds-checked, so you get more safety using those instead of buffer offsets. Of course it's possible for people to use buffer offsets when they should use slices, but slices are built-in and have nice syntax so they're the idiomatic approach.

A main reason people use manual object recycling in languages like Java is to reduce load on the garbage collector, reducing memory or throughput overhead depending on how it's tuned. With Rust this is a non-issue because you can stack-allocate in many cases, and for heap objects a good allocator like jemalloc uses free lists to give you most of the benefits of recycling.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

josh — Sat, 25 Feb 2017 19:45:35 +0000

Depends on the approach you use. If the whole DDoS uses the same set of credentials, it'd be easy enough to detect and drop that traffic at the web server or load balancer.

But yes, otherwise agreed. If you actually do need DDoS mitigation, rather than just normal bandwidth/scaling improvements, then that becomes a much harder problem to solve without interposing a whole-site CDN.

Please

corbet — Sat, 25 Feb 2017 19:39:57 +0000

This comment is neither respectful nor informative; no more of these, please?

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

clintpatton — Sat, 25 Feb 2017 19:26:42 +0000

I like the Cloudflare advertising slogan, "Making the Internet Work the Way It Should". As a Computer Scientist, I know Cloudflare is actually "Breaking the Internet and Making It Worse".

I bet Cloudflare is run by Chinese Communists who are former Google China employees? Or maybe Cloudflare has outsourced its programming to India and the clueless idiots who break HTML 5 every 24 hours?

Long live HTML 4 and the Free and Open Internet. I'm now cancelling my W3C membership.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

Cyberax — Sat, 25 Feb 2017 19:21:07 +0000

A typical DDoS simply overwhelms _any_ dynamic functionality, even simple password check.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

josh — Sat, 25 Feb 2017 18:55:53 +0000

You can point to the CDN domain for cacheable items. That'll even speed up loading, via sharding: the browser can make another set of concurrent requests to the CDN.

As for DDoS protection: depends on the nature of the DDoS. If it's request-based, it may suffice to cache all the static content, and if someone is DDoSing authenticated content, revoke those authentication credentials temporarily while investigating.

However, if it's a non-request-based DDoS, with attackers just throwing raw data at you to drown you in bandwidth consumption, that's a bigger problem. Sometimes the requests can be blackholed at the routing level, if you have a good technical relationship with your bandwidth provider. If the requests are coming from multiple otherwise-legitimate ASes and can't easily be blackholed at the routing level, then you don't have much choice other than "go offline" or "have more bandwidth than the attack can consume". CDNs take the latter approach, but only if you put the CDN in front of your site rather than to the side, which is exactly the problem here.

Short of becoming big enough to be your own mini-CDN (with load balancers and caching layers in various data centers), I don't know. I can imagine some technical solutions, but they'd certainly require more work.

However, how many sites using Cloudflare actually needed the DDoS protection, and how many just wanted to save bandwidth? The latter is much easier to solve.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

zlynx — Sat, 25 Feb 2017 17:17:14 +0000

This requires assembly coding or knowing that you should use a security library. Regular memory clears are optimized away, almost always.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

luto — Sat, 25 Feb 2017 16:26:05 +0000

I'll add a third lesson: when using a language like C, clear sensitive buffers when you're done with them. Heck, even with a memory-safe language, clear your buffers if only to avoid exposure in coredumps, swap files, and the like.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

HelloWorld — Sat, 25 Feb 2017 14:14:30 +0000

And besides, even "safe" wrappers around arrays don't help against all other kinds of mistakes that basically happen only in C, like dereferencing invalid pointers (e. g. pointers to freed memory or to a local variable that has gone out of scope).

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

HelloWorld — Sat, 25 Feb 2017 14:04:03 +0000

> The only fault of C is that people use arrays directly instead of having wrappers that checks out-of-range access.
The reason people often don't use them is that C doesn't provide good abstractions to do that. Without language support for generics or templates, that sort of thing will invariably suck.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

pizza — Sat, 25 Feb 2017 12:56:09 +0000

> You could interject that no one would implement anything like that, but that is ignoring reality. I've seen much worse.

I agree, and I've personally written "worse" in Java.

And it was absolutely necessary too, because when profiling our code to identify performance bottlenecks, I discovered that over half our server load was due to overhead in constructing messages using the "correct" JavaWay(tm). I roughly quadrupled the number of users each server could handle by using shared buffers with offset pointers because we were being absolutely murdered by the java allocator/gc overhead. Just by itself, that saved the company enough money to more than pay my salary the entire time I worked there.

(The next major Java revision added features that would have obviated the whole mess, but by then the dotcom crash was long over and I had very happily abandoned Java middleware in favor of bare metal C)

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

job — Sat, 25 Feb 2017 12:09:12 +0000

> This bug read off the end of a buffer into memory. That can't happen in memory-safe code.

Of course it can. If the buffer is not cleared between requests for example. Or if it is a shared buffer managed with offset pointers. It couldn't happen in exactly the same way, but there are countless similar potential screw ups possible.

You could interject that no one would implement anything like that, but that is ignoring reality. I've seen much worse. One could just as easily argue that surely no one deploys something handling sensitive data in 2017 without Valgrind (which on the face of it would seem to have caught this) yet here we are.

The poster child for managed languages is Java. No pointers so we're safe, right? Yet Tomcat has had nasty bugs with implications almost identical to this. A memory safe language may help but won't get rid of the problem. You simply have to be careful with production code, design things to fail safe where possible and use whatever tools at your disposal to discover problems early. (Cloudflare seems to have been running this for at least half a year until Google called them out on it.) It's much too easy to post facto declare your favorite language superior, but it has no basis in reality.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

sorokin — Sat, 25 Feb 2017 11:52:15 +0000

I have a story about how people using safe language (.NET) got out-of-range accesses, double frees and degraded performance at the same time.

They needed to store some data in arrays. And arrays were quite big. And big arrays aren't kept in gen0 heap, they are created in gen2. And this causes full GC. And full GC causes UI to hang. The solution? Split a big array into an array of small arrays of fixed-sizes (great idea: each access does 2 indirections, 2 branches, 1 division). And to make anything fast and smooth let's make a pool of these small fixed size arrays. Surely the container doesn't check for index out-of-range condition, so you can access a few element after the last and won't get any error. And of course they got double free errors. And for .NET you have no valgrind/sanitizer to detect errors like this.

The only fault of C is that people use arrays directly instead of having wrappers that checks out-of-range access. And I would like to note, that sometimes you really do want to access array without these checks.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

gioele — Sat, 25 Feb 2017 10:53:08 +0000

> Stop using Cloudflare. By all means use CDNs to cache non-sensitive, non-customer-specific data, but don't use a CDN that effectively acts as an HTTPS MITM on half the Internet.

In the current "everything should use HTTPS" Web, how do you provide a CDN service without acting as a MITM? Also, how do you provide DDoS protection?

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

phg — Sat, 25 Feb 2017 09:40:58 +0000

> The Incident report on Cloudflare's blog quite clearly states "the bug is not in Ragel itself. It is in Cloudflare's use of Ragel. This is our bug and not the fault of Ragel."

Not initially. One of Ragel’s authors had to reach out first
for Cloudflare to add the clarifying remarks:
https://www.reddit.com/r/programming/comments/5vtv16/clou...

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

josh — Sat, 25 Feb 2017 03:59:03 +0000

> A bug like this that can be trivially made in most memory-safe languages as it is not about memory safety. Rather it is about not enforcing proper partitions for accessing data allowing other users to read it.

This bug read off the end of a buffer into memory. That can't happen in memory-safe code.

The broken parsing could still have happened, but the data exposure wouldn't have.

That doesn't mean *other* bugs couldn't happen in memory-safe code; you can write bugs in any language. But this particular bug couldn't have happened in memory-safe code.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

ibukanov — Sat, 25 Feb 2017 02:51:35 +0000

A bug like this that can be trivially made in most memory-safe languages as it is not about memory safety. Rather it is about not enforcing proper partitions for accessing data allowing other users to read it.

One needs at least dependable types that turns code in a messy mathematical proofs even for simple things.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

bronson — Sat, 25 Feb 2017 02:02:16 +0000

Maybe combine those two suggestions: "CloudFlare"

Sounds scary!

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

rgmoore — Sat, 25 Feb 2017 01:03:25 +0000

If this doesn't wind up in next week's "quotes of the week", somebody is slipping.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

geek — Sat, 25 Feb 2017 00:43:31 +0000

well, certainly type-safe languages are available that have been used for large programs and have no known errors. C is not one of them.

Cloudflare Reverse Proxies are Dumping Uninitialized Memory

tialaramex — Sat, 25 Feb 2017 00:21:33 +0000

Unfortunately for Cloudflare the advice to avoid them probably makes more sense for their larger, paying customers than the small people on a "free" plan. This is because a lot of those "free" Cloudflare sites are also running in a cheap hosted environment anyway, they're vulnerable to equivalent goofs in their hosting environment which, like Cloudflare, they share with random people from the Internet but unlike Cloudflare it's not a huge Internet brand that will attract massive press attention & react quickly to problems. So Cloudflare probably isn't the weakest link for them.