LWN: Comments on "Announcing Arti, a pure-Rust Tor implementation (Tor blog)"

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

immibis — Thu, 22 Jul 2021 10:43:40 +0000

No, it simply means "or". As in:

> So you are either one of the very rare programmers whose mental capacity is sufficient to write C or C++ code without introducing subtle memory errors.

These are both languages where it's possible to introduce subtle memory errors, and the statement suggests that the person in question is capable of understanding their program's memory management well enough to not introduce them.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

immibis — Thu, 22 Jul 2021 10:42:09 +0000

This sounds like it has exactly the same problem as Python. Does the build script work on every Linux distribution?

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

marcH — Wed, 21 Jul 2021 22:25:09 +0000

> > Is the former acceptable?

> "Acceptable" is just a point on the cost/benefit curve.

Sure, what I meant was: should anything at all be done about memory corruption? The answer is yes of course - and it is being done in some places and not just with Rust. Exciting times.

> Until then, it will be ignored unless an external entity (eg government regulation or payment card or insurance carrier requirement) forces an organization to proactively care.

In general yes of course but there a few exceptions like Microsoft, Google and a few other "SmallTech". Check the Microsoft slides linked above.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

pizza — Wed, 21 Jul 2021 17:27:22 +0000

> Is the former acceptable?

"Acceptable" is just a point on the cost/benefit curve.

Clearly it has been acceptable. And, to be blunt, it will continue to be, for the same reason that the financial industry still runs on COBOL -- Rewriting the billions of lines of existing "inherently unsafe" code into "memory-safe" languages will cost a *lot* more money (and introduce far more new bugs along the way) than the current practice of (semi-proactively) plugging holes and cleaning up messes after the fact.

The costs of "security vulnerabilities" are nearly always external (ie someone else's problem). That vulnerability only becomes an "incident" is when an organization directly incurs some cost. Until then, it will be ignored unless an external entity (eg government regulation or payment card or insurance carrier requirement) forces an organization to proactively care.

Ownership and lifetimes

anton — Tue, 20 Jul 2021 10:25:53 +0000

Concerning integer overflow, the result with a given bit-width is the same for twos-complement arithmetic across all hardware (they all can perform modulo arithmetic, aka wrapping on overflow). And all architectures introduced since 1970 use twos-complement arithmetic exclusively (while, e.g., . So for integer overflow, the behaviour could be just standarized, no need for cop-outs like implementation-defined or implementation-specified. As an example modulo arithmetic has been standardized in Java from the start.

Ownership and lifetimes

anton — Tue, 20 Jul 2021 09:58:02 +0000

Backwards-compatible C compilers in the relaxed Linus Torvals sense (if no one notices, it's not breakage) are possible and desirable.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

anton — Tue, 20 Jul 2021 08:55:30 +0000

Yes, religion is a more appropriate analogue for the attachment that programmers feel to programming languages than Stockholm syndrome. Another phenomenon that may explain the attachment is choice-supportive bias (and that may also be related to religion).

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

marcH — Mon, 19 Jul 2021 02:36:52 +0000

This is a very interesting presentation, thanks for sharing it! I enjoyed it because you really don't need to be a security expert to make the most of it, you "only" need to be a software engineer.

Among others it explains why attacks rely less often on vulnerabilities and more on other techniques like social engineering. It's not because there was much coding and quality progress, in fact it rather depressingly states that developers keep making the same mistakes as before. The main reason is because Microsoft fixes them much faster than ever and made it very difficult to turn off auto updates. This gives a very strong incentive to minimize usage of vulnerabilities and reserve them for discrete, targeted attacks. Auto-updates have basically changed the market of vulnerabilities, their value has increased to the point where only nation states or very rich actors can afford them. I'm over simplifying of course, open the PDF if you're interested in the actual numbers.

So I stand corrected: the connection between ransomware and memory safety is probably tenuous... _if_ you use aggressively auto-updated software exclusively!

https://www.theverge.com/2021/1/6/22217052/microsoft-wind...
(Windows 7 is still running on at least 100 million PCs)

As long as you're not a high profile target or you don't mind getting spied on by any government or other powerful actor, keep using software written in a memory corruption language. Just make sure there's a billion dollars company constantly racing to keep it up to date.

Also keep in mind this analysis is based on Microsoft's data only.

Another interesting point is how effective but still limited mitigations like CFG, CET,... are and how the only way to make them 100% effective "decomposes to needing to solve memory safety".

ncm

tialaramex — Sun, 18 Jul 2021 21:53:05 +0000

It seems plausible that ncm is Nathan Myers, who was, and perhaps still is, an active JTC1/SC22/WG21 (ie C++ Standards Committee) participant. In which case what you've got there is two experts disagreeing.

But we are on the Internet, and so it is of course also possible ncm is a dog (I have verified that the committee member was not a dog).

In the former case, maybe we can say Nathan is trying to get C++ to a better place. After all, I think implicit constructors were a bad idea, they're the wrong default, but they're only a default at all because of the "explicit" keyword, which is apparently Nathan's idea. So once the situation was "C++ constructors are inexplicably dangerous" and Nathan improved it to "C++ constructors are inexplicably dangerous by default". The correct fix (an "implicit" keyword) violates C++ backwards compatibility promises and (which of more practical upshot) breaks a bunch of working code. So "explicit" means as long as every C++ programmer is conscientious and never makes a mistake they avoid this particular footgun. "Hooray".

Ownership and lifetimes

marcH — Sun, 18 Jul 2021 18:49:42 +0000

> Stroustrup never, ever rants. When he writes, every word counts. To perceive ranting only demonstrates confusion.

Whether it's true or not, I'm afraid you don't realize how cultish that is perceived.

Ownership and lifetimes

marcH — Sun, 18 Jul 2021 18:44:03 +0000

> > I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.

I think this is because when you read a piece of code, the difficult and error-prone part is the "stateful" part. A child can understand a one page-long function as long it's "pure" with only explicit inputs and outputs and no side-effect; it's "just maths"!
https://en.wikipedia.org/wiki/Referential_transparency

But good luck when that one page-long is updating some complex data structure that persists before and after the code is run. Then you need to keep in your head a lot more context and information that is not directly visible on the screen. If on top of that you also need to remember what other pieces code could be performing on the same data _at the same time_ then it's game over for most people and even for the smartest is much more cognitive strain. See excors's comment about locks etc. at https://lwn.net/Articles/862591/

This is also why writing hardware code is IMHO "harder" than software: hardware design is all about managing a huge number of concurrent states.

It's about time low-level software takes some elevation and safe (!) distance from hardware design and stops mimicking it because unlike hardware, software has no reason to burden developers with so much micro-management of data states and side effects. Instead, tell the compiler or static analyser who owns or borrows what data and when and let it find the bugs. Yes, that makes compilation much longer but that time is nothing compared to error-prone code reviews that often don't happen because no time has been planned for them. Let the machines do the tedious work, they're so much better at it.

Ownership and lifetimes

marcH — Sun, 18 Jul 2021 18:10:19 +0000

> To believe that C++ is not rapidly growing and changing is to believe a fantasy that C++20 is hardly different from C++17, from C++14, from C++11, from C++03, from C++98.

OK, so language theory discussions are fascinating (thank you everyone) but now let's get real a bit. Let's pretend I'm managing a software team and I've been asked to get rid of most safety issues in the C++ codebase of our mission-critical application. What -Wold-unsafe-c++ compiler flag do I turn on to make sure my not-all-perfect developers don't make any mistake and stick to "modern C++"? If that compiler flag is not available, what static analyzer(s) should I use? I have practically unlimited budget so I don't mind commercial products. (Even an unlimited budget is unfortunately not enough to hire an elite team of C++ developers who never make any mistake)

> there is no worry about lunch.

I don't think anyone doubts C++ will still make a lot of money after even the youngest people in this discussion are dead, no reason to feel threatened.
There is a shortage of developers, not a shortage of code to write. Apparently you can still get paid to fix COBOL code:
https://www.theverge.com/2020/4/14/21219561/coronavirus-p...

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

marcH — Sun, 18 Jul 2021 17:54:26 +0000

> There's merit in selecting the good parts of prior art and mixing them in a useful way, the same way there's merit in meta analysis even when they don't create new knowledge.

I would call this "engineering".

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

marcH — Sun, 18 Jul 2021 16:48:32 +0000

> Ah, okay, so that's "70% of security vulnerabilities" -- which is *not* the same as "security incidents".

Is the former acceptable?

Ownership and lifetimes

flussence — Sun, 18 Jul 2021 13:13:00 +0000

Congratulations on becoming the new IBM OpenOffice Guy.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

pizza — Sun, 18 Jul 2021 01:39:05 +0000

> Graph shows a period of 12 years, 2006-2018, showing that around 70% of "security vulnerabilities addressed by a security update"

Ah, okay, so that's "70% of security vulnerabilities" -- which is *not* the same as "security incidents".

(As the 7th & 8th slide in that deck demonstrates, there is a considerable gap between "vulnerabilities" and "exploits", and the 9th slide says the "market" has moved predominantly to social engineering-based attacks instead)

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

ms-tg — Sat, 17 Jul 2021 22:43:11 +0000

Further googling also reminds us of the following additional citations that may be of interest:

* Microsoft Security Response Center: A proactive approach to more secure code
https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/
(repeats same graph and statistic, introduces the case for Rust in that context)

The connection between the cited statistic, and Microsoft's exploration of the Rust language, appear to have been widely covered in all sorts of media, for example:

* https://www.zdnet.com/article/microsoft-to-explore-using-...

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

ms-tg — Sat, 17 Jul 2021 22:34:29 +0000

Hi pizza,

> I'd love to see an actual citation for that "70% of all security incidents" thing.

Here's the citation I am aware of:

* Matt Miller, an engineer at Microsoft, presented the following talk at the conference Bluehat 2019 in Israel:
- VIDEO: https://www.youtube.com/watch?v=PjbGojjnBZQ
- SLIDES: https://github.com/Microsoft/MSRC-Security-Research/blob/...

As an aside: this appears to be the same conference at which Andrew "bunnie" Huang delivered the talk "Supply Chain Security: If I were a Nation State…", which I think I also read about here at LWN.

It appears that the citation for the 70% statistic starts at 14:05 into the talk, which corresponds to Slide 11.

Graph shows a period of 12 years, 2006-2018, showing that around 70% of "security vulnerabilities addressed by a security update" each year correspond to a memory safety issue. Subsequent slides break those down further into classes of memory safety issues.

Hope this helps!

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

pizza — Sat, 17 Jul 2021 21:31:43 +0000

> For free time projects you're right. But when all businesses tired of ramsomware and other malware finally connect 70% of security incidents with the language of memory corruption and finally ask their providers "Wait, what language did you code this in?", then it won't be up for programmers themselves to decide. Money will talk. Very few programmers get to decide what language is used in $DAYJOB.

I'd love to see an actual citation for that "70% of all security incidents" thing.

For example. The largest data breach in history (Equifax) was due to an unpatched Apache Struts deployment, written in "no memory corruption" Java.

Phishing and general credential theft? Social engineering, not memory corruption.
Ransomware? Social engineering and MS Windows treating "opening documents" the same as "executing a binary" (along with awful UI design so the user genuinely can't tell the difference most of the time), not memory corruption.
SQL injection & XSS vulnerabilities? Improper validating/escaping inputs, not memory corruption
SSH/SSL downgrade attacks? Protocol misdesign, not memory corruption
IoT botnets? Hard-coded passwords/backdoors, not memory corruption

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

marcH — Sat, 17 Jul 2021 20:52:26 +0000

It depends how programmers are "filtered out". For free time projects you're right. But when all businesses tired of ramsomware and other malware finally connect 70% of security incidents with the language of memory corruption and finally ask their providers "Wait, what language did you code this in?", then it won't be up for programmers themselves to decide. Money will talk. Very few programmers get to decide what language is used in $DAYJOB.

BTW Rust is of course not the only memory safe language and this "no memory corruption" trend has started already, after all Java/Kotlin, Javascript, Swift and golang are already very popular. What's new with Rust is filling the last gap where C and C++ had no serious competition. That's why its zealots feel threatened.

Ownership and lifetimes

mrugiero — Sat, 17 Jul 2021 16:06:25 +0000

Then you use an older system I guess. If you were fine with no updating your software you're probably fine with that too.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

marcH — Fri, 16 Jul 2021 22:30:19 +0000

Exactly.

C++ is safe, you're just "holding it wrong". Well, too bad a couple lines of code "held wrong" are enough for a vulnerability or elusive concurrency crash.

Standard committee-level discussions matter, but how that translates into what happens in the trenches matters even more.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

roc — Fri, 16 Jul 2021 00:20:16 +0000

Python is a mess.

Rust libraries that wrap a C library usually come with a "build script" (build.rs) that is capable of downloading, configuring, building and linking the C library. This works very well in practice, and is not a hack at all. The cc library (https://crates.io/crates/cc) gives you a nice API to help with this. A lot of these Rust libraries also give you the option of using the system version of the library if you want.

If the C code itself has dependencies that aren't available on the system then that would get complicated. I haven't encountered this problem in my Rust work. If I did, I guess I'd probably choose a different library.

> suddenly the distro system library/dependency model doesn't look so bad after all.

No, the model where distros package all libraries simply doesn't work at all for me, and lots of other developers, for the reasons I mentioned.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

pizza — Thu, 15 Jul 2021 23:46:12 +0000

> When I'm writing software I'll decide what libraries and what versions of libraries I want to depend on. I'm not going to ask the Ubuntu maintainers to decide which libraries are OK for my project.

Sure, you need pyFoo == 1.33.1 (or whatever)

Meanwhile, pyFoo includes a big pile of native C (or Rust, or whatever) code that itself has other dependencies.

How are those dependencies and sub-dependencies handled? (The answer: Quite badly, as in "It only works on a specific Ubuntu point release." Go ahead, ask me how I know this)

These language-specific repositories are great, until they inevitably have to reach outside the language ecosystem. Then it's hack upon hack until suddenly the distro system library/dependency model doesn't look so bad after all.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

mss — Thu, 15 Jul 2021 23:14:20 +0000

For many people it's far easier to trust (and occasionally randomly verify) just one distro package repository rather than the distro one plus all the programming-language-related ones.
Especially that there were already some backdoored packages found in, for example, NPM and PyPI, so the threat is very real.

> When I'm writing software I'll decide what libraries and what versions of libraries I want to depend on.

Distros sometimes (often?) need to build and run a package against different library versions than its upstream supports (for various reasons).

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

rgmoore — Thu, 15 Jul 2021 17:11:12 +0000

C++ can express abstractions that Rust (still) cannot.

This is trivially true, since one of the goals of Rust is to make it difficult or impossible to express some kinds of bugs. Greater expressivity is desirable only to the extent the things you're expressing are themselves desirable. Adding lots of foot guns makes the language more expressive, but in a way that's likely to make the final output worse rather than better.

Ownership and lifetimes

khim — Thu, 15 Jul 2021 16:42:55 +0000

This is all well and good, but this requires some form of consensus.

And in C/C++ world discussions are just not happening. Why have the coveted “pointer provenance” proposals were not incorporated into standard in 15 years (and counting)?

Because not even C/C++ compiler writers can agree with each other. -std=friendly-c proposal had the exact some fate.

And when someone tries to resolve the issue by “starting from scratch”… the end result is D, Java, C# or Rust… never a “friendly C”…

Rust is just the first such “new C” language which is low-level enough to actually be usable for all kinds of usages. Starting from lowest-level just above assembler.

Ownership and lifetimes

farnz — Thu, 15 Jul 2021 16:26:19 +0000

If such platforms exist and need to be cared about (like you, I'm not sure if they did, or did not), an easy solution would be to make the behaviour of signed integer overflow unspecified, rather than undefined.

In the C language spec, there are four groups of behaviour (from strongest definition to weakest):

Defined. To be a confirming implementation, you must behave in the way the Standard says you should, and there are no choices here. For example, the behaviour of the free(void * ptr) standard library function in C is defined if ptr is NULL or a pointer returned by malloc.
Implementation defined. The Standard sets out restrictions on what this can be defined as, but the implementation gets to choose one concrete behaviour and stick to it. For example, the number of bits in int is implementation defined in C - any value greater than or equal to 16 is permissible, and the implementation must document which value it has cchosen.
Unspecified. The Standard sets out restrictions on what behaviour is acceptable (e.g. "abc" == "abc" must be either 1 for true or 0 for false depending on whether the compiler merges identical string literals, but it cannot be 42 under any circumstances). The compiler can choose any behaviour it likes from the set the standard allows, and can choose different behaviour every time it sees the unspecified construct (e.g. printf("%d %d\n", "abc" == "abc", "abc" == "abc"); can print "0 0", "0 1", "1 0", or "1 1", and can print a different one each time the program is run); it does not have to stick to one choice, as it does for implementation-defined behaviour.
Undefined. If a program run would follow a path with undefined behaviour, the meaning of the program as a whole is not set out by the Standard; it can do anything the compiler implementer wishes, even if the user of the compiler thinks that's a crazy stupid outcome.

If signed integer overflow became unspecified, such that the result of a signed integer overflow could be any integer value, then we're in a much better place. The compiler can just use the machine instruction (assuming it doesn't trap), and we don't have the pain where the compiler goes "well, if this is signed overflow, then behaviour is undefined, ergo I can assume it's not, ergo I can optimise out a check"; instead, signed overflow has to produce an integer, but which integer is not known by the code author.

Ownership and lifetimes

matthias — Thu, 15 Jul 2021 15:56:18 +0000

Even if there is a platform that lacks an overflow happens flag. Then on this platform the implementation defined behavior is overflow with the semantics of what overflow means on this platform. Implementation defined means you can choose different semantics for very combination of compiler/platform.

If you do not know which platform your code will run on, you can still be not sure what the result of x+100 is in case of overflow, but you can at least be sure what (x+100)&0 is. And you can be sure that if you do an assertion x+100>x to test for an overflow that the assertion is not optimized away. Ok, it can be optimized away of the compiler can prove that there is not overflow, but this is no problem.

Ownership and lifetimes

mathstuf — Thu, 15 Jul 2021 14:12:19 +0000

> That is, if it's implementation defined the compiler you use needs to define a given behavior for the platform it'll run and never violate it. So, gcc would be mandated to tell you what value will result from overflowing, even if such a value is one for SPARC and a different one for x86. I fail to see why it UB made more sense to them than that at the time.

I may be misremembering, but didn't some platforms not provide a way to tell? That would require the compiler to emit manual checks for every possible overflowing operation in order to guarantee *a* behavior. This would also mean that optimizing to, e.g., FMA instructions is basically impossible because if the intermediate ops overflow, that needs to be handled as well.

But if there are no platforms that lack an "overflow happened" flag…meh.

Ownership and lifetimes

mathstuf — Thu, 15 Jul 2021 13:46:43 +0000

Well, the fun thing here is that the problem appeared on macOS arm64. Previously, everything had been working on all platforms (our library does use `-fvisibility=hidden` to emulate Windows behavior on the other platforms because we *do* export our symbols properly). Which makes me wonder why the typeinfo wasn't already duplicated…but whatever. Anyways, something is different with macOS arm64 and this problem came up and now it's fixed by improving the code, so I'm happy at least (and it's one more "ghostly linker fun" bug under my belt :/ ).

Ownership and lifetimes

mathstuf — Thu, 15 Jul 2021 13:43:57 +0000

These snipe-y comments sound awfully embarrassing by themselves to me too. If the post is so embarrassing, why not just leave them to stand on their own if you're not going to contribute anything to the discussion in your reply?

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

mathstuf — Thu, 15 Jul 2021 13:42:15 +0000

> C++ can express abstractions that Rust (still) cannot.

The one I know of is "template template" parameters (basically higher kinded types). I suspect concepts can get one higher-ranked trait bounds as well, but I'm not sure about that. FWIW, both of these are being worked on.

On the other hand, C++ lacks useful destructuring (pattern matching is being worked on, but I don't see it being anywhere near as ergonomic), is stuck with terrible macro expansion rules, bad move semantics, and a blissful unawareness of lifetime analysis at the language (as opposed to the documentation/review) level.

I know which set *I* find more useful, but that's obviously not all that universal since there are many niches in the programming space.

Ownership and lifetimes

foom — Thu, 15 Jul 2021 11:49:27 +0000

The problem you describe here is that windows (PE/COFF) DLLs don't support C++'s vague linkage symbols across DLLs properly. This makes windows DLLs effectively incompatible with the C++ standard's requirements.

Neither MacOS (mach-o) nor Linux (elf) shared libraries have this issue. They both work fine in this situation, by ensuring that vague linkage symbols are exported and deduplicated across the shared library boundaries by default.

One might expect this would have been addressed in Windows at some point in the last few decades...but it hasn't been.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

roc — Thu, 15 Jul 2021 06:21:37 +0000

If you're going to contradict a C++ standards committee member when making a claim about the capabilities of C++, you'd better have some evidence to back it up. So, how do you achieve equivalence to Rust memory and thread safety with C++? (Don't say ASAN or TSAN, because those tell you nothing about untested paths.)

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

roc — Thu, 15 Jul 2021 06:10:34 +0000

Rust "cargo" definitely is not trying to "reimplement the distro package repository". Distro-packaged libraries come nowhere near cargo's capabilities --- e.g. feature selection, library version selection, the ability to substitute patched versions of libraries, the ability to run multiple versions side by side, the ability to publish libraries whenever you want and use them in any project immediately regardless of what Linux distro your users are running --- or, of course, if they're not running Linux at all.

When I'm writing software I'll decide what libraries and what versions of libraries I want to depend on. I'm not going to ask the Ubuntu maintainers to decide which libraries are OK for my project.

Ownership and lifetimes

ncm — Thu, 15 Jul 2021 05:52:40 +0000

You have failed to keep up with events, so you embarrass yourself by publishing falsehoods.

Announcing Arti, a pure-Rust Tor implementation (Tor blog)

ncm — Thu, 15 Jul 2021 05:41:39 +0000

Yet, you can. The propaganda machine loves an underdog, but the great bulk of real work, by several orders of magnitude, is still done in C++, and will be for a long time to come.

C++ can express abstractions that Rust (still) cannot. But both languages are evolving rapidly.

Ownership and lifetimes

mrugiero — Thu, 15 Jul 2021 02:16:47 +0000

I find your post very educational, however, it doesn't address my point: all the problems in making signed integer overflow defined behavior could have been fixed by making it implementation defined rather than undefined. Implementation defined means it's still hard to write *portable* code, but that when you do know where it'll run, you know there exists one behavior that is appropriate. That is, if it's implementation defined the compiler you use needs to define a given behavior for the platform it'll run and never violate it. So, gcc would be mandated to tell you what value will result from overflowing, even if such a value is one for SPARC and a different one for x86. I fail to see why it UB made more sense to them than that at the time.

But yeah, talking about artificial puzzles when you have all those kinds of behavior and you don't even mandate warnings when you hit them is a bit hypocritical.

Ownership and lifetimes

mrugiero — Thu, 15 Jul 2021 02:10:08 +0000

Stupidly enough, I didn't think of references. It shows how much time passed since the last time I wrote C++ :)