LWN: Comments on "A longstanding GnuTLS certificate validation botch"

RAII in C

psevon — Wed, 09 Apr 2014 13:02:04 +0000

The problem with cleanup is that it doesn't allow for passing the allocated object to an outer scope and stil be subject to automatic cleanup, which is one of the main strengths of smartpointers in C++. It also doesn't work nicely together with setjmp/longjmp based exception mechanisms, since the intermediate scope (i.e., functions between the function doing setjmp and function doing longjmp in the call stack) cleanups will never be triggered. See my project https://github.com/psevon/exceptions-and-raii-in-c for an implementation of smartpointers and exceptions in C that avoids the above mentioned problems.

A longstanding GnuTLS certificate validation botch

Cyberax — Tue, 25 Mar 2014 18:50:42 +0000

After struggling with packaging application on Linux for multiple platforms I've learned to like Windows. It keeps developers honest - there is no single global prefix for libraries, so each dependency has to be referred specifically from the build files. So assembling all the dependencies in one installer package is pretty much trivial.

On Linux it's superficially easy to just install all the dependencies using apt-get or similar package managers. Everything will even work on developer's machine. But then comes the packaging time and extracting all these dependencies (even accidental ones) becomes a task which can easily result in a non-working package if you forget something.

And that's even without going into the !@#&!&*!^@ glibc versioning that forces me to use an ancient RHEL image to get a 'universal' binary.

A longstanding GnuTLS certificate validation botch

khim — Tue, 25 Mar 2014 17:21:47 +0000

Unfortunately that's not true. SONAMEs and symbol versioning could fix the “DLL hell” in theory, but they are not used consistently (libraries are not bumping SONAMEs when they change API and/or they don't use symbol versioning consistently thus you can not use different versions in the same process). In the end situation for third party developers is better in Windows: at least there you can use boost both in application and it's plugin even if versions of boost are different. On Linux… it does not really work.

A longstanding GnuTLS certificate validation botch

nix — Tue, 25 Mar 2014 15:50:18 +0000

Uh, drag, Linux's DLL hell was largely (though not entirely) obviated by SONAMEs and symbol versioning. You seem to have become so obsessed with the claim that distributions are evil that you're allowing it to twist your view of the world into zones best described as mendacious reasoning.

A longstanding GnuTLS certificate validation botch

jwakely — Mon, 17 Mar 2014 12:23:28 +0000

Second paragraph of the article:
https://bugzilla.redhat.com/attachment.cgi?id=867911&...

A longstanding GnuTLS certificate validation botch

DavidMoffatt — Mon, 17 Mar 2014 11:36:16 +0000

I don't get it! "First, ensuring that check_if_ca() returned zero" makes sense but why " != 1 rather than == 0"? When is a negative number == 0?
Did they do some kind of overload of "=="?

Could be get a repo URL and change # so we could see the diff?

A longstanding GnuTLS certificate validation botch

jkt_ — Fri, 14 Mar 2014 22:03:25 +0000

Travis-CI doesn't check every commit, but every push - at least their zero-cost version tied into GitHub.

A longstanding GnuTLS certificate validation botch

ms-tg — Wed, 12 Mar 2014 18:10:10 +0000

> (Look at GPSd for a case of TDD applied to a primarily-C project)

I think this is the HEAD version of the build file for GPSD:
http://git.savannah.gnu.org/cgit/gpsd.git/tree/SConstruct

Not for nothing, but:
1. Where is the build target that that runs all the tests?
2. Where is the target that generates coverage stats?
3. Is there a link to where Travis CI, or another automated
system, is running the tests on each commit?

Perhaps I am confused, but it doesn't appear to me that even GPSD is doing these things. Please help me out if I've missed it!

A longstanding GnuTLS certificate validation botch

ms-tg — Wed, 12 Mar 2014 18:00:58 +0000

> > Specifically, they're about encouraging the development
> > and use of languages which make such things easier and
> > more reliable, and thus more likely to be implemented.
>
> I believe the point nybble41^H^H^H^H ms-tg was attempting
> to make is that test-driven development (and other such
> "modern" ideas) is independent of the underlying language
> used. (Look at GPSd for a case of TDD applied to a
> primarily-C project)

Yes, thank you, exactly!

> No amount of "encouragement" will make higher-level
> languages suitable for system/low-level tasks -- the
> features (reflection/introspection, dynamic compilation,
> "package repository") that make those languages more
> easily testable at the module level make them unsuitable
> for low-level tasks.

This may be true, but I think it's worse than that.

Based on the comments here and elsewhere on the web, it
seems like there's a widespread message that one must
leave the C language in order to adopt modern ideas about
testing and test coverage!

This suggests a self-reinforcing phenomenon where the
majority of the folks interested in learning test-driven
development leave the C language to do it! And therefore,
the bulk of our existing C code remains inadequately
covered by automated tests, and continues to be written
in ways that make coverage difficult to add.

Couldn't one argue that the choice to advocate
for a modern type system (Haskell, ML, Rust, etc), when
the immediate issue is that C code has no test coverage,
is a textbook example of "The Perfect is the Enemy of the
Good"?

(apologies for the length of this rant ;)

A longstanding GnuTLS certificate validation botch

ms-tg — Wed, 12 Mar 2014 17:50:17 +0000

> While I agree that these concepts are equally applicable
> to any language, even C, my point was that the proper
> choice of language can significantly reduce the testing
> burden

Considering the *vast* quantity of C language code that makes up the modern software stack which was developed outside the culture of unit testing, can't I please persuade you (and others reading, perhaps) that by putting your language suggestions on a separate track, the community might optimize its efforts to introduce test coverage the vast bulk of our linux software?

For example, I suspect that the reason "C culture" seems impervious to adopting the lessons of test-driven development has a lot to do with the masses of developers who are interested in it, by following your advice, are moving to other languages and practicing it there.

In other words, by complecting the issue of unit testing and test coverage with the choice of language, are we not actively *contributing* to the continuing absence of these ideas from C culture, and thus from the bulk of our existing systems?

Food for thought, at least, I hope!

A longstanding GnuTLS certificate validation botch

nybble41 — Wed, 12 Mar 2014 03:28:13 +0000

> I believe the point nybble41 was attempting to make is that test-driven development (and other such "modern" ideas) is independent of the underlying language used.

I think you meant "ms-tg" rather than "nybble41".

While I agree that these concepts are equally applicable to any language, even C, my point was that the proper choice of language can significantly reduce the testing burden by making stricter guarantees regarding the behavior of the program at compile-time. Speaking as someone who works with system-level C code on FAA-certified embedded systems, C just give you way too much rope to hang yourself with, and it shows in the amount of testing required to obtain full coverage.

As for not being able to do "system/low-level tasks" in high-level languages, I think the authors of House[1] and Kinetic[2] would disagree. While these two OS projects are not written entirely in Haskell, neither is the Linux kernel written entirely in C. Certain core operations require lower-level access to the system, via C and/or assembly, but drivers, network stacks, window systems, and command shells seem "low-level" and "system" enough to me.

[1] http://ogi.altocumulus.org/~hallgren/ICFP2005/house.pdf
[2] http://intoverflow.wordpress.com/kinetic/

A longstanding GnuTLS certificate validation botch

pizza — Tue, 11 Mar 2014 23:47:40 +0000

> Specifically, they're about encouraging the development and use of languages which make such things easier and more reliable, and thus more likely to be implemented.

I believe the point nybble41 was attempting to make is that test-driven development (and other such "modern" ideas) is independent of the underlying language used. (Look at GPSd for a case of TDD applied to a primarily-C project)

No amount of "encouragement" will make higher-level languages suitable for system/low-level tasks -- the features (reflection/introspection, dynamic compilation, "package repository") that make those languages more easily testable at the module level make them unsuitable for low-level tasks.

A longstanding GnuTLS certificate validation botch

nybble41 — Tue, 11 Mar 2014 21:13:59 +0000

> I find it frustrating that discussions of these issues continually
> drift towards language wars, rather than towards modern ideas about
> unit testing, software composability, test-driven development, and
> code coverage tracking.

These "language wars", as you put it, are pretty much all about modern ideas regarding unit testing, composability, test-driven development, and code coverage tracking. Specifically, they're about encouraging the development and use of languages which make such things easier and more reliable, and thus more likely to be implemented.

A longstanding GnuTLS certificate validation botch

ms-tg — Tue, 11 Mar 2014 20:01:17 +0000

> In addition to the culture of limited testing you alluded to,
> I think there are some language issues here as well

Yes, true. But I wonder if discussing type systems is also a
distraction from the more pressing issue here? After all, even
with all the help of Haskell's type system, you *will* still
have bugs.

It seems to me that the lack of rigorous testing was:
(a) The most immediate cause of these bugs
(b) More common in projects written in C

I find it frustrating that discussions of these issues continually
drift towards language wars, rather than towards modern ideas about
unit testing, software composability, test-driven development, and
code coverage tracking.

Aren't these the more pressing questions?
(1) Where are the GnuTLS unit tests, so I can review and add more?
(2) Where is the new regression test covering this bug?
(3) What is the command to run a code coverage tool on the test
suite, so that I can see what coverage is missing?

Say what you will about "toy" languages, but that is what would
happen in any halfway mature Ruby or Python or Javascript project,
and I'm happy to provide links to back that up.

Say what you will about the non-systems languages on the JVM, but
that is also what would happen in any halfway mature Scala, Java,
or Clojure project.

It's only in C, the systems language in which so many of these
vital libraries are written, that this is not the case. Isn't it
time to ask why?

A longstanding GnuTLS certificate validation botch

nybble41 — Tue, 11 Mar 2014 15:37:48 +0000

>> that happened to a C program, but the connection to C
>> as language or culture ends there.

> I wonder if this is true?

In addition to the culture of limited testing you alluded to, I think there are some language issues here as well. C will let you do pretty much anything in a function, which is one of its strong points in certain kinds of code. The flip side to this, however, is that it means you have to _test_ for pretty much anything. You got an enum value... how do you know it's actually one of the defined values, when anyone can pass in whatever integer value they want? You got a pointer... how do you know that it points to valid memory of the correct type? How do you test for the absence of unexpected side-effects? Does the result depend on inputs other than the parameters?

In certain other languages (like Haskell) the type system ensures that thing like out-of-range enums, invalid pointers, and undeclared side-effects simply can't happen unless you go out of your way to bypass the system (e.g. with something like unsafeCoerce or unsafePerformIO, which set off major warning flags). If a function is declared with type "MyEnum -> MyDataStructure -> String" then you only need to test it on valid enum values and data structures; the result is guaranteed to be a well-formed string dependent only on the two parameters, and there won't be any side-effects. This makes testing far simpler even before you consider libraries like QuickCheck.

A longstanding GnuTLS certificate validation botch

ms-tg — Tue, 11 Mar 2014 13:28:34 +0000

> Both the Apple and GnuTLS bugs were a testing problem

Amen to that. Even for simple web applications, we now
test every known error case, we have integration and unit
level tests, we measure code coverage on each build on
Travis CI... how can this not be done for widely-used
SSL libraries?

> that happened to a C program, but the connection to C
> as language or culture ends there.

I wonder if this is true? Until I see one of these
libraries, out of the hundreds of vital C-language
libraries on my system, approach the level of 100%
code coverage testing that "toy language" cultures
enforce, I am going to have to suspect that this
may actually be a C culture issue.

A longstanding GnuTLS certificate validation botch

zblaxell — Mon, 10 Mar 2014 21:26:47 +0000

There's a lot of amusing language-bashing in here--everything from C++ RAII to toy languages is proposed as a solution, complete with buggy examples in the comments--but this was never a language problem. Both the Apple and GnuTLS bugs were a testing problem that happened to a C program, but the connection to C as language or culture ends there.

There needs to be a set of test inputs in the test suite that exercise every instruction of the code in every meaningfully distinct state so they can all be verified. A basic code coverage test should have caught these bugs easily as soon as someone noticed how hard it was (i.e. impossible) to form a test input capable of touching code on both sides of some of those gotos.

gcov tries to do coverage analysis for C code (although it's a huge pain to use in practice). callgrind tries to do it for arbitrary C and C++ programs (although it has several problems and isn't much easier to use than gcov). Other languages have the instrumentation built in (and the bondage-and-discipline to make sure that every branch is on a separate line of code, so line-based code coverage analysis tools work). A crude but effective ad-hoc coverage tool can even be built out of preprocessor macros with a bit of cunning and discipline.

A longstanding GnuTLS certificate validation botch

zblaxell — Mon, 10 Mar 2014 21:14:44 +0000

I don't see how early bailout is a "C pattern." Touching tainted data from the Internet is risky in every language. We want to stop doing it as soon as we can determine a negative result to reduce our attack surface (unless we are defending against a timing attack). In a language that isn't C we might throw an exception or use some other idiom instead of gotos or cleanup functions, but we'd still stop processing early to avoid exposing further code to attack.

Yes, we can do all sorts of wonderful analysis of invalid certificates if we keep going through all the parsing stages; however, at the end of the day a malformed certificate is still invalid, and needs only to be rejected.

A longstanding GnuTLS certificate validation botch

vonbrand — Mon, 10 Mar 2014 20:44:22 +0000

It might well be dangerous to have e.g. f7() working on data known bad from before, some checks will have to be repeated in such a case (and f7() becomes more complex and fragile). And using this to be able to say that tests covered x% of f7() is nonsense, what is interesting isn't coverage in cases that don't do anything (because it failed before).

A longstanding GnuTLS certificate validation botch

ibukanov — Mon, 10 Mar 2014 19:45:40 +0000

> In the second case, there are fewer branches at the top level, but the invariants, preconditions, and postconditions of the functions are weaker.

Preconditions and postconditions stay the same. What is different is a reaction to a violated precondition. In the first case that follows a typical C pattern the reaction is to bailout. In the second case the reaction is to recover via guessing what would be the right data while tainting the results.

> Imagine fixing six slightly different string-quoting bugs because every function after f3() has to check for conditions that f3() already identified as part of an error state.

There is no error state besides a pointer to error indicator and logging facilities. If f3() is responsible for enforcing the quotation, then in case of errors it should just change the data while tainting the calculation so the data as seen by the rest of code contain the proper quote. This is similar to what some parsers do to report multiple errors when they rewrite lookahead buffer as a part of recovery.

A longstanding GnuTLS certificate validation botch

drag — Mon, 10 Mar 2014 19:05:14 +0000

> Does it mean "DLL hell" is now a problem on Linux and not on Windows?

It's always been a problem on both systems. Microsoft largely solved their issue by taking various approaches to versioning DLLs, preventing applications from overwriting library files, static compiling, heuristics involving locating and using libraries based on what was bundled with the applications and a few other things I only have a distant and vague memory of.

Linux distributions fixed it's own version this by simply making things really difficult for users that try to install anything that isn't carefully compiled, versioned, and controlled by whatever distribution they happen to be using. Then when users run into problems with missing libraries or conflicts they can are called idiots in online forums and such places for not using one of the applications that can be installed by the local package management software... and if that didn't work then the normal advice is to reformat the disk drive and install a different Linux OS that probably had a working version of whatever application they are struggling with.

> That's something to think about, especially if it hinders C++ adoption.

C++ ABI breakage caused myself and many other Linux users a huge number of headaches in the past. Now it's been a long time since I've ran into issue.

However I still consider it a very bad sign whenever I run into a applications or APIs that makes use of 'boost' or any similar binding generation thingy. If it happens to work, great, but if it doesn't then it's going to be a nightmare.

A longstanding GnuTLS certificate validation botch

zblaxell — Mon, 10 Mar 2014 18:59:02 +0000

The second form is better than the first, but the first was bad to start with. C code with gotos is better than the first one.

Even then, the transformation doesn't reduce complexity. It just moves complexity around the program in a way that may make practical risks worse.

In the first case, there are lots of branches in the top-level function, but the preconditions of each function are stronger. This means f10() doesn't have to recheck f1()'s work, and neither do f9(), f8(), etc. A bug in f3() that fails to detect a violation of f4()'s preconditions could ruin our day, but f4() is simpler in this case, and if we get f3() wrong we'll probably get f4() wrong at the same time anyway.

In the second case, there are fewer branches at the top level, but the invariants, preconditions, and postconditions of the functions are weaker. f10() has to cope with everything f1() let through, and so does f9(), f8(), etc. This can breed code duplication and maintenance mistakes, and increases the attack surface considerably. Imagine fixing six slightly different string-quoting bugs because every function after f3() has to check for conditions that f3() already identified as part of an error state. We can avoid this failure mode by having every function check *errors and return early, but then we've just created a variation on the first case that the compiler can't optimize as easily, and that requires wrapper functions around libraries that don't use this convention.

Either one has a linearly expanding set of states inside example(), assuming you don't make a dumb semantic mistake. There's pretty much no difference from a formal coverage testing point of view.

A longstanding GnuTLS certificate validation botch

jezuch — Mon, 10 Mar 2014 08:38:31 +0000

> if (... && (x == SOME_CONSTANT || SOME_OTHER_CONSTANT))

Looks like they were programming in Perl 6 before Perl 6 even existed :)

A longstanding GnuTLS certificate validation botch

nix — Mon, 10 Mar 2014 07:35:46 +0000

... hm, there was gnu-versioned-namespace, in 2005, which is clearly not baked yet because its mangling was changing as recently as 2011.

But yes, you're right, the last major soname change to the non-gnu-versioned-namespace seems to have been in 2003, incorporated into 3.4.0; still years after I thought it was. Fallible human memory etc etc:

2003-01-23 Benjamin Kosnik <bkoz@redhat.com>

* configure.in (libtool_VERSION): To 6:0:0.

A longstanding GnuTLS certificate validation botch

vonbrand — Mon, 10 Mar 2014 00:46:22 +0000

In a compiler "bad untainting" leads to an error cascade, a sadly well-known phenomenon. But people just fix the obvious errors and compile again, they will rarely report that as a compiler bug.

A longstanding GnuTLS certificate validation botch

vonbrand — Mon, 10 Mar 2014 00:37:25 +0000

Bailing out early is what cuts down exponential growth. Finding out that after the first sanity test fails the thirtieth does (or doesn't) adds no useful data (state is tainted, i.e., known broken anyway).

A longstanding GnuTLS certificate validation botch

ibukanov — Sun, 09 Mar 2014 22:34:27 +0000

> But if an error means curtains, better bail out early.

in a security-sensitive code it is critical to be able to test all the branches if a formal verification is not feasible. An early bailout not only hinders that via exponential growth of branch space but also subjects the software for timing attacks.

As for accidental untainting, consider that GnuTLS and Apple bugs are exactly the examples of this occurring in the code that follows established practice of error handling in C. Yet I have not heard of bugs caused by untainting errors reported by a parser - typically an infrastructure to support reporting of multiple errors naturally minimizes the number of places in the code where the error state can be cleared. It is just harder to wipe out an error array than to clear or ignore a return value flag.

A longstanding GnuTLS certificate validation botch

vonbrand — Sun, 09 Mar 2014 22:12:27 +0000

If you'd like to find all errors that is right. But if an error means curtains, better bail out early. In security-sensitive code, you'd better make sure there is no way the tainted state gets somehow untainted (by mistake or by nefarious intent).

A longstanding GnuTLS certificate validation botch

ibukanov — Sun, 09 Mar 2014 19:48:20 +0000

> handling cases where an error in the first part allows to do (part of) the rest are a mess.

Consider what a good parser does when it detects a syntax error. First it reports it. Second it tries to recover from it guessing if necessary to allow to report *other* errors during single pass. The end result is that it produces a valid parsed tree reflecting its guesses but that tree is tainted with errors so the code generation would never be performed.

Effectively this replaces all the error checks in all callers in the parser implementation by a single check in the code generator for presence of errors.

A longstanding GnuTLS certificate validation botch

ibukanov — Sun, 09 Mar 2014 16:39:09 +0000

> This complicates the callee for no benefit, it turns out to have to check for errors before doing anything.

In that model the callee has to check for errors only as a debugging tool so the origin of the error and its propagation through the code can be logged. In general the idea is to treat errors similar to NaN that does not affects the data flow but only the result. That minimizes the amount of rarely taken branches potentially allowing to test all code paths.

> how do I know all of the callees are doing their job?

There is no need for that as long as the code that detected the error properly taints the evaluation with error condition.

> handling cases where an error in the first part allows to do (part of) the rest are a mess.

I do not see the source of supposed extra mess. If one can recover from a particular error, the error state can be cleared.

A longstanding GnuTLS certificate validation botch

paulj — Sun, 09 Mar 2014 16:04:12 +0000

I don't know why it isn't used more. I've only really seen it in network protocol code, where the code is implementing an RFC that explicitly specifies a state machine. Which makes it seem the authors didn't think too much about the other applications.

I wonder if maybe many programmers just aren't aware of this approach.

Especially with security sensitive parsers of external input (e.g. network message) I absolutely cringe when I see hand-crafted, irregular parsers, twiddling pointers directly into buffers in C. It's 2014....

I've seen a PhD that extended the Java language with support for specifying the allowed state transitions on objects. This allowed the compiler to do extra checks - sanity checks on the FSM and on users of the code. I think the author has moved on to other things though, I think.

I'd really like to see languages provide more support for more restricted abstractions, like FSMs, that can be layered over the code and be more easily checked for problems than that code.

A longstanding GnuTLS certificate validation botch

vonbrand — Sun, 09 Mar 2014 15:31:02 +0000

This complicates the callee for no benefit, it turns out to have to check for errors before doing anything. The error testing is hidden (how do I know all of the callees are doing their job?), and moreover handling cases where an error in the first part allows to do (part of) the rest are a mess.

Just not calling the other tasks if they make no sense trades a call and a branch inside for a simple branch. But who's worrying about performance in error paths... it does cut down on code size if the tasks are used repeatedly (only the branch inside, not each time it is called upon). Perhaps that was the real reason for this coding pattern?

A longstanding GnuTLS certificate validation botch

ibukanov — Sun, 09 Mar 2014 12:00:59 +0000

The technique described in your blog effectively changes the code pattern:

int example() {
    ...
    int errors = f1();
    if (errors) {
         error_cleanup1();
         return errors;
    }
    int errors = f2();
    if (errors) {
         error_cleanup2();
         return errors;
    }
    int errors = f3();
    if (errors) {
         error_cleanup3();
         return errors;
    }
    ...
    normal_cleanup();
    return 0;
}

into

void example(int *errorp) {
    if (*errorp)
        return;
    ...
    f1(errorp);
    f2(errorp);
    f3(errorp);
    ...
    cleanup();
    return 0;
}

That is, this moves the error check into the callee freeing the caller from doing the repeated error checks. This is possible as long as the error state idempotent and indeed minimizes the number of branches in the code.

This pattern does not require using finite state machines for the whole code. I have read about it in a book published over 30 years ago about embedded systems programming and always wondered why this have never became mainstream.

A longstanding GnuTLS certificate validation botch

HelloWorld — Sun, 09 Mar 2014 07:11:31 +0000

> I know many who love boost and cannot live without it, but IMO it is the worst type of hacks it is a language-level hack to foist C++ to something that it is not.
This is Not Even Wrong. It's so vague it's meaningless and you're talking as if boost were a single library while in reality it's a collection of libraries, some of which are unquestionably great.

> This is besides the fact that one has to make sure the compiler itself knows C++ quirks enough to compile boost correctly!
The solution to broken compilers is to not use them.

A longstanding GnuTLS certificate validation botch

HelloWorld — Sun, 09 Mar 2014 07:00:07 +0000

C++ is an abomination, and yet it's still much better than C.

A longstanding GnuTLS certificate validation botch

mjg59 — Sun, 09 Mar 2014 01:56:59 +0000

Didn't 3.4.0 bump the libstdc++ soname? That was 2004. Debian's last C++ ABI transition was 2005, IIRC - http://lwn.net/Articles/160330/ seems to agree.

A longstanding GnuTLS certificate validation botch

jbailey — Sun, 09 Mar 2014 01:40:52 +0000

That may be the most graceful way anyone has ever called me old. Thanks. :)

A longstanding GnuTLS certificate validation botch

nix — Sat, 08 Mar 2014 23:26:59 +0000

If by 'a decade ago' you mean 'about 1998', then yes. But that's nearly two decades ago now. :)

A longstanding GnuTLS certificate validation botch

nix — Sat, 08 Mar 2014 23:26:10 +0000

That's still not a lovely syntax, but it's *much* better.

I've been meaning to learn more about ragel. Now I have an excuse :)

A longstanding GnuTLS certificate validation botch

nix — Sat, 08 Mar 2014 23:15:49 +0000

Quite. I'm not saying it's impossible to make it work, obviously it isn't. It's just not at all easy, and it's not obvious when you got it wrong.

I like exceptions, but I'm wary of them in much the same way as I would be of a gun that has a habit of firing spontaneously and exploding when fired. :)