A longstanding GnuTLS certificate validation botch

Posted Mar 11, 2014 15:37 UTC (Tue) by nybble41 (subscriber, #55106)
In reply to: A longstanding GnuTLS certificate validation botch by ms-tg
Parent article: A longstanding GnuTLS certificate validation botch

>> that happened to a C program, but the connection to C
>> as language or culture ends there.

> I wonder if this is true?

In addition to the culture of limited testing you alluded to, I think there are some language issues here as well. C will let you do pretty much anything in a function, which is one of its strong points in certain kinds of code. The flip side to this, however, is that it means you have to _test_ for pretty much anything. You got an enum value... how do you know it's actually one of the defined values, when anyone can pass in whatever integer value they want? You got a pointer... how do you know that it points to valid memory of the correct type? How do you test for the absence of unexpected side-effects? Does the result depend on inputs other than the parameters?

In certain other languages (like Haskell) the type system ensures that thing like out-of-range enums, invalid pointers, and undeclared side-effects simply can't happen unless you go out of your way to bypass the system (e.g. with something like unsafeCoerce or unsafePerformIO, which set off major warning flags). If a function is declared with type "MyEnum -> MyDataStructure -> String" then you only need to test it on valid enum values and data structures; the result is guaranteed to be a well-formed string dependent only on the two parameters, and there won't be any side-effects. This makes testing far simpler even before you consider libraries like QuickCheck.

to post comments

A longstanding GnuTLS certificate validation botch

Posted Mar 11, 2014 20:01 UTC (Tue) by ms-tg (subscriber, #89231) [Link] (7 responses)

> In addition to the culture of limited testing you alluded to,
> I think there are some language issues here as well

Yes, true. But I wonder if discussing type systems is also a
distraction from the more pressing issue here? After all, even
with all the help of Haskell's type system, you *will* still
have bugs.

It seems to me that the lack of rigorous testing was:
(a) The most immediate cause of these bugs
(b) More common in projects written in C

I find it frustrating that discussions of these issues continually
drift towards language wars, rather than towards modern ideas about
unit testing, software composability, test-driven development, and
code coverage tracking.

Aren't these the more pressing questions?
(1) Where are the GnuTLS unit tests, so I can review and add more?
(2) Where is the new regression test covering this bug?
(3) What is the command to run a code coverage tool on the test
suite, so that I can see what coverage is missing?

Say what you will about "toy" languages, but that is what would
happen in any halfway mature Ruby or Python or Javascript project,
and I'm happy to provide links to back that up.

Say what you will about the non-systems languages on the JVM, but
that is also what would happen in any halfway mature Scala, Java,
or Clojure project.

It's only in C, the systems language in which so many of these
vital libraries are written, that this is not the case. Isn't it
time to ask why?

A longstanding GnuTLS certificate validation botch

Posted Mar 11, 2014 21:13 UTC (Tue) by nybble41 (subscriber, #55106) [Link] (6 responses)

> I find it frustrating that discussions of these issues continually
> drift towards language wars, rather than towards modern ideas about
> unit testing, software composability, test-driven development, and
> code coverage tracking.

These "language wars", as you put it, are pretty much all about modern ideas regarding unit testing, composability, test-driven development, and code coverage tracking. Specifically, they're about encouraging the development and use of languages which make such things easier and more reliable, and thus more likely to be implemented.

A longstanding GnuTLS certificate validation botch

Posted Mar 11, 2014 23:47 UTC (Tue) by pizza (subscriber, #46) [Link] (5 responses)

> Specifically, they're about encouraging the development and use of languages which make such things easier and more reliable, and thus more likely to be implemented.

I believe the point nybble41 was attempting to make is that test-driven development (and other such "modern" ideas) is independent of the underlying language used. (Look at GPSd for a case of TDD applied to a primarily-C project)

No amount of "encouragement" will make higher-level languages suitable for system/low-level tasks -- the features (reflection/introspection, dynamic compilation, "package repository") that make those languages more easily testable at the module level make them unsuitable for low-level tasks.

A longstanding GnuTLS certificate validation botch

Posted Mar 12, 2014 3:28 UTC (Wed) by nybble41 (subscriber, #55106) [Link] (1 responses)

> I believe the point nybble41 was attempting to make is that test-driven development (and other such "modern" ideas) is independent of the underlying language used.

I think you meant "ms-tg" rather than "nybble41".

While I agree that these concepts are equally applicable to any language, even C, my point was that the proper choice of language can significantly reduce the testing burden by making stricter guarantees regarding the behavior of the program at compile-time. Speaking as someone who works with system-level C code on FAA-certified embedded systems, C just give you way too much rope to hang yourself with, and it shows in the amount of testing required to obtain full coverage.

As for not being able to do "system/low-level tasks" in high-level languages, I think the authors of House[1] and Kinetic[2] would disagree. While these two OS projects are not written entirely in Haskell, neither is the Linux kernel written entirely in C. Certain core operations require lower-level access to the system, via C and/or assembly, but drivers, network stacks, window systems, and command shells seem "low-level" and "system" enough to me.

[1] http://ogi.altocumulus.org/~hallgren/ICFP2005/house.pdf
[2] http://intoverflow.wordpress.com/kinetic/

A longstanding GnuTLS certificate validation botch

Posted Mar 12, 2014 17:50 UTC (Wed) by ms-tg (subscriber, #89231) [Link]

> While I agree that these concepts are equally applicable
> to any language, even C, my point was that the proper
> choice of language can significantly reduce the testing
> burden

Considering the *vast* quantity of C language code that makes up the modern software stack which was developed outside the culture of unit testing, can't I please persuade you (and others reading, perhaps) that by putting your language suggestions on a separate track, the community might optimize its efforts to introduce test coverage the vast bulk of our linux software?

For example, I suspect that the reason "C culture" seems impervious to adopting the lessons of test-driven development has a lot to do with the masses of developers who are interested in it, by following your advice, are moving to other languages and practicing it there.

In other words, by complecting the issue of unit testing and test coverage with the choice of language, are we not actively *contributing* to the continuing absence of these ideas from C culture, and thus from the bulk of our existing systems?

Food for thought, at least, I hope!

A longstanding GnuTLS certificate validation botch

Posted Mar 12, 2014 18:00 UTC (Wed) by ms-tg (subscriber, #89231) [Link]

> > Specifically, they're about encouraging the development
> > and use of languages which make such things easier and
> > more reliable, and thus more likely to be implemented.
>
> I believe the point nybble41^H^H^H^H ms-tg was attempting
> to make is that test-driven development (and other such
> "modern" ideas) is independent of the underlying language
> used. (Look at GPSd for a case of TDD applied to a
> primarily-C project)

Yes, thank you, exactly!

> No amount of "encouragement" will make higher-level
> languages suitable for system/low-level tasks -- the
> features (reflection/introspection, dynamic compilation,
> "package repository") that make those languages more
> easily testable at the module level make them unsuitable
> for low-level tasks.

This may be true, but I think it's worse than that.

Based on the comments here and elsewhere on the web, it
seems like there's a widespread message that one must
leave the C language in order to adopt modern ideas about
testing and test coverage!

This suggests a self-reinforcing phenomenon where the
majority of the folks interested in learning test-driven
development leave the C language to do it! And therefore,
the bulk of our existing C code remains inadequately
covered by automated tests, and continues to be written
in ways that make coverage difficult to add.

Couldn't one argue that the choice to advocate
for a modern type system (Haskell, ML, Rust, etc), when
the immediate issue is that C code has no test coverage,
is a textbook example of "The Perfect is the Enemy of the
Good"?

(apologies for the length of this rant ;)

A longstanding GnuTLS certificate validation botch

Posted Mar 12, 2014 18:10 UTC (Wed) by ms-tg (subscriber, #89231) [Link] (1 responses)

> (Look at GPSd for a case of TDD applied to a primarily-C project)

I think this is the HEAD version of the build file for GPSD:
http://git.savannah.gnu.org/cgit/gpsd.git/tree/SConstruct

Not for nothing, but:
1. Where is the build target that that runs all the tests?
2. Where is the target that generates coverage stats?
3. Is there a link to where Travis CI, or another automated
system, is running the tests on each commit?

Perhaps I am confused, but it doesn't appear to me that even GPSD is doing these things. Please help me out if I've missed it!

A longstanding GnuTLS certificate validation botch

Posted Mar 14, 2014 22:03 UTC (Fri) by jkt_ (guest, #90352) [Link]

Travis-CI doesn't check every commit, but every push - at least their zero-cost version tied into GitHub.