LibreSSL languishes on Linux

Posted Jan 5, 2021 10:02 UTC (Tue) by marcH (subscriber, #57642)
Parent article: LibreSSL languishes on Linux

> In particular, he complained about ... and the constant stream of regressions that the project must deal with.

Can't all these projects share some common test suite? It's not like they require mocking up hardware or complex dependencies, do they?

LibreSSL languishes on Linux

Posted Jan 5, 2021 13:28 UTC (Tue) by idra (guest, #36289) [Link] (4 responses)

It's not easy, but you can help with https://github.com/tlsfuzzer/tlsfuzzer to improve protocol coverage for any of these libraries.

LibreSSL languishes on Linux

Posted Jan 5, 2021 18:39 UTC (Tue) by marcH (subscriber, #57642) [Link] (3 responses)

Fuzzing is great but "regressions" hinted at more basic, functional testing at the API level.

Could you summarize why a test suite portable across all these implementations is not easy? I naively believe they implement the same API.

LibreSSL languishes on Linux

Posted Jan 6, 2021 10:24 UTC (Wed) by hkario (subscriber, #94864) [Link] (2 responses)

from the project's repo: "SSL and TLS protocol test suite and fuzzer"

it works both as a test suite and a fuzzer, with existing test cases placing far more emphasis on the test suite and compliance parts than on fuzzing part. In other words, it's can be a TLS fuzzer, some test scenarios behave like a fuzzer, but it's not only a TLS fuzzer.

Full disclosure: I'm the primary developer of tlsfuzzer

> Could you summarize why a test suite portable across all these implementations is not easy? I naively believe they implement the same API.

they don't

while few libraries implement OpenSSL API, it's not complete reimplementation, usually limited only to a subset of calls necessary to perform TLS connections, nothing more. Also it's only some of the libraries. For example NSS (library used by Firefox) doesn't do it, BoringSSL is also API incompatible now with OpenSSL.

so, the libraries in question implement the same protocol, and need to interoperate with each-other, but that doesn't mean they implement the same API for applications to use them

LibreSSL languishes on Linux

Posted Jan 6, 2021 23:20 UTC (Wed) by marcH (subscriber, #57642) [Link] (1 responses)

> from the project's repo: "SSL and TLS protocol test suite and fuzzer". it works both as a test suite and a fuzzer,

My assumption that it was just a fuzzer was based on the README there; you may want to rephrase it a bit. I missed the title in the top-right corner and scroll straight down to the README which starts like this:

> Fuzzer and test suite for TLS (SSLv2, SSLv3, v1.0, v1.1, v1.2, v1.3) implementations.

... which I misread as "Fuzzer and [fuzzing] test suite for...".

The name "tlsfuzzer" does obviously not help either.

> they don't

Thanks for all the info!

LibreSSL languishes on Linux

Posted Jan 7, 2021 15:12 UTC (Thu) by hkario (subscriber, #94864) [Link]

> you may want to rephrase it a bit.

I did, please check if it's clear now.

> The name "tlsfuzzer" does obviously not help either.

there are 2 things that are hard in computer science: cache invalidation, naming things and off by one errors ;)

on more serious note: you can use it as a simple, dumb, fuzzer, but all included scripts (with exception of like 1 or 2) don't; they expect very well defined behaviour from the server. Recently we're extended test coverage to testing even the timing of server responses.

also, it's more about the future scope of the project; I want to make it actually mutate the existing scripts to create new test scenarios randomly

so it's a more like "TLS property based tester" but that doesn't exactly roll off the tongue

LibreSSL languishes on Linux

Posted Jan 6, 2021 13:04 UTC (Wed) by ametlwn (subscriber, #10544) [Link]

The OpenSSL API is evolving, staying compatible with LibreSSL requires either staying with the older functions or supporing (and testing) both, complicating the code.
See e.g. https://lists.exim.org/lurker/message/20200706.050711.4b8...