Fighting the last war [LWN.net]

Fighting the last war

Posted May 21, 2008 16:42 UTC (Wed) by stephen_pollei (guest, #23348)
In reply to: Fighting the last war by tialaramex
Parent article: Open Source Security Report

There is nothing that said that you couldn't do both at the same time. make the time etc
static and test the statistical properties of the output. In fact if you did both you might
find marginal cases which a test of one without the other would have missed.

(Log in to post comments)

Not even fighting the last war

Posted May 22, 2008 11:39 UTC (Thu) by tialaramex (subscriber, #21167) [Link]

So your proposal is to create an unchanging environment in which OpenSSL can run, and then run
it several times, using statistical tests to ensure that the random output is statistically
independent between runs despite holding all of the environment (except /dev/random
presumably) constant. That sounds like quite a serious piece of work, how much development
time do you think it would take to build a robust and portable version of that test ?

You can come up with all sorts of sufficiently arbitrary tests that would so happen to be
tripped by this error but they all incur a maintenance cost and don't seem to really justify
it with a rationale as to what proportion of real world bugs they'll catch other than this one
which we already fixed.

Running MD5 over the released OpenSSL source and having a unit test fail with "Stop messing
with things you don't understand" if you've changed it would also have been an effective way
to detect this bug, but I don't think we're really considering that.