Handling variability in test outputs

The testsuite I develop as part of PyLETS also has various hacks to cope with test results generating different files or strings than expected and still being valid results. E.G. auto-generated passwords, session cookies, date/time strings and related numeric values, URLs based on where the program is installed, and a growing list of other things. The approach my suite adopts is to normalise all of these values by using regexes and symbolic substitutions in both the expected stored values and actual values to replace variable expected values with symbolic constants prior to diff comparing actual against expected strings and files.

So I would also tend to regard this kind of breakage caused by an upstream dependancy as being the kind of thing I would expect to have to fix myself anyway in the course of upgrading the testsuite between major Python versions. Certainly wouldn't want the security of my application to depend upon whether I'd read and understood all the release notes of thousands of packages which run on my server.