Semi-closing a hole

Let me give some details about my approach. My test suite is dead simple: simply a bash script that runs the Python program against a predefined collection of test documents. For each one it generates the result file and compares (using plain old diff) this result with a pre-generated correct file. No output means that the result is correct. If there is any justified change in the code, I just mv the test file to the correct file. It is fast and works great, but it does not allow for runtime munging of dict keys.

In my program sometimes (not often) elements are output in hash order. If runtime hash generation was random I would have to output elements always sorted, which would penalize performance without any user-visible gains. But I would prefer having to sort everything than changing my test suite. And I would prefer not to change element output order unless it was strictly necessary.

I am not complaining about the change in the Python runtime because it invalidates my testing techniques, but I worry about possible side-effects: someone running the test suite on their machines with PYTHONHASHSEED=random and seeing failures would have a hard time diagnosing these failures. Just one more pain point for Python development.

One of the principles of Test-Driven Development is "do the minimal thing that passes the tests". I do not fully agree with it, but it can be useful sometimes. And especially when writing tests!