Fixing tests

Posted Apr 12, 2012 12:45 UTC (Thu) by dskoll (subscriber, #1630)
In reply to: Semi-closing a hole by man_ls
Parent article: Python 2.6.8, 2.7.3, 3.1.5, and 3.2.3 security release

However, making the hashing order random would make it nearly impossible to keep static test files and my whole scheme would not so much fly but plummet.

Surely your test code should not depend on the order of elements? The Perl module Test::Deep has routines for comparing hashes (equivalent of Python's dicts) that don't depend on the order of elements.

to post comments

Fixing tests

Posted Apr 12, 2012 14:56 UTC (Thu) by man_ls (guest, #15091) [Link] (10 responses)

I am not comparing dicts, but files. My package generates files so I compare those to canonical versions. You can check it out for yourself, it is called eLyXer. Perhaps it is not a canonical technique but it works well enough: less than a minute on a netbook for 48 rather sophisticated test files. Hash order has given me certain trouble in the past, but for a different task: parsing a config file and generating a Python preferences file. I am not even sure if my tests would run correctly or not, but I am not eager to try it.

And it seems that I am not alone: in the bug discussion people discuss about the change breaking a lot of the Python unit tests and their doctests. This kind of work is not hard to solve but a bit obnoxious; I would myself prefer to order all items instead of performing deep comparisons. With my testing technique of comparing result files I would have no choice anyway.

Fixing tests

Posted Apr 12, 2012 16:02 UTC (Thu) by dskoll (subscriber, #1630) [Link] (9 responses)

My package generates files so I compare those to canonical versions.

I see. Wouldn't it make sense, then, to modify the code that generates the files to sort the keys first so the generated files are always in a known order? This would cause some performance hit, I guess.

Fixing tests

Posted Apr 12, 2012 19:07 UTC (Thu) by man_ls (guest, #15091) [Link] (8 responses)

Yes, that would be a good solution. The performance hit would be suffered by everyone but nice, ordered, predictable output is always a valuable thing, even when outputting a config file. Sigh. Better to bite the bullet and avoid random hash order whenever possible!

On this subject, I have been very surprised by javascript because it seems to output dict keys in alphabetical order. I don't know if it's only the Gecko and WebKit engines or it's a feature of the language though.

Fixing tests

Posted Apr 12, 2012 19:57 UTC (Thu) by dskoll (subscriber, #1630) [Link] (6 responses)

I believe it's an implementation artifact. I don't think JavaScript hashes guarantee any kind of order.

PHP (yuck) arrays do seem to guarantee an order. PHP is completely broken and confusingly combines hashes and arrays into the same data structure. Internally, it keeps extra pointers around so you can pull out elements in the same order you put them in. So a PHP hash bucket costs at least two pointers worth of space more than it should, not to mention the constant time overhead of maintaining the two extra pointers.

Fixing tests

Posted Apr 12, 2012 20:22 UTC (Thu) by man_ls (guest, #15091) [Link] (5 responses)

My mistake, Gecko and WebKit return keys in insertion order, not alphabetical. And it is not javascript arrays but objects; as you probably know, Javascript Does Not Support Associative Arrays

By the way, PHP also returns keys in insertion order, this time with real associative arrays. It is a nice feature IMHO; it may have some overhead but the results are much more intuitive than with Python's dicts. In a certain sense these other languages are hiding the details of the implementation, so that e.g. primitive testing schemes do not rely on the particular hashing function used.

Fixing tests

Posted Apr 12, 2012 20:41 UTC (Thu) by spaetz (guest, #32870) [Link] (2 responses)

Why dont you use OrderedDicts from python's collections module, if that is what you want?

Fixing tests

Posted Apr 12, 2012 20:50 UTC (Thu) by man_ls (guest, #15091) [Link] (1 responses)

Why, OrderedDicts didn't exist when I began coding eLyXer (and Python in general). Even if they had, they are new in Python 3.1 and I am still using Python 2. And I wanted to maintain compatibility back to Python 2.4 (Debian oldstable at the time), now back to Python 2.5.

Also, I did not know that I might eventually need insertion order at the time. I am guessing that also pure inertia might have played a role, although if OrderedDicts keep the same interface (and it appears that they do) that would make the switch quite painless. So I have at least three good reasons and two good excuses :)

Fixing tests

Posted Apr 13, 2012 6:54 UTC (Fri) by cathectic (guest, #40543) [Link]

There is a backport of it for older Python releases all the way back to 2.4:

http://pypi.python.org/pypi/ordereddict

Fixing tests

Posted Apr 13, 2012 13:10 UTC (Fri) by dskoll (subscriber, #1630) [Link] (1 responses)

By the way, PHP also returns keys in insertion order, this time with real associative arrays. It is a nice feature IMHO

I don't like the fact that PHP treats arrays and hashes as the same thing. I would prefer a different syntax for a numerically-indexed array than for an associative array. I would also rather that hashes don't return keys in insertion order by default. PHP should provide a separate data type similar to Python's OrderedDict for those times you want insertion order respected and are willing to pay the overhead.

Fixing tests

Posted Apr 13, 2012 14:17 UTC (Fri) by cortana (subscriber, #24596) [Link]

Let's not get ahead of ourselves. I'd much prefer that PHP provide basic things like a working < operator before moving on to such advanced features!

Fixing tests

Posted Apr 24, 2012 23:43 UTC (Tue) by steffen780 (guest, #68142) [Link]

The performance hit doesn't have to suffered by everyone. Either sort the hashes after the output is completed by using e.g. the sort utility, or add a test flag that then causes the hashes to be sorted before output. The latter will probably cause a little slow down due to the extra variable, parsing an extra CLI parameter and the necessary if - but the hit will be entirely negligible I should think.