A backdoor in xz

In a similar area, for a long time we tried to make the libctf and ld.ctf testsuites in binutils work the same way the DWARF ones mostly do: describe a binary CTF dict via a .S file and just assemble it. But a mixture of there being no cross-arch-portable way to represent arbitrary binary data in .S files (.long, .byte4 etc, are all arch-dependent) and a desire to always test the latest CTF format except in a few error cases meant that nearly all of the tests end up compiling a .c file with -gctf just to get its CTF, then running the tests on that. It turned out to be much nicer to work with that, and much easier to see what the tests were actually *doing* as well: .S files are a pretty opaque way to describe a test input! (For corrupted CTF tests, the same problem arises, though at only a few dozen bytes each they're hardly likely to be malicious. There's just no room.)

This is mostly what I was thinking of - have a human-readable language designed to specify the binary formats, and use that to generate the test data. It would work for both good and bad test data, though - rather than simply copying "known bad" test files for testing, you'd pull them apart and write a spec for the test case that generated the same kind of bad data. Doing that would actually specify the badness in great detail, far more so than simply having a binary test file that happens to break things; it could also help drive highly targeted fuzzing whenever particular failures were identified.

The biggest problem I see is that for something like a compression format the difference in complexity between a specification of the binary format and it's possible contents and an actual implementation of the compression algorithm probably isn't all that big, and the specification is certainly going to be subject to as many bugs as the code it's supposed to be testing. This is why I'd hesitate to suggest trying to make this sort of thing into any kind of generalised "best practise" - there are probably going to be too many cases where the payoff just isn't worth the effort required.

That said, a compression algorithm and its on-disk format are probably just about the worst case scenario - I expect a lot of software that reads or writes a binary format would have much less difficulty creating this kind of test data spec. And in a more generalised sense the idea of generating test data rather than simply embedding it in the repository is something that's probably worth encouraging, along with tooling that would support it . . .