A backdoor in xz

This is mostly what I was thinking of - have a human-readable language designed to specify the binary formats, and use that to generate the test data. It would work for both good and bad test data, though - rather than simply copying "known bad" test files for testing, you'd pull them apart and write a spec for the test case that generated the same kind of bad data. Doing that would actually specify the badness in great detail, far more so than simply having a binary test file that happens to break things; it could also help drive highly targeted fuzzing whenever particular failures were identified.

The biggest problem I see is that for something like a compression format the difference in complexity between a specification of the binary format and it's possible contents and an actual implementation of the compression algorithm probably isn't all that big, and the specification is certainly going to be subject to as many bugs as the code it's supposed to be testing. This is why I'd hesitate to suggest trying to make this sort of thing into any kind of generalised "best practise" - there are probably going to be too many cases where the payoff just isn't worth the effort required.

That said, a compression algorithm and its on-disk format are probably just about the worst case scenario - I expect a lot of software that reads or writes a binary format would have much less difficulty creating this kind of test data spec. And in a more generalised sense the idea of generating test data rather than simply embedding it in the repository is something that's probably worth encouraging, along with tooling that would support it . . .