I've done this kind of thing twice in the dim past, and would like to find spare time to do it again because I find it quite rewarding. I have a feeling that most people don't feel the same way.
In SANE I fed just slightly unexpected values into the buffer size parameters, finding (as I expected given the mysterious crashes reported) that several backends wrongly assumed they would be asked for buffers that were at least a certain size, or were a multiple of some small integer like 4 even though the specification does not require this.
Last time I looked my test code still lives in the SANE command line tools, hopefully new driver authors are testing their code with it.
For LADSPA I wrote a tool named 'demolition' which sees what happens when legal but extraordinary values are fed into a LADSPA audio plugin, either as parameters or as audio data. It judges which values will be considered extraordinary in part by examining the plugin's built-in metadata. A compliant plugin should at worst run very slowly (and a watchdog timer moves on to the next test in this case) but often they crash or exhibit other undesirable behaviour.