Coverage for important strings

Posted Oct 29, 2024 19:37 UTC (Tue) by mathstuf (subscriber, #69389)
In reply to: Coverage for important strings by iabervon
Parent article: A new approach to validating test suites

I've started running `cargo-mutants` on some of my crates and this thought crossed my mind. However, some strings just…don't matter. They might be the base name of a temporary file that just needs to agree in every location. In fact, if something like that *does* trip the test suite, it feels like a phantom invisible dependency that I'd rather move out to a single constant to share among the important places. But using `cargo mutants` to *find* these strings would be nice.

Other strings should certainly *not* be permuted into some test string though (e.g., changing `cp` to `rm` in something doing commands sounds…very risky). Alas, even in Rust, strings end up in many places (usually wrapped in some type-scented decoration for type checking during compilation), but mutation of them is probably something needing typed string literals of some kind.

Coverage for important strings

Posted Oct 31, 2024 16:48 UTC (Thu) by mbp (subscriber, #2737) [Link] (1 responses)

It doesn't mutate strings at the moment, but it's an interesting idea. At least some of them are probably important, although probably many . Perhaps byte strings or C strings are more likely to be critical that they stay exactly as they are.

> Other strings should certainly *not* be permuted into some test string though (e.g., changing `cp` to `rm` in something doing commands sounds…very risky).

Mutating the program is just unavoidably, intentionally, going to cause new behaviors. If you have a function that, for example, takes a path to delete and with "" deletes your home directory, then cargo mutants might cause that to execute, but so might any other bug. For programs containing code that might cause side effects beyond the program's own execution you probably want to run the tests in a container or VM: https://mutants.rs/limitations.html?highlight=delete#caut...

Coverage for important strings

Posted Nov 4, 2024 13:16 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

I think some way to mark "these strings are content" versus "these strings are control" would be useful. That way things like comment format strings could be mutated but things like `["git", "apply", …]` command arrays would not be. But I'm not sure how to do that without sprinkling attributes everywhere (and wanting expression attributes to boot).