Coverage for important strings

Posted Oct 29, 2024 20:46 UTC (Tue) by tialaramex (subscriber, #21167)
In reply to: Coverage for important strings by iabervon
Parent article: A new approach to validating test suites

I'm pretty sure that my tests would not catch a significant proportion of such changes, and that maybe half of those uncaught changes are significant (the others being cases where the string didn't matter, for example it's fine if my panic spells the word "available" wrong)

On the other hand, I always check what I'm about to commit, because I know I will read the changes sometimes, and if they're full of stray garbage that's a problem. Likewise if I turn out to have changed more than I realised, my log message should reflect the actual work, not just the last thing I did before I typed commit. So if I made a correct change but also mangled a string, that string is likely to stick out when examining the git diff.

Coverage for important strings

Posted Oct 29, 2024 21:27 UTC (Tue) by iabervon (subscriber, #722) [Link] (3 responses)

I've hit a key with the cursor in the wrong window while writing my commit message, so I look at the commits after making them, before pushing them anywhere. I'd say it's likely to get my attention, too, but then I worry about just how unlikely I am to miss it, particularly on a day when my typing is subpar.

Coverage for important strings

Posted Oct 30, 2024 12:15 UTC (Wed) by epa (subscriber, #39769) [Link] (2 responses)

That's one thing I would like AI to handle for me: checking the code changes match the commit message. In some cases you don't even need AI; if the commit message is "Whitespace." then the diff should be empty modulo whitespace, and for "Comment." the old and new should be identical after stripping comments.

Coverage for important strings

Posted Oct 31, 2024 9:50 UTC (Thu) by taladar (subscriber, #68407) [Link] (1 responses)

Why would you use AI (presumably a LLM) for something that a few lines of shell script in a pre-commit hook can do for you?

Coverage for important strings

Posted Oct 31, 2024 13:08 UTC (Thu) by epa (subscriber, #39769) [Link]

"Whitespace" and "Comment" yes a shell script can check. More complex but still trivial changes like "Renamed variable" or "Moved code into new subroutine" can be checked automatically but only with some knowledge of the language's syntax and semantics. At this point, rather than writing a parser for each language you might reach for an 80% solution by asking a LLM to work out roughly what's going on and whether there were any accidental code changes. If I had the option to get my commits and commit messages reviewed by AI, I'd certainly try it, although I am reluctant to have the AI write the code itself.

Coverage for important strings

Posted Oct 29, 2024 23:30 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

"Every developer should be as rigorous as me with such things." Alas, a team of size one does not typically scale so well.

I'd be interested to also drive mutation two ways: CI first smoke tests those that have been discovered to be sensitive to the code in question (you still want the full test suite eventually). Also, run the tests under mutation of the patched code to find issues. I think it's a tool in the toolbox and, with some plumbing between git and CI, could be very powerful to help find issues before maintainer time is spent looking at diffs.

Coverage for important strings

Posted Oct 31, 2024 17:00 UTC (Thu) by mbp (subscriber, #2737) [Link]

There are some features for running under CI, including focusing on the changed code: https://mutants.rs/ci.html

It does not yet comment on your PRs, but I'd like to add that.

Coverage for important strings

Posted Oct 30, 2024 8:56 UTC (Wed) by taladar (subscriber, #68407) [Link]

> for example it's fine if my panic spells the word "available" wrong

There is a tool called https://crates.io/crates/typos-cli that you can put into your pre-commit hook to avoid committing mistakes like that. The main issue with it is that it only supports English so if you have other languages in your source code you might get false positives (but there are ways to add to the list of words accepted or to exclude files).

Coverage for important strings

Posted Oct 31, 2024 23:09 UTC (Thu) by himi (subscriber, #340) [Link]

`git commit -v` is very useful there, though I find it's easier to skip re-checking the diff at that point if it's too long and involved . . . Which has the side-effect of encouraging me to make my commits smaller, so I don't have that niggling sense of discomfort from seeing the diff and going "naah, not today - tl;dr"

I've also found gitui a useful tool for giving me a different view of what I've changed - I rarely end up using it to drive the actual commit process (I use it more often as a friendly way of doing `git add -i`), but it gives a different perspective which can help get past that tl;dr issue.

For the specific case of mutating strings in the code, this seems like a situation where fuzzing would be useful - it'd need to be aware of things like format string mini-languages as well as potentially the distinction between a "plain" string and something like a path (in Python you could target mutations at pathlib.Path objects, for example), but just throwing mutations at any constant/hard-coded string value would be a useful starting point. Though you'd need to run your tests in a sandboxed environment, since the semantics of `subprocess.run(["nm", objfile])` and `subprocess.run(["rm", objfile])` are kind of different . . .