LWN: Comments on "Fully automated bisecting with "git bisect run""

Fully automated bisecting with "git bisect run"

bfields — Wed, 03 Oct 2018 14:27:32 +0000

Seems to me that a lot of the dumb build bugs I've introduced over the years have required a particular combination of config options.

This is a case where I should do more testing--a lot of those bugs would probably be caught by just a few extra builds that exercise some of the most commonly problematic combinations.

But I'll never catch all of those just by doing enough builds. I mean, the kernel has thousands of build options. Combinatorics is against us.

Fully automated bisecting with "git bisect run"

bfields — Wed, 03 Oct 2018 14:08:52 +0000

Completely agreed on the value of regression testing. We could always use more. But that will never catch every bug!

Take any random bug that's been discussed on lwn, say, I don't know, this mutex_unlock/free race or the famous e1000e bug. Without knowing about the bug ahead of time, what kind of testing would be guaranteed to catch one of those in a reasonable amount of time?

It's not impossible, but it's hard, and something will always slip through, absent some change more fundamental than just adding more regression tests.

Fully automated bisecting with "git bisect run"

neilbrown — Wed, 03 Oct 2018 07:15:25 +0000

> Why would the kernel group allow code to be committed without running a clean build?

Maybe someone started using a new compiler.
Or maybe I'm doing a git-bisect on my own private tree prior to submitting it. I just noticed a bug and now I want to find out where I introduced it, so I can fix and rebase.

> The point I was getting to here is that with build tests and regression tests, there would be little or no need for bisection, automated or manual.

"Testing shows the presence, not the absence, of bugs". Still true 50 years after Dijksrta wrote it.

Fully automated bisecting with "git bisect run"

samiam95124 — Wed, 03 Oct 2018 01:37:27 +0000

From my experience, the answer is "none of them and all of them". Let me explain.

What VMs (and simulations) allow is the finding of "some bugs" related to hardware. The reason is simple. Hardware generates timing bugs. VMs and simulations do find some of those issues, but they also tend to model hardware in an overly simplified way that will not trip many of the most serious timing bugs. What they bring to the table, however, is important in another way, which is they are generally repeatable (this is not always true by the way). Just like a pseudo-random sequence, they can deliver the same bug over and over again. I think a lot of us have enough experience chasing hardware timing bugs to realize how valuable that is.

Thus VM/Sim has a place, and that is usually first before you enter the "wild and woolly world" of hardware. The nice part is that typically matches hardware design flow, because sim is typically done before the hardware is available, especially in silicon/ic design world.

Fully automated bisecting with "git bisect run"

samiam95124 — Wed, 03 Oct 2018 01:28:34 +0000

My 2 cents:

First, love the fact that this has been automated. I have been doing this procedure automatically for years (decades).

However, this is a bit like closing the barn door after the horse has left. I was a bit shocked to find there is a procedure to find make errors *after they have been committed*??? Why would the kernel group allow code to be committed without running a clean build?

Second, Linux is a pretty mature product these days. Your "clients" the hardware manufacturers, all run (or should all run) extensive regression tests. In fact, I don't think it is a tall order to submit regression tests along with drivers, which is not a requirement at the moment. Driver groups develop extensive regression tests, which Linux never sees, since they aren't upstreamed. So the drivers are open source, but their tests are secret?

Finally, for the extensive kernel code running outside of driver land, this also needs regression testing.

The point I was getting to here is that with build tests and regression tests, there would be little or no need for bisection, automated or manual. The reason build servers are popular now is to prevent make errors from reaching repositories. And it should be no different for functional errors, as well.

Fully automated bisecting with "git bisect run"

christian_couder — Tue, 10 Feb 2009 04:41:38 +0000

The long usage message is:

$ git bisect help
Usage: git bisect [help|start|bad|good|skip|next|reset|visualize|replay|log|run]

git bisect help
        print this long help message.
git bisect start [<bad> [<good>...]] [--] [<pathspec>...]
        reset bisect state and start bisection.
git bisect bad [<rev>]
        mark <rev> a known-bad revision.
git bisect good [<rev>...]
        mark <rev>... known-good revisions.
git bisect skip [(<rev>|<range>)...]
        mark <rev>... untestable revisions.
git bisect next
        find next bisection to test and check it out.
git bisect reset [<branch>]
        finish bisection search and go back to branch.
git bisect visualize
        show bisect status in gitk.
git bisect replay <logfile>
        replay bisection log.
git bisect log
        show bisect log.
git bisect run <cmd>...
        use <cmd>... to automatically bisect.

Please use "git help bisect" to get the full man page.

So you have to use "git bisect run <cmd>..." to automatically run the command you investigate. And if you don't want to automatically run the command, you can test by yourself at each step of the binary search and then use "git bisect good" or "git bisect bad" or "git bisect skip" depending on the result of your tests.

Fully automated bisecting with "git bisect run"

droundy — Mon, 09 Feb 2009 18:19:22 +0000

I'd imagine that very few of the "hard" bugs could be found on a virtual machine, almost by definition. I'd think that a "hard" bug would be one that can't easily be reproduced by a kernel developer.

Fully automated bisecting with "git bisect run"

wfranzini — Sun, 08 Feb 2009 16:06:48 +0000

I noticed only reading the article that 'git bisect' was unable to run the command being investigated.

People interested in a workflow that includes tests can look at Aegis (http://aegis.sf.net/). It also has an aebisect(1) command that run the command being investigated.

Fully automated bisecting with "git bisect run"

oak — Sat, 07 Feb 2009 18:19:39 +0000

In general, how many of the kernel bugs need to be tested on real HW and
how many could be found in a virtual machine?

Bisect on patchset boundaries ?

lbt — Fri, 06 Feb 2009 10:33:11 +0000

The "buggy intermediate versions" problem is one that I would have expected more attention to be drawn to. I've done a few bisects and had my filesystem fail to mount on several cuts during one run.

A bisect that throws you into the middle of a patch set that messes with your filesystem is a dangerous place to be; especially when you ask 'normal' users to run a bisect on their everyday machines without warning them that they are potentially about to expose their data to random collections of code.

The problem is that all patches are supposed to be non-toxic - and git is deliberately not good at revisionist history ;)

That means that marking 'safe' bisect points is hard - but maybe a step in the right direction would be worth cutting on patchset boundaries?

Or maybe an external (well, inside .git/) list of 'good' commits eg rc releases?

Obviously this would make the bisect slightly less efficient but it may reduce the risk.

Other similar tools

epa — Thu, 05 Feb 2009 10:30:28 +0000

'git bisect' is a kind of delta debugging <http://www.st.cs.uni-saarland.de/dd/>: where you have a broken version of the code and a known working version, and you find the change between them that introduces the bug. It's constrained to the version history in the repository, so it will find the guilty commit.

An alternative is to randomly generate in-between files to find what difference causes the change. I believe DD.py <http://www.st.cs.uni-saarland.de/dd/ddusage.php3> is a tool for doing this.

Finally, a plug for delta <http://delta.tigris.org/> which isn't quite the same thing, but will automatically generate a minimal test case given a larger one.

Fully automated bisecting with "git bisect run"

dlang — Thu, 05 Feb 2009 07:44:55 +0000

note that to cause problems the buggy intermediate versions need to be buggy in a way that your test script can't tell from the failure you are looking for.

how big a problem this is depends on what your failure condition is.

if you know that when it fails it generates message X then you just look for message X and mark everything else as 'good' (it may actually crash and not do anything useful, but it's not the bug you are looking for)

if you are looking for a hang (or failure to boot like Ingo did in one example) then it's harder, you may end up going down the wrong path becouse some other bug is causing the problem (failing to boot in this example)

Fully automated bisecting with "git bisect run"

christian_couder — Thu, 05 Feb 2009 05:58:41 +0000

Yes this is the idea.

Exit code 125 tells "git bisect" to use "git bisect skip". "git bisect skip" marks the current commit as untestable and checks out another one nearby to be tested.

See the "git bisect" documentation for more information:

http://www.kernel.org/pub/software/scm/git/docs/git-bisec...

Fully automated bisecting with "git bisect run"

dtlin — Thu, 05 Feb 2009 03:48:25 +0000

The article did mention that exit code 125 signaled "skip this commit" -- if you have your git run script return that for known-toxic versions, that would do the trick. Or so it seems, from reading the article; I've never tried it myself.

Fully automated bisecting with "git bisect run"

ncm — Thu, 05 Feb 2009 02:13:33 +0000

Perhaps I am showing my ignorance here, but doesn't this require that buggy intermediate versions of the code be scrubbed out of the repository, so that the bisection may choose any random intermediate version? Or must known-toxic versions labeled to deflect bisect from choosing them?