Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

[Posted May 4, 2021 by corbet]

The Red Hat Developer Blog has posted an introduction to the rr debugger. "rr records trace information about the execution of an application. This information allows you to repeatedly replay a particular recording of a failure and examine it in the GNU Debugger (GDB) to better investigate the cause. In addition to replaying the trace, rr lets you run the program in reverse, in essence allowing you 'rewind the tape' to see what happened earlier in the execution of the program."

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 15:59 UTC (Tue) by SLi (subscriber, #53131) [Link] (15 responses)

This is intruguing (though it's not the first time I hear of rr). I wonder how unmanageably large the traces become? I currently run KiCad *always* under GDB, because I usually hit something. If the overhead is not much worse than GDB and the traces of sessions less than terabytes, this might be an option that allows me to reproduce transient problems...

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 17:37 UTC (Tue) by tux3 (subscriber, #101245) [Link] (8 responses)

The traces are manageable, as long as you try to keep it to a few minutes max. It's not fast by any means (has to run single threaded), but it's not valgrind levels of slow. It's made to be usable even for browsers (which should give a good workout to any tool ptracing them).

AAIU rr takes a snapshot every second or so, so that reverse-continuing can rust fast. So that _would_ be a lot of data, but they compress it and you gain back about an order of magnitude.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 17:44 UTC (Tue) by SLi (subscriber, #53131) [Link] (2 responses)

Ok. Probably a bit unmanageable for me then data amount wise (GDB already slows things down; I guess rr might not be that much worse?); I tend to work with KiCad for hours.

I tried running KiCad under it. It at least started fast, but died almost immediately, apparently because it uses shared memory, which is IIUC fundamentally incompatible with rr:

```
$ rr record kicad spkr_interface.kicad_pro
rr: Saving execution to trace directory `/home/sliedes/.local/share/rr/kicad-0'.
shared memfd open() failed: Function not implemented
[... some kicad messages]
Trace/breakpoint trap
```

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 23:38 UTC (Tue) by roc (subscriber, #30627) [Link] (1 responses)

Can you file an rr issue and paste the entire console log? It might not be related to the "shared memfd open" message.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 5, 2021 13:05 UTC (Wed) by SLi (subscriber, #53131) [Link]

Done: https://github.com/rr-debugger/rr/issues/2849

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 18:27 UTC (Tue) by MattBBaker (guest, #28651) [Link] (2 responses)

How much detail does RR retain? Like, if I step back 10,000 instructions then what will I see from 'info registers'?

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 19:44 UTC (Tue) by bjacob (guest, #58566) [Link]

rr should show you the exact correct values in `info registers` at any time, yet that does not require storing that much information. The basic idea of rr is that most instructions in a program are deterministic, the only non-determinism is localized at relatively few points in the program's execution (typically: I/O). So in order to allow you to rewind back to any point and show the correct values in `info registers`, rr only needs to state from those relatively few non-determinism sources, rewind back to there, then re-play forward from there.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 23:50 UTC (Tue) by roc (subscriber, #30627) [Link]

rr replay reproduces the exact memory and register contents* of the recording at every point during program execution.

* There are some very small exceptions in our accelerated syscall interception machinery that are irrelevant in practice.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 23:39 UTC (Tue) by roc (subscriber, #30627) [Link] (1 responses)

FWIW rr recording does not take snapshots. rr *replay* with gdb does, but we only keep around a few at a time and we use fork() so they're mostly copy-on-write, so their memory overhead is never an issue in practice.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 5, 2021 11:58 UTC (Wed) by tux3 (subscriber, #101245) [Link]

Thank you for the clarification!

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 20:13 UTC (Tue) by acolin (guest, #61859) [Link] (2 responses)

> I currently run KiCad *always* under GDB, because I usually hit something.

I must be an lite user then -- never hit a single crash over months of laying out and schematic editing in KiCad. Though, that was all on macOS with some Windows, none of that work happened on Linux. Incidentally, with Altium, grew accustomed to a couple deterministic crashes. Maybe I just remember Altium crashes more because of the $$$ that were paid for it.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 20:20 UTC (Tue) by SLi (subscriber, #53131) [Link] (1 responses)

Oh, it works relatively fine. It's not even KiCad (and now that I realize I implied that, I'm sorry). It's just that I seem to have an IT breaking aura. Also works on, for some inexplicable reason, bicycles. In theory I don't believe in things like that. In practice, it's so obvious...

Well, and a large part of it is because I moved it to the nightlies (5.99), though I did generate lots of bug reports when I used the stable, too. At this late point in the release cycle, and given how fast the amazing team fixes bugs, I'd say that's been a good deal even if I didn't want to put my talent into use by testing it :-)

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 20:22 UTC (Tue) by SLi (subscriber, #53131) [Link]

And by "nightlies", I of course mean locally built debug version.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 4, 2021 23:41 UTC (Tue) by roc (subscriber, #30627) [Link]

Yeah, rr is generally too high overhead to just run an application under it all the time, for hours at a time ... unfortunately.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 5, 2021 12:37 UTC (Wed) by daenzer (subscriber, #7050) [Link]

If you're not familiar with coredumpctl yet, I recommend checking it out. These days I mostly rely on "coredumpctl gdb" instead of running stuff that might crash in GDB.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 6, 2021 13:35 UTC (Thu) by gabrielesvelto (guest, #83884) [Link]

rr was originally built to debug Firefox. A Firefox trace takes several gigabytes of storage per minute of execution. That can be considered pretty much the worst case given Firefox complexity and size; most other applications will generate significantly smaller traces.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 5, 2021 14:15 UTC (Wed) by mcatanzaro (subscriber, #93033) [Link] (2 responses)

Unfortunately it looks like it still only works on Intel processors. :(

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 5, 2021 14:36 UTC (Wed) by niner (subscriber, #26151) [Link]

That's not true. I'm using it on a Ryzen 5950X and it's of tremendous help.
https://github.com/rr-debugger/rr/wiki/Zen for details on running it on AMD Zen. Personally, I'm using the kernel module.

Instant replay: Debugging C and C++ programs with rr (Red Hat Developer)

Posted May 5, 2021 22:17 UTC (Wed) by roc (subscriber, #30627) [Link]

As the commenter above says, it works on Zen. There is also experimental Aarch64 support (for userland that uses CAS instead of LDREX/STREX).