LWN: Comments on "How to fix an ancient GDB problem"

How to fix an ancient GDB problem

Hi-Angel — Fri, 18 Nov 2022 21:04:28 +0000

> Back when v2 was posted to the list last year, someone tested it with PipeWire specifically, and confirmed it works there.

That was me. The patchset still haven't been merged…? Oh, my :c

Since only 2007?

rmano — Thu, 13 Oct 2022 20:40:35 +0000

This one wins... https://bugzilla.mozilla.org/show_bug.cgi?id=377621 ;-)

How to fix an ancient GDB problem

palves — Tue, 04 Oct 2022 11:52:03 +0000

> When attaching to a running process, you just have to hit Ctrl+C in the terminal running GDB, as before. This mode of
> operation hasn't really suffered from these problems, as far as I understand it.

When you attach to process running in another terminal, and hit Ctrl-C in the terminal running GDB, that Ctrl-C is turned into a SIGINT sent to GDB. So that half of the problem does not exist in that scenario, GDB sees the SIGINT first, not the inferior. However, currently, in "I am attached" scenario, GDB forwards that SIGINT to the target process, using plain "kill(pid, SIGINT)", and then relies on ptrace intercepting that SIGINT. If the target process blocks SIGINT, then you're back to square 1. To get that scenario working properly, we stil need part of the proposal in place, specifically the part about pausing the target process in a different way, with SIGSTOP.

How to fix an ancient GDB problem

palves — Mon, 03 Oct 2022 18:30:05 +0000

> My personal experience with a possibly-related bug: when debugging programs which handle Ctrl+C themselves
> (like PipeWire and Audacious), pressing Ctrl+C will trigger the app's graceful shutdown process
> (terminating execution) rather than breaking into the debugger like I want. Will the changes to gdb fix this issue?

Back when v2 was posted to the list last year, someone tested it with PipeWire specifically, and confirmed it works there.

From https://inbox.sourceware.org/gdb-patches/1c54ccee2e4a2980... :

>> "I was recently fixing some bug in Pipewire. To be exact, I was working on
>> `pipewire-media-session` (one of daemons that Pipewire is comprised of). Process
>> of debugging basically included setting breakpoints somewhere, then hopefully
>> triggering the breakpoint, then inspecting the state. Nothing unusual.
>>
>> So, the thing that was slowing me down and annoying is that I couldn't just
>> pause the debuggee through the usual means of ^C, then add a breakpoint.
>> Pressing ^C was causing the process to exit!
>>
>> Pipewire project seems to have a number of various daemons, and all of them set
>> SIGINT handlers (judging by git-grepping for SIGINT). So basically, it is hard
>> to debug Pipewire with GDB due to GDB not being able to interactively pause the
>> process.
>>
>> Now, I tested the patchset, and I confirm it solves the problem! GDB built from
>> the palves' branch pauses pipewire-media-session correctly on ^C! "

> Also is "a GDB command that allows the user to specify which key should interrupt the process
> and return to the GDB prompt" already present, or will it be added in the rewrite?

It does not exist yet. It will be added in the next version of the patches.

why blocking sigint ?

iabervon — Mon, 03 Oct 2022 18:06:23 +0000

If you're really just going to set a flag, the fact that a blocked signal is pending is effectively a flag that the kernel handles for you without any fuss and without interrupting syscalls. If you install a signal handler, you get to find out how well your program handles getting a ton of short writes or EINTR in a critical section when the user hammers ctrl-c. Your program should handle it well, but that path is probably not being exercised at all under normal operation.

why blocking sigint ?

NYKevin — Fri, 30 Sep 2022 19:52:33 +0000

Signal handlers suck, because they can only call fully reentrant functions. It is often desirable to run code on a "real" stack (i.e. not from the signal handler), at which point you might decide to forego the signal handler altogether, by doing one of the following instead:

* Make a signalfd(2) and add it to your main event loop's select(2)/poll(2)/etc. call.
* Make a dedicated "handle the signals" thread, block all the signals you care about (including SIGINT), and call sigwaitinfo(2) from that thread.

You might also block signals for a short time while the thread or process is holding a mutex, in cases where this could cause deadlock or violate an invariant.

Frankly, there is absolutely nothing in either POSIX or Linux to suggest that blocking signals is "wrong" so long as the process actually responds to them in a valid and appropriate way. SIGINT means "please stop what you're doing," not "please push a weird extra frame on the stack so the debugger can break in."

How to fix an ancient GDB problem

ballombe — Fri, 30 Sep 2022 15:58:02 +0000

> My personal experience with a possibly-related bug: when debugging programs which handle Ctrl+C themselves (like PipeWire and Audacious), pressing Ctrl+C will trigger the app's graceful shutdown process (terminating execution) rather than breaking into the debugger like I want. Will the changes to gdb fix this issue?

Unlikely. Usually 'graceful shutdown' is implemented by forking and waiting for the child to terminate (or crash).
This means that you are running gdb against the wrong program, so of course you do not get anything useful.
You need to tell gdb to follow the fork.

why blocking sigint ?

farnz — Fri, 30 Sep 2022 15:17:37 +0000

Yes, but this also requires competent programmers, as opposed to people who flail around until they find something that fixes the reported bug (users killing the program with Ctrl-C when it can't safely be interrupted without triggering a full DB recheck).

why blocking sigint ?

ballombe — Fri, 30 Sep 2022 15:15:41 +0000

> A proper fix was to replace this with a signal handler, and teach the critical sections to check to see if there had been a SIGINT while they were in progress (exiting immediately once you were out of the ).

Yes, that is how my project does it too. Just adds a signal handler that set some flag, and check the flag when leaving the critical section. Much nicer than blocking SIGINT.

why blocking sigint ?

farnz — Fri, 30 Sep 2022 14:46:21 +0000

I've worked on a codebase that blocked SIGINT whenever it entered a critical section, and unblocked it afterwards. This was a poor way to avoid being killed by the end user hitting Ctrl-C to escape the program at an inopportune moment (not least because it didn't account for Ctrl-\ or Ctrl-Z), but it worked well enough for the use case in question, since users just spammed Ctrl-C until it died.

A proper fix was to replace this with a signal handler, and teach the critical sections to check to see if there had been a SIGINT while they were in progress (exiting immediately once you were out of the critical section). But that took maintenance work being done for other reasons - it wasn't something that was going to get fixed until I had to do some work to port to a new OS.

How to fix an ancient GDB problem

leromarinvit — Fri, 30 Sep 2022 14:43:15 +0000

That would actually be useful I think. Some time ago, I half-seriously looked into in-kernel ways to colorize stderr (so as to include programs that don't use glibc, so LD_PRELOAD tricks don't work), and BPF would be a nice, non-intrusive (at least for the actual colorization logic) solution if it had the capabilities.

Since the tty code can't be built as a module, the only other way to achieve what I wanted would either require rebuilding the kernel (which I'm usually too lazy for) or ugly runtime patching.

why blocking sigint ?

ballombe — Fri, 30 Sep 2022 14:13:00 +0000

Why would a program blocks SIGINT ?
I can understand installing a signal handler, but blocking it outright ?
This does not seems very user friendly.

How to fix an ancient GDB problem

fw — Fri, 30 Sep 2022 13:51:40 +0000

When attaching to a running process, you just have to hit Ctrl+C in the terminal running GDB, as before. This mode of operation hasn't really suffered from these problems, as far as I understand it.

How to fix an ancient GDB problem

gray_-_wolf — Fri, 30 Sep 2022 13:19:12 +0000

I would assume this will work only when starting new process and not when attaching to existing one?

How to fix an ancient GDB problem

gray_-_wolf — Fri, 30 Sep 2022 13:17:25 +0000

I've encountered this too (when trying to debug segfaulting of guile itself). Yet another option (that I've used) is to just continue over the initial segfault. Worked for me.

How to fix an ancient GDB problem

kleptog — Fri, 30 Sep 2022 08:24:06 +0000

Sounds like what we really need is is to be able to attach an eBPF program to the PTY that can manipulate the I/O and customise the signal handling/generation.

I'm only half joking.

How to fix an ancient GDB problem

rwmj — Fri, 30 Sep 2022 07:53:40 +0000

Perhaps I misunderstand how this works, but if GDB connects to or disconnects from an existing process is it able to change the process's terminal to the new pty and then reset it back to the regular terminal later?

How to fix an ancient GDB problem

fw — Fri, 30 Sep 2022 07:36:55 +0000

The segfault when running GDB under GDB might come from Guile support, or more precisely, from the Boehm-Demers-Weiser garbage collector library Guile uses: in some version, that library probes mapping boundaries, deliberately triggering segfaults, rather than enumerating shared objects using the dl_iterate_phdr function. If that's indeed the problem you could build GDB without Guile support, or apply this garbage collector patch.

How to fix an ancient GDB problem

nyanpasu64 — Fri, 30 Sep 2022 02:53:04 +0000

My personal experience with a possibly-related bug: when debugging programs which handle Ctrl+C themselves (like PipeWire and Audacious), pressing Ctrl+C will trigger the app's graceful shutdown process (terminating execution) rather than breaking into the debugger like I want. Will the changes to gdb fix this issue?

Also is "a GDB command that allows the user to specify which key should interrupt the process and return to the GDB prompt" already present, or will it be added in the rewrite?

How to fix an ancient GDB problem

dullfire — Thu, 29 Sep 2022 22:41:51 +0000

Wow. I've long felt gdb to be extremely unstable. I suspect much of it was this bug. I have tried digging into the issues... however getting into cases were gdb segfaults... And attempting to run gdb under gdb yields the top level gdb segfault... is NOT a happy place to be (the segfaults are definitely unrelated to this... and sadly I never managed to figure out was causing them).

Anyhow, This is sure to make my debugging time less of "fury/anguish at my debugging tools". Thanks for the explanation and write up.