Merging kdb and kgdb

It was something of a surprise when Linus Torvalds merged kgdb—a stub to talk to the gdb debugger—back in the 2.6.26 merge window, because of his well-known disdain for kernel debuggers. But there is another kernel debugging solution that has long been out of the mainline: kdb. Jason Wessel has proposed merging the two solutions by reworking kgdb to use the "kdb shell" underneath, which would lead to both solutions being available for kernel hackers.

The two debuggers serve different purposes, with kdb having much less functionality, but they both have uses. Kgdb allows source-level debugging using gdb over a serial line, but that requires a separate system. For systems where it is painful or impractical to set up a serial connection, kdb may provide enough capability to debug a problem. In addition, things like kernel modesetting (KMS) allow for additional features that kdb has lacked. Wessel described one possibility:

While kgdb allows access to all of the standard debugging commands that gdb provides, kdb has a much more limited command set. One can examine and change memory locations or registers, set breakpoints, and get a backtrace of the stack, but those commands typically require using addresses, rather than symbolic names. Currently, the best reference for kdb commands comes from a developerWorks article, though Wessel plans to change that. There is some documentation that comes with the patches, but a command reference will depend on exactly which pieces, if any, actually land in the mainline.

It should be noted that one of the capabilities that was removed from kdb as part of the merger is the disassembler. It was x86 specific, and the new code is "99% platform independent", according to the FAQ about the merged code. Because kgdb is implemented for many architectures, rewriting it atop kdb led to support for many more architectures for kdb. Instead of just the x86 family, kdb now supports arm, blackfin, mips, sh, powerpc, and sparc.

In addition, kgdb and kdb can work together. From a running kgdb session, one can use the gdb monitor command to access kdb commands. There are several that might be helpful like ps for a process list or dmesg to see log output.

The FAQ lists a number of other advantages that would come from the merge, beyond just getting kdb into the mainline so that its users no longer have to patch their kernels, The basic idea behind the advantages listed is to unite the users and developers of kgdb and kdb so that they are all pulling in the same direction, because "both kdb and kgdb have similar needs in terms of how they integrate into the kernel." There have been arguments in the past about which of the two solutions is best, but, since they serve different use cases, having both available would have another benefit: "No longer will people have to debate which is better, kdb or kgdb, why do we have only one... Just go use the best tool for the job."

Wessel notes that Ubuntu has enabled kgdb in recent kernels, which is something he would like to see done by other distributions. If kdb is available, that too could be enabled, which would make it easier for users to access the functionality:

    echo kms,kbd > /sys/module/kgdboc/parameters/kgdboc
    echo g > /proc/sysrq-trigger
    dmesg
    bt
    go

In the example above, Wessel shows how to enable kdb (for keyboard (kbd) and KMS operation), then trap into it using sysrq-g (once enabled, kdb will also be invoked if there is a panic or oops). The following three commands are kdb commands for looking at log output, getting a stack backtrace, and continuing execution.

The patches themselves are broken up into three separate patchsets: the first and largest adds the kdb infrastructure into kernel/debug/ and moves kgdb.c into that directory, the second adds KMS support for kdb along with an experimental patch to do atomic modesetting for the i915 graphics driver, and the third allows kernel debugging via kdb or kgdb early in the boot process; starting from the point where earlyprintk() is available. Wessel is targeting 2.6.34 and, at least so far, the patches have been well received. The most recent posting is version 3 of the patchset, with a long list of changes made in response to earlier comments. Furthermore, an RFC about the idea last May gained a fair number of comments that clearly indicated there was interest in kdb and merging it with the kgdb code.

Sharp-eyed readers will note some similarities between this proposal and the recent utrace push. In both cases, an existing debugging facility was rewritten using a new core, but there are differences as well. Unlike utrace, the kdb/kgdb patches directly provide some lacking user-space functionality. Whether that is enough to overcome Torvalds's semi-hostile attitude towards kernel debuggers—though the inclusion of kgdb would seem to indicate some amount of softening—remains to be seen.