Debugging kernel modules

Linus is famously against the use of interactive debuggers on the kernel, but many developers use them anyway. Debugging a running kernel is a little harder than working with a typical application, but it can be done in a couple of ways. It is relatively easy to query kernel data structures in the current running kernel by running gdb with /proc/kcore as the "core" file. More extensive debugging, allowing the use of breakpoints and such, can be done by using gdb on a remote machine and controlling the target via a serial line or a network interface. The -mm tree contains the necessary patches for using gdb in this mode for a few architectures.

One limitation with using gdb this way is that it can't be used to work with loadable modules. The debugger can query the memory used by loadable modules, set breakpoints there, etc. The problem is that it does not know what addresses get assigned to functions and variables when a module is loaded. Those addresses, obviously, are not in the core kernel executable, and there is no real way to find them at run time. The developer can thus work by typing in hex addresses directly, but that gets tiresome fairly quickly.

Your editor was recently finishing out the debugging chapter for Linux Device Drivers, Third Edition (which is getting closer to ready - honest) when he ran up against the loadable module problem. The kernel knows where all of the symbols go when it loads a module; it really seemed like it should be possible to communicate that information to a debugger. A bit of digging revealed that, in fact, the relevant information gets dropped once the module gets loaded. So it was time for a fix.

Like any other ELF executable, a loadable module is divided up into several sections. The section called .text contains (most of) the module code itself; .data and .bss contain most of the variables. The module loader looks at all of the sections and lays them out sequentially in (vmalloc) memory; after relocating symbols it forgets about where the sections went. If the positions of the sections could be recovered, however, they could be passed to gdb in the same add-symbol-file command which tells the debugger about the module code. The section offsets are all that gdb needs to figure out where the module's variables live.

Your editor, rather than tell LDD3 readers that symbolic debugging of kernel modules was impossible, chose to do a little hacking. The result was this patch, which hangs a new kobject onto each loadable module and populates it with a set of attributes containing the section offsets. Those attributes will show up under /sys/module. Thus, for example, after module foo is loaded, /sys/module/foo/sections/.data will contain the beginning of the .data section. The foo developer can then fire up gdb and, after connecting to the target kernel, use the section offset information to issue a command like:

    add-symbol-file /path/to/module 0xd081d000 \  # .text
 		-s .data 0xd08232c0 \
		-s .bss  0xd0823e20

Thereafter, debugging the module is just like debugging the rest of the kernel. There is a little script (included with the patch) which generates the add-symbol-file command, reducing the operation to a simple cut-and-paste.

The patch has been merged into Linus's BitKeeper tree, and will be part of 2.6.8.

Index entries for this article
Kernel	Debugging
Kernel	Modules/Debugging

---

to post comments

Debugging kernel modules

Posted Jun 24, 2004 3:01 UTC (Thu) by jdaily (guest, #604) [Link] (1 responses)

Kudos, Jon. This reminds me of one of the key benefits of man pages (back when software developers used to write man pages): writing the BUGS section would often shame the programmer into fixing the bug.

Debugging kernel modules

Posted Jun 24, 2004 3:15 UTC (Thu) by allesfresser (guest, #216) [Link]

Ay, bravo. Very nice.

Impromptu module debugging?

Posted Jun 24, 2004 3:39 UTC (Thu) by speedster1 (guest, #8143) [Link]

An interesting use of sysfs... I solved this same problem with 2.4 kernels by modifying the module loader to output the appropriate info. Do you know if it is possible with the -mm gdb stubs to start up the kernel normally, then attach a gdb session via serial port later? That would be awesome coupled with your ever-present module address map -- perfect for tracking down intermittent module weirdness that shows up once in a blue moon and NEVER when you're purposefully looking for it. That's the best thing about xmon debugger, that you can run with it enabled and never notice till a module oopses and you're dropped into xmon.
-- S. Lockwood-Childs

P.S. I bet a lot of ppl would love an article that was a case study on using a kernel debugger to track down some problem (maybe that one is already in the works, as a natural follow-up to this one?)

How we use GDB on LKMs today

Posted Jun 25, 2004 18:11 UTC (Fri) by giraffedata (guest, #1954) [Link] (1 responses)

This is a welcome usability enhancement to GDB with the kernel, but the situation isn't as dire today as the article makes it sound.

People debug loadable kernel modules with GDB all the time by using the -m option on 'insmod' to get that section location information. kgdb.sourceforge.net has a program that does the insmod and generates a convenient gdb script you can invoke to do the appropriate add-symbol-file.

The drawback is that you have to plan in advance -- and have to do a special load of the module. With Jon's work, the information is always there.

Well, not exactly. You still have to plan ahead and look at the sysfs file before gdb gets entered and the system stops.

With current Kgdb (kgdb.sourceforge.net), There's a mechanism where Gdb automatically knows where the modules are loaded. But something about it makes Gdb annoyingly slow, so I abandonned it and can't say much about it.

How we use GDB on LKMs today

Posted Jun 25, 2004 18:37 UTC (Fri) by corbet (editor, #1) [Link]

"People debug loadable kernel modules with GDB all the time by using the -m option on 'insmod' to get that section location information."

Yes, people do that when developing for 2.4 kernels, where insmod has a clue of where the sections live. In 2.6, the section layout is done inside the kernel itself, so insmod could not tell you where the sections ended up even if it did still support the -m option, which it doesn't...

Debugging kernel modules

Posted Oct 12, 2004 11:29 UTC (Tue) by mithlesh_t (guest, #13591) [Link]

The modules which are loaded on machine can very well be debugged using KGDB which can be found at http://kgdb.sourceforge.net . There is a patch for GDB which makes it understands that its debugging modules and kernel instead of normal user land programs. This patch and other module debugging utilities are available at http://kgdb.linsyssoft.com . With these you can debug modules as well as kernel as they are user land programs :)

Debugging kernel modules

Posted Mar 17, 2014 3:22 UTC (Mon) by bazis (guest, #96006) [Link]

You can also use an IDE like VisualKernel to handle the symbol loading automatically. See this tutorial for a detailed step-by-step example.