kerneloops.org records its 100,000th oops

[Posted November 17, 2008 by jake]

From:		Arjan van de Ven <arjan-AT-infradead.org>
To:		linux-kernel-AT-vger.kernel.org
Subject:		kerneloops.org database passes the 100.000 mark
Date:		Sat, 15 Nov 2008 19:48:01 -0800
Message-ID:		<20081115194801.7a257704@infradead.org>
Cc:		torvalds-AT-linux-foundation.org, tytso-AT-mit.edu, akpm-AT-linux-foundation.org, alan-AT-lxorguk.ukuu.org.uk

Today, just a few days short of the first anniversity of the project, 
the kerneloops.org database of oopses and warnings has passed 
the 100.000 stored oopses/warnings.

Number one hundred thousand is....

WARNING: at drivers/net/wireless/iwlwifi/iwl-tx.c:1196 iwl_tx_cmd_complete+0x57/0x22d [iwlcore]()

stored at http://www.kerneloops.org/raw.php?rawid=100000

(and yes there is some irony in this)

Looking back at the first 100.000 entries, the top 10 files and top 10 oopses 
of these 100.000 are shown below:

Per file statistics
6986	external/madwifi/wrapper
5838	fs/jbd/journal.c
5633	drivers/parport/procfs.c
3581	external/fireglx/binary
3473	external/madwifi/binary
3078	sound/core/init.c
2640	net/sched/sch_generic.c
1872	drivers/ata/libata-sff.c
1615	kernel/time/tick-broadcast.c
1536	drivers/ssb/main.c
					Count	Last version	First version 
1. journal_update_superblock		6045 	2.6.27.5	2.6.24-rc6-git1 
	A patch for this is -mm for a while already
2. parport_device_proc_register		588	2.6.27-rc7-git1	2.6.24-rc5 
	Alan has a partial fix for this one pending
3. ath_dynamic_sysctl_register(P)	5408	2.6.25.9	2.6.24 
4. init_ath_hal (P)			3300	2.6.26.3	2.6.24 
5. snd_card_register			3076	2.6.25.14	2.6.24 
6. firegl_ioctl (P)			2891	2.6.26.6	2.6.25 
7. dev_watchdog()			2631	2.6.27.4	2.6.26-rc3 
8. tick_broadcast_oneshot_control	1609	2.6.25.14	2.6.24-rc4 
9. ath_sysctl_register	(P)		1551	2.6.26.3	2.6.24-rc4-git4
10. ssb_tmslow_reject_bitmask		1536 	2.6.26.6	2.6.24


Other than the top 2 items, which have patches, we've done a pretty good job of fixing
the high occurance bugs (excluding the binary drivers which we obviously cannot fix)

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

oops logging during module load/unload

Posted Nov 17, 2008 21:39 UTC (Mon) by abatters (✭ supporter ✭, #6932) [Link] (1 responses)

I just noticed today that oopses that happen during insmod/rmmod/modprobe are printed to the console but not logged by syslogd/klogd. I wonder how many oopses have been missed because of this.

oops logging during module load/unload

Posted Nov 18, 2008 0:12 UTC (Tue) by arjan (subscriber, #36785) [Link]

if it's still in dmesg, then the kerneloops client will still pick it up.
The database has quite a few examples of these already.

kerneloops.org records its 100,000th oops

Posted Nov 17, 2008 21:51 UTC (Mon) by zdzichu (subscriber, #17118) [Link] (10 responses)

So 3 of 5 top issues are caused by off-the-tree/binary drivers?

kerneloops.org records its 100,000th oops

Posted Nov 17, 2008 23:20 UTC (Mon) by kirkengaard (guest, #15022) [Link] (9 responses)

No, three of the five top *locations* for issues are off-the-tree/binary drivers, and two of those are madwifi. More specifically, two of those are in the binary part of the madwifi atheros driver. That driver seems responsible for three of the top ten oopses, one for initializing the binary-only HAL and one each for the dynamic and static sysctl registration methods, which together account for 10,259 separate reports. Which is what you get when legal issues bind coding issues and you're stuck navigating around a blob with the software you can see.

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 4:07 UTC (Tue) by jwb (guest, #15467) [Link] (8 responses)

This makes me feel a lot better about having removed the Atheros wifi from my ThinkPad and replacing it with iwl3945. The statistics remind me of the crash stats from the Mozilla project, which are dominated by Adobe Flash, Google Desktop, and various Windows antivirus systems. It's quite difficult to look through Mozilla crash stats to find something =not= caused by proprietary crapware.

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 14:29 UTC (Tue) by nowster (subscriber, #67) [Link] (5 responses)

There wasn't much need to do that. The in-tree ath5k driver is now proving to be stable.

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 17:26 UTC (Tue) by kirkengaard (guest, #15022) [Link]

Quite right. ath5k and the new ath9k are where the real action is, and they have been done in order that the binary-only HAL might be avoided. In fact, madwifi project is involved in all three, though the serious effort is apparently going into the in-tree drivers. They were just stuck with Atheros' decision about the proprietary HAL because of radio regulations, and the madwifi code was designed to work with the Atheros HAL. They're a good example, I think, of how solutions can be found to proprietary binary sections (as opposed to certain other binary-only "benefactors" of the community).

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 18:29 UTC (Tue) by jwb (guest, #15467) [Link] (3 responses)

Well it wasn't stable enough to make it into Ubuntu 8.10, so I tossed the hardware.

kerneloops.org records its 100,000th oops

Posted Nov 19, 2008 12:22 UTC (Wed) by Janne (guest, #40891) [Link] (2 responses)

The environment thanks you. We could always use some more e-waste!

kerneloops.org records its 100,000th oops

Posted Nov 19, 2008 17:14 UTC (Wed) by jwb (guest, #15467) [Link] (1 responses)

I've heard plenty of stupid proposals in my time, but keeping unworking peripherals in my laptop due to environmental concerns is in the 99th percentile of stupid.

kerneloops.org records its 100,000th oops

Posted Nov 25, 2008 13:49 UTC (Tue) by Janne (guest, #40891) [Link]

Instead of tossing it away, how about giving/selling it to someone who could use it?

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 17:32 UTC (Tue) by bkoz (guest, #4027) [Link]

ditto, times two.

kerneloops.org records its 100,000th oops

Posted Nov 19, 2008 8:50 UTC (Wed) by pabs (subscriber, #43278) [Link]

I'm considering going the other way, iwl3945 -> atheros, simply because Intel requires non-free binary-blob firmware.

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 1:43 UTC (Tue) by johill (subscriber, #25196) [Link] (3 responses)

Incidentally, I think I may just have found the cause for that #100,000. Anyone affected look here: http://marc.info/?l=linux-wireless&m=122696931311854&...

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 5:38 UTC (Tue) by xoddam (guest, #2322) [Link] (2 responses)

Could you maybe also explain why Arjan thinks it's ironic that this particular oops marked the move into six digits?

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 8:01 UTC (Tue) by biehl (subscriber, #14636) [Link]

Because it is an Intel-bug?

http://intellinuxwireless.org/?p=iwlwifi

kerneloops.org records its 100,000th oops

Posted Nov 19, 2008 7:52 UTC (Wed) by harinath (subscriber, #47697) [Link]

Maybe because the 100,000th report got posted by _not_ tickling the same bug that caused the oops -- it's in a network transmit routine, afterall. (Assuming, of course, that the oops was reported by the same kernel :-)

kerneloops.org records its 100,000th oops

Posted Nov 18, 2008 17:58 UTC (Tue) by mb (subscriber, #50428) [Link]

I want to note that the ssb_tmslow_reject_bitmask WARN_ONs in drivers/ssb/main.c are nothing we can really avoid before they hit users.
Broadcom keeps silently changing the hardware bits and we can only fix this _after_ we got the WARN_ON "bugreports". So let's hope they stopped changing the crap now, so the warning will disappear from the top 10.

Debian Etch does not support kerneloops.org collection utility?

Posted Nov 19, 2008 19:38 UTC (Wed) by cpeterso (guest, #305) [Link] (1 responses)

Debian Etch's apt-get repository does not include kerneloops.org's kernel oops collection utility. But Debian Lenny and Sid do: kerneloops packages

Is there another easy way to report kernel oopses if I'm using Etch?

Debian Etch does not support kerneloops.org collection utility?

Posted Nov 20, 2008 3:00 UTC (Thu) by arjan (subscriber, #36785) [Link]

using the client is obviously the easiest way
but kerneloops.org also gets the oopses from LKML or from bugzilla.kernel.org......