LWN: Comments on "Tracing: no shortage of options"

oprofile

xoddam — Wed, 18 Mar 2009 05:57:47 +0000

Forgive me if it's an obtuse question, but if it's performance metrics you're after and not investigating a particularly thorny race condition, isn't oprofile enough?

Why trace the kernel?

compudj — Tue, 05 Aug 2008 17:01:20 +0000

Currently, LTTng implements support for user-space markers on x86 32 and 64 bits. It's a bit
slow, since it goes through a system call each time an event record must be recorded, and the
API is subject to change, but one can currently add markers to their userspace program or
library. See the package : http://ltt.polymtl.ca/packages/markers-userspace-0.5.tar.bz2. It
depends on the LTTng patchset to enable/disable markers and to record data. The tarball
contains examples telling how to modify the makefiles and linker scripts to use markers in
userspace.

Being the LTTng project lead, I dream about a simple in-kernel API to manage the performance
counters, which would aim at managing these limited resources for the various users (watchdog,
user-space perfmon-like API, in-kernel LTTng). The tracer is itself easily extensible and can
record new events which include performance counters either in an interrupt mode or at
specific events occuring on the system (system call entry/exit, interrupt handler entry/exit,
trap entry/exit...). I just need something to setup these counters and let them run free on
the system without changing them when switching from one task to another : this is something
really annoying when gathering system-wide information. I haven't looked at the perfmon code
lately, but I think that most of the user-space system call API is useless to an in-kernel
user like LTTng. A minimalistic perfmon would be welcome.

Tracing: no shortage of options

renox — Thu, 31 Jul 2008 09:13:23 +0000

[[Plus, may dtrace scripts use type names/structs/fields that do not correspond to those in
linux, so one would need to change those scripts or create a solaris emulation layer
someplace.]]

Those issue are the same for FreeBSD, MacOS X, no?
And AFAIK they have not made a Solaris emulation layer, so Solaris-specific dtrace's script
won't work here either..

A compatibility layer with dtrace portable scripts would be nice, but compatibility with
OS-specific dtrace scripts doesn't matter much IMHO.

Why trace the kernel?

nix — Wed, 30 Jul 2008 22:22:28 +0000

That's true. (Particularly true in this case, where quite a bit of the 
magic is handled at ld.so startup time.)

Why trace the kernel?

oak — Wed, 30 Jul 2008 14:37:59 +0000

Thanks, this looks useful!

(Although dynamic linker based things are faster than ptrace, they require 
restarting the monitored process to enabled the tracing so they are not 
very suitable for ad-hoc system monitoring.)

Why trace the kernel?

nix — Wed, 30 Jul 2008 01:05:09 +0000

Regarding your ltrace comments, yes, it's a kludge. Thankfully glibc 
provides LD_AUDIT which is much more capable, because the dynamic linker 
really can tell what dynamic calls are being made. It would be even better 
if there was even the slightest hint of documentation about it, but 
thankfully we don't need that as Jiri Olsa has done it for us: 
<http://latrace.sf.net/>.

Why trace the kernel?

oak — Tue, 29 Jul 2008 21:01:58 +0000

Another point is that unlike strace, SystemTap's system-wide[1], and being 
kernel side instead of going through the fragile (threads/signals...) & 
slow ptrace() interface it can be much faster, especially after it starts 
using the static markers / tracepoints.

[1] One can attach strace to multiple processes, but it really slows down 
the system and is pretty inconvenient (ptrace has side-effects).  If one 
wants to do full system tracing, currently LTT is much better alternative 
than SystemTap though due to its much smaller overhead (especially with 
multiple probes).  LTT and SystemTap are a bit for different purposes 
though.

Currently neither LTT nor SystemTap provide user-space tracing like DTrace 
does, but I think Frysk does some of that.  Frysk doesn't do kernel 
tracing though and it would also profit from utrace as currently it uses 
ptrace().  (Note: ltrace tool is pretty useless for user-space tracing as 
it doesn't trace library->library calls and it doesn't support dlopen() 
which is used almost by anything a bit more complicated on Linux desktop).

Once one has system wide monitoring, one could have something like this:
- http://idea.opensuse.org/content/ideas/filemon-like-syste...
- Or the RedHat Google SOC about getting BootChart type of visuals from 
SystemTap...

PS. This is not just DTrace envy, it would be nice to have a GUIs like 
these:
http://developer.apple.com/tools/performance/optimizingwi...
http://developer.apple.com/documentation/DeveloperTools/C...
(Instruments uses DTrace, Shark doesn't) :-)


Btw. This is more about performance analysis framework stuff than tracing, 
but what is happening with perfmon2?  Is that going to be used e.g. by 
LTT?

Why trace the kernel?

corbet — Fri, 25 Jul 2008 01:33:36 +0000

This kind of tracing is wanted by people trying to track down problems which happen in production environments. You need to see why the system is behaving poorly (or crashing) while operating under its real workload. That requires the ability to hook into the system almost anywhere without perturbing its operation. Tracing can be nice for normal kernel debugging, but it's the folks wondering why their monster trading systems are bogging down that really want it.

The "DTrace envy" is there (1) because Sun has been using it as one of its primary anti-Linux marketing tools, and (2) because what we have isn't as good. We'll get there.

Why trace the kernel?

pr1268 — Thu, 24 Jul 2008 23:26:58 +0000

Forgive me for sounding ignorant, but why all the fuss about tracing the kernel? And more specifically, why is there DTrace "envy" (referring to a previous week's LWN article)?

My impressions are that in the Linux kernel, the code is already well-optimized and (relatively) error-free, so why is there such a demand as of late to insert all these probes like stuffing pins in a pincushion? Granted, I'm familiar with using tools like strace(1) (and similar tools, both open-source and proprietary), but I'm still unsure why this has become such an important issue with regards to the kernel lately.

And, what's the envy all about over Sun Microsystems' DTrace? Linux kernel devs are certainly capable of creating their own tracing tools (as this article explains).

Thanks in advance for any comments--I'm just trying to feed my curiosity by asking here.

Tracing: no shortage of options

fuhchee — Wed, 23 Jul 2008 12:32:25 +0000

It might not be hard for systemtap to parse dtrace's smaller scripting
language, but there's more to it than that.  A variety of dtrace
kernel and userspace features have no equivalent yet, so many real
dtrace scripts would not work.  Plus, may dtrace scripts use
type names/structs/fields that do not correspond to those in
linux, so one would need to change those scripts or create a
solaris emulation layer someplace.

Tracing: no shortage of options

willy — Wed, 23 Jul 2008 04:05:33 +0000

Four -- Linux, Solaris, MacOS and FreeBSD.  The other three already use DTrace.

Tracing: no shortage of options

SEJeff — Tue, 22 Jul 2008 23:57:10 +0000

DTrace on Linux is basicly a noop due to licensing conflict. Now what would happen if someone
wrote a syntax compatible "dtrace interpreter" that used systemtap and utrace, once it comes
out?

That would be a win/win because you have 1 language, D, for tracing events on two of the most
popular posix platforms.

Tracing: no shortage of options

bronson — Tue, 22 Jul 2008 21:40:37 +0000

Also, Paul Fox is porting DTrace to Linux: http://www.crisp.demon.co.uk/blog/index.html  I
haven't tried it yet but I'm sure I will soon.

Tracing: no shortage of options

fuhchee — Tue, 22 Jul 2008 18:59:16 +0000

> James Bottomley has been playing some with the SystemTap code; one result 
> of that work is changes to SystemTap's internal relocation code in an 
> attempt to make it more acceptable for mainline kernel inclusion.

The part of systemtap code that James is proposing to modify lives
entirely in user-space, so mainline kernel inclusion of that part
hasn't even come up.