LWN.net Logo

Factual errors about DTrace

Factual errors about DTrace

Posted Aug 7, 2007 18:50 UTC (Tue) by cajal (subscriber, #4167)
Parent article: On DTrace envy

I'd like to correct several factual errors about DTrace in this article.

DTrace does not depend upon static probes in the kernel. In fact, the "D" in DTrace stands for dynamic; DTrace uses dynamic probing. It does allow developers to add statically defined probe points, and the Solaris kernel does have many statically defined probes.

But it is inaccurate to state that DTrace depends on static probes. In fact, the overwhelming majority of the in-kernel probes on Solaris are dynamic (they're provided by the function boundary tracing (fbt) DTrace provider). This is where most of the "tens of thousands" of in-kernel DTrace probes come from (they're tracing kernel function entry and return points). In userspace, the pid provider allows probes to be dynamically inserted at arbitrary offsets in a program.

The D language does have (limited) control flow: It has the ternary operator.

Concerning user-space probes, it is not required that application developers insert DTrace probe points before DTrace can use them. The pid provider, which I mentioned above, can trace user-space function entry and return (as well as at any arbitary offset within an application). However, it is possible for developers to insert static probes points into their applications. There are several examples of this:

* PostgreSQL 8.2 (http://www.postgresql.org/docs/8.2/interactive/dynamic-tr...)
* Sun's Java 6 (http://java.sun.com/javase/6/docs/technotes/guides/vm/dtr...)
* mod_dtrace for Apache (http://prefetch.net/projects/apache_modtrace/)
* Ruby (https://dtrace.joyent.com/projects/ruby-dtrace/wiki/Ruby+...)
* X.org 7.3 (http://wiki.x.org/wiki/Releases/7.3)
* PHP (http://pecl.php.net/package/DTrace)
* Sendmail (http://mail.opensolaris.org/pipermail/dtrace-discuss/2006...)
* Python
(http://blogs.sun.com/levon/entry/python_and_dtrace_in_build)

Granted, the last two are still experimental.


(Log in to post comments)

static probes

Posted Aug 7, 2007 19:24 UTC (Tue) by fuhchee (subscriber, #40059) [Link]

> DTrace does not depend upon static probes in the kernel.

That is true at a certain level, but almost all of the interesting
data that is gathered comes from providers other than fbt, and these
are implemented with static probes. Check out the dtt code.

In linux, the closest thing we have are markers, which are slowly
making their way through LKML for upstream inclusion. Systemtap
already interfaces to these effortlessly.

static probes

Posted Aug 7, 2007 20:57 UTC (Tue) by bcantrill (guest, #31087) [Link]

DTrace does not depend upon static probes in the kernel.

That is true at a certain level, but almost all of the interesting data that is gathered comes from providers other than fbt, and these are implemented with static probes. Check out the dtt code.

That's a pretty gutsy assertion from someone who, to the best of my knowledge, has never used DTrace. (Because it was not explicit in his comment, Frank Ch. Eigler -- a.k.a. fuhchee -- is the SystemTap lead at Red Hat.) And indeed, you're falling prey to a bit of a logical fallacy: we invented SDT/USDT because it allowed scripts to not have to rely on the implementation, and as the DTraceToolkit wishes to avoid depending on the implementation, it has of course implemented everything in terms of SDT/USDT; looking at the scripts in the DTraceToolkit is therefore not representative (necessarily, anyway) of how people use DTrace.

So how do people use DTrace? When first looking at a problem with DTrace, much investigation does indeed start with probes from SDT- and USDT-provided probes -- but it often finishes with probes from fbt and pid. That is, one often descends from the most abstract ("Why are we doing I/O?") through to the implementation ("Why is FT_New_Face inducing I/O? That font should be cached!") So it's not one or the other -- one needs both methodologies to be able to comprehensively instrument the system, and that is why we have taken both approaches in DTrace.

static probes

Posted Aug 7, 2007 22:15 UTC (Tue) by fuhchee (subscriber, #40059) [Link]

> That's a pretty gutsy assertion

And a correct one, as we seem to be in vigorous agreement. Many scripts
use the static probes, especially the ones one might use to start analyzing
a problem. There is nothing wrong with "depending" on static probes in this
sense, so I don't see why you would object.

> from someone who, to the best of my knowledge, has never used DTrace.

The best of your knowledge needs to get better.

Vigorous disagreement

Posted Aug 7, 2007 22:34 UTC (Tue) by bcantrill (guest, #31087) [Link]

Frank, you write:
That's a pretty gutsy assertion

And a correct one, as we seem to be in vigorous agreement.

No, no we're not -- not at all. You wrote this:
...but almost all of the interesting data that is gathered comes from providers other than fbt, and these are implemented with static probes.
That is wrong, and I am not in agreement with it whatsoever. It's wrong in two dimensions. The first being that "all...interesting data comes from providers other than fbt"; as I said (clearly, I thought) fbt and pid often are used as an investigation proceeds from the symptoms of a suboptimal system to the root-causes of that suboptimality in the implementation. So it's a gross mischaracterization to dismiss their role in DTrace. The second way in which your statement is incorrect is the implication that "providers other than fbt ... are implemented with static probes." This shows complete ignorance of the pid provider, which can instrument any instruction in any running process, and is the workhorse of user-level instrumentation.

You also write:

from someone who, to the best of my knowledge, has never used DTrace.

The best of your knowledge needs to get better.

Fair enough; allow me to rephrase: "someone who, if they have used DTrace at all, appears to have learned nothing from the experience." Better?

Please don't misquote

Posted Aug 12, 2007 18:49 UTC (Sun) by felixfix (subscriber, #242) [Link]

Your credibility in this argument just took a big dive. fuhchee said "almost all of the interesting data" and you left out the "almost", which changes the meaning considerably. Besides which, you yourself say "much investigation does indeed start with probes from SDT- and USDT-provided probes -- but it often finishes with probes from fbt and pid" which certainly implies that a majority of investigations begin with SDT and USDT, which also implies that SDT and USDT are "more" used than anything else. Maybe "almost all" is an exageration of "most", but leaving out the "almost" is a misquote twisting his words to match your thoughts.

static probes

Posted Aug 8, 2007 8:06 UTC (Wed) by njs (guest, #40338) [Link]

Not the interrupt the Flame-War of the Titans, but...

When Frank sez "almost all of the interesting data that is gathered comes from providers other than fbt", I don't think he's saying "no-one ever uses fbt". (Nor is he trying to estimate total number of in-production probe hits on different providers or anything like that.) He's replying to a claim that static probes are irrelevant, or uninteresting, or at least not as important as the original article claimed. Maybe cajal didn't mean his comment that way, but it certainly can be read that way.

Based on that, to me it basically sounds like Frank is saying "static providers are critical to dtrace's usability in practice", and you're saying "no, no, static providers are only critical to do initial investigation of problems so you can figure out how to deploy the other parts of dtrace". These are not exactly contradictory statements. I can't imagine trying to do anything with dtrace without relying on proc, sched, io, even if I used fbt and pid too.

So cajal's comment does seem a bit misleading, and it seems fair to correct that part of it.

(FTR, I haven't used dtrace either, just envied it intensely.)

static probes

Posted Aug 8, 2007 13:22 UTC (Wed) by cajal (subscriber, #4167) [Link]

I was responding to the original article's claim that "SystemTap, instead, does not depend on static probe points within the kernel". That claim implies that DTrace depends on static probe points in the kernel. I was only trying to point out that DTrace provides both dynamic and static trace points (since, as Bryan points out, both are needed).

Very, very experimental Perl DTrace support

Posted Aug 16, 2007 8:39 UTC (Thu) by richdawe (subscriber, #33805) [Link]

Alan Burlison from Sun wrote some experimental support for DTrace in Perl and blogged about it at <http://blogs.sun.com/alanbur/entry/dtrace_and_perl>. I've reproduced his work -- I have a patch <http://rich.phekda.org/perl-dtrace/>, which needs some more work.

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds