On DTrace envy

On DTrace envy

Posted Aug 7, 2007 18:55 UTC (Tue) by bcantrill (guest, #31087)
Parent article: On DTrace envy

First, thanks for what is largely an accurate comparison -- it's especially gratifying to see understanding of (and appreciation for) the design decisions that we made in DTrace around safety. One very important correction, however: DTrace does not rely solely on programmer insertion of probes. In the kernel, DTrace can instrument every function entry and return without any programmer involvement whatsoever (which is the reason that tens of thousands of probes exist on any system -- there are two for most kernel functions). At user level, DTrace goes one better and can instrument every instruction in every process -- again without programmer involvement.

Now, all of that said, DTrace benefits from programmer involvement: programmers may add static points of instrumentation that export the semantics of the system instead of its implementation. As a concrete example, there are three functions in the Solaris kernel in which we enqeue a thread to run: setkpdq, setfrontdq and setbackdq. An expert in Solaris internals could therefore determine when a thread is scheduled to run with the following DTrace fragment:

setkpdq:entry,
setfrontdq:entry,
dispdeq:entry
{
        printf("%s scheduled to run...\n", args[0]->t_procp->p_user.u_comm);
}

This doesn't (or didn't) require any changes to the Solaris kernel itself, which is good. But this also requires knowing quite a bit about the implementation of Solaris: where threads are enqueued to run, what the thread structure looks like, etc. And it's brittle: if we change the kernel's implementation, the script is broken.

To address these shortcomings, we added a statically-defined tracing (SDT) provider, the sched provider, which makes available higher-level semantics. When rephrased in terms of sched, the above becomes:

sched:::enqueue
{
        printf("%s scheduled to run...\n", args[1]->pr_fname);
}

This is something that is simpler to understand, and represents stable, documented abstractions -- and we can change the kernel without breaking the script.

While originally a kernel-level notion, we brought this same technology to user-land, with what we call user-level statically-defined tracing (USDT). USDT allows one to instrument, say, one's Ruby, Python, Java, JavaScript, etc. in terms of the language instead of the implementation; the reader is directed to google + [language of choice] + "I'm feeling lucky" for details.

Anyway, thanks again for the careful look at DTrace. And if anyone finds themselves suffering from excessive DTrace envy, you might want to chime in on Adam's blog about how a Linux port could be made possible...

---

(Log in to post comments)