Minor factual errors regarding systemtap [LWN.net]

Minor factual errors regarding systemtap

Posted Aug 7, 2007 19:44 UTC (Tue) by fuhchee (subscriber, #40059)
Parent article: On DTrace envy

Here are some nits to correct in the body of the otherwise well-done article. I'm a systemtap developer.

It is true that systemtap relies on dynamic probes almost exclusively. Since these are expressed in terms of functions / source code lines, they indeed rely on an understanding of the kernel. We are well aware that this is too much for an ordinary user, which is why from day 1, systemtap has included a "tapset" facility. This method allows kernel experts to encode their knowledge about subsystems into higher level probe points, so that end-user scripts can refer to higher level abstractions. So, a knowledgeable person can probe kernel.function("sys_open") and look at local variables or parameters; a less knowledgeable one can rely on the automatic tapset searching and probe syscall.open. Such higher level tapset probes are intended to allay the "must be a kernel hacker" worry, and are generally documented accordingly.

The documentation not only mutters about a blacklist. There are kernel-side and systemtap-side blacklists. (There exists a similar blacklist in dtrace for analogous low-level kernel code.) While these are actively maintained, they are not complete, thus the caution in the docs.

Regarding the relative safety of a virtual-machine-based interpreter versus the compiled-in checks of systemtap, as far as we can see, it's a toss-up. One can consider the systemtap method to be equivalent to inlining all the same checks that the virtual machine would do. All the difficult areas of safety assurance come from other places - the runtime, the choice of probe points.

Regarding the difficulty of getting started with systemtap due to its installation requirements, we agree, and are working on improving this aspect. Similarly, we are aware of the need for more polish, fuller documentation, a bigger library of samples. Our small group is working on these things, and would appreciate community assistance. All of our work (code, bugs, documents, mailing lists) have been in the open since day 1.

Let me finish off with a plug at a systemtap-dtrace comparison page. We have invited Adam Leventhal from Sun to help edit it to keep us all honest.

(Log in to post comments)

Minor factual errors regarding systemtap

Posted Aug 7, 2007 21:19 UTC (Tue) by ahl (guest, #40497) [Link]

As I've conveyed to you privately, I appreciate the invitation to edit your comparison, but I thought
my role was limited to rectifying inaccuracies. I'll be happy to add some material to make the
comparison more complete e.g. stack backtraces, tracing Ruby, Perl, Python, Java, JavaScript, PHP,
etc. -- all of which are absent from SystemTap in its current form. I had though that would be
overstepping my role.

Minor factual errors regarding systemtap

Posted Aug 7, 2007 21:20 UTC (Tue) by oak (guest, #2786) [Link]

A colleague did a quick/ad-hoc test of the recent ARM port of Systemtap
and a single Systemtap syscall probe seems to have about as much
performance hit for the system as the whole LTT(ng) thing. Does this
result sound right (LTT uses static probe points whereas Systemtap uses
dynamic kprobe + has its own overhead on top of that)?

Minor factual errors regarding systemtap

Posted Aug 7, 2007 22:10 UTC (Tue) by fuhchee (subscriber, #40059) [Link]

> [on ARM a] single Systemtap syscall probe seems to have about as much
> performance hit for the system as the whole LTT(ng) thing.

Please post some details to the mailing list.

> Does this result sound right (LTT uses static probe points whereas Systemtap
> uses dynamic kprobe + has its own overhead on top of that)?

Yes, dynamic probes are significantly costly than static probe points,
but overall cost is a function that includes rate-of-hits and other
quantities. Note that the LTT static probe points will be transparently
exploitable from systemtap once this part of LTT is merged upstream,
and this should greatly reduce the performance differences.

DTrace blacklist?

Posted Aug 7, 2007 21:25 UTC (Tue) by bcantrill (guest, #31087) [Link]

The documentation not only mutters about a blacklist. There are kernel-side and systemtap-side blacklists. (There exists a similar blacklist in dtrace for analogous low-level kernel code.) (Emphasis added)

What are you referring to? We dynamically decide what can be safely instrumented; there is no "blacklist of functions" to maintain in DTrace. This is not to say that we can instrument every function, just that we determine whether or not we can safely instrument a function on-the-fly. And I might add that because of our safety-centric architecture, there are very few contexts that one cannot safely instrument...