LWN: Comments on "On DTrace envy"

Be careful of the implicit context

oak — Thu, 06 Mar 2008 20:41:27 +0000

IOgrind has been released a while ago.  Its advantage over live system 
profiling is that the results are deterministic whereas live system 
performance measurements can (according to Meeks) differ as much as 10% 
(on Linux) from run to run.  On a properly designed system, you don't 
(anymore) find that large bottlenecks, they are smaller.

If the bottlenecks are larger, I would assume one could catch them even 
with strace (just strace all applicable processes at the same time).

Be careful of the implicit context

renox — Wed, 29 Aug 2007 20:18:29 +0000

Sorry but 'can be done' is very different from 'can be done usefully' i.e. with DTrace a sysadmin can do the disk profile on a live production system, AFAIK Valgrind cannot do this..

Note that I'm not criticising Valgrind which is a very useful tool, just that the great selling point of DTrace is that it can do systemic tracing on live production system, so sure you can do many things with CPU simulators but it's quite out of topic/context..

On DTrace envy

renox — Wed, 29 Aug 2007 20:06:03 +0000

> Follow AHL's recipe (see his blog, linked to several times in this article)

Well AHL's "recipe" is the one followed by FreeBSD I think to use DTrace and they won't integrate DTrace in FreeBSDv7 because some of them have doubts about the legality of linking BSD code with CDDL headers.

The issue would be of course the same for the Linux kernel.

Is the legal issue real or not? Some say yes, some say no, I have really no clue..

Hopefully this legal mess will be cleared at some point, otherwise maybe Sun could relicense just the header under the BSD license to clear this mess once for all?

Very, very experimental Perl DTrace support

richdawe — Thu, 16 Aug 2007 08:39:20 +0000

Alan Burlison from Sun wrote some experimental support for DTrace in Perl and blogged about it at <http://blogs.sun.com/alanbur/entry/dtrace_and_perl>. I've reproduced his work -- I have a patch <http://rich.phekda.org/perl-dtrace/>, which needs some more work.

Please don't misquote

felixfix — Sun, 12 Aug 2007 18:49:02 +0000

Your credibility in this argument just took a big dive. fuhchee said "almost all of the interesting data" and you left out the "almost", which changes the meaning considerably. Besides which, you yourself say "much investigation does indeed start with probes from SDT- and USDT-provided probes -- but it often finishes with probes from fbt and pid" which certainly implies that a majority of investigations begin with SDT and USDT, which also implies that SDT and USDT are "more" used than anything else. Maybe "almost all" is an exageration of "most", but leaving out the "almost" is a misquote twisting his words to match your thoughts.

dtrace clone

man_ls — Sun, 12 Aug 2007 12:13:21 +0000

As you probably know, any non-trivial piece of code probably infringes on a lot of patents. That hasn't stopped kernel developers from improving the kernel, and rightly so. Maybe the situation with specific patents is different.

dtrace clone

dlang — Sat, 11 Aug 2007 22:09:41 +0000

please name anything in the kernel that has been put in if there is any doubt about the legality of it.

the license and patent status of dtrace is very much a reason for it not being considered (in fact, due to these issues most of the kernel developers don't even look at it to avoid any accusations of them being tainted)

dtrace clone

fuhchee — Sat, 11 Aug 2007 20:53:02 +0000

> Merging upstream has to be done with SystemTap too.

Not as much as one might think. Systemtap relies only on existing
externalized kernel interfaces and hooks - no changes just on our
account. That's not to say it wouldn't be nice to get some extra
interfaces/hooks, or to offload some version drift maintenance to other
folks. It means that we can work toward their development and upstream
inclusion gradually, without blocking the rest of the work.

DTrace real port

man_ls — Fri, 10 Aug 2007 23:13:21 +0000

Is there any specific problem you see with this approach? Not that it is an easy task, but after all there is a working implementation with source code to study. It would probably be easier than working around the limitations you people have exposed in SystemTap, wouldn't it?

dtrace clone

man_ls — Fri, 10 Aug 2007 23:09:18 +0000

Merging upstream has to be done with SystemTap too. Is there any reason why it would be harder with a dtrace clone?

And patent trouble has not stopped kernel developers before...

On DTrace envy

oak — Fri, 10 Aug 2007 19:08:08 +0000

> Right now on Linux there is no way to take an app and profile its disk
seeks in the same way that oprofile lets us profile its i-cache misses;

Disk seeks can already be done, just not at the kernel side (what's cached
and what's not). There was a demonstration in GUADEC 2007 about a new
(yet to be released) Valgrind extension which catches both file accesses
(open etc) AND accesses to memory mapped files. As Valgrind is a CPU
simulator, it can catch these. You can then have another tool that maps
these file/mmap accesses to the actual disk geometry. There was also UI
for visualizing this.

dtrace clone

fuhchee — Fri, 10 Aug 2007 14:48:03 +0000

There are several obstacles. Rewriting the code is a relatively small part. Merging upstream is huge and controversial. There may also be legal (patent) barriers.

On DTrace envy

bfields — Thu, 09 Aug 2007 19:55:49 +0000

I wonder if Linus et al. would object to hooks for an open source component, and, if they did, what the grounds for those objections would be.

That sort of thing has always met a lot of resistance. Currently any in-kernel API can be changed as long as you take care to fix up all the in-tree users. Obviously that makes certain kinds of changes much easier. And having in-tree users for API's makes those API's easier to understand and maintain.

DTrace real port

bcantrill — Thu, 09 Aug 2007 15:47:27 +0000

Good luck with that.

On DTrace envy

paulj — Thu, 09 Aug 2007 11:47:40 +0000

I don't want to sound too humbugish about attribution, but that was my point with "(remember, they're both free software licences)" from my point b) (get linux devs to agree CDDLed Dtrace modules are ok). Any points about potential odd-standards are implied in that (particularly as I had already referred to proprietary modules earlier in my post, highlighted in bold too to make it obvious..).

DTrace real port

man_ls — Thu, 09 Aug 2007 10:25:23 +0000

I'm surprised that nobody has asked the question, so it is probably an obvious stupidity. Anyway there it goes.

Why bother to port? Why not reimplement everything (including the kernel modules) and put it under GPLv2? Building a compatible implementation shouldn't require that much effort, even to the point where DTrace scripts can be shared (that is, those which are not too tightly coupled to Solaris internals).

Drag-n-drop DTrace into Linux Kernel?

nsoranzo — Thu, 09 Aug 2007 08:59:36 +0000

> There was some talk this past spring that Sun was looking at GPLv3 as a worthy successor for or alternative to CDDL. If Sun really does relicense OpenSolaris (and hence XFS and DTrace) under GPLv3 (or dual license it)

Obviously you mean ZFS, not XFS...

Nicola

On DTrace envy

lysse — Thu, 09 Aug 2007 01:58:47 +0000

What he said.

On DTrace envy

madscientist — Wed, 08 Aug 2007 21:30:34 +0000

You're right, I misspoke. I wonder about ahl's point as well: is the Linux devs' position that they shouldn't create special hooks for code that is not GPL-compatible, and hence stands no chance of ever being integrated with the mainline kernel? Or is it only binary blobs they don't like? They really push hard the goal of getting everything promoted up to the mainline kernel.

Of course if it's the case that the in-kernel hooks can be made generic between SystemTap (et.al.) and DTrace, then it doesn't matter anyway.

Drag-n-drop DTrace into Linux Kernel?

khim — Wed, 08 Aug 2007 20:22:37 +0000

Does there seem to be any indications that this could happen sometime in the future?

Today there are no need for this: the parts of DTrace which can be easily lifted are already reimplemented in SystemTap, the stuff that remains is important, but tied to Solaris too tightly to be easily lifted...

On DTrace envy

ahl — Wed, 08 Aug 2007 19:30:57 +0000

That's a very interesting point. I wonder if Linus et al. would object to hooks for an open source component, and, if they did, what the grounds for those objections would be.

On DTrace envy

JoeBuck — Wed, 08 Aug 2007 17:28:14 +0000

In this case, that's not the purpose; DTrace is free software, not proprietary software, even though the license isn't compatible. Furthermore SystemTap could use the same hooks.

Drag-n-drop DTrace into Linux Kernel?

madscientist — Wed, 08 Aug 2007 17:17:50 +0000

A really interesting twist to this would be GPLv3.

There was some talk this past spring that Sun was looking at GPLv3 as a worthy successor for or alternative to CDDL. If Sun really does relicense OpenSolaris (and hence XFS and DTrace) under GPLv3 (or dual license it), I wonder if that's the "killer app" that would get the Linux kernel devs to re-examine the GPLv2 / GPLv3 issue. The buzz is that Linus and Co. are not as disdainful of the final GPLv3 draft as they were of some earlier versions, but that the effort involved with relicensing was widely seen as a waste of time with no real benefit.

Personally I doubt it would change anything on the Linux side; the people who would have to make this decision and do much of the work don't really seem to care much about DTrace (at least).

On DTrace envy

madscientist — Wed, 08 Aug 2007 17:12:02 +0000

> That gets you GPLed reimplementation of certain DTrace bits for the linux
> kernel (perfectly redistributable)

While it is redistributable, I should point out that as described there's very little chance that those changes will ever be accepted into the mainline kernel. Linus et.al. have a long standing, and firm, position that no code will be accepted (even if it's perfectly legal GPL code) into the mainline kernel if its only purpose is to enable proprietary modules of one sort or another.

Of course, there's no reason why individual distributors like Red Hat, SuSE, Debian, etc. couldn't apply that patch themselves to their kernel packages, but this becomes a pain.

Of course, another option would be for the DTrace port to use existing, generic kernel facilities or, if none such exist, work on getting them added. This would be a lot more work, I expect.

On DTrace envy

paulj — Wed, 08 Aug 2007 13:32:55 +0000

"If the bulk of Linux devs of significance do so, then the problem is solved."

Hmm, I should re-state that, as in the above form it allows for possibly morally-bankrupt situations (e.g. some developers objecting, but who do not have resources to take action), which wasn't my intention. So:

"If no Linux kernel developers object, then there is no problem".

static probes

cajal — Wed, 08 Aug 2007 13:22:07 +0000

I was responding to the original article's claim that "SystemTap, instead, does not depend on static probe points within the kernel". That claim implies that DTrace depends on static probe points in the kernel. I was only trying to point out that DTrace provides both dynamic and static trace points (since, as Bryan points out, both are needed).

On DTrace envy

paulj — Wed, 08 Aug 2007 13:07:30 +0000

DTrace is licenced under the CDDL, the Linux kernel is licenced under the GPLv2. Those two licences are incompatible, so any wholesale lifting of DTrace into Linux would give you a result which is (legally) undistributable.

It's pretty simple.

a) Follow AHL's recipe (see his blog, linked to several times in this article)

That gets you GPLed reimplementation of certain DTrace bits for the linux kernel (perfectly redistributable) and CDDLed DTrace bits for the linux kernel (which a user is perfectly entitled to download and combine together with the GPLed bits - many Linux users already have long availed of this right to download and install completely proprietary Linux modules from their distributions repositories).

For 100%-shiny-white redistribution, you need:

b) Linux kernel developers to state they're fine with redistribution of CDDLed Linux kernel modules together with GPLed kernel and/or modules (remember, they're both free software licences).

If the bulk of Linux devs of significance do so, then the problem is solved.

On DTrace envy

njs — Wed, 08 Aug 2007 08:20:00 +0000

I don't understand this. The "predefined trace points" that DTrace provides are things like "thread blocked waiting on IO", "thread resumed after waiting on IO", "cpu went idle", "module was loaded", "page was swapped out". Sure Linux changes really fast, but... is it going to stop having threads that block on IO, cpus that go idle, modules that get loaded, and pages that get swapped out? Of course there is some work in maintaining high-level trace points like these as the actual code implementing the high-level events changes, but I don't see how it's an unreasonable amount of work, given the benefits.

Right now on Linux there is no way to take an app and profile its disk seeks in the same way that oprofile lets us profile its i-cache misses; on Solaris dtrace makes it trivial, including userspace stack traces (another oprofile feature, so still a fair comparison).

static probes

njs — Wed, 08 Aug 2007 08:06:56 +0000

Not the interrupt the Flame-War of the Titans, but...

When Frank sez "almost all of the interesting data that is gathered comes from providers other than fbt", I don't think he's saying "no-one ever uses fbt". (Nor is he trying to estimate total number of in-production probe hits on different providers or anything like that.) He's replying to a claim that static probes are irrelevant, or uninteresting, or at least not as important as the original article claimed. Maybe cajal didn't mean his comment that way, but it certainly can be read that way.

Based on that, to me it basically sounds like Frank is saying "static providers are critical to dtrace's usability in practice", and you're saying "no, no, static providers are only critical to do initial investigation of problems so you can figure out how to deploy the other parts of dtrace". These are not exactly contradictory statements. I can't imagine trying to do anything with dtrace without relying on proc, sched, io, even if I used fbt and pid too.

So cajal's comment does seem a bit misleading, and it seems fair to correct that part of it.

(FTR, I haven't used dtrace either, just envied it intensely.)

On DTrace envy

comay — Wed, 08 Aug 2007 04:49:39 +0000

whose pace of development is as glacial as Solaris's is.

Although I'm sure the number of commits into the Linux kernel source base is larger than that of OpenSolaris, to call the pace of development of OpenSolaris "glacial" is misleading. Have you taken a look at the commits going into the OS/Net consolidation to see the changes taking place?

Of course, it might be interesting to break down the reason for all of the various commits taking place in both projects. How many are new features and other enhancements? How many are due to earlier changes introducing a regression or not being complete?

On DTrace envy

bgregg — Wed, 08 Aug 2007 02:06:34 +0000

Any predefined trace points are going to have their placement and semantics change on a week to week if not a day to day basis. It wouldn't help users at all.

I'm not sure I follow. That last sentence makes no sense to me at all.

The vast majority of people I've met (and also taught DTrace to), either don't have the time to browse kernel source and figure things out, or the assumed programming knowledge to do so. For example, if you had an NFS server latency issue and no stable probes, how many regular users would be able to pick their way through the NFS code, plus the TCP/IP stack code, plus the filesystem cache code, plus the disk driver code (or abstractions), and finally be able to point to the source of the latency? This would span around 300,000 lines of kernel code (in Solaris).

Stable probes allow users to trace complex systems without spending weeks untangling the code, or needing the programming background to understand the kernel. It also allows them to write stable scripts that will work for years to come. So, if the code is changing on a day to day basis, then there is an even greater need for these stable probes - if customers are expected to use the facility.

whose pace of development is as glacial as Solaris's is

Much of the kernel does change frequently, such as the TCP/IP code. A couple of years ago I wrote some dynamic probe based scripts to monitor TCP traffic (tcpsnoop, etc), which have broken several times during those years due to kernel changes. It's a pain for customers trying to use them, and me trying to maintain them. The solution? I've written stable TCP/IP providers for Solaris (not yet putback), which will allow scripts similar to tcpsnoop to be both simple to write and robust.

On DTrace envy

davem — Wed, 08 Aug 2007 00:30:34 +0000

Built-in predefined trace points are a great tradeoff for a system
whose pace of development is as glacial as Solaris's is.

This same tradeoff simply does not traslate to the pace at which the
linux kernel core innards are changing. Any predefined trace points
are going to have their placement and semantics change on a week to week
if not a day to day basis. It wouldn't help users at all.

On DTrace envy

ahl — Wed, 08 Aug 2007 00:21:50 +0000

Fair point:

I'm a member of the DTrace team. Development started with two people in October 2001; I joined about 6 months later. We integrated into Solaris in September 2003 so the original effort took about 5.5 man years of work. From 2003-2005 (when Solaris 10 shipped), the focus for the three of us continued to be DTrace, but we had several other projects on the side as well.

On DTrace envy

clugstj — Tue, 07 Aug 2007 23:36:12 +0000

Not really a fair comparison unless we are told how much much effort has been expended on each project, not just how much time.

On DTrace envy

bgregg — Tue, 07 Aug 2007 22:51:30 +0000

Great article. I have some points to add from by background as a customer using DTrace, writing most of the scripts in the DTraceToolkit, and more recently joining Sun.

making this tool usable by a much wider set of system administrators

This is also important with DTrace, which I've helped encourage by creating a collection of scripts called the DTraceToolkit DTraceToolkit (inspired by the SE Toolkit), which also contains man pages, example files, notes, and a list of one-liners. It would be great if regular sysadmins wrote their own scripts, but if they don't have the time or skill to do that, then using the DTraceToolkit is better than not using DTrace. People can also use the scripts and one-liners in the toolkit as examples from which to learn DTrace.

Another way to reach a larger audience for DTrace is the DTrace Topics wiki, where I hope to write documentation specific to different roles (sysadmin, DBA, etc). This is in addition to the Dynamic Tracing Guide on docs.sun.com, which (DTrace aside) is an example of outstanding technical documentation.

SystemTap's language, instead, has conditionals, loops, and the ability to define functions.

Loops keep being mentioned as a feature of SystemTap's language, perhaps as it's a well known programming construct from other languages. If you write a few hundred DTrace scripts, you'll discover that loops aren't actually that important - perhaps 1 in 40 scripts that I write I think that loops would be nice, then use a workaround. If they did become more important, I imagine they could be added after carefully addressing safety concerns.

As for functions: again, sounds nice, but write a few hundred DTrace programs and note how many times you wanted them. Most DTrace scripts are short, a dozen lines or so; and the DTrace providers make concise abstractions available that shouldn't need too much digging to get to what you want (such as needing functions or loops). There are also workarounds for functions, such as using #defines.

So not having loops or functions would, for other languages, be a big deal. It isn't really the case here, at least in my experience.

DTrace has the ability to work with user-space probes.

This is a big deal; it is allowing providers to be created for languages such as Java, JavaScript, Ruby, Python, etc, and is what makes DTrace complete. DTrace is about observing your application as it runs and interacts with the system libraries, syscalls, kernel and device drivers - the entire software stack. This means if a customer has a performance issue, almost anywhere, DTrace can find it. This is the miracle solution to performance disputes - the application developers, database vendor and os vendor can all use the same tool and see the same numbers.

A while back I wrote a JavaScript DTrace provider, and now with much help from engineers at both Sun and Mozilla, we are looking at how best this could be made a permanent part of Mozilla. Great for JavaScript developers on either Solaris or MacOS, and Linux too - if it ports DTrace.

Drag-n-drop DTrace into Linux Kernel?

pr1268 — Tue, 07 Aug 2007 22:34:53 +0000

> If Sun were to dual-license DTrace under GPLv2, the situation would be different.

Does there seem to be any indications that this could happen sometime in the future? I'm not trying to immediately discount SystemTap; but rather, I'm curious if Sun would/could be so persuaded.

Thank you, Jon, for a wonderful article. Your research articles always seem to get the liveliest discussions! :-)

Vigorous disagreement

bcantrill — Tue, 07 Aug 2007 22:34:22 +0000

Frank, you write:

That's a pretty gutsy assertion
And a correct one, as we seem to be in vigorous agreement.

No, no we're not -- not at all. You wrote this:

...but almost all of the interesting data that is gathered comes from providers other than fbt, and these are implemented with static probes.

That is wrong, and I am not in agreement with it whatsoever. It's wrong in two dimensions. The first being that "all...interesting data comes from providers other than fbt"; as I said (clearly, I thought) fbt and pid often are used as an investigation proceeds from the symptoms of a suboptimal system to the root-causes of that suboptimality in the implementation. So it's a gross mischaracterization to dismiss their role in DTrace. The second way in which your statement is incorrect is the implication that "providers other than fbt ... are implemented with static probes." This shows complete ignorance of the pid provider, which can instrument any instruction in any running process, and is the workhorse of user-level instrumentation.

You also write:

from someone who, to the best of my knowledge, has never used DTrace.
The best of your knowledge needs to get better.

Fair enough; allow me to rephrase: "someone who, if they have used DTrace at all, appears to have learned nothing from the experience." Better?

static probes

fuhchee — Tue, 07 Aug 2007 22:15:45 +0000

> That's a pretty gutsy assertion

And a correct one, as we seem to be in vigorous agreement. Many scripts
use the static probes, especially the ones one might use to start analyzing
a problem. There is nothing wrong with "depending" on static probes in this
sense, so I don't see why you would object.

> from someone who, to the best of my knowledge, has never used DTrace.

The best of your knowledge needs to get better.

Minor factual errors regarding systemtap

fuhchee — Tue, 07 Aug 2007 22:10:14 +0000

> [on ARM a] single Systemtap syscall probe seems to have about as much
> performance hit for the system as the whole LTT(ng) thing.

Please post some details to the mailing list.

> Does this result sound right (LTT uses static probe points whereas Systemtap
> uses dynamic kprobe + has its own overhead on top of that)?

Yes, dynamic probes are significantly costly than static probe points,
but overall cost is a function that includes rate-of-hits and other
quantities. Note that the LTT static probe points will be transparently
exploitable from systemtap once this part of LTT is merged upstream,
and this should greatly reduce the performance differences.

On DTrace envy

tfheen — Tue, 07 Aug 2007 21:53:56 +0000

Can anyone mention a single good reason DTrace can't be lifted as-is, name kept and everything, into the Linux kernel? The Sun people seem to be pushing for this while the Linux people seem intriguingly silent on the issue.

Given that Linux has so many contributors, it is for all practical purposes impossible to change the licence or add an exception allowing linking with the CDDL. Sun owns (afaik) the copyright to DTrace and could with a sufficient effort relicence DTrace under the GPLv2 which would allow lifting the code into Linux, barring any technical difficulties.

Note that I am in no way saying that GPL is better or worse than the CDDL; they are just incompatible due to the GPL's requirement on "no further restrictions" and the CDDL's patent termination clauses. Also please note that I am in no way saying that Sun should relicence DTrace, only that if they want it to end up in the Linux kernel, as-is, relicencing is probably the easiest way to make that happen.