Oracle offering DTrace for Linux [LWN.net]

Oracle has announced the availability of a beta release of the DTrace tracing framework, ported to its "Unbreakable Enterprise Kernel." There is not a lot of information currently about how the port works or how to use it; the DTrace on Linux forum contains only a "welcome" message. There is a usage example in this weblog post by Wim Coekaerts.

to post comments

Oracle offering DTrace for Linux

Posted Feb 22, 2012 23:32 UTC (Wed) by kragil (guest, #34373) [Link] (27 responses)

I really hope this will become the defacto tracing solution for Linux (and Solaris, BSDs, XNU) or is Oracle doing some licensing shenanigans again?

Nothing against systraceperfmonUtraceNG, but DTrace seems a lot nicer and better thought out.

Oracle offering DTrace for Linux

Posted Feb 22, 2012 23:38 UTC (Wed) by dlang (guest, #313) [Link] (3 responses)

even assuming that they aren't doing anything questionable with the licensing. The fact that they are shipping with dtrace doesn't mean that the dtrace patches are going to be acceptable to the upstream kernel (that's been a large part of the problem with other tracing solutions)

I do look forward to the discussion though.

Oracle offering DTrace for Linux

Posted Feb 22, 2012 23:52 UTC (Wed) by tialaramex (subscriber, #21167) [Link] (2 responses)

A big part of the supposed benefit of DTrace has been "You can just do this stuff and it's safe and will do exactly what it says on the tin". If they deliver that on Linux then it will be far harder to argue that it shouldn't get merged. If they can't deliver that on Linux then it will be far harder to argue that DTrace is serving any purpose other than putting an Oracle-flavour stamp on things.

The middle option is that they deliver some of it, but only gradually "it's beta, of course it crashes but that'll get fixed" and meanwhile the alternatives mature to a similar level of robustness and production-worthy reliability.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 9:42 UTC (Thu) by robert_s (subscriber, #42402) [Link] (1 responses)

"A big part of the supposed benefit of DTrace has been "You can just do this stuff and it's safe and will do exactly what it says on the tin". If they deliver that on Linux then it will be far harder to argue that it shouldn't get merged."

DTrace managed to do that for Solaris largely because it has a much more closed-box kernel.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 13:40 UTC (Fri) by davecb (subscriber, #1574) [Link]

I'd suggest it did it not because the kernel was closed, but rather that the compiler shoved the functionality into a really cramped little closed box, so it had trouble causing the traced programs problems, either directly or even by side-effect.

--dave

Oracle offering DTrace for Linux

Posted Feb 23, 2012 1:56 UTC (Thu) by nevets (subscriber, #11875) [Link] (19 responses)

I can't see how they could do anything that the state they do without a GPL license. They would either have to modify the kernel core (which must be GPL) or if they use tracepoints, the interface is only EXPORT_SYMBOL_GPL().

Oracle offering DTrace for Linux

Posted Feb 23, 2012 7:17 UTC (Thu) by lkundrak (subscriber, #43452) [Link] (17 responses)

When they announced they're working on DTrace, I was curious about it and isolated a patch from their "UEK" beta kernel. Apart from there being practically no useful tracepoints, this is what I saw:

-EXPORT_SYMBOL_GPL(print_context_stack);
+EXPORT_SYMBOL(print_context_stack);

And yes, they actually use it from their MODULE_LICENSE("CDDL"); module.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 12:40 UTC (Thu) by rich0 (guest, #55509) [Link] (16 responses)

I don't get how that is illegal, as long as they post their modified sources to the kernel proper. They distribute a modified GPL kernel, and a non-GPL module. I don't see how the GPL dictates who can change the interface between them...

Oracle offering DTrace for Linux

Posted Feb 23, 2012 13:18 UTC (Thu) by nevets (subscriber, #11875) [Link] (7 responses)

-EXPORT_SYMBOL_GPL(print_context_stack);
+EXPORT_SYMBOL(print_context_stack);

Is a clear attempt to circumvent the GPL. I don't know the CDDL license but just posting their modified source is not enough to satisfy the GPL license. You must also allow those that you distribute the code to, the same rights that you have with the code. Free to modify themselves, and they too must follow the rules as well.

Needing the print_context_stack makes their module a derived work. Not something that was ported straight from Sun OS.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:17 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (1 responses)

Needing print_context_stack is an indication that the kernel developers feel it's likely to be a derived work. Oracle's legal department may have disagreed.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:38 UTC (Thu) by nevets (subscriber, #11875) [Link]

As the author of print_ftrace_graph_addr() I would want them to remove that from their version. There's no need for a module to use it unless it was a derived work. I'll let others talk about the rest. It looks like the rest of the function originated from Arjan.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:52 UTC (Thu) by kvh (guest, #83085) [Link] (4 responses)

I am not sure what you diff'd the source code against, but the 0.1 beta release of the DTrace code for Linux was against 2.6.32, and print_context_stack was not an exported symbol in that release. We needed to be able to override the default walk stack functionality (which in turn calls print_context_stack) and therefore I added an export for print_context_stack, and made it match the export for dump_trace (which is not GPL-only) since it is part of the same code. The ability to specify a walk stack function was added as well.

As soon as we realized that in later kernels print_context_stack was exported as GPL-only rather than the non-GPL-only we had chosen, we changed our code to match that, and we moved the part of the DTrace code that uses it GPL as well. This is evidenced in the source code drop for 0.2 that will be placed on oss.oracle.com very soon.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 2:47 UTC (Fri) by clump (subscriber, #27801) [Link] (2 responses)

Given that DTrace's copyright owner now distributes DTrace for Linux, should there be any reason not to simply dual-license DTrace as GPL? I'm assuming inclusion/harmony with kernel.org is a goal, in which case the GPL seems rather obvious.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 3:29 UTC (Fri) by nevets (subscriber, #11875) [Link] (1 responses)

Well if they would like any changes into the kernel their module would have to be shipped GPL (dual license or not), otherwise you can expect big NAKs to any changes that help them out.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 19:56 UTC (Fri) by clump (subscriber, #27801) [Link]

Precisely. Working around deliberately incompatible licenses might be necessary if Oracle didn't own the copyrights to DTrace. I hope the goal isn't to make DTrace harder to use on other versions of Linux.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 7:03 UTC (Fri) by lkundrak (subscriber, #43452) [Link]

Thanks for the reply.

Sorry for not using the version that will be released "very soon" :) Thanks for the oss.oracle.com pointer though; I've been comparing the SRPMs. Seems like I did compare 2.6.32-201.0.1.el5uek to Red Hat's 2.6.32, where they seem to have backported the print_context_stack() independently. Certainly my fault, let me apologize.

Having recompared against 2.6.32-200.19.1.el6uek, on top of which was it based according to the change log, the following seems to have been exported:

+EXPORT_SYMBOL(syscall_nr_to_meta);

Now -- I don't get this; using symbols that are GPL-only from a non-GPL module is wrong, while using ones that have not been exported at all is fine?

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:07 UTC (Thu) by robert_s (subscriber, #42402) [Link] (1 responses)

Then you clearly don't understand the idea of marking symbols GPL and non-GPL.

Marking symbols non-GPL is effectively the kernel developers saying "we don't consider this symbol as an internal core function of the kernel" and as such a module using just those symbols would not be considered a derived work. i.e. "we won't sue over use of these symbols"

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:15 UTC (Thu) by khim (subscriber, #9252) [Link]

Actually it's the other way around: if symbol is declared EXPORT_SYMBOL_GPL then it means that linux developers believe it's impossible to use it without making module derived work of Linux kernel because it's hook to such an intimate part of it.

It's still perfectly possible to create module which is derived work of Linux kernel and which only uses EXPORT_SYMBOL hooks.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:11 UTC (Thu) by lambda (subscriber, #40735) [Link] (5 responses)

Oracle is not the copyright holder on the kernel; they hold copyright on some parts, but not others. Each developer or company holds the copyrights to the portions which they contributed.

The GPL does not allow someone downstream to link and distribute modules to the GPLed software unless that code is also available under the GPL. However, as an exception, kernel developers have granted the ability for non-GPLed kernel modules to link against the kernel, as long as they only use certain symbols that are exported via EXPORT_SYMBOL instead of EXPORT_SYMBOL_GPL. Anyone can apply the GPL to their own work, and then grant additional freedoms beyond what the GPL grants, and that is what the kernel developers (and copyright holders) have done whenever they apply EXPORT_SYMBOL to an interface.

However, someone else cannot come along, and grant themselves additional freedoms. I can't take a piece of GPLed software, and add a few declarations saying "this interface is OK for non GPLed software to link against," and then add my own proprietary code to the software and distribute it. I do not own the copyright, so I cannot grant myself the additional freedom to do this.

That is what Oracle seems to be doing here; they seem to be trying to grant themselves additional freedom beyond what the GPL license of the kernel grants. And I can imagine that some kernel developers might wind up being fairly unhappy about that.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:32 UTC (Thu) by khim (subscriber, #9252) [Link] (4 responses)

However, as an exception, kernel developers have granted the ability for non-GPLed kernel modules to link against the kernel, as long as they only use certain symbols that are exported via EXPORT_SYMBOL instead of EXPORT_SYMBOL_GPL.

Citation needed™. And it's needed really badly because historicaly Linux developers had the opposite POV. It's hard to interpret the infamous Linus's

If an author claims your work is derivative, you should step very very lightly.

And I claim that binary-only kernel modules ARE derivative "by default", by virtue of having no meaning without the kernel.

except as "if you lawyer believes your work is not derivative and if you are ready defend this POV in court, then maybe, just maybe your work is indeed not a derivative and thus can be non-GPL licensed".

However, someone else cannot come along, and grant themselves additional freedoms.

Yes, they can. This is just a marker. But yes, what Oracle did is a clarion call: they took symbol which was marked with "no way in hell can you use it without creating derived work" symbol and removed this mark. Their lawyers should be prepared to defend removal of said mark quite seriously.

And I can imagine that some kernel developers might wind up being fairly unhappy about that.

Not necessarily. If it can be shown that DTrace indeed uses this hook in way which does not imply intimate knowledge of kernel internals then this will be strong signal that this symbol was marked as EXPORT_SYMBOL_GPL by mistake.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 19:28 UTC (Thu) by lambda (subscriber, #40735) [Link] (3 responses)

Sorry, you are right. I got the sense backwards in terms of which one is more explicit. EXPORT_SYMBOL is not explicit about what it is declaring; using it, you may or may not be creating a derived work. EXPORT_SYMBOL_GPL is a way of documenting "if you use this interface, you are clearly mucking about in the internals of the kernel and thus are clearly creating a derived work." The only explicit disclaimer is that any interface exported to user space is disclaimed from creating a derived work for purposes of the GPL.

That doesn't really change my point about the intent of changing an interface from EXPORT_SYMBOL to EXPORT_SYMBOL_GPL; that change says "I know that the kernel developers have said that what I'm doing will create a derived work, but I am going to do that anyhow, and I am not releasing my module under the GPL." That does not automatically mean that what they are doing is a copyright violation; but it shows that they are aware that there are kernel developers who believe that use of that interface constitutes creating a derived work, and they are proceeding anyhow.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 0:51 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (2 responses)

If you look at this[1] comment, it isn't a proper diff. The kernel that DTrace is based on didn't export print_context_stack at all and their patches exported it as non-GPL, whereas more recently, kernel developers exported it as GPL. A grep through the 2.6.32.57 tarball doesn't show print_context_stack being exported at all, so this seems, to me, to be taken out of context.

[1]https://lwn.net/Articles/483322/

Oracle offering DTrace for Linux

Posted Feb 24, 2012 13:49 UTC (Fri) by vonbrand (guest, #4458) [Link] (1 responses)

AFAIU, there are three cases here:

Not exported: For in-kernel use only; use is automatically GPLed
EXPORT_SYMBOL: For use in modules; just using it doesn't make your module GPLed
EXPORT_SYMBOL_GPL: For use in GPLed modules

If it wasn't exported, it's the first case; adding EXPORT_SYMBOL is shady.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 21:14 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

I would imagine if it still weren't exported today, there would be discussion of making it non-GPL enforcing export for merging. I imagine other symbols gained an EXPORT_SYMBOL after being unexported for a time. In any case, it seems a moot point since (apparently) the 0.2 version obeys the EXPORT_SYMBOL_GPL.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 18:45 UTC (Thu) by renox (guest, #23785) [Link]

Or distribute DTrace as a patch for which endusers have to recompile their kernel.
Of course this makes it much less useful for Linux..

Oracle offering DTrace for Linux

Posted Feb 23, 2012 9:40 UTC (Thu) by robert_s (subscriber, #42402) [Link] (2 responses)

Nothing against systraceperfmonUtraceNG, but DTrace seems a lot nicer and better thought out.

So a system designed for a whole other kernel is better than what the actual linux kernel developers designing this stuff for years could come up with?

Oracle offering DTrace for Linux

Posted Feb 23, 2012 13:30 UTC (Thu) by nevets (subscriber, #11875) [Link] (1 responses)

Note, the tracing infrastructure in the kernel has to take a second seat to the rest of the kernel. That is, tracing must not impact the normal flow of the kernel. If this was a company that owned Linux, and the upper management said, we want the tracing infrastructure to be #1 priority, then we could add a strong infrastructure even if the maintainers of the subsystems that we modified disagree.

But this is not how Linux works. Linux is a community, there's a lot of give and take. The subsystem maintainers are responsible for their code, and the tracing folks can't just come in and add hooks where they want without good rational. It is really up to the maintainers to add the hooks.

Then there's more than one player in this game. And tracing is a lot like personal editors. Everyone has their favorite and no one will back down. This slows down the development but hopefully in the long term the best of all utilities will be incorporated. Yes it takes time. This is part of the way open source works.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 17:03 UTC (Thu) by robert_s (subscriber, #42402) [Link]

Indeed - I was rather alluding to what you said in your more comprehensive answer rather than showing disbelief in the kernel development process.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 9:48 UTC (Thu) by and (guest, #2883) [Link] (7 responses)

To me this news seems like Oracle wants to stop the development of solaris in the medium term. Now that btrfs is almost production-ready, there's not too much in the solaris kernel that has an edge on linux (well, except DTrace, maybe)

Oracle offering DTrace for Linux

Posted Feb 23, 2012 11:28 UTC (Thu) by nye (guest, #51576) [Link] (4 responses)

>Now that btrfs is almost production-ready

If you read linux-btrfs it will soon become clear that btrfs is a long, long way from production-ready.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 15:12 UTC (Thu) by Pawlerson (guest, #74136) [Link] (2 responses)

Oracle claims they will use btrfs in their next release of enterprise Linux as production ready file system. They're known for marketing lies, but in this case they're probably saying truth. It looks they're saying goodbye to solaris.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 16:25 UTC (Thu) by SEJeff (guest, #51588) [Link] (1 responses)

<snark>Do you blame them?</snark>

I wouldn't say getting rid of so much as re-closing it back up and making it more for appliances. My perception is that Oracle is keeping Solaris for their really high end stuff such as the Exadata beast[1]. They'd rather be in the business of supporting everything from databases or crm to financials vs generic operating systems.

[1] http://www.oracle.com/us/products/database/exadata/overvi...

Oracle offering DTrace for Linux

Posted Feb 23, 2012 20:35 UTC (Thu) by Pawlerson (guest, #74136) [Link]

This makes sense for now, but I was talking about the future when btrfs will be marked stable. There won't be any reasons to keep Solaris while you will have everything it offers in Linux + much more.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 22:08 UTC (Thu) by jengelh (subscriber, #33263) [Link]

Maybe they have secret plans about fuse-zfs :)

Oracle offering DTrace for Linux

Posted Feb 24, 2012 3:10 UTC (Fri) by shmerl (guest, #65921) [Link] (1 responses)

Latest developments in ZFS happen outside Oracle (namely in the illumos project), key engineers left Oracle already.

Oracle offering DTrace for Linux

Posted Feb 24, 2012 20:10 UTC (Fri) by ESRI (guest, #52806) [Link]

Not entirely accurate. There are now two divergent versions of ZFS however, but for example Oracle has added support for encryption and shadow migration in ZFS whereas the Illumos sponsored version has not.

Oracle offering DTrace for Linux

Posted Feb 23, 2012 23:01 UTC (Thu) by SEJeff (guest, #51588) [Link]

Adam Leventhal, one of the creators of DTrace commented on this: http://dtrace.org/blogs/ahl/2012/02/23/dtrace-oel-update/

Kernel space under GPL now

Posted Feb 25, 2012 16:46 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

Seems like Oracle is dual licensing it.

http://oss.oracle.com/git/?p=linux-2.6-dtrace-modules-bet...

Kernel space under GPL now

Posted Feb 25, 2012 16:48 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link]

I was mistaken. It is just the header files.