Gregg: bpftrace (DTrace 2.0) for Linux 2018

[Posted October 9, 2018 by corbet]

Brendan Gregg introduces the bpftrace tracing tool. "bpftrace was created as an even higher-level front end for custom ad-hoc tracing, and can serve a similar role as DTrace. We've been adding bpftrace features as we need them, not just because DTrace had them. I can think of over a dozen things that DTrace can do that bpftrace currently cannot, including custom aggregation printing, shell arguments, translators, sizeof(), speculative tracing, and forced panics".

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 11, 2018 6:44 UTC (Thu) by kloczek (guest, #6391) [Link] (15 responses)

Yet another tool with NIH syndrome.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 11, 2018 9:32 UTC (Thu) by cyphar (subscriber, #110703) [Link]

I don't think that's a fair comment to make. DTrace has been massively useful to Solaris and FreeBSD developers (and administrators) for many years, and bpftrace is just a wrapper around existing kernel functionality with a DTrace-like language (and given that Brendan Gregg worked on DTrace this should not be surprising at all). If you wanted to play the NIH game, SystemTap and ktap would be the NIH projects -- DTrace predates both and is better in many respects.

Personally I'm super excited to use bpftrace because it will allow for much easier dynamic tracing on Linux -- if you've ever had to process a struct member you'd know how much of a pain it is. Also this provides providers and other DTrace niceties purely through combining all of the various *existing* kernel bits that allow for dynamic tracing.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 11, 2018 21:11 UTC (Thu) by flussence (guest, #85566) [Link] (1 responses)

So you're saying DTrace isn't worth copying?

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 12, 2018 4:41 UTC (Fri) by kloczek (guest, #6391) [Link]

This is what bpftrace says.
IMO it is worth to copy in pure form.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 11, 2018 23:45 UTC (Thu) by Paf (subscriber, #91811) [Link] (10 responses)

That would imply it’s possible to use DTrace with Linux. Since it’s at least partly NOT, then NIH can’t apply. They’re copying something good they’re not allowed to have, not just making their own out of pride.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 12, 2018 3:35 UTC (Fri) by pabs (subscriber, #43278) [Link]

It is possible to use DTrace with Oracle Linux:

http://www.oracle.com/technetwork/server-storage/linux/do...

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 12, 2018 4:42 UTC (Fri) by kloczek (guest, #6391) [Link] (8 responses)

Please tell this to Oracle developrs which are porting DTrace to Linux and offer this as co *commercial product* (part of the Oracle Linux)

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 12, 2018 5:50 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

Well, yeah. Oracle developers are NIH-ing likely GPL-illegal product.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 12, 2018 11:24 UTC (Fri) by ehiggs (subscriber, #90713) [Link] (4 responses)

Oracle GPL'd kernel components of dtrace in 2017 and user space components are under the GPL compatible UPL:

https://oss.oracle.com/projects/DTrace/

As seen in the actual files in tree:

https://oss.oracle.com/git/gitweb.cgi?p=dtrace-linux-kern...

I'm not sure why there doesn't seem to be a push to move this upstream. Maybe people are waiting for Oracle to make the first step.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 13, 2018 7:10 UTC (Sat) by cyphar (subscriber, #110703) [Link] (3 responses)

I think it's because, at this point, it's too late. Linux already has effectively all of the kernel-side tracing knobs that DTrace would add -- and so adding DTrace would effectively just be duplication. There are differences of course, but eBPF actually provides some features that DTrace does not (I don't recall the list -- Brendan Gregg has talked about this in the past). The only thing you might get is the userspace tooling, but since that compiles D-script for the DTrace VM, you would need to rip out all of that code as well and rewrite it (also I believe the userspace tooling is still CDDL).

Maybe if DTrace had an "acceptable license" from the outset it would've been merged and we wouldn't have needed SystemTap or ktap or most of the bpf_trace code we have now (then again, there are many examples to the contrary where the Linux community has decided to not merge acceptably-licensed code from other kernels -- such as kqueue for instance).

I used to pine for DTrace on Linux, but what we have today with bpftrace is more than adequate.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 13, 2018 7:37 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

DTrace has string manipulation in its language, which is impossible in eBPF.

Additionally, DTrace seamlessly integrates kernel and userspace tracing. This is missing from bpftrace as far as I can see.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 14, 2018 17:05 UTC (Sun) by kloczek (guest, #6391) [Link] (1 responses)

So as I wrote .. NIH syndrome.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 14, 2018 21:29 UTC (Sun) by anselm (subscriber, #2796) [Link]

Not really. Since DTrace until not very long ago wasn't available under a GPL-compatible license, the choice was between cloning it (which especially with a litigation-happy company like Oracle at the other end doesn't sound like a great idea, ask Google) or coming up with something similar but independent. It's no great surprise that the Linux developers opted for the latter.

Right now the Linux-native approaches are so well established that even if Oracle submitted DTrace for inclusion in the mainline kernel it would probably not be accepted as-is, not because of NIH but because there is no real necessity to have two invasive subsystems that basically do the same thing.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 14, 2018 17:03 UTC (Sun) by kloczek (guest, #6391) [Link] (1 responses)

They GPLed DTrace more than half year ago
https://www.theregister.co.uk/2018/02/19/oracle_open_sour...

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 14, 2018 17:39 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

So basically after the bpftrace started. If Oracle were serious about DTrace they should have submitted their code for the mainline.

Gregg: bpftrace (DTrace 2.0) for Linux 2018

Posted Oct 12, 2018 12:04 UTC (Fri) by jubal (subscriber, #67202) [Link]

oh, go solaris yourself again; you're the prime example of a mindless advocacy drone