On DTrace envy

Posted Aug 8, 2007 0:30 UTC (Wed) by davem (guest, #4154)
Parent article: On DTrace envy

Built-in predefined trace points are a great tradeoff for a system
whose pace of development is as glacial as Solaris's is.

This same tradeoff simply does not traslate to the pace at which the
linux kernel core innards are changing. Any predefined trace points
are going to have their placement and semantics change on a week to week
if not a day to day basis. It wouldn't help users at all.

On DTrace envy

Posted Aug 8, 2007 2:06 UTC (Wed) by bgregg (guest, #46639) [Link]

Any predefined trace points are going to have their placement and semantics change on a week to week if not a day to day basis. It wouldn't help users at all.

I'm not sure I follow. That last sentence makes no sense to me at all.

The vast majority of people I've met (and also taught DTrace to), either don't have the time to browse kernel source and figure things out, or the assumed programming knowledge to do so. For example, if you had an NFS server latency issue and no stable probes, how many regular users would be able to pick their way through the NFS code, plus the TCP/IP stack code, plus the filesystem cache code, plus the disk driver code (or abstractions), and finally be able to point to the source of the latency? This would span around 300,000 lines of kernel code (in Solaris).

Stable probes allow users to trace complex systems without spending weeks untangling the code, or needing the programming background to understand the kernel. It also allows them to write stable scripts that will work for years to come. So, if the code is changing on a day to day basis, then there is an even greater need for these stable probes - if customers are expected to use the facility.

whose pace of development is as glacial as Solaris's is

Much of the kernel does change frequently, such as the TCP/IP code. A couple of years ago I wrote some dynamic probe based scripts to monitor TCP traffic (tcpsnoop, etc), which have broken several times during those years due to kernel changes. It's a pain for customers trying to use them, and me trying to maintain them. The solution? I've written stable TCP/IP providers for Solaris (not yet putback), which will allow scripts similar to tcpsnoop to be both simple to write and robust.

On DTrace envy

Posted Aug 8, 2007 4:49 UTC (Wed) by comay (guest, #46649) [Link]

whose pace of development is as glacial as Solaris's is.

Although I'm sure the number of commits into the Linux kernel source base is larger than that of OpenSolaris, to call the pace of development of OpenSolaris "glacial" is misleading. Have you taken a look at the commits going into the OS/Net consolidation to see the changes taking place?

Of course, it might be interesting to break down the reason for all of the various commits taking place in both projects. How many are new features and other enhancements? How many are due to earlier changes introducing a regression or not being complete?

On DTrace envy

Posted Aug 8, 2007 8:20 UTC (Wed) by njs (subscriber, #40338) [Link] (3 responses)

I don't understand this. The "predefined trace points" that DTrace provides are things like "thread blocked waiting on IO", "thread resumed after waiting on IO", "cpu went idle", "module was loaded", "page was swapped out". Sure Linux changes really fast, but... is it going to stop having threads that block on IO, cpus that go idle, modules that get loaded, and pages that get swapped out? Of course there is some work in maintaining high-level trace points like these as the actual code implementing the high-level events changes, but I don't see how it's an unreasonable amount of work, given the benefits.

Right now on Linux there is no way to take an app and profile its disk seeks in the same way that oprofile lets us profile its i-cache misses; on Solaris dtrace makes it trivial, including userspace stack traces (another oprofile feature, so still a fair comparison).

On DTrace envy

Posted Aug 10, 2007 19:08 UTC (Fri) by oak (guest, #2786) [Link] (2 responses)

> Right now on Linux there is no way to take an app and profile its disk
seeks in the same way that oprofile lets us profile its i-cache misses;

Disk seeks can already be done, just not at the kernel side (what's cached
and what's not). There was a demonstration in GUADEC 2007 about a new
(yet to be released) Valgrind extension which catches both file accesses
(open etc) AND accesses to memory mapped files. As Valgrind is a CPU
simulator, it can catch these. You can then have another tool that maps
these file/mmap accesses to the actual disk geometry. There was also UI
for visualizing this.

Be careful of the implicit context

Posted Aug 29, 2007 20:18 UTC (Wed) by renox (guest, #23785) [Link] (1 responses)

Sorry but 'can be done' is very different from 'can be done usefully' i.e. with DTrace a sysadmin can do the disk profile on a live production system, AFAIK Valgrind cannot do this..

Note that I'm not criticising Valgrind which is a very useful tool, just that the great selling point of DTrace is that it can do systemic tracing on live production system, so sure you can do many things with CPU simulators but it's quite out of topic/context..

Be careful of the implicit context

Posted Mar 6, 2008 20:41 UTC (Thu) by oak (guest, #2786) [Link]

IOgrind has been released a while ago.  Its advantage over live system 
profiling is that the results are deterministic whereas live system 
performance measurements can (according to Meeks) differ as much as 10% 
(on Linux) from run to run.  On a properly designed system, you don't 
(anymore) find that large bottlenecks, they are smaller.

If the bottlenecks are larger, I would assume one could catch them even 
with strace (just strace all applicable processes at the same time).