KHB: Dynamic Instrumentation of Production Systems (a.k.a. DTrace)

KHB: Dynamic Instrumentation of Production Systems (a.k.a. DTrace)

Bryan Cantrill and his cohorts are certainly to be congratulated in raising awareness for this type of technology. But whether it can be claimed that they have earned a place in the "Kernel Hacker's Bookshelf for describing the first system that lives up to this ideal" is arguable. Preceding dtrace is a lineage of evolving technologies that go back to the dawn of commercial mainframes. Their paper references the immediately preceding work on Dynamic Probes for Linux, which was published 2001 at FREENIX. That work acknowledges its immediate predecessor, DTRACE for OS/2 (ca 1994) - yes that's right the name isn't even original. OS/2's DTRACE provided a comprehensive low-level scripting language that allow system-wide instrumentation to be applied to a running system. And that language permitted data gathering, rudimentary statistics and triggers for other debugging capabilities. DTRACE arose from an earlier implementation in OS/2 Version 1, the user interface to which was provided by a high-level scripting language and a language interpreter called TRCUST. Besides providing dynamic tracing, TRCUST also provided dynamic profiling. All of that has its origins in similar technologies developed in the 1970s. An example being the Dynamic Support System, which was part of IBM's OS/VS2 (precursor to MVS, great-grandfather of zOS) which ran of the IBM S/370 mainframe. But the story doesn't stop there. DSS came from RSS which was present in some embryonic predecessor to VS2 back in the 1960s.

The problem with debugging technologies is that they have always been, and still are to some extent, considered to be poor relations to "the real operating system features". Debugging has been regarded by many and having no rightful place in a production system. This view as a general statement is nonsense and Sun's DTRACE has greatly helped dispel that nonsense view. Once could say that it has done Linux a great favour by bringing the need to a broader public arena. The work done by Sun has added to the debate and the work already done IBM, SGI, HP, Intel, Red Hat, SuSE and many others in this arena on Linux. The infrastructure to support dynamic instrumentation was accepted into the Linux kernel during the 2.5 development cycle. We now have comparable tool to DTRACE - namely System Tap, that exploits this infrastructure.

Richard J Moore - IBM Linux Technology Center