More DTrace envy

Nearly a year ago, we looked at the status of SystemTap in the context of Sun's much-hyped DTrace tool. Since that time there has been progress, but the basic problem still remains: Linux does not have a good, ready-to-run answer to those wanting the equivalent functionality of DTrace. Due to an apparent disconnect between the developers of SystemTap and the kernel hackers, tracing for the Linux kernel—never mind user space programs—is not up to the competition.

Both SystemTap and DTrace are tools meant to help administrators track down performance and other problems on production systems by instrumenting the kernel. Because SystemTap has not matured to the point of easy usability, DTrace is often seen as a prime differentiator between Linux and Solaris. In a posting to the ksummit-2008-discuss mailing list—where Kernel Summit topics are considered—Matthew Wilcox brought up the subject based on his experience at a recent PostgreSQL conference:

Typically Linux has been in the forefront of interesting new technologies for free operating systems. When Sun opened up Solaris, though, a few features jumped ahead of their Linux counterparts, in particular the ZFS filesystem and DTrace. SystemTap is supposed to provide the tracing functionality while Btrfs is the leading candidate for a "next generation" filesystem. But, so far, SystemTap has not lived up to its potential.

There are a few reasons for disappointment with SystemTap, some of which were pointed out by James Bottomley:

Those are all valid concerns, but the biggest problem for users is that, unless they are knowledgeable about kernel internals, it is difficult to know how to use SystemTap. A more simplified interface, one that is less reliant on kernel internals, needs to be created; the way to do that is through the placement of static trace points in the kernel and the creation of "tapsets" to make them easily usable. The SystemTap developers think the kernel hackers are in the best position to do that work. Ted Ts'o agrees but sees some barriers:

If the kernel developers have trouble using SystemTap, they are unlikely to add the tapsets that would make it more usable for system administrators and others who have some general kernel knowledge but not enough to sensibly instrument it. For people using distribution kernels—at least for the enterprise distributions and Fedora—it is only somewhat painful to get SystemTap up and running. But kernel hackers tend to run their own kernels, often many different versions in a short period of time, so they need to be able to be easily build one that works with SystemTap and includes all of the debugging information that it requires.

SystemTap developer Frank Ch. Eigler has a long reply to many of the complaints in the thread. It seems clear that the SystemTap folks and the kernel hackers have not been communicating—there are solutions to many of the problems that were cited. They are in various states of readiness, but are mostly working. So SystemTap is most of the way there for kernel tracing as long as you are well-versed in kernel internals, but that has been true for some time.

In order to get SystemTap to where it needs to be, the kernel hackers need to be involved. Building the infrastructure and waiting for tapsets to magically appear is not a recipe for success. The SystemTap hackers need to be engaging the kernel community, as well as distributions, to make the tool into something that gets used.

SystemTap can use static probe points, kernel markers—merged into 2.6.24—but it is notable that no one has, as yet, made use of them. A concerted effort needs to be made to make the tool more usable for the kernel developers who can, in turn, help make it more usable for others. There is a clear problem when folks like Ts'o regularly try, but find it too difficult to be useful:

It is a commonly heard complaint that while SystemTap is difficult to use, DTrace "just works" for Solaris; Eigler responds:

Many administrators and other users of tracing facilities are not necessarily interested in kernel-level tracing, but would really like to be able to use the instrumented versions of things like PostgreSQL. That is in the plan according to Eigler: "We aim to piggyback on these efforts by reusing the dtrace instrumentation calls embedded into postgres etc., if at all possible."

Until the rough edges can be smoothed on the kernel side, Bottomley wonders if it even makes sense to start considering user space:

DTrace sounds like a nice working solution that has many uses and many happy users. If one can ignore the self-congratulatory postings from its lead developer, it might be worth having in Linux, but that simply is not going to happen. Paul Fox is working on a port of DTrace to Linux, but that ignores the licensing realities that would never allow it to become part of Linux. It also ignores the difficult path a DTrace port would face getting merged into the mainline. (We hope to have an article from Mr. Fox on his DTrace porting work soon, stay tuned).

For all of the talk out of Sun about how they would love to make DTrace a part of Linux, they clearly made a choice to ensure that could not happen. Even if any technical barriers were lifted, the CDDL is not compatible with the GPL. It is perfectly fine as a free software license, but if you wish to get things into Linux, they must be licensed in a GPL-compatible way. This was well understood at the time Sun freed Solaris, so this must have been a conscious decision. Given how much their marketing organization likes to tout DTrace, it would seem to be a choice that Sun is quite happy with.

Linux will eventually get the tracing support it needs, in a way that is easily accessible to users, but it may take some time. Conversations like the recent one on ksummit-2008-discuss are an important part of getting there. It would appear that better support for the use cases of kernel developers will be forthcoming. It is mostly a matter of documentation along with simplifying some of the building and installation issues. Once the kernel hackers actually start using it, progress is likely to be fairly swift.

This is the way free software development works; it generally does not track a straight path to a solution, but often wanders about in the solution space for a while. It is highly unlikely that a development like DTrace could have come about in the way that it did in a true community-developed operating system. For that you need everyone pulling in the same exact direction, which may be why Sun is reluctant to turn over much of the governance of Solaris to the community. That may help them develop things more quickly, because there will be fewer barriers, but it won't help them to foster the kind of development community that characterizes Linux.

To an extent, it's unfair comparing Solaris with desktop Linux distributions. If you compare
Solaris to RHEL or SLES, it doesn't fare as badly. For instance, desktop Linux distributions
have power management functionality that's still nowhere near present in any shipping versions
of Solaris. Enterprise Linux distributions aren't likely to ship that functionality until next
year, and they're the market that Solaris is competing with.

(Though, really, I expect we'll still be ahead on power management for a long time. Solaris
isn't anywhere near supporting a tickless kernel and that's one of the major factors in the
idle power consumption on modern hardware)

It's frustrating that Bryan Cantrell apparently reacts to all mentions of license issues as "I
want to talk about technology, not law, legal issues are unimportant".

Well, no, they're not.  If they were, that'd be great, but as it happens, we have to care
about under what terms we use the code that we use, and if the licenses aren't compatible, we
can't use it.

He goes on and on about how foolish the SystemTap people are for trying to reimplement DTrace
when DTrace already exists, while refusing to even discuss the SOLE reason why they saw the
need to bother!

I didn't say that the issues are unimportant, I said that I didn't want to discuss them in the
comments section of my blog (which I don't).  I also happen to think that (1) the licensing
issues are largely a smokescreen for good 'ole NIH in the Linux community and (2) the law
(that is, the case law) in the domain of software licensing is so ill-defined that one can
often conclude what one wishes to conclude.  Take these two together, and you have a plague of
amateur legal opinions, contorted to justify rampant, metastasized NIH -- and I personally
want no part of it.

A Sun developer accusing the Linux community of NIH. Great.

Eben Moglen is not an amateur.  And he vets the FSF's legal opinions on license compatibility,
so...if they say it's not compatible, it probably isn't.

As for NIH, well, no, Linux has often adopted ideas from other systems.

Licences being incompatible is a moot, philosophical matter (least until the *AA succeed in
making copyright infringment a criminal matter) if no one thinks the infringment matters. Who
exactly is going to sue if someone distributes a DTrace modified kernel? Sun would have no
standing, as the CDDL seems to allow combining with GPL works. 

So it'd have to be a Linux kernel contributor, sueing someone for distributing free software
derived from free software.

But it isn't a "moot, philosophical matter". As one of the people who worked for Sun on the
CDDL said the license was explicitly designed to be GPL incompatible on request of the
engineers: http://en.wikipedia.org/wiki/CDDL#cite_note-5

It is simple to make this matter really moot if Sun would want to. All they have to do is
change the CDDL so that there is an explicit permission to distribute a larger combined work
under terms compatible with the GPL.

Since there are records of the incompatibility being deliberate it isn't even just a legal
matter. It would actually be rude and impolite to go against the wishes of the copyright
holder at this time. You do have to respect their wishes even if you could find some legal
loophole to go against it.

But if they really change their minds and put that in writing by updating the CDDL that would
be wonderful of course. Lets hope they mean it and indeed make this a legally mood matter for
real. And then we can all happily work together without having to have any doubts about it.

Look, the CDDL appears to be absolutely fine with copyright holders of GPL works incorporating
CDDL software and distributing it. All the GPL copyright holders have to do is promise not to
sue themselves or others for it. Pretty simple. (Consult your own counsel, of course!).

Essentially, you're complaining that Sun released free-software under a licence not quite to
your liking. Further, given that Sun legal had some involvement in the GPLv3 process, it's
possible that even if Sun had chosen the GPL, Sun'd have chosen v3 (course, v3 took another
few years to be finalised..), and people would still be complaining.

Further, in good humour, you're speculating yourself toward conspiracy-theory kook land. Read
what Danese said, she said "partially because". It's therefore entirely reasonable to think
there were other factors involved. E.g. one of the major differences between the CDDL and the
GPLv2 concerns patent rights and Sun is subject to patent infringment suits on a
not-infrequent basis. It seems reasonable to put and two and two together and speculate that
Suns' lawyers really wanted a licence with some kind of patent cross-licensing language. I.e.
it's possible GPL-licensing simply would not have been acceptable, regardless of what you take
from Danese's comment.

NB: I work for Sun, but I was no more privy to the "which licence for opensolaris?"
discussions than you were (indeed, they would have predated my employment largely ;) ).

No, I am not complaining about which free software licenses Sun choose. I am happy they
release software under a free license. And I hope they will do more of it. However pretending
that legal uncertainty and explicitly distributing source code under GPL-incompatible terms
aren't a (the) major issue, while claiming to want wider adoption of that code base under
GNU/Linux and suggesting it isn't adopted because of NIH is just silly.

I have worked on several occasions with Sun, and I have spoken on several occasions to Danese
and other people at Sun about free software, CDDL, GPL,  communities and legal issues to know
they really understand that the legal uncertainty around the CDDL is the only issue preventing
adoption of the technology on a wider scale in GNU/Linux.

As GNU Classpath maintainer I have been more than impressed with how Sun handled Java the last
couple of years. The GNU and Sun java communities happily work together now (after 15 years of
separation) exactly because Sun made sure to do the right thing legally to ensure the widest
possible adoption and cooperation with the existing libre-java communities. I have only praise
for how that was handled. And that is really the reason I am so disappointed about the mess
they made with dtrace/cddl preventing cooperation and adoption on a wider scale, while
claiming to want cooperation that is impossible to achieve because of these messy legal
issues.

Well, Danese left Sun quite a few years ago. Also, be sure to read Simon Phipps response
(linked from the Wiki article you linked to).

I accept there are differences and difficulties. However, it is Suns' prerogative to define
how the software it created is to be derived from. Again though, Suns' licence does *not*
(AIUI+IANAL) prevent other OSes from re-using that code - not even Linux (even if it does
require collective action from Linux copyright holders). Reading the tea-leaves to find
evidence of Suns' "evil" intentions is however not at all productive, no matter whose side
you're on. Then projecting a moral duty of having to respect these *divined intentions* is,
again with all due respect, not very sensible.

If there are problems on some side, then the only way understanding can increase and so
solutions (if any) are ever be found is through dialogue.

You have it backwards. 

It's not that the GPL is incompatible with the CDDL... The CDDL was designed, specifically, to
be incompatible with the GPL. Even if all the Linux kernel developers decided that CDDL was
acceptable, then it still wouldn't matter. Sun _doesn't_want_it's_code_in_the_linux_kernel_. 

That's it. They don't want Dtrace to be in Linux. They don't want ZFS in Linux. Get it? They
did it on purpose.  So all the philosophical or political are the arguments that are MOOT. Sun
made sure of that. Sun says their code isn't going into Linux, then their code isn't going in.
There is absolutely no question about this.

Your the only one mentioning 'evil' here or anything like that. You can see their intentions
very clearly written in between the lines in the CDDL license even if you don't believe
Danese. All the legal ramifications have already been worked out years in advanced for
Mozilla-like licenses and Sun could of easily made it compatible with Linux, but they chose
not to do that. EASILY. No question about that at all. 

Most everybody, unless they are loonies, is happy that Sun has their code under a open
license, even if it's intentionally incompatible with the (pre-existing and very well
established) GPL license.

They may change their mind in the future and make the license compatible, but until then it's
not going in. 

You see, that's how you know Sun wouldn't mind having it's code in Linux. If they made the
license compatible then you'd know it was ok. Everything else is handwaving.

> what you think the CDDL was designed specifically for, is ... largely irrelevant to anyone
considering porting CDDL code.

Do you seriously believe this?

Do you agree that the CDDL is incompatible with the GPLv2?  If so, then don't you think that
someone considering porting CDDL code to a GPL project had better consider whether his actions
are even legal?  That hardly sounds irrelevant!

If there's estoppel that would allow the CDDL to be used with the GPL, do you suppose you
share a link?  Every communication from Sun that I've seen has very specifically avoided this.

If Sun actually does want its CDDL code to be incorporated into GPLv2 projects, it would take
only tiny changes.  Since the GPL is older and more popular (http://freshmeat.net/stats/), one
would expect that Sun carefully considered researched GPL compatibility.  Even if the
incompatibility is an honest mistake, then why doesn't Sun make the tiny changes required now
to rectify the situation?

> Sun has published its code under the CDDL. It's a free software licence that's quite
permissive. At least one other OS (if not two) has incorporated code (the crown jewels no
less) from Solaris, by way of the CDDL licence. Take that + whatever statements you want from
its officers that seem interpretive or estopping and ask your lawyer for an opinion.

They don't care if FreeBSD or OS X adopts ZFS or Dtrace because those operating systems are
irrelevant and are not competitive.  Sun doesn't care if OS X adopts ZFS because they won't do
anything interesting with it. At most it will make OS X users feel better because they can
finally have a file system that doesn't suck worse then Vfat.

If Sun has it's software adopted by the Linux folks then they lose control over it. They will
improve it and incorporate it into Linux and it will be yet another reason why nobody has a
reason to choose Solaris over Linux. 

Linux is now the dominate Unix system out there. Sun wants to maintain control over their
'ip'. They've done that with most everything they've released.  They are learning to let go,
but it's slow coming.


Then explain to me why the hell Sun has not extended it's patent protections (use our IP and
we won't sue you) only to the users of CDDL-licensed code and not open source programmers and
users in general? Other people have done this (Redhat, Novell (not talking about the MS-Novell
agreement, but previous things they've done), it's certainly possible.


> What you feel Sun wants, or what you think the CDDL was designed specifically for, is
amazingly speculative at best and largely irrelevant to anyone considering porting CDDL code. 

_read_the_license_.  The license expresses intent. That's the point of it.

70%-90% of open source software used in a commercial environment is going to be licensed GPL.
Your telling me that Sun made CDDL problematic to be used in the majority of popular open
source software by ACCIDENT?

All they had to do was make a exception. They can still do it now. Mozilla did it. 

They took and copied a license that everybody, including the original authors (Mozilla),
realizes is problematic when combined with other open source software and Sun didn't realize
it?

You're the second person to have misread my (I thought fairly clear) comment, so I guess I
need to reply and clarift: I specifically said that *your* opinion of Suns' motivations in
choosing the CDDL was largely irrelevant (not least because motivations, even when generally
acknowledged, aren't that relevant).

Next (sorry to restate, but its important to the coherence of this post) you have been given
several reasonable, practical and good reasons why the GPLv2 would not have been acceptable to
Sun, regardless of Suns' motivations towards Linux, namely:

- Sun did not want to dictate to ISVs that they must GPL their code. (This one you have heard
directly from Bryan)

- Sun does not want licence-forking (to cover any rebuttal to the previous point of "so use
LGPL", as the LGPL can be converted to GPL, and ignoring the fact that the LGPL is far from
pretty)

- The GPLv2 is out of date with regard to patents. That Sun added patent cross-licensing+MAD
terms to CDDL suggests this may have been a factor. (Why it applies only to CDDL? I've no
idea.. I suspect there may be legal, technical difficulties in drafting the language to have
many other licences - particularly with regard to the 'MAD' aspect of the CDDL patent
language. Can you provide a pointer to the grants you refer to? Be interesting to read).

Next, ignoring the above and if we accept your argument: Exactly how did you develop this
massive sense of entitlement that you think it is your automatic right to Suns' code on
licensing terms favourable to your chosen OS? I'm a Free Software Foundation supporter and I
always understood that the ethical reasoning for companies to choose the free software was out
of respect for their *users*...

Finally, can you tell me with a straight face that had Sun chosen the GPL but it had chosen
the GPLv3 (imagining the GPLv3 had existed), that you'd then not be complaining here today
about Suns' choice of licence? Even though it too is incompatible with Linux to the *same
extent* as the CDDL is (namely: it'd require linux copyright holders to agree to update the
Linux not-quite-GPLv2 licence)?

You have deliberately omitted the statements immediately following the ones you've quoted
which qualify those quotes. The first quote is particularly out of context, as the qualifying
statement is in the same sentence: "... if no one thinks the infringment matters.".

That's quite trollsome behaviour, something I would not have expected from an LWN subscriber.

Ignoring your attacks, myself and some others here obviously think infringement matters.
That's kind of the point... It's also ironic that you've gone out of your way to condemn others
for thinking they know why Sun chooses the licenses it does, yet you consider it a deliberate
attempt to troll when someone points out what appears to be inconsistency in your logic.

By all means, disagree with my point if I am wrong but don't pretend you're taking the high
road.

Be careful using the notion of "Sun" as a person which can have intentions and plans and so
forth.  Even though the people who chose CDDL did so deliberately in order to avoid their
technologies being added into Linux (according to Danese Cooper's public comments and other
private communications that I've had with other Sun employees), that doesn't mean that Bryan
Cantrill so chose, or even that he is aware of or believes that it was so chosen.

Anyway, Bryan Cantrill's knowledge, opinions, and intentions are necessarily different than
those of other Sun employees, and neither is he wholly responsible for what the company does
nor is the company wholly responsible for what he does.

There is a sense in which companies cannot actually have intentions.  Fortunately, in our
society we allow them to enter into binding agreements, such as by publishing their software
under an open source licence, so that we no longer have to form a mental model of the
company's intention in order to know what we can do with their source code.

P.S.  I'm glad that free and open Solaris exists, as well as other free and open operating
systems such as the BSD's and apparently eventually Symbian.  If every computer in the world
ran Linux, we would all be poorer for it -- monoculture leads to stagnation and fragility.
Diversity and competition is the fount of creativity and excellence.

Brian, if you want DTrace to be part of Linux, just dual-license it.  You can release your
code under "CDDL or GPL" and then it can happily become part of Linux.  I, for one, would
welcome that.

FreeBSD, MacOS X, QNX...  So all we have to do is relicense the Linux kernel under BSD?  Well,
why didn't you say so!  That sounds so easy!

Perhaps you're interpreting a specific licensing incompatibility as a lack of desire from the
Linux community?  I can assure you, the community as a whole really does want to integrate
DTrace.  A lot!  TONS.  It would happen in a matter of weeks if only the licenses allowed it
to be possible.

Now there's some big talk!  Do you have any idea of the technical details involved in this, or
are you one of the ESR groupies who believes that given "enough eyeballs" pixies and fairies
magically solve all problems?  I can assure you from working with Paul that the Linux
idiosyncrasies are making the DTrace port rougher going than most -- and porting DTrace is a
deeply technical endeavor on the best of days.  So it would not show up in a "matter of
weeks", even if it were GPLv2 and Torvalds himself were doing the port.  (Sacrilege, I know.)

I was careless and I'm sure you're right.  What I *should* have said is, "lots of Linux devs
would spend a lot of their time incorporating DTrace, if only the licenses allowed it."

But my point remains: you seem to be interpreting license problems as a lack of desire.  The
desire is there, no question!  If the licensing problems could just be fixed somehow, I'm
confident the Linux camp would devote amazing hours to DTrace.

As it is, they have to devote amazing hours to replicating DTrace.  :(

Wow, more disingenouousness.  I'm impressed.

> know that many of your users would like to see you embrace the technology, and happen to
care much more about solving their business problems than about legal arguments about why it's
"impossible"

It works the other way around too. Many people would like to use solaris but can't since the
drivers are missing and care more about solving their business problems than legal arguments
why it's "impossible" to solaris to copy those drivers from Linux to solaris.

And to remind you, this problem was not created by Linux community or GPLv2, but by SUN. The
more paranoid side of community believes the problem was created on purpose.

In a hindsight it might seem it's not a problem to sun's customers, since Sun is surely
capable of rewriting (who was it again calling NiH?) all drivers needed to run the Sun servers
being sold, even if there already is a working GPLv2 driver. However, even Sun systems have
these USB/PCI-express expansion slots, where many people would like to attach interesting
hardware...

>but know that many of your users would like to see you embrace the 
>technology, and happen to care much more about solving their business 
>problems than about legal arguments about why it's "impossible"...

I'm sure it wasn't what you intended, but the tone of that comment makes it appear (at least
to me) like you're suggesting legal restrictions don't matter. Could you clarify that
slightly...?

To be honest, I'm having some difficulty with your post altogether. The suggestion that the
Linux developers are being deliberately obtuse in not integrating software which legally they
cannot integrate is, um, curious. Or do Sun not bother with such restrictions? ;-)

(Joking, I'm joking...)

Pedantically, it's a Linux kernel copyright holder rather than a Linux kernel contributor.
Some portions of code have been lifted from other projects, and some portions of code have
been signed away to other organisations. But to the extent that I think it's unlikely that
anyone would sue, I agree. "Unlikely" probably isn't good enough for the large companies that
could be potential targets, though, and as a result it's a practical matter as well as a
philosophical one.

As an example, SCO own copyright to various pieces of Linux. Microsoft could buy one of the
small companies that owns various files. Stepping into obvious legal uncertainty like this
would allow for the generation of huge quantities of FUD ("Linux violates its own copyright
license. Do you really trust that it doesn't violate others?"), DMCA takedowns and lawsuits.
Paranoid? Yes. But no major Linux vendor is going to leave themselves open to that for the
sake of a single feature, no matter how attractive it is.

Even worse: Apple might buy all the copyrights from Linus tommorow to all of his works and
relicense them as proprietary (let's make it even worse: CDDL ;-) ). The sky will fall, right?

Not quite.

http://www.fsf.org/licensing/licenses/gpl-faq.html#CanDev...

Any other legal uncertainty?

You've missed the point. If Microsoft (say) end up holding the copyright on part of the
kernel, they get to enforce that copyright. If they interpret the GPL as preventing the
incorporation of the CDDLed dtrace then they get to sue everyone distributing the Linux/dtrace
combination. With that as a possibility, I just don't see any major Linux vendor taking the
risk.

If it was already published as GPL, then they cannot change that. (Otherwise they would have
done so already). That's the nice thing about free software.

BTW: look at the Tentacles of Evil test in
http://en.wikipedia.org/wiki/Debian_Free_Software_Guideli...
. We try to assume the worst. We try to assume that the copyright owners
will suddenly become a second Caldera/SCO. The software must remain usage,
distributable, modifiable etc. even in that case. 

If a software fails that test it won't make it into Debian.

In fact, there are those who claim that the CDDL does not qualify because it fails exactly
that test (due to the choice of venue clause).

No, you're still missing the point. For the sake of argument, we'll assume that using the
CDDLed dtrace code in Linux is a violation of the GPL[1]. Let's also assume that none of the
Linux copyright holders are going to sue anyone who distributes this infringing combined work.
Now let's imagine that Microsoft buy a small Linux contracting company that holds the
copyright for part of the Linux kernel. Microsoft now get to enforce the GPL against anyone
shipping Linux in a way that violates it, and so file takedown notices against Red Hat and
Novell. The only defence these companies would have is "It's not a derivative work of the
kernel", which runs counter to arguments that they've made in the past. Do you really think
anyone wants to be open to that situation?

[1] As Bryan points out, this is a grey area. But I don't see any of the major Linux vendors
being keen on being the first to have that tested in court.

You have completely ignored the "if the Linux copyright holders agree" condition repeated
several times across, at least, my comments.

It is entirely within the power of the copyright holders to define the licence used, including
adding their exceptions to the GPL, such as "incorporating CDDL code is ok". My understanding
is that the Linux licence has even been modified in the past, and unilaterally at that!

We could imagine otherwise, sure. We could also imagine Jonathan Schwartz as He-Man battling
it out with Linus as Skeletor over who is master of Greyskull*... (feel free to mentally
reverse the roles according to preference).

* NB: humour..

Some of the Linux copyright holders are dead. Some have merely vanished. One of them's a
company that tried to kill Linux a few years ago. In some cases we probably don't even have a
solid idea who the copyright holder is. Changing the license isn't a real possibility.

(The last time the license changed was in 1992, when it probably was possible to identify
everyone who'd contributed any code to Linux using a single person's fingers. Since then Linus
has clarified what his interpretation of the license is at various points, but these
clarifications aren't binding)

ISTR some kind of brouha, discussed on LWN I think, not so long ago about Linus changing the
Linux copyright to exclude the possibility of upgrading its GPLv2 to GPLv3 (it was claimed he
hadn't actually changed the licence, but that wasn't accepted all? I dont quite remember).

Huh, nm. You covered the GPLv2 clarification. My bad for not reading.

Oh.. dead copyright holders either have successors in interest, or else they don't matter, is
what I suspect a lawyer would tell me.

Indeed, my understanding is that the copyright will generally end up with whoever holds the
estate. That doesn't make it easy to track them down.

I bet there are ways to solve this problem, such as publicising a proposed change and asking
for objections. Perhaps the law has already dealt with cases where some minority of copyright
holders in a collective work can't be found and/or don't take an interest...

It's not Suns' problem though..

I agree - it's unfortunate for Linux that Sun chose a license that the GPL is incompatible
with, but they were entirely within their rights to do so. My only objection is to the
repeatedly raised "Linux people hate dtrace for irrational reasons" type claims. Linux vendors
feel they can't ship dtrace for justifiable legal reasons, which means that there's little
incentive to work on the technical details. Whatever NIH tendencies the Linux community may
have, they're not the reason for ignoring dtrace.

No, it's not Sun's problem.  Unfortunately, it's not Linux's problem either -- the GPLv2 was
written 15 years before the CDDL.

It's a mutual problem.

The Linux team is unable to modify the Linux Kernel's license, and the DTrace team sounds
quite unwilling to amend/modify/dual license DTrace, so I guess we're at an impasse.  Can
anything be done?

I'm sure we could find licences which predate the GPL which also would *not* have served
OpenSolaris. E.g. Solaris engineering has a fairly strong BSD background and you can bet that
licence was at least mentioned..

So, sorry, that's just a daft argument.

Hm, I don't understand.  What is a daft argument?  That it's a mutual problem?  That combining
CDDL and GPL code appears to be at an impasse?

There are probably over 1000 Linux copyright holders. Getting 100% agreement out of 1000
people is rare if not impossible. One of the bonuses Sun can say as a business argument is
that OpenSolaris is 100% theirs and they can do whatever they want when they want without
having to get such agreement. 

Now we've drifted into the absurd.  What are the supposed damages in this hypothetical case?
In the historical conflict between open and closed source, the case for damages is clear:  if
Hacker's GPL code is swiped by Meglocorp and ships as a (closed-source) component in a
Meglocorp product, the revenues from that product are (arguably) damaging Hacker.  But when
all of the components are open source, I don't really see the case for damages -- how could a
Linux copyright holder possibly be harmed by the presence of another open source component in
the same address space?  This is not a theoretical point, by the way:  if there are no
damages, there is no tort.

In terms of the law, I think that the GPL is in a dimly lit area to begin with -- and the
issue of free software license incompatibility extinguishes whatever light remains.  So
strictly in terms of case law, no, it doesn't exist. Not that this is desirable, of course; I
think software desperately needs legal precedent on the linking issue in particular.  If (as I
personally believe) dynamic linking does not create a derived work, there is no real
"incompatibility" issue to speak of, and we can stop wasting our collective time huffing and
puffing about it.  Unfortunately, it is in nearly everyone's best interests (FSF, companies,
lawyers) for this issue to stay murky -- and I expect it to remain so for the foreseeable
future...

The DMCA allows for damages to be set at the profit realised as a result of the infringement
(504(b)). An injunction against Red Hat or Novell would almost certainly be rather more
crippling for them than the actual monetary damages.

> if there are no damages, there is no tort.

Erm, it's fairly easy to get injunctive relief and attorney's fees even without damages.

On top of this, there are plenty of reasons to sue other than tort.  Unfair competition,
deceptive business practices, breach of contract, etc.  Any of these might apply.

More generally, while article 3 of the constitution leaves this open for interpretation, a few
of the recent Supreme Court decisions have allowed large suits with some awfully vague damages
(Mass v. EPA).  And many states have their own laws, like CA's recent prop 64, that
essentially do allow tort without damages in certain circumstances.

So, depending the state where the lawsuit is brought and the current makeup of the Supreme
Court, your statement ranges from somewhat incorrect to completely false.

So, just because two entites are making no profits, that doesn't mean they can wantonly
violate their mutual licenses and contracts.  Now *that* would be absurd!

So who is going to sue whom on what grounds? Say Novell ships a Linux-DTrace and SCO file for
injunction relief based on their copyright interest in Linux (see mjg's comments). What are
the grounds exactly?

If you really want to find reasons why you can't port DTrace, knock yourself out. Seems a
productive use of energy..

The argument would be that any Linux implementation of dtrace is going to end up being a
derived work of the Linux kernel, and therefore would have to be available under the terms of
the GPL. The CDDL includes restrictions not present in the GPL, making it impossible to
satisfy both licenses simultaneously. Shipping the combined work would therefore be a
violation of the GPL. In the absence of the GPL there's no further permission to distribute
the Linux (and Linux derived) code, and therefore doing so constitutes a copyright
infringement.

Various people have various opinions on the validity of that argument. I'm aware of various
legal opinions that have been professionally offered. To the best of my knowledge, though,
there's no especially useful case law and so it's difficult to know which way the courts would
go. People tend to err on the side of caution when the potential cost (injunctions against
distributing their primary product) outweigh the potential benefit (a single, even if useful,
feature).

> So who is going to sue whom

Doesn't matter.

> on what grounds?

From my previous post: unfair competition, deceptive business practices, breach of contract,
etc.  Depends on the circumstances and parties.

> If you really want to find reasons why you can't port DTrace, knock yourself out.

Nobody's trying to create reasons not to port DTrace if that's what you're implying.  But
here's the thing: if it isn't clearly legal and moral to snake someone else's code, then it's
only considerate and safe to assume that you can't!

I've been searching for reasons to believe that they're compatible but I haven't found any
clear legal statement from Sun or anyone else, nor even any tiny nugget of estoppel.  Did Sun
ever produce the Devleoper's Bill of Rights?
http://www.cbronline.com/article_news.asp?guid=3B8EDD57-1...

I showed my engineer-attemptls-law post to a friend and she said:

> Sometimes when you sue for breach of contract, you can be awarded restitutionary damages
which means that even if you can't prove that you have been damaged, you can be awarded money
equivalent to the other party's "ill gotten gain".  So if the other party unfairly got some
advantage that was, for lack of a better term "unfair", you could recover their gain just
because it is unfair for them to keep it.

> That's one reason why it is dangerous for profitable companies to use open source code.  You
use code in a way that breaks your contract with the programmer ... that programmer could
recover your "ill gotten gain".

Wow.  Now I'm curious if this has been used in any of the GPL violation cases.  I'll have to
watch them a little closer!

You (or rather your friend) is making my point:  the restitutionary damages that she mentions
are the case of Meglocorp shipping GPL'd goober in proprietary product, and Hacker suing for
damages.  That's not the case that was being floated; the assertion was that Meglocorp would
become a Linux copyright holder and then could sue when CDDL'd source was distributed with
the(ir) GPL'd source.  I don't see what the damages are in this case -- compensatory or
restitutionary.  I think if it's flipped around -- Meglocorp ships GPL'd+CDDL'd goober, Hacker
sues because GPL has been violated -- one at least has a case, but I happen to think it's an
extraordinarily weak case: it relies on both an expansive definition of a derived work, and a
creative approach to damages.

Nobody knows how far the derived work clause of the GPL would actually reach. Staking your
future business to a legal gamble isn't a move many people are willing to make without a large
resulting benefit, and dtrace simply isn't perceived as a sufficiently large benefit. It's
perfectly rational reasoning.

> Meglocorp ships GPL'd+CDDL'd goober, Hacker sues because GPL has been violated

Agreed.  This is the only one that seems plausible.  But proving damages seems to be rather
easy nowadays.  And just about anybody involved with Meglocorp could sue; it doesn't have to
be a Linux kernel hacker.

Here's the big-picture problem as I see it: after the SCO disaster, the Linux kernel devs will
not incorporate any code of questionable license status into the kernel.  They want to keep
the entire tree completely unencumbered by legal issues.  It was a good lesson at a good time.

So, until it can be demonstrated that CDDL and GPL code absolutely can be mixed, they won't
accept CDDL-licensed code into the kernel.  it's just too risky.

As far as derived work, while I agree that those parts of the GPL are horribly murky and
untested, I don't think it's very relevant here...  Even if someone *could* claim that DTrace
is a derived work (a stretch!), as long as the CDDL+GPL combination is "licensed as a whole at
no charge to all third parties under the terms of [the GPL]" everything is good.  If the CDDL
and the GPL are compatible, the terms of the GPL are met, and the whole derived work issue is
moot.  (Right?)

To make CDDL+GPL compatible, IIUC the patent issues just need to be resolved the way Mozilla
did with the MPL.  Everything else in the CDDL and the GPL is compatible today.  (I'm no
expert though so I'd sure value someone else's opinion.  That almost sounds too easy!)

It would be nice to see someone reject license prejudices and try to maintain an out-of-tree
DTrace-Linux but...  can you imagine?  The git-rebases would make the even the heartiest
programmer cower in fear!

Yes, the authors did intend it to be a license -- but the attorneys that I have spoken with on
this issue believe that the expansive views of the GPL require it to be interpreted as a
contract.  Again, some case law here would be quite helpful...

http://lwn.net/Articles/61292/

That's a relief, although it does diminish the chances of me finding my code in the Google
Toolbar and making a quick hundred million.  :)

Various parts of Sun went at each other with iron tongs when SunView was looked at being
opensourced back in the early 1990's. It is just harder to sue each other inside of a
corporation. [I think parts of IBM tried that in the 1960's.]

NIH: Exactly what technology from Linux kernel has Solaris adopted instead of re-implementing
it yourself?
License: If the licensing is so unclear, how come there is no GPL code mixed into Solaris? I'm
fairly certain it's not a smokescreen or amateur opinion, but something professional Sun
lawyers said would be best...

Don't get me wrong, I'm happy for all the things sun has contributed, but the amount of
licensing misery caused by inventing (NiH!) yet another license, CDDL, is very real.

@Jake, the article writer:
How can you mention SystemTap's lack of userspace tracing support when it does support this
feature? Roland McGrath of redhat is working on a new ptrace() named utrace[1]. Using utrace,
someone created a project named uprobes [2]. Using uprobes, tapsets including userspace
tracing were written at [3].

Roland made grumbles of submitting utrace for mainline again not so long ago. Once utrace is
in mainline we will soon see full userspace + kernelspace tracing support in SystemTap. The
systemtap + utrace userspace tracing support was put into RHEL5.2 beta [4].

So please get the facts straight. While it would be *REALLY NICE* if sun were to allow DTrace
in Linux by relicensing it gplv2, that doesn't look like it will happen.

So eventually...
Linux's answer to DTrace = SystemTap + uprobes
Linux's answer to ZFS = btrfs

As fast as things are developing we aren't a long way off. This isn't fud, only facts people.


[1] http://people.redhat.com/roland/utrace/2.6-current/
[2] http://ols.108.redhat.com/2007/Reprints/keniston-Reprint.pdf
[3] http://lwn.net/Articles/255429/
[4] http://www.redhat.com/archives/rhelv5-announce/2008-March... CTRL F SystemTap

> As fast as things are developing we aren't a long way off. This isn't 
fud, only facts people.

I don't want to be pessimistic, but Utrace seems still work in progress, 
only the very first patchset from it has gotten to upstream, it supports 
well only x86 & powerPC and rest of it's still being designed.  See the 
latest status report:
http://www.redhat.com/archives/utrace-devel/2008-July/mht...

I think it's at least a year until it's fully (API stabilized etc) in 
upstream kernel with Systemtap supporting it well, a good set of tapsets, 
useful scripts, documentation and some GUIs (Mortadelo and Stapgui lack a 
lot of things).

This is what Apple is already doing with DTrace:
http://developer.apple.com/documentation/developertools/C...

And before that their developers already had Shark (which AFAIK doesn't 
use DTrace):
http://developer.apple.com/tools/performance/optimizingwi...

Btw. the Tracing Summit in Canada earlier this year produced fairly good 
overview of the state of tracing tools currently (also elsewhere than on 
x86):
https://ltt.polymtl.ca/tracingwiki/index.php/TracingBook

Probably because the author know the state of the user tracing where DTrace support has
been/is being incorporated in many tools: Ruby, etc which simplify a lot the tracing by the
end-user..

Whereas currently SystemTap doesn't have this 'support' of all the enduser tools.

>we aren't a long way off.

Depends of your measurements! If you look at Ted Tso complaints, support of SystemTap by
endusers tools, the schedule, SystemTap appears to be very, very late compared to DTrace..

Personally, the DTrace thing looks like a clear case where the Linux development model fell
flat on its face. I first released the Linux Trace Toolkit back in July '99 -- close to 10
years ago now. From very early on I met a tremendous amount of resistance to it,  despite
having made clear that this was made for users (not kernel developers) to understand how their
system behaves. Maybe I didn't do a good job of communicating that view, and, hey, maybe even
the implementation wasn't the best out there (though the record would show that implementation
quality was hardly a reason for refusing to merge the code), but it remains that interest in
kernel tracing for *users* only came to the forefront and remains on the forefront for one
simple reason: the existence of DTrace. And that, it seems to me, is the wrong reason for
things to happen. If Linux (kernel and system) is to lead the pack in what it delivers to its
users, it must be able to give  functionality to users that other OSes can't. But as the
recent "Ubuntu girlfriend experiment" has shown, we're still trailing other OSes (DTrace is
but a symptom) and that needs to change.

Thanks, seems like LTTng is more than worth a look.

I find most of the comments to this article 'interesting' in that they debate the licensing
issues and who owns what.

Nearly everyone is missing the point. Whilst you all debate the merits of a license, nothing
much happens.

Dtrace is needed by real people in real production environments. Environments designed for
dtrace - production critical, and by people who dont know the kernel.

There are lots of issues with dtrace - which relate to competence of a user in a scenario
where they need or could use dtrace.

Does dtrace have to be in the kernel? Absolutely not. I dont know why people are hung up on
that issue. We dont put mplayer or firefox in the kernel, and dtrace doesnt need to be either.

By *not putting this in the kernel simply increases the work for the dtrace maintainer
(presently, me)* to handle the Linux development model.

When I buy an off the shelf product - I expect it to work, irrespective of my platform or
version of OS. Dtrace should be like that. That is the issue - can it be platform independent
to an end user. (The code may end up with horrid #ifdefs but the end user wont/shouldnt care).

The more i get into dtrace the more I am convinced of the above. I have not had to modify a
Linux kernel to make it work and have no intention in the future - the difficult bit (proving
it can be done) is over.

The next phase is looking at the end-user experience and looking for compatibility and
least-surprise.

paul fox