Though Harald Welte's contributions to the free software community are
many, the work he is best known for may well be the gpl-violations.org
pursuing those who ship his code (and that of others he represents) without
complying with the source requirements of the GPL, Harald has secured the
release of much code into the community, established a precedent upholding
the GPL in German court, and greatly increased the respect many companies
have for the GPL. Thanks to Harald, the GPL has some teeth.
Back in February, Harald complained that
the number of non-compliant products on the market was exploding, and that
he did not have the time to deal with them all. He suggested that the
time was right to incorporate gpl-violations.org into a nonprofit
organization which could pursue violators while allowing Harald to get back
to hacking. Those plans are moving forward, with the possibility that this
new organization could be created by August, and operating by the end of
the year. We were recently able to talk with Harald about this effort; so,
without further ado...
LWN: How many GPL violations have you found over the last year? How
many of those have been brought to some sort of resolution?
There were 158 reports during the last 12 months, of which about 100
were real violations, about 50 have been addressed, and 40 of them
resolved, others are still going on.
The difference between 'reported real violations' and 'addressed
violations' is due to:
- lack of time
- bad enforcement prospect due to difficult jurisdiction and no sale
in EU countries
Up to today, since the project was started, there was not a single
legally unsuccessful enforcement. By legally unsuccessful I want to
say that legally those formerly infringing companies are now clear.
However, a small number (about 3) have decided to withdraw the product
from the market rather than releasing source code. So those cases,
while legally successful, haven't been successful with regard to the
ideas of Free Software.
You seem to be unique in carrying out this effort. Do you know of
others who have been digging out GPL violations (in the kernel or
There are two 'others' that I'm aware of: The FSF in the US, where
David Turner from the FSF compliance lab is enforcing the GPL
(out-of-court) for software that the FSF holds copyright.
The other one is MySQL, which only enforces the GPL on their DB
software in order to motivate people to buy alternative licenses. It
still is GPL enforcement, though ;)
The FSF has a "GPL Compliance Lab" which only rarely draws attention
to itself. Rather than incorporating a separate nonprofit, might
there be an advantage in folding this effort into the work the FSF
does? Why, or why not?
There are a number of reasons. First, the FSF only enforces (and can
only enforce) the GPL on software which they hold copyright on. So
joining efforts with the FSF GPL Compliance Lab would also mean that
I (and other copyright holders that I represent) would have to transfer
their rights to the FSF.
Secondly, the FSF has a quite different enforcement strategy. They are
doing enforcement in a "softer" way, meaning that they don't pull as
many legal strings as gpl-violations.org does. This difference is
partly due to a difference in the US / German legal system and legal
culture, but also intentional. My whole reason for starting
gpl-violations.org was that I think a different strategy is more helpful
in the end, since publicizing GPL violations will actually prevent new
Third, the FSF is based in the US, whereas gpl-violations.org is based
in Germany. There are many legal differences in copyright law, and also
many differences in the kind of companies we can take action against in
our local jurisdiction.
Having said that, I can assure you that there is a very friendly
cooperation between the FSF GPL Compliance Lab and gpl-violations.org.
We're passing on cases between each other, sometimes get active
independently in the same violation and share information, etc.
Would you be seeking funding to get this operation off the ground?
What sort of individual or company, do you think, might be interested
in funding this effort?
Obviously some initial funding would help to get moving more quickly.
However, I don't think it will be required for making it work.
As for your second question, I think a lot of individuals, both
developers and users within the Free Software community, are very
sympathetic to what gpl-violations.org does. I think some of them were
willing to show their support by donating. However, I've discouraged
them from doing it so far, since they would basically donate 'to me',
and I would have to treat it like regular income, i.e. pay taxes on it,
etc. Also, since there is no separate legal entity yet, there is no
public accountability, i.e. you cannot audit the books, verify that your
donation has only been spent in "the right way", etc.
As for companies, there also are companies supporting the work we do at
the project. I'm not sure whether I would be able to name them here,
but let's say companies who do oblige to the GPL and take it seriously,
and who think their competitors are gaining an illegal competitive
advantage by using GPL licensed software but not following the GPL.
Would you anticipate this effort being self-funding in the long term?
Yes, not only in the long-term. Looking at the rate of new violations
that we now have consistently for a number of years in the embedded
market, it should very much be possible to make it self-funding.
gpl-violations.org has been able to obtain various donations to
charitable organizations such as EDRi, FoeBuD, CCC, FSF Europe, Bridge
Foundation, ... during enforcement. Those donations are usually part of
a settlement that allows the respective vendor to sell already-produced
products (without a GPL license text or written offer) during a grace
So the idea is to redirect those donations (or at least part of it) to
the newly established gpl-violations.org organization. This way we can
hire somebody to take care of the administrative and paper work.
If that kind of self-funding stops for some time, then apparently we
don't have as many GPL violations anymore, and the purpose of
gpl-violations.org does no longer exist. That's the ideal case, and we
can suspend or even dissolve the organization :)
What do you think are the prospects of expanding the GPL compliance
work beyond Germany?
We're actually doing GPL enforcement outside Germany already. We have
been able to obtain declarations to cease and desist from a number of
formerly-violating companies in Taiwan and Korea, for example.
To the casual observer, it looks like the rate of GPL violations is
not decreasing - if anything, the opposite is happening. So far, the
community has been quite accommodating to those who violate the GPL,
being (for the most part) satisfied if the company involved brings
itself into compliance. Might it be that the risk involved with
violating the GPL is simply not high enough to deter people? Should
the community start seeking damages against GPL violators?
The absolute rate is definitely increasing. But you have to set this
in relation with the overall massive growth of the Linux embedded
market. I don't have any figures on this (and I doubt anyone can have
good figures), but I think that the percentage of Linux-using embedded
devices that ship out of compliance is decreasing, or at most: steady.
There are people suggesting that the penalty should be higher, and we
should seek damages. I think for 95% of all cases this would be the
wrong decision. The vast majority of GPL violations happens because
some Taiwanese or Korean OEM/ODM does something (sometimes even in clear
violation with the contract to their customer!) that the Vendor that
we're approaching isn't really aware of.
Also, most of the companies who once had a GPL problem actually have a
good record ever since. Yes, there are occasional "problem companies",
such as D-Link or Sitecom. But in general, I have the feeling they take
gpl-violations.org quite seriously.
If we start asking for huge amounts of damages and try to raise the bar,
then we will frighten vendors from using/buying embedded Linux at all.
I am definitely not in favor of Linux adoption without GPL compliance.
But we have to carefully draw the line between legally indicating that
we don't accept GPL compliance, and on the other hand not frightening
people who fear to make a mistake at some time from using Linux / GPL
licensed software at all.
Also, when you ask for (and actually get) damages, you have the problem
of what to do with it. Distributing it between all the authors is
virtually impossible, because in most cases the transaction fees will be
higher than whatever the individual developer will get. Donating it to
some organization? To which? Who decides on that? ...
As a summary: I think for now, gpl-violations.org draws that line at a
reasonable position. In the mid-term future that might be different,
and for individual cases I might share the view that higher penalties
are justified. But not in general.
Anything else you think a clueless LWN writer should know about this
What is most interesting about having some organization backing this
project, is that we can actually do "more interesting" legal action than
I can do now. So far, we've only enforced very clear cases, from a legal
point of view. Until now, gpl-violations.org has not helped to
produce any legal precedents on important questions such as derivative
works or binary-only kernel modules. However, after funding the
organization later this year, and thus the legal risk landing on that
organization rather than me personally, I could very much imagine that
we would look into getting some court decisions on that area, too. So
stay tuned, there is probably an exciting time ahead in the next couple
of years ;)
I would like to thank Armijn Hemel who is basically doing almost as much
work in gpl-violations.org than me these days, and I would like to thank
JBB Rechtsaenwaelte, the Law firm that has so far helped us win all the
cases we did :)
So do you anticipate taking an action based specifically on binary-only
I'm not planning anything concretely. But I expect sooner or later we
will face such an issue. And I think that matter needs clarification -
whether or not we (as in the Free Software enthusiasts) will like the
results. At least afterwards, there is some precedent either way, and a
much more clean situation for anybody doing software development in
mixed Free / proprietary environments.
Many thanks are due to Harald for taking the time to answer all of these
Comments (26 posted)
Back in March, your editor received some not-entirely-friendly
communications from a prominent OpenSSH developer. This person was unhappy
about a number of things found in the article about OpenBSD's
, as well as one thing that was absent: a discussion of
OpenSSH alternatives. The point which was supposed to emerge from such a
discussion is that there are
no viable alternatives. Your editor
has set out to try to determine if that is truly the situation or not. To
that end, this article will look at SSH server implementations; the client
side of the picture will be addressed in a future article.
There are a number of things one can look at while evaluating an SSH
server. Features, for example: which ciphers are supported, port
forwarding features, control over what users can do, PAM integration, etc.
One can also look at performance issues; data-heavy SSH sessions can put a
significant load on the host system. But the issue which must dominate the
others is security. An SSH server is designed to give access - perhaps
full, root access - to a suitably authorized user coming in from an
arbitrary location on the net. Any vulnerabilities in this server thus
have a high probability of turning into a full compromise of the system.
Evaluating security is hard. Certainly one can look at security-oriented
features found in a given implementation, and there will be useful
information there. But features do not make security; that requires
careful coding, extensive code review, and quick response to security
issues as they come up. It requires an active development community which
continually works to tighten the security of the server. An SSH server
which is the subject of a large number of security advisories would make
your editor nervous, but a server with a moribund mailing list and no
advisories at all would be worse.
With these thoughts in mind, your editor set out to play with the three SSH
server implementations he found which are free and under some sort of
Dropbear is an
SSH server and client implementation available under an MIT-style license.
It runs on just about every Unix-like system, including Cygwin. Dropbear
development places a strong emphasis on small size; it is intended for use
in embedded systems and other space-constrained situations. The current
version of Dropbear is 0.48.1, released on March 12, 2006.
As might be expected in a program which is meant to be small, Dropbear
offers fewer features than some others. It can perform X11 connection
forwarding (and port forwarding in general), and has options for
controlling whether password authentication may be used to log in. There
is no configuration file, however, and many of the options available with
certain other servers are not implemented in Dropbear.
Dropbear can do passwordless login using RSA or DSA keys. It understands
OpenSSH-style authorized_keys files, allowing the same keys to be
used with both servers. The key format for host keys is different,
however; a script is provided to convert OpenSSH keys into Dropbear's format if
needed. Dropbear can be configured to perform password authentication
through PAM, though one gets the sense that most installations don't
There is little information available on the ciphers supported by
Dropbear. A look at the code, however, shows options for AES-128, AES-256,
triple DES, Blowfish, Twofish-128 and Twofish-256.
Dropbear appears to have an active developer and user community. There is
a fairly long list of distributions listed as using Dropbear, including
OpenWRT, OpenZaurus, Trinux, and Motorola A780 phones. The volume on the
mailing list is steady but low - Dropbear users apparently have little to
talk about. The last publicly-acknowledged security issue was in March,
2006, when a denial of service problem (which also affected a wide variety
of other network servers) was fixed.
Prior to that, fully remotely exploitable format string vulnerability was
(and very quickly fixed) in 2003. Another remote vulnerability was disclosed in 2004 and yet
another was fixed
in early 2005. In December of 2005, a "buffer sizing error" which could
enable root access for authenticated users was fixed.
The code base is small - a little over 23,000 lines for both the server and
client - but not particularly well commented. The Dropbear code should be
relatively easy to audit; the extent to which anybody has done so is
Lsh comes billed as "a
GNU implementation of the secure shell protocols." So, unsurprisingly, it
is released under the GPL. Lsh provides both client and server
implementations. The current release of lsh is 2.0.3, from May 9,
The lshd server daemon, like Dropbear, lacks a configuration file;
it does have a number of command-line options for controlling options like
password authentication and port forwarding. There is support for
public-key authentication in lshd, but OpenSSH-format keys must be
converted into the lsh format first. The converted key must then be fed to
lsh-authorize before the server will recognize it. There does not
appear to be an lsh-unauthorize command, making it more
challenging than it should be to revoke access for a specific key.
Documentation for lsh is more complete than for Dropbear. From that
documentation, one sees that the supported ciphers are AES-256, triple-DES
(though it is listed as "3dec"), Blowfish, and ARCFOUR.
Disclosed vulnerabilities in lsh include a file
descriptor leak enabling a local denial of service attack (January,
2006), a denial of service
problem (March, 2005), and a remotely exploitable buffer
overflow (September, 2003). While lsh releases do continue to happen,
it is not clear how large the user and developer community really is. The
lsh mailing list is dominated by spam, with legitimate messages seemingly
being carried at a rate of less than one per month.
Lsh is written in C, but a look at the code gives the impression that the
author would rather be using something else. Some sort of preprocessor is
used on the code, a memory garbage collector has been implemented, there
appears to be some sort of exception mechanism in place, etc. As a whole,
the code is harder to read than the Dropbear code, and it is not clear that
this code has seen much attention from anybody other than its original
All told, your editor would hesitate before committing to lsh; it is far
from clear that this tool has the user and developer communities needed to
keep it alive and secure into the future.
OpenSSH is clearly the dominant offering
in this area. All available evidence indicates that almost every publicly
reachable SSH server is running OpenSSH. This implementation is maintained
by the OpenBSD developers; the current release is 4.3 (or 4.3p2 for systems
other than OpenBSD) from February, 2006.
If you are looking for features, OpenSSH is the way to go. The sshd_config
man page lists a vast number of options controlling authentication
mechanisms, ciphers used, user restrictions, file locations, port
forwarding, and more. The list of supported ciphers includes ARCFOUR,
blowfish, CAST, and several variants of AES. OpenSSH is clearly the most
feature-complete of the SSH server implementations; it is also, in many
ways, the best documented.
Vulnerabilities disclosed in OpenSSH include a root compromise in 2001
(but only when an obscure configuration option was set to a non-default
value), a set of
integer and buffer overflow vulnerabilities in 2002 which affected
relatively few sites, a remotely exploitable
heap corruption bug in 2003, an access restriction bypass
vulnerability in 2003, a remotely
exploitable PAM-related vulnerability in 2003 (non-default
configurations only). The nastiest of these will be the 2003 heap
corruption bug, which is thought by some to have been actively exploited
for some months prior to being fixed.
It would appear that no OpenSSH server vulnerabilities have emerged since
2003 (there has been one client-side vulnerability since then). As this
article is being written, there is some discussion on the OpenSSH list of a
number of bugs found by a Coverity scan. Fixes are in circulation, but
there does not appear to be much concern that these bugs are exploitable.
The OpenSSH developers clearly take security seriously. The code base is
probably the most heavily reviewed of the three implementations discussed
here. The OpenSSH server also has a "privilege separation" feature,
wherein the bulk of the protocol code (prior to the establishment of the
user's session) runs in a separate, unprivileged process. This mechanism
will, it is hoped, contain the damage should an exploitable vulnerability
turn up in that code in the future.
The handling of the 2002 integer and buffer overflow vulnerability raised
some eyebrows; the developers refused to disclose specifics on the
vulnerability, insisting, instead, that all users perform a significant
upgrade to the current release. They have made
it clear that they would do so again:
If there is ever a security problem (again :) in OpenSSH we will
disclose it exactly like we want, and in no other way, and quite
frankly since noone has ever paid a cent for it's development they
have nothing they can say about it. Dear non-paying user --
please remember your place.
The fact remains that the OpenSSH developers have earned a high level of
trust, and that most users are entirely happy in their place. The OpenSSH
mailing list is active, with a steady flow of questions (and patches) from
the user community.
OpenSSH is implemented with a significant amount of C code. The code base
is written for OpenBSD in particular; the version the rest of us use is the
"portable" release which has seen added tweaks to make it run elsewhere.
There is a set of regression tests packaged with the code as well.
Your editor began this project with the idea of determining whether there
are truly no alternatives to OpenSSH on the server side. Of the two
discussed here, only Dropbear looks even remotely viable. For
resource-constrained applications dropbear may even be the preferred
choice, but it can also be used in any other setting that does not require
the larger feature set of OpenSSH. As noted above, your editor is made a
little nervous by lsh, and would choose to avoid it.
There are two schools of thought on the OpenSSH monoculture - for that is
essentially what it is. Some, including your editor, find the situation a
little scary; a serious vulnerability in OpenSSH could be the opening
needed for a devastating Internet worm. To these people, some diversity in
the OpenSSH ecosystem could only help to make the net as a whole more
secure. Others, however, feel that we are better off with a single code
base which can benefit from the concentrated auditing and hardening efforts
of the entire development community.
One can only hope that there is some merit in the latter view, given that,
for most systems, OpenSSH is the only viable choice available.
Comments (32 posted)
It has been a while since we have posted on the status of LWN.
Now seems like a good time to catch up, especially since your editor is
traveling and can write this article ahead of time.
Subscriptions to LWN continue to grow, but that growth continues to be
slow. Very slow. Various schemes for improving the situation are in the
various complications have impeded the process. We are contemplating hiring
another editor to help to expand and improve the LWN content mix; those
plans remain vague at this time.
We are always looking for writers, however. To that end, we have raised
our (still inadequate) pay scales a bit. If you have something to say to
the community, and you are willing to write for demanding editors and even
more demanding readers, please have a look at the writing for LWN page and
Readers of the RSS feeds may have noticed some changes which have been made
there. It has (slowly) occurred to us that RSS seems to be the primary
interface to the site for many readers, and that maybe we should pay a bit
of attention to it. There is also a new feed which tracks the most
recently posted comments; anybody who is interested in tracking the LWN
discussion across the site is encouraged to subscribe. See the LWN headlines page for a full list
of available feeds; expect to see some others before too long.
Maybe someday we'll implement an Atom feed and be properly buzzword
compliant, but that is rather lower on the list of priorities.
When LWN first started allowing comment posting, some readers predicted that
one result might be the death of the "Letters to the Editor" page. Those
readers may well have been right; the Weekly Edition almost never includes
a Letters page anymore, because there are no letters to publish on it. So
we are considering just dropping that page altogether. The alternative,
for those who would like to see that page retained, would be to start
sending us letters.
Occasionally we get queries from people who would like to reuse content
published on LWN, often translated into other languages. We have never yet
refused such a request. We are still evolving a complete policy on
licensing of LWN content, but it will look something like this:
This policy is still under development; we're interested in any suggestions
or advice that anybody might have.
Finally, for those of you who will be at the Ottawa Linux Symposium this
year, LWN editor Jonathan Corbet will be talking on the state of Linux
kernel development. It is, at this point, almost as traditional as the
Black Thorn party.
Comments (37 posted)
Page editor: Rebecca Sobol
Next page: Security>>