of the CVE database
turns up 80 CVE numbers related to kernel
vulnerabilities so far this year. At one recent conference or another,
while talking with a prominent kernel developer, your editor
confessed that he found that number to be discouragingly high. In an era
where there is clearly an increasing level of commercial, criminal, and
governmental interest in exploiting security holes, it would be hard to be doing
enough to avoid the creation of vulnerabilities. But, your editor
wondered, could we be doing more than we are? The response your editor got
was, in essence, that the
bulk of the holes being disclosed were ancient vulnerabilities which were
being discovered by new static analysis tools. In other words, we are
fixing security problems faster than we are creating them.
That sort of claim requires verification; it is also amenable to being
verified by a researcher with sufficient determination and pain
resistance. Your editor decided to give it a try. "All" that would be
required, after all, was to look at each vulnerability and figure out when
it was introduced. How hard could that be?
So, the basic process followed was this: pick a CVE entry, find the patch
which closed the hole, then dig through the repository history and other
resources in an attempt to figure out just when the problem was first
introduced into the kernel. In some cases, the answer was relatively easy
to find; others were sufficiently hard that your editor eventually gave up.
One especially valuable resource in the search turned out to be the Red Hat
bugzilla; the developers there (and Eugene Teo in particular) go out of
their way to document the particulars of vulnerabilities. Sometimes, the
commit which introduced the bug was simply listed there. The "git gui
blame" utility is also quite useful when doing this kind of research.
About 60 of the 80 vulnerabilities listed above were dealt with in this way
before your editor's eyes crossed permanently. The results can be seen in
the following table. Let it be said from the outset that there will
inevitably be some errors in the data below; the most likely mistake will
be assigning blame to a commit which actually just moved the vulnerability
from somewhere else. That may lead to a bias that makes vulnerabilities
look more recent than they really are. That said, a best effort has been
made, and things should not be too far off.
Some other notes relevant to the table:
- No attempt was made to find the origin of vulnerabilities which were
present in the initial commit which began the git era during the
2.6.12 development cycle. Anything which was already present then can
certainly be said to be an old bug.
- Some parts of the code have been changed so many times that it can be
truly hard to determine when a vulnerability was introduced; places
where your editor give up are marked as "unknown" above. One could
maybe come up with a real answer by bisecting and trying exploits, but
your editor's dedication to the task was not quite that strong.
- A couple of these bugs are old in a different way - CVE-2010-1188
was fixed in 2008, but was only understood to be a security issue in
2010. Anybody running a current kernel would not be vulnerable, but
bugs like this can be nicely preserved in enterprise kernels for many
Looking at when the vulnerabilities were introduced yields a chart like
So, in a sense, the above-mentioned kernel hacker was correct - an awful
lot of the vulnerabilities fixed over the last year predate the git era,
and are thus over five years
old. It seems that security bugs can lurk in the kernel for a very long
time before somebody stumbles across them - or, at least, before somebody
According to the information above, we have fixed dozens of vulnerabilities
since 2.6.33 without introducing any. The latter
part of that claim might be charitably described as being unlikely to stand
the test of time. There were (at least) 13 vulnerabilities fixed in the
2.6.35 cycle, 21 in the 2.6.36 cycle. We can hope that fewer
vulnerabilities were added in that time; it seems certain, though, that
(1) the number of vulnerabilities added will not be zero, and
(2) it will probably take us five years or more to find many of
There may be some comfort in knowing that a large proportion of 2010's known
security vulnerabilities are not a product of 2010's development. Indeed,
assuming that a fair number of the old vulnerabilities are a bit older yet,
one can also claim that they are not a product of the "new" kernel
development model adopted in the early 2.6 days. That claim
could be tested by extending this research back into the BitKeeper era;
that is a task for a future project.
Your editor remains concerned, though, that it is too easy to put
insecure code into the kernel and too hard to discover the vulnerabilities
that are created. Analysis tools can help, but there really is no
substitute for painstaking and meticulous code review when it comes to
keeping vulnerabilities out of the kernel. At times, it is clear that the
amount of review being done is not what it should be.
There may well come a day when we'll wish we had found a
way to be a bit more careful.
to post comments)