LWN: Comments on "How old is our kernel?"

How were the moved/renamed files accounted for ?

nye — Tue, 02 Mar 2010 12:47:28 +0000

So -M (detect renames) seems from a user's point of view to act differently depending on whether you're trying to log a file or the whole project.

This seems like a good, specific example of one of those usability issues people are always handwaving about.

How were the moved/renamed files accounted for ?

jnareb — Tue, 02 Mar 2010 01:13:59 +0000

In my experience, 'git log -M' did never give the full log of a single file. I have now discovered that 'git log --follow' does.

Actually the problem is that in "git log -M filename" the filename part is path limiter, and is applied (for history simplification) before rename detection, and that is why you need "git log --follow filename". "git log -M" (no pathspec), or "git log -M directory" should work as expected.

Loose objects, packfile, deltification and the cost of disk space

jnareb — Tue, 02 Mar 2010 01:10:15 +0000

There was no deltification at all iirc Argument from Linus back in git early days was that disk space is cheap. Others come up with the whole xdelta pack thing later.

Actually packfiles and deltification (LibXDiff, not xdelta) was, from what I remember and understand, originally because of network bandwidth (which is much more costly than disk space), and I/O performance of using single mmapped file instead of very large number of loose objects.

How were the moved/renamed files accounted for ?

phdm — Mon, 01 Mar 2010 03:42:29 +0000

In my experience, 'git log -M' did never give the full log of a single file.
I have now discovered that 'git log --follow' does.

How were the moved/renamed files accounted for ?

dlang — Fri, 26 Feb 2010 18:31:15 +0000

I believe that there is a -M flag that tells these tools to follows moves and renames (and IIRC you can give it an argument to tell it how hard to look for moves and renames)

this can also track when parts of a file are copied to another file.

How were the moved/renamed files accounted for ?

phdm — Fri, 26 Feb 2010 16:33:26 +0000

I didn't know that 'git blame' did such a good job. Does 'git log' also do such a nice job now (following moves and renames) ?

Older trees

eli — Fri, 26 Feb 2010 14:44:45 +0000

Thank you, sir!

'Course, now I need to stare at it for an hour looking for all the interesting things it's trying to tell me. ;)

How were the moved/renamed files accounted for ?

corbet — Fri, 26 Feb 2010 14:28:20 +0000

Git tracks renames nicely and retains the history for the moved files. So no, a rename does not, alone, cause any lines in the file to be considered to be changed.

Blank lines

efexis — Fri, 26 Feb 2010 08:51:16 +0000

Unless the blank lines were proportionally evenly distributed, in which case the effect would simply be cancelled out, but even so... a character changed in a long line of code counts the same as the whole line changing, but it's enough to give a feel for what's going on, as long as it's understood that these types of stats serve to guide towards reality rather than define it.

How were the moved/renamed files accounted for ?

phdm — Fri, 26 Feb 2010 08:46:50 +0000

Just a question : did you count moved or renamed files as changed or unchanged lines ?

Older trees

robert_s — Fri, 26 Feb 2010 00:26:32 +0000

With this graph, sir, you have read my mind.

How old is our kernel?

ariveira — Thu, 25 Feb 2010 22:15:16 +0000

There was no deltification at all iirc

Argument from Linus back in git early days was that disk space
is cheap.

Others come up with the whole xdelta pack thing later.

Any chance of increasing the resolution?

KGranade — Thu, 25 Feb 2010 19:46:54 +0000

How difficult would it be to increase the resolution to -rc releases? I think it would be very interesting to see what kind of pattern there is for them, if any. I would assume that late rcs, while smaller would have better longevity percentage-wise than early rcs.

Changing Licence

Felix_the_Mac — Thu, 25 Feb 2010 00:54:28 +0000

Back when GPL3 was being worked on, and it was a (slightly) open question whether the kernel would adopt it, it seemed to me that this type of historical analysis of code history was exactly what was required.

With the ownership of each line of code traced to its author, and an agreed definition of what constitutes original work (rather than just a minor change to somebody else's work) it would be possible to determine who held rights in each file.

If, for example, the GPL2 became unenforceable in China or India, and it was decided that a license change to GPL2++ was required, then permission could be sought from identified license holders and the code belonging to those who were unavailable or unwilling to give assent could be rewritten.

Older trees

Aissen — Tue, 23 Feb 2010 00:34:24 +0000

Is your tree online? How does this compare to http://lwn.net/Articles/285366/ ?
I contacted the original author about 3 months ago and built a tree using his ocaml program. It seems to gather data from dave, tglx and linus' tree.

If anyone is interested, I can forward the ~210k archive of the program building the tree.

Older trees

nix — Mon, 22 Feb 2010 17:40:26 +0000

Oh, that's a lovely graph.

I wonder if the inflection point can be attributed to the staging tree?
(That's certainly a lot of new crap^Wcode to be deleted...)

Older trees

aegl — Mon, 22 Feb 2010 17:04:25 +0000

Here's the "strata" picture you asked for:

http://www.kernel.org/pub/linux/kernel/people/aegl/codest...

The lowest line is the 2.6.12-rc2 git origin. Count up from there
to 2.6.32 at the top. Scripts were run with current tip of "linus"
tree at v2.6.33-rc8-113-gf8b55f2 so it doesn't take into account the
320 lines of code added and 149 deleted over the weekend.

Visually there does seem to be an inflection point around 2.6.27
where we slowed down at deleting old code (perhaps because there
was so much new code to be deleted instead?)

Older trees

aegl — Fri, 19 Feb 2010 22:32:19 +0000

The current git history looks too short to tell whether the speed of code
removal follows a "half-life" curve. We still have 58.6% of the git origin
(2.6.12-rc2) code present in the current kernel (it only makes up 30.5% of
the current code because the kernel is almost twice as large now).

Here's a graph showing growth of the kernel, and decline of the original
code:

http://www.kernel.org/pub/linux/kernel/people/aegl/codede...

How old is our kernel?

hmh — Fri, 19 Feb 2010 12:11:36 +0000

http://lkml.org/lkml/2010/1/21/426

drivers/block/floppy.c
1 files changed, 618 insertions(+), 601 deletions(-)

Blank lines

hummassa — Fri, 19 Feb 2010 10:33:13 +0000

From the POV of the development process, yes, you are right.
But if you are measuring the age of the code, I would weed out anything that
ends up generating the same object code. Just my opinion there.

How old is our kernel?

aegl — Fri, 19 Feb 2010 05:03:48 +0000

There have only been about 169000 non-merge commits since git time began (the 180,000 figure quoted in the article includes all the merges).

All of the code in the current tree is attributed to just 131,681 commits by "git blame". So, not counting the merges, it would appear that about 23% of all commits are ultimately completely superceeded (or just plain reverted).

Size

corbet — Fri, 19 Feb 2010 00:49:23 +0000

The early git repositories - one file per object - were truly huge. There was a lot of griping at the time. Obviously, things have gotten a lot better since.

How old is our kernel?

jengelh — Fri, 19 Feb 2010 00:34:27 +0000

Interestingly, the full history currently takes one CD (my repo is 652M), much less than Linus's projected 3.2GB (perhaps that was a worse deltification back then?).

How old is our kernel?

mezcalero — Thu, 18 Feb 2010 15:28:57 +0000

Listing sound/ here is not entirely fair. The fact that sound/ is in the kernel top directory makes it show up in the stats here, although drivers/sound/ would be a much more appropriate place for it.

It would be really interesting to see the directories beneath drivers/ showing up in these stats at the same level as sound/.

Blank lines

nix — Thu, 18 Feb 2010 14:11:24 +0000

I'd call comment and string changes significant. The only reason I singled out blank lines is because they really *cannot* change in a meaningful manner.

How old is our kernel?

error27 — Thu, 18 Feb 2010 11:22:20 +0000

It would be interesting to see how much of the original SCO kernel is still included.

He he.

Blank lines

hummassa — Thu, 18 Feb 2010 10:16:10 +0000

This seems to me a speculative, rather than a factual, answer.
I'd suggest cleaning all and anyy comments, blank lines and strings from the
measurements, and comparing. Does not seem too difficult to do.

How old is our kernel?

awils1 — Thu, 18 Feb 2010 08:40:02 +0000

David, I'd definitely be interested!

(Heh, and here I was thinking I was the only crazy-reformatter -- it may have not been t-philes, but I'm guilty of reformatting some of the GNU documentation.)

How old is our kernel?

nix — Thu, 18 Feb 2010 07:57:44 +0000

Yeah! An excuse to refactor floppy.c!

(there are some things not even kudos can do)

WoPDaSD 2010

PO8 — Thu, 18 Feb 2010 07:48:26 +0000

The acronym is kind of awkward, but the workshop is pretty cool. It works that way sometimes. :-)

How old is our kernel?

neilbrown — Thu, 18 Feb 2010 02:23:35 +0000

Now that these numbers have been published, we risk the XKCD effect - developers will check which of "their" files appear to still be "old" and find an excuse to change them, just to make "their" subsystem seem more current, cool, hip, and up-to-date.
Further proof that you cannot observe a system without changing it....

WoPDaSD 2010

felixfix — Thu, 18 Feb 2010 00:02:11 +0000

It's related to INTERCAL ...

"The full name of the compiler is "Compiler Language With No Pronounceable Acronym", which is, for obvious reasons, abbreviated "INTERCAL"."

Blank lines

xoddam — Wed, 17 Feb 2010 23:04:37 +0000

The measurement is entirely relative -- LOCs ain't LOCs, except in the special case where the code is written in the same language, in the same style, even in the same codebase.

If it is a valid, apples-to-apples comparison to compare the line count without blank lines (debatable, but I think it is in this case : it's all Linux kernel code after all), I reckon it's equally valid to compare it with them.

Older trees

viro — Wed, 17 Feb 2010 22:38:24 +0000

FWIW, I've got a slightly more complete tree (several versions missed by davej added to his) plus the CVS-exported 2.4 BK tree; need to convert the latter to git, then we'll get full tree with all branches (right now there's stuff up to 2.4.0 + 2.4.0--2.6.12 + 2.4.31--2.4.current + 2.6.12--2.6-current, with gap between 2.4.15 and 2.4.31).

Older trees

corbet — Wed, 17 Feb 2010 22:31:25 +0000

That's davej's repository, yes. I have it. The tools will require some tweaks to work well with that data source, but it's all certainly doable.

Older trees

dlang — Wed, 17 Feb 2010 22:24:15 +0000

there is a historical git archive that can be grafted onto the current 2.6.12 archive.

once this is done the combined archive can be treated as a single archive and I expect that the scripts used for this report could be used as-is (although it will obviously take longer)

IIRC, the historical git archive goes all the way back to the 0.0x days (although not without gaps)

David Lang

No worries!

jzbiciak — Wed, 17 Feb 2010 22:10:13 +0000

No worries! The new number makes much more sense. Since I've seen elsewhere that drivers seem to make up a pretty big portion of the tree, it makes sense that the center of gravity for change-rate would land near there too.

I guess I wasn't too far off (in orders of magnitude) on the number of unaccounted-for lines in the data, although I did undercall it by a factor of 4.

Man, the kernel's gotten huge! I actually tried to download the latest kernel and untar it in a 300MB partition, and it b0rked out, filling the disk only part-way into the "drivers/" directory. Next time I'll do it on a machine with a more generous disk space allotment. :-)

Yep, I blew it

jcm — Wed, 17 Feb 2010 21:30:54 +0000

Nah, you obviously didn't blow anything - awesomely informative stats, thanks Jon :)

WoPDaSD 2010

ajross — Wed, 17 Feb 2010 20:50:58 +0000

Were all the pronouncable acronyms already taken?

Yep, I blew it

corbet — Wed, 17 Feb 2010 20:18:18 +0000

The 41% number was wrong; the article has been edited with the correct data.

For the curious: the bug was a combination of the tool descending into the .git directory (which has no history of its own) and it attributing unknown lines to the oldest release. That essentially inflated the number of "old" lines in the full kernel by about 2 million - enough to make a big difference.

Please accept my apologies for the screwup. I'd taught the program to avoid .git a while back, but I thought it was an efficiency improvement only; I didn't realize it had messed up the full-kernel numbers. Embarrassing.