How git affects kernel.org performance

[Posted January 10, 2007 by corbet]

From:		"H. Peter Anvin" <hpa-AT-zytor.com>
To:		"H. Peter Anvin" <hpa-AT-zytor.com>, git-AT-vger.kernel.org
Subject:		How git affects kernel.org performance
Date:		Sat, 06 Jan 2007 21:24:02 -0800
Cc:		nigel-AT-nigel.suspend2.net, "J.H." <warthog9-AT-kernel.org>, Randy Dunlap <randy.dunlap-AT-oracle.com>, Andrew Morton <akpm-AT-osdl.org>, Pavel Machek <pavel-AT-ucw.cz>, kernel list <linux-kernel-AT-vger.kernel.org>, webmaster-AT-kernel.org

Some more data on how git affects kernel.org...

During extremely high load, it appears that what slows kernel.org down 
more than anything else is the time that each individual getdents() call 
takes.  When I've looked this I've observed times from 200 ms to almost 
2 seconds!  Since an unpacked *OR* unpruned git tree adds 256 
directories to a cleanly packed tree, you can do the math yourself.

I have tried reducing vm.vfs_cache_pressure down to 1 on the kernel.org 
machines in order to improve the situation, but even at that point it 
appears the kernel doesn't readily hold the entire directory hierarchy 
in memory, even though there is space to do so.  I have suggested that 
we might want to add a sysctl to change the denominator from the default 
100.

The one thing that we need done locally is to have a smart uploader, 
instead of relying on rsync.  That, unfortunately, is a fairly sizable 
project.

	-hpa