LCA: How to destroy your community [LWN.net]

LCA: How to destroy your community

Posted Jan 24, 2010 15:47 UTC (Sun) by zooko (guest, #2589)
In reply to: LCA: How to destroy your community by johnflux
Parent article: LCA: How to destroy your community

Yes, I'm quite sympathetic to the sentiment, if not to the terminology, that too many security engineers don't understand the concept of "trade-off", or if they do they seem to think that the security knob should be turned to "11" regardless of the consequences.

I don't fault Linus for being impatient with security worriers in the sense of "not wasting his time listening to them", but I do fault him for being impatient in the sense of insulting them.

One thing that has always bugged me about git's use of SHA-1 is that there was very little engineering cost, and probably not too much cost in CPU cycles, to making it secure -- just use SHA-256 instead of SHA-1. (I believe that the reason git uses SHA-1 is the Monotone did. The earliest prototype of git used MD5 because BitKeeper did.)

The engineering cost for upgrading git from SHA-1 to a new algorithm is much higher. I'm not sure how it can be done well. First of all we probably need to deploy a version of git (let's call it version 2 for this conversation) which allows there to be a slot for a new hash value even though it doesn't read or use that space -- it just uses the SHA-1 hash value which is in the other slot. That way once we *eventually* deploy yet a newer version of git, let's call it version 3, which produces SHA-3 hashes in addition to SHA-1 hashes, people using git version 2 will be able to continue to interoperate. (Although, per this discussion, they may be vulnerable and people who rely on their SHA-1-only patches may be vulnerable.)

As far as I understand people using today's git, git version 1, will not be able to exchange patches in any way with people using some future version (which I called "version 3" above) that uses a new algorithm.

to post comments

LCA: How to destroy your community

Posted Jan 24, 2010 16:09 UTC (Sun) by johnflux (guest, #58833) [Link] (5 responses)

Looking at: http://www.cryptopp.com/benchmarks.html

It seems, roughly, that SHA256 takes about 40% more time than SHA1. From what I understand, the speed of git is determined most by the speed of the SHA1 implementation (Based on a long thread called 'Linus' sha1 is much faster!'). So roughly, switching would make everything 40% slower.

I think that's a trade-off that they wouldn't be willing to make. However, just playing with the C code of the SHA1 code by the git developers ended up making it nearly twice as fast, so I don't know what the optimal speed difference is against SHA256.

If the numbers stay about the same, I think the git guys wouldn't accept a 40% speed decrease.

(On a side point - subscribing so LWN was worth every penny. How I love to have civil conversations with intelligent people.)

LCA: How to destroy your community

Posted Jan 24, 2010 17:14 UTC (Sun) by zooko (guest, #2589) [Link] (4 responses)

Yeah, I was guessing that git is actually network-bound or disk-bound often enough that the CPU hit doesn't matter, but I'm not sure. (According to http://cryptopp.com/benchmarks-amd64.html , SHA-1 is 192 MB/s, SHA-256 is 139 MB/s. Faster than either your disk or your network?)

If git holds out for SHA-3 then hopefully SHA-3 will turn out to be faster than SHA-256. There's even a chance that it will turn out to be faster than SHA-1 on modern CPUs!

LCA: How to destroy your community

Posted Jan 25, 2010 0:32 UTC (Mon) by johnflux (guest, #58833) [Link]

http://www.mail-archive.com/bug-coreutils@gnu.org/msg1729... That's the thread I was thinking of.

LCA: How to destroy your community

Posted Jan 25, 2010 3:56 UTC (Mon) by njs (subscriber, #40338) [Link] (1 responses)

SHA-1 can definitely be a bottleneck in some situations. The most extreme -- though I'm not sure whether git does this -- is verifying hashes on an initial clone. (The idea is to prevent one person's disk corruption on an old file or whatever from spreading throughout the network of clones.) Here the disk and network cost is proportional to the size of the delta-compressed repository, but the SHA-1 cost is proportional to the size of the uncompressed repository, which can easily be in the terabyte range.

It can also easily be the bottleneck on, say, committing a large merge (many modified files, all in cache because they were just written).

LCA: How to destroy your community

Posted Dec 29, 2015 18:11 UTC (Tue) by Spitfire19 (guest, #106038) [Link]

I feel that you would also have a big hit when you are performing CI tasks, as for some scenarios you may want your automated tool to delete everything in given directory before pulling and checking out the latest commit.

LCA: How to destroy your community

Posted Jan 27, 2010 11:58 UTC (Wed) by broonie (subscriber, #7078) [Link]

In normal workflows you end up with the files you're accessing in cache so there's no I/O to physical devices and most operations do get CPU bound, especially read only ones. For performance purposes git pretty much assumes that most of the time you're running from hot cache.