LWN: Comments on "Whatever happened to SHA-256 support in Git?"

At this point...

JeffBai — Wed, 14 Feb 2024 07:57:48 +0000

(Very late reply, but!) https://git-scm.com/docs/hash-function-transition/ mentions that blake2sp-256 was a contender. I have no interest in looking through the big mail thread to find out what happened to it. Could be something about OpenMP.

Whatever happened to SHA-256 support in Git?

geert — Wed, 04 Oct 2023 13:44:46 +0000

Yes it is. Both hashes are (eventually[*]) made from the same input.

[*] For blobs (files), this is obvious, as the hash is calculated from the file contents plus some file metadata.
For all other objects (trees, commits), the hash is calculated from tree or commit info plus hashes calculated before.

Whatever happened to SHA-256 support in Git?

xnox — Wed, 04 Oct 2023 02:37:12 +0000

Is conversion from sha1 to sha256 git format reproducible?

Whatever happened to SHA-256 support in Git?

luto — Sat, 31 Dec 2022 07:29:40 +0000

I don’t understand why interoperability can’t exist. Imagine a hybrid repository: objects can be hashed with any algorithm (SHA-1 or SHA-256). Objects hashed with SHA-256 can refer to any object, but objects hashed with SHA-1 can only refer to other objects hashed with SHA-1. You can also add a rule, per-repo, that commits claiming to be after a certain date or (transitively) referring to commits past a certain date can’t refer to trees that (transitively) refer to SHA-1 objects.

This means that conversion to SHA-256 is more or less an all-in affair. You start adding any SHA-256 objects, and you very quickly can’t add any new SHA-1 objects. But you could convert.

Compliance nonsense

epa — Fri, 15 Jul 2022 15:31:40 +0000

That's an interesting point. If the hash function is known to be weak (or you want to hedge against it becoming broken in future) then you could add an extra defence with a 'normalized hash'. If the file looks like C source code then strip out the comments, normalize the whitespace, and perhaps rename all the variables that aren't visible from outside the compilation unit. Then both the original content and the normalized one are hashed separately and both of these go into the final commit id.

IPv6

oldtomas — Sat, 09 Jul 2022 05:53:02 +0000

Inspired by yours, I tried an equally precise and rigorous experiment. Context: bog standard (Debian Gnu-)Linux box. I moved a couple of weeks ago. In my old flat, I gave up on IPv6 (your bog standard DSL, one of ghe Big Providers around here). In my new flat (same setup, the other of the Big Providers, yes, we have more than one)... surprise:

tomas@trotzki:~$ ping lwn.net
PING lwn.net(prod3.lwn.net (2600:3c03::f03c:91ff:fe82:68b2)) 56 data bytes
64 bytes from prod3.lwn.net (2600:3c03::f03c:91ff:fe82:68b2): icmp_seq=2 ttl=56 time=102 ms
64 bytes from prod3.lwn.net (2600:3c03::f03c:91ff:fe82:68b2): icmp_seq=3 ttl=56 time=102 ms
...

So it seems it's slowly coming, not just for smartphones

Whatever happened to SHA-256 support in Git?

koh — Thu, 07 Jul 2022 11:06:13 +0000

If I understand correctly: Merkle DAG + automated choice of the "mainline" branch to add nodes to + automated distribution of all nodes/commits in a network.

Not sure about the "all" in the last part, but that helped, many thanks!

Whatever happened to SHA-256 support in Git?

farnz — Wed, 06 Jul 2022 15:46:01 +0000

To be fair, that's an issue because you're choosing between two different blockchains, each of which does the trust thing automatically.

And that sort of problem is what I meant by saying that it's mathematically neat, but not necessarily practical - being able to form a consensus without trust is cool, but there are other dimensions involved beyond simply forming a consensus, such as which blockchain to trust.

Whatever happened to SHA-256 support in Git?

excors — Wed, 06 Jul 2022 15:21:08 +0000

> in a blockchain, consensus is formed automatically and does not depend on humans choosing trusted people

...except when, say, the core developers can't agree on a technical change for the project and so they fork the blockchain and now you've got two versions that both claim to be authoritative, and they have to fight it out on social media to convince users/miners/exchanges/etc to support their side. Maybe the mathematical model is trustless but that's because it's modeling an unrealistically abstract version of the problem - the practical implementation is never trustless, it's just obscuring who you're having to trust. (And as demonstrated over and over again, users often end up having to trust people who really don't deserve that trust.)

Whatever happened to SHA-256 support in Git?

farnz — Wed, 06 Jul 2022 14:34:38 +0000

The distinction is that in the Linux development model, Linus is a single point of failure - the consensus algorithm in the federated git tree world is "we trust Linus". In a blockchain, the consensus algorithm will choose a tree from the set in the federation such that no individual tree in the federated set is "more trusted" than others - if Linus were to go rogue or go on vacation, a blockchain development model would choose someone else's tree as "mainline Linux" automatically.

This is the key to the blockchain's difference from other Merkle trees - in a blockchain, consensus is formed automatically and does not depend on humans choosing trusted people, while in most Merkle trees, the consensus decision depends on humans making trust decisions.

It's mathematically neat that we can have consensus without needing trust, but it's not necessarily a practical result.

Whatever happened to SHA-256 support in Git?

geert — Wed, 06 Jul 2022 14:04:33 +0000

Sounds like Linux kernel development, where (ideally) all forks end up being merged into Linus' tree, eventually...

See James Bottomley's closing keynote at OLS2007 (https://www.linux.com/news/ols-closes-keynote/).

Whatever happened to SHA-256 support in Git?

nybble41 — Wed, 06 Jul 2022 03:18:21 +0000

The key element which sets a blockchain apart from an arbitrary Merkle tree (or DAG) is the Byzantine consensus system which ensures that there is only *one* dominant chain in the distributed system. Git repos are organized into one or more Merkel trees, but it's a federated system where each node is a silo with its own data, not a distributed one where all the nodes (eventually) come to share a single Merkle root with new "blocks" being added to a common "chain".

Whatever happened to SHA-256 support in Git?

koh — Tue, 05 Jul 2022 22:17:18 +0000

Please do all of us a favour and enlighten us as to what technically (usually) are those fundamental differences you're hinting at.

Plan is to make it easy to transition.

gmatht — Thu, 30 Jun 2022 13:53:25 +0000

As jthill already mentioned, we can already do `git init --object-format=sha256`. The difficulty is making it easy to transition to a new hash. Once it is easy to transition to a new hash, there is less need to pick a future-proof hash, and in the meanwhile sha256 has more widespread hardware acceleration. See: https://stackoverflow.com/questions/60087759/git-is-movin...

Whatever happened to SHA-256 support in Git?

Karellen — Thu, 30 Jun 2022 10:35:34 +0000

My first thought was to prefix hashes with the hash type and a separator, e.g. "sha256:0123...". That way you could add new hashes whenever the devs wanted, by just adding the algorithm and assigning a new prefix. Any hash already stored inside a git repo that is not preceded by a type is automatically assumed to be sha1. A commit with an sha256 hash could have parents with either sha256 or sha1 hashes, or even both!

The way they decided to go about things (last I checked) did seem a bit constraining.

Compliance nonsense

cortana — Tue, 28 Jun 2022 11:49:58 +0000

Eagerly awaiting a tool that produces collisions by adding commented ascii art to source code... :)

IPv6

ceplm — Mon, 27 Jun 2022 19:25:44 +0000

I think the bias is that more readers are IT professionals sitting on old IPv4 networks who use IPv6 only as a back-stop.

Whatever happened to SHA-256 support in Git?

smammy — Mon, 27 Jun 2022 15:24:44 +0000

Git people are so into shortened hashes that I doubt they'd go for a format that requires a four-digit prefix. Multihash has been discussed but obviously that never went anywhere.

Whatever happened to SHA-256 support in Git?

KaiRo — Mon, 27 Jun 2022 14:02:05 +0000

Please do all of us a favor and don't use the word "blockchain" when you obviously don't know what makes something one. While blockchain usually use merkle trees, that doesn't mean a merkle tree is a blockchain, it's (usually) part of one. There is enough FUD and scamming around this area, it's neither useful to you to join into that muddy crowd nor is it useful to those of us who are trying to make decent and honest use out of the social engineering and technology combination that actual blockchains represent.

At this point...

kilobyte — Mon, 27 Jun 2022 08:56:50 +0000

BLAKE3 instead? Much faster, an earlier version of it was a SHA-3 finalist, there's no risk the NSA picked an algorithm they know how to break (there were some irregularities during the competition), can be arbitrarily parallelized.

Whatever happened to SHA-256 support in Git?

taladar — Mon, 27 Jun 2022 08:47:24 +0000

And spout nonsense like "never change a running system" because that is what some old person told them in the 70s.

At this point...

jd — Sun, 26 Jun 2022 18:34:24 +0000

...Switch to SHA-3. It won't, apparently, slow adoption at all and, at least, will still be secure by the time everyone uses it.

Compliance nonsense

jthill — Sun, 26 Jun 2022 14:49:57 +0000

Except that exists already, it's just Git. `git init --object-format=sha256` and your repo uses sha256 only and can't talk to sha1 repos. I'd be curious how easily the web frontends' private-server options can be made to use the new object format if they don't have to talk to any poor left-behind sha1 repos either.

As a side note, afaik all known or suspected collision-generating methods require some place to hide gobs of carefully-chosen noise bits in both colliding texts. pdf is a binary format and can hide arbitrary noise. source code can not. There is no possibility that anyone get an engineered source file past even the most cursory code review. The garbage would appear the first time anyone so much as glanced at the diffs.

IPv6

Sesse — Sun, 26 Jun 2022 11:29:01 +0000

This is very much true, and it is largely due to the incumbent ISPs lagging. Most countries' status is usually very much driven by what key people in a few select ISPs choose to care about. :-/

IPv6

jem — Sun, 26 Jun 2022 11:27:16 +0000

Deployment of IPv6 in Norway is lagging six years behind the global average, though. (Based on the aforementioned Google stats: https://www.google.com/intl/en/ipv6/statistics.html#tab=p...)

Whatever happened to SHA-256 support in Git?

gdt — Sun, 26 Jun 2022 03:16:43 +0000

As a worked example, Australia's Information Security Manual states

Only hashing algorithms from the SHA-2 family are approved for use. When using SHA-2 for hashing, an output size of at least 224 bits is used, preferably SHA-384.

To use Git you cannot make any claim that hashing in Git contributes to addressing your organisation's threat model: say the threat of subversion of the repository. Then apply for an exception, arguing that Git's use of SHA-1 is out-of-scope as it is not implicated in any threat model. You may then be asked to show how the threat of subversion of the repository is countered, which could be GPG-signing each commit from a key only held on a trusted processor (eg, a Yubikey).

Of course this application for an exception may not be successful: not every organisation's security policy makers may have a deep technical understanding; not every application for an exception may fully address the threat model; and there may be a overarching policy of limiting exceptions in fields with large and widespread consequences, such as the supply chain threat from subversion of software builds.

Whatever happened to SHA-256 support in Git?

salimma — Sat, 25 Jun 2022 23:13:36 +0000

I'm curious too. Mercurial also hasn't moved

https://www.mercurial-scm.org/wiki/SHA1TransitionPlan

IPv6

Sesse — Sat, 25 Jun 2022 20:05:52 +0000

Here (Norway), the copper network (POTS, ISDN, DSL) is simply left to die; a little of it is still left, but if it breaks, it won't be fixed. Nearly all voice is 2G/4G/5G (3G has largely been turned down). Data is DOCSIS (cable), FTTH or 4G/5G.

IPv6

Wol — Sat, 25 Jun 2022 18:51:25 +0000

> Mobile technology is newer and faster moving, old landlines are in the category "they work, so don't fix them".

In the UK, "old landlines" will soon be history. Our POTS here has already been upgraded to VOIP - my old POTS phone is now plugged into my broadband router and works fine (for a somewhat jaded definition of "fine" :-(

Dunno about other countries in Europe, though ...

Cheers,
Wol

Whatever happened to SHA-256 support in Git?

ms-tg — Sat, 25 Jun 2022 17:46:45 +0000

> There's also Multihash, for what it's worth.

How can we get this amplified? From my understanding, adopting multihash would go a long way to future-proofing git, as there would be a single “before multihash” case to account for, and then all future iterations would be signaling the encoding in-band with the ability to add future options cleanly? Wouldn’t it?

Whatever happened to SHA-256 support in Git?

jezuch — Sat, 25 Jun 2022 17:27:02 +0000

> problems with it historically

...and people still recommend against XFS because of a data-eating bug that was fixed in 2005 *sigh*

Whatever happened to SHA-256 support in Git?

farnz — Sat, 25 Jun 2022 16:20:03 +0000

One fun thing about that is that you can expect to see more IPv6 traffic by byte volume or packet count than IPv4, but not necessarily by connection count.

A thing that drives IPv6 adoption in mobile is that data-intensive services like Netflix and YouTube are IPv6-enabled - so by enabling IPv6 for your customers, you can use stateless routing to get that traffic off your backbone and onto the video provider network nearer the cell site, whereas for CGNAT (including NAT64 and 464XLAT here), you have the complexity of maintaining distributed state to handle.

Whatever happened to SHA-256 support in Git?

alison — Sat, 25 Jun 2022 04:46:24 +0000

> Every git commit is the root of a merkle tree, or as the kids call it, a “blockchain”. A git
> commit object id is the hash of a string which includes among other things the commit’s
> immediate parent object ids, and the commit’s root tree object id.

Is the algorithm used by TPMs also a merkel tree?

IPv6

alison — Sat, 25 Jun 2022 04:41:31 +0000

LWN is the funniest website that I read regularly, mostly intentionally. Keep up the good work!

Whatever happened to SHA-256 support in Git?

dvdeug — Fri, 24 Jun 2022 19:24:36 +0000

My boss insists on turning off IPv6 on all computers we install. I'm not a fan, but he's the boss, and he's had problems with it historically.

Whatever happened to SHA-256 support in Git?

khim — Fri, 24 Jun 2022 17:54:21 +0000

You can even do both at the same time: “h” would mean “normal SHA256-bit hash in 65 letters” and “stuvwxyz” would start “40-letters long SHA256-bit hash” (if you take ASCII, excluse 37 “really bad” symbols, e.g. first 32, 127, “ ”, “%”, “$” and “.” then you can encode 13 bits in two characters).

This way you would even have an option for these old tools where 40-letters space is reserved for hash.

This should not be a default because at some point even longer hash would be needed, most likely.

IPv6

Lennie — Fri, 24 Jun 2022 17:04:20 +0000

Maybe lwn.net has a certain regional bias when it comes to IPv6 traffic.

Whatever happened to SHA-256 support in Git?

smammy — Fri, 24 Jun 2022 16:18:15 +0000

There's also Multihash, for what it's worth.

Whatever happened to SHA-256 support in Git?

angelsl — Fri, 24 Jun 2022 14:17:04 +0000

Yes, so you could replace the contents of a file (blob object), or the tree object, or the commit object itself, by finding a hash collision.

Whatever happened to SHA-256 support in Git?

dbnichol — Fri, 24 Jun 2022 12:15:43 +0000

This part of the article is what stuck me, too. Git has been around for nearly 20 years now. There are vast amounts of existing git repos with sha1 identifiers in them.

I'd say the project is at best 50% done if there's no interoperability with sha1 repos. Even if you switched git to default to sha256 on new repos and convinced all the major hosting providers to rewrite the history on all their repos to sha256 today, it would be years of pain before that trickled down through all the repos in the wild.

Unless there's compatibility with sha1 repos and a nearly automatic way to rewrite existing repos to sha256 in a compatible way, then it's essentially unusable. That seems like just as big a problem if not bigger than making git capable of using a different hashing algorithm.