shallow clones are bad? [LWN.net]

shallow clones are bad?

Posted Aug 31, 2026 20:45 UTC (Mon) by jtaylor (subscriber, #91739)
Parent article: Ryabitsev: Creepy crawlies

The post mentions

> ... but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. ...

I expected shallow clones to be better for parties involved as its less data to transfer, that its bad is unexpected and the manpage doesnt mention anything about it.
Best information I found is a vague statement here https://github.blog/open-source/git/get-up-to-speed-with-...

> These clones also put undue stress on later fetches, so they are strongly discouraged for developer use. They are helpful for some build environments where the repository will be deleted after a single build.

Though this also kind of indicates it is ok for the CI use case.

Thinking about it I guess shallow clones may need to unpack old files from pack files which is more costly than just sending the whole pack to the client.
So if I'm not limited by bandwidth or disk space I should always use full clones also for single use?
A mention on the manpage would probably help to avoid this misconception.

to post comments

shallow clones are bad?

Posted Aug 31, 2026 20:52 UTC (Mon) by mricon (subscriber, #59252) [Link] (7 responses)

They are bad on the server side. If someone's doing a regular clone of stable/linux.git, most of the pack is already prebuilt and we just send it down the pipe. Whatever loose objects are missing, it's only a few and we can quickly make a pack of of them.

Now, if someone is cloning with --shallow, this means we need to:

- figure out which objects those are out of several million (could be --depth 10, for all we know)
- compress them on the fly (that's gzipping 1G+ of stuff)
- keep it in RAM as we send it down the pipe until the client finishes

One or two shallow clones is no big deal, but if a CI system decides to spin up 20 nodes that all decide to do --depth 1, that's a problem.

shallow clones are bad?

Posted Sep 1, 2026 7:19 UTC (Tue) by taladar (subscriber, #68407) [Link] (3 responses)

For high traffic servers like kernel.org, would it perhaps make sense to have the shallow clone pack already prepared for the head of commonly shallow clones branches?

shallow clones are bad?

Posted Sep 1, 2026 14:05 UTC (Tue) by mricon (subscriber, #59252) [Link] (2 responses)

There's no reasonable way to do this, because everyone's shallow clone is different. Some will want --depth 0, some --depth 10, others just commits and no blobs. The only predictable pack we can create is when someone making a full clone.

shallow clones are bad?

Posted Sep 2, 2026 7:38 UTC (Wed) by taladar (subscriber, #68407) [Link] (1 responses)

I would assume the vast majority of CI wants exactly the content of the current commit a branch points to to build that. Plus maybe some metadata on the commit itself (like its commit hash and tags) to include in the built --version output.

shallow clones are bad?

Posted Sep 2, 2026 14:30 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

CI should not point to branches. The forge should reserve a ref for a CI pipeline to use based on the branch (e.g., GitLab uses refs/pipelines/PIPELINE_ID). Just because the branch was on commit X when the pipeline was created doesn't mean that a job started an hour from now won't get commit Y when asking for the same branch name.

Tags are also not populated (by default) when forking, so you may also need a step to fetch tags from the target project in addition to the code from the source project.

shallow clones are bad?

Posted Sep 1, 2026 11:48 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

For CI builds I've been recommending --filter=tree:0 to avoid issues with git describe and other history issues. Is that more economical on the server side? Just curious.

shallow clones are bad?

Posted Sep 1, 2026 14:06 UTC (Tue) by mricon (subscriber, #59252) [Link] (1 responses)

If it's just a single system doing CI, we're fine with whatever legitimate things you're doing with git. If, however, you're doing CI at any kind of scale, please set up your own mirror, don't hammer the public resource.

shallow clones are bad?

Posted Sep 1, 2026 14:58 UTC (Tue) by marcH (subscriber, #57642) [Link]

I'm not asking about hitting your servers specifically. I was only trying to leverage your git server expertise.