shallow clones are bad? [LWN.net]

shallow clones are bad?

Posted Aug 31, 2026 20:52 UTC (Mon) by mricon (subscriber, #59252)
In reply to: shallow clones are bad? by jtaylor
Parent article: Ryabitsev: Creepy crawlies

They are bad on the server side. If someone's doing a regular clone of stable/linux.git, most of the pack is already prebuilt and we just send it down the pipe. Whatever loose objects are missing, it's only a few and we can quickly make a pack of of them.

Now, if someone is cloning with --shallow, this means we need to:

- figure out which objects those are out of several million (could be --depth 10, for all we know)
- compress them on the fly (that's gzipping 1G+ of stuff)
- keep it in RAM as we send it down the pipe until the client finishes

One or two shallow clones is no big deal, but if a CI system decides to spin up 20 nodes that all decide to do --depth 1, that's a problem.

to post comments

shallow clones are bad?

Posted Sep 1, 2026 7:19 UTC (Tue) by taladar (subscriber, #68407) [Link] (3 responses)

For high traffic servers like kernel.org, would it perhaps make sense to have the shallow clone pack already prepared for the head of commonly shallow clones branches?

shallow clones are bad?

Posted Sep 1, 2026 14:05 UTC (Tue) by mricon (subscriber, #59252) [Link] (2 responses)

There's no reasonable way to do this, because everyone's shallow clone is different. Some will want --depth 0, some --depth 10, others just commits and no blobs. The only predictable pack we can create is when someone making a full clone.

shallow clones are bad?

Posted Sep 2, 2026 7:38 UTC (Wed) by taladar (subscriber, #68407) [Link] (1 responses)

I would assume the vast majority of CI wants exactly the content of the current commit a branch points to to build that. Plus maybe some metadata on the commit itself (like its commit hash and tags) to include in the built --version output.

shallow clones are bad?

Posted Sep 2, 2026 14:30 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

CI should not point to branches. The forge should reserve a ref for a CI pipeline to use based on the branch (e.g., GitLab uses refs/pipelines/PIPELINE_ID). Just because the branch was on commit X when the pipeline was created doesn't mean that a job started an hour from now won't get commit Y when asking for the same branch name.

Tags are also not populated (by default) when forking, so you may also need a step to fetch tags from the target project in addition to the code from the source project.

shallow clones are bad?

Posted Sep 1, 2026 11:48 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

For CI builds I've been recommending --filter=tree:0 to avoid issues with git describe and other history issues. Is that more economical on the server side? Just curious.

shallow clones are bad?

Posted Sep 1, 2026 14:06 UTC (Tue) by mricon (subscriber, #59252) [Link] (1 responses)

If it's just a single system doing CI, we're fine with whatever legitimate things you're doing with git. If, however, you're doing CI at any kind of scale, please set up your own mirror, don't hammer the public resource.

shallow clones are bad?

Posted Sep 1, 2026 14:58 UTC (Tue) by marcH (subscriber, #57642) [Link]

I'm not asking about hitting your servers specifically. I was only trying to leverage your git server expertise.