LWN.net Logo

Support large repositories!

Support large repositories!

Posted Apr 5, 2010 22:18 UTC (Mon) by marcH (subscriber, #57642)
In reply to: Support large repositories! by RCL
Parent article: A proposed Subversion vision and roadmap

> That's so common that I'm suprised that you are asking "why".

That's so common that only perforce seem to handle large binaries well?

I am afraid what is actually common is to generate large binaries from source.


(Log in to post comments)

Support large repositories!

Posted Apr 6, 2010 1:19 UTC (Tue) by bronson (subscriber, #4806) [Link]

It's clear you don't work on projects with any appreciable graphics or sound. Really, given the number of mobile, game, and Flash devs nowadays, you might want ot rethink your use of the word "common".

Besides, when it takes six hours for an optimized compile (this was the 90s), or when the dev tools cost $25,000/seat, then hell yes you check binaries into revision control. Right next to the source code.

Support large repositories!

Posted Apr 6, 2010 9:09 UTC (Tue) by marcH (subscriber, #57642) [Link]

> Besides, when it takes six hours for an optimized compile (this was the 90s), or when the dev tools cost $25,000/seat, then hell yes you check binaries into revision control. Right next to the source code.

As a matter of fact, I work daily with binaries that I cannot compile myself. Hell no they are not checked in right next to the source code, not to make revision control operations unnecessarily slow down to a crawl.

Support large repositories!

Posted Apr 6, 2010 18:52 UTC (Tue) by avik (guest, #704) [Link]

The correct way to handle generated binaries is with a ccache-style shared repository. This way the first person to compile takes the hit, the rest reuse the generated binaries, and the source control doesn't need to be aware of it.

Support large repositories!

Posted Apr 7, 2010 23:19 UTC (Wed) by cmccabe (guest, #60281) [Link]

> The correct way to handle generated binaries is with a ccache-style shared
> repository. This way the first person to compile takes the hit, the rest
> reuse the generated binaries, and the source control doesn't need to be
> aware of it.

Amen to that.

Checking in large blobs of "mystery meat" on the honor system just leads to chaos.

Support large repositories!

Posted Apr 8, 2010 23:13 UTC (Thu) by bronson (subscriber, #4806) [Link]

Mystery meat is mystery meat no matter where it's stored. When you give someone commit rights it becomes an honor system no matter what software you're using.

I notice you guys are ignoring my main points about audio and video files, and cross compilers that cost a lot of dough per seat. OK, fine, let's restrict this discussion to just native compiling. Even in this specialized case, anyone who's kept a distributed ccache up and running might be skeptical of Avi's advice.

Executables are WAY more backward compatible than object files. If you can ensure that everyone is running the exact same minor version of gcc and libraries, ccache would probably work. In most dev shops, where there's a crazy mix of personal favorite Linux distros is plus a bunch of custom-compiled shared libs, I'm pretty sure trying to keep everyone on ccache will cost you a lot more time than it saves. (spoken from my bitter experience of trying to do this in 2006).

Different strokes, right? You will to use whichever technique is best for your shop. That might be ccache, custom scripts pulling binaries off fileservers, or just checking them right into source control. Each one has its place.

Support large repositories!

Posted Apr 30, 2010 18:44 UTC (Fri) by cmccabe (guest, #60281) [Link]

> Mystery meat is mystery meat no matter where it's stored. When you give
> someone commit rights it becomes an honor system no matter what software
> you're using.

When you check _code_, a skilled coder can look at your change and figure out what it is doing. When you check in a _binary_, there is no obvious way to figure out how it differs from the binary that was previously there. Sure you could disassemble it and run a detailed anaylsis, but realistically, that's not going to happen. Hence, it's "mystery meat."

> I notice you guys are ignoring my main points about audio and
> video files

No, I totally agree with your points regarding audio and video. I hope that git will be extended to support working with these large files more effectively.

> Executables are WAY more backward compatible than object files. If
> you can ensure that everyone is running the exact same minor version
> of gcc and libraries, ccache would probably work. In most dev shops,
> where there's a crazy mix of personal favorite Linux distros is plus
> a bunch of custom-compiled shared libs, I'm pretty sure trying to
> keep everyone on ccache will cost you a lot more time than it saves.
> (spoken from my bitter experience of trying to do this in 2006).

You are doing it wrong. Set up a chroot environment with the proper libraries and compiler. Look up "cross compiling with gcc."

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds