Not logged in
Log in now
Create an account
Subscribe to LWN
Pencil, Pencil, and Pencil
Dividing the Linux desktop
LWN.net Weekly Edition for June 13, 2013
A report from pgCon 2013
Little things that matter in language design
SHA calculation is what kills it. As I wrote, I didn't succeed in creating
a 22GB (modest by gamedev standards) repo with git/bzr/hg...
Versioning really-big files
Posted Apr 4, 2010 20:33 UTC (Sun) by smurf (subscriber, #17840)
The larger problem, however, is that you want a way to carry multiple versions of slowly-changing multi-GB files in your repo -- without paying the storage price of (a compressed version of) the whole blob, each time you check in a single-byte change. Same for network traffic when sending that change to a remote repository.
This is essentially a solved problem (rsync does it all the time) and just needs integration into the VCS-of-the-day. This problem is quite orthogonal to the question of whether said VCS-of-the-day is distributed or central, or whether it is named git or hg or bzr or whatever.
Yes, I know that the SVN people seem to have gotten this one mostly-right ("mostly" because their copy of the original file is not compressed). Hopefully, somebody will do the work for git or hg or whatever. It's not exactly rocket science.
Versioning really-big (binary) files
Posted Apr 6, 2010 18:47 UTC (Tue) by vonbrand (subscriber, #4458)
git uses delta compression by default (and has done so for a long time now), so the "huge binary files that change a bit" shouldn't be a problem. Please check with the latest version.
Posted Apr 6, 2010 23:12 UTC (Tue) by dlang (✭ supporter ✭, #313)
even for images and audio, if you were to check them in uncompressed the git delta functionality would work well and diff the files against each other, but if you compress the file (jpeg, mp3, or even png) before checking it in, a small change to the uncompressed data results in a huge change to the compressed data. If it's a lossless compression (i.e. png) then it would be possible to have git uncompress it before checking for differences, but if it's a lossy compression you can't do this.
Posted Apr 7, 2010 7:53 UTC (Wed) by paulj (subscriber, #341)
Posted Apr 12, 2010 1:14 UTC (Mon) by vonbrand (subscriber, #4458)
If the contents needs version control, it should be handled by a VCS. The size or format of the files could be a technical hurdle, sure; but it shouldn't be an excuse for not solving the problem.
Subversion considered obsolete
Posted Apr 4, 2010 20:55 UTC (Sun) by simlo (subscriber, #10866)
Where I work, we use subversion (and I use git svn :-). We have scripts which pulls the tar.gz files for various packages in specific versions from a server, unpacks, patches and crosscompiles them to our target. The only thing we have in subversion is the scripts and the files we have changed.
For the Linux kernel we tried to have the full thing in subversion, but it took way too much for subversion, so now we only have a makefile, which clones a git repository, when the source is needed.
Posted Apr 5, 2010 3:12 UTC (Mon) by RCL (guest, #63264)
It's almost like having very large firmware blobs in Linux kernels, much larger than they are currently...
See comments below where I elaborate on this interdependency between code and data in games.
Posted Apr 5, 2010 14:00 UTC (Mon) by simlo (subscriber, #10866)
On the other hand you can have data like maps and icons, which is also "source code" and belongs in the VCS if you edit them by using some program (some map editor, GIMP or whatever). But the overall system is badly designed if these files are "large". They ought to be seperated into small files, each containing seperate parts of the information and then "compiled" into larger files. This will usually make a more flexible and maintainable system (besides making life easier for the VCS). It is the same with C code: You don't make one big file but smaller ones, seperated by functionality.
Posted Apr 5, 2010 3:22 UTC (Mon) by martinfick (subscriber, #4455)
Posted Apr 8, 2010 10:08 UTC (Thu) by epa (subscriber, #39769)
Posted Apr 5, 2010 12:12 UTC (Mon) by peschmae (guest, #32292)
Looks like you're stuck with perforce :-p
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds