LWN.net Logo

Support large repositories!

Support large repositories!

Posted Apr 4, 2010 20:07 UTC (Sun) by dlang (✭ supporter ✭, #313)
In reply to: Support large repositories! by RCL
Parent article: A proposed Subversion vision and roadmap

as I understand it the problem is not large repositories.

There are two separate problems

1. large files (individual files > 4G or larger than you can reasonably mmap on your system)

this is a real, acknowledged problem that is discussed every 6 months or so on the git list.

2. files that don't diff and therefor make the repository large and therefor slow to copy, etc.

shallow clones (i.e. don't pull all the history) are the work-around for this.


(Log in to post comments)

Support large repositories!

Posted Apr 5, 2010 11:36 UTC (Mon) by cortana (subscriber, #24596) [Link]

Large repositories are a problem, as well (as least with Git). Git occasionally decides to 'repack' a
large number of objects in its database into a smaller number of much larger 'packfiles'. As soon
as the size of one of these packfiles goes over 2.1GB, it fails with all kinds of obtuse error
messages.

Support large repositories!

Posted Apr 5, 2010 17:25 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]

if you rename a pack file and add .keep to the end of the name, git will never try to repack that file.

Support large repositories!

Posted Apr 5, 2010 19:02 UTC (Mon) by nix (subscriber, #2304) [Link]

Uh, not quite. You want to create a *new* file with the same name as the
packfile but ending in .keep instead of .pack. (The content can be
anything; a reason why you never want to repack this packfile, or nothing
at all).

Support large repositories!

Posted Apr 5, 2010 19:01 UTC (Mon) by nix (subscriber, #2304) [Link]

Yeah, big packfiles pretty much require a 64-bit box right now (and, given
the near-zero popularity of new 32-bit boxes in non-embedded roles,
probably always).

Support large repositories!

Posted Apr 5, 2010 23:15 UTC (Mon) by cortana (subscriber, #24596) [Link]

And I am *eagerly* awaiting a 64-bit git client for Windows. :)

Support large repositories!

Posted Apr 5, 2010 23:36 UTC (Mon) by dlang (✭ supporter ✭, #313) [Link]

the code is 64 bit clean, does it not work if you compile it with a 64 bit compiler on windows?

or are you saying that git needs to provide pre-compiled binaries for 64 bit windows?

Support large repositories!

Posted Apr 6, 2010 8:49 UTC (Tue) by cortana (subscriber, #24596) [Link]

Pre-compiled binaries are necessary. I really don't have enough time to work out how to navigate
the maze that is preparing an mingw/msys environment followed by working out how to build git,
package it into an installer, etc.

Support large repositories!

Posted Apr 8, 2010 16:39 UTC (Thu) by Spudd86 (guest, #51683) [Link]

err compiling git on msys should be just

./configure
make
make install

you may or may not have to call configure with --prefix=/usr to get a usable result... but I don't think git has much in the way of dependencies... unless you want things like gitk or git-svn

('course I am assuming you already have MSYS installed...)

Support large repositories!

Posted Apr 8, 2010 16:40 UTC (Thu) by Spudd86 (guest, #51683) [Link]

Oh yea if you do what I said you don't need an installer, just fire up your MSYS shell and use git from there

Support large repositories!

Posted Apr 8, 2010 16:48 UTC (Thu) by cortana (subscriber, #24596) [Link]

But then I'm using git in msys, not the nicely-packaged msysgit. And I would need to have a 64-bit
msys environment installed on the developer's systems. Does that even exist?

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds