Large projects and decentralized development

Development using Git, with its decentralized model, is gaining proponents for projects beyond its Linux kernel heritage. Some recent threads on the kde-core-devel mailing list have been discussing how Git might be used by some developers without disrupting the Subversion (svn) infrastructure that is used by KDE. That conversation has broadened to consider how a large project like KDE might reorganize to take advantage of Git's strengths. It does not look like KDE is really considering a switch – they converted from CVS a little over two years ago – but the discussion is useful to anyone thinking about using Git.

There are really two separate discussions taking place, the first concerns using Git without disrupting svn, while the second covers the larger issues of how to structure and use Git for a larger project. The two are intertwined as the "best practice" for a KDE-sized project is to convert incrementally. Smaller sub-projects, a particular KDE application for example, would use Git while still committing the changes back to the svn repository. Trying to do a wholesale conversion of a project the size of KDE, with many developers, testers, translators and users – not to mention millions of lines of code – would be something approaching impossible.

For tracking an svn repository, while using Git locally, the git-svn tool is indispensable. It uses any of the svn protocols to check out a repository, optionally including branches and tags, and installing them as a Git repository. A developer then uses Git commands locally, using git-svn again when ready to update from or push changes to the svn repository. It is not a perfect fit, complaints about losing history in the conversion have been heard, but it does provide Git users a way to interact with svn.

The decentralized nature of the Git development model is always a stumbling block for projects that are used to the single, central, repository model of svn and other revision control systems. Adam Treat invited a rather well-known expert on Git, with some small experience in applying it to large projects, to comment on some of the questions he and others had. Linus Torvalds, who is also a KDE user, responded, at length, with some very useful insights.

Breaking the project into sub-projects is the first step:

Using the git-submodule command, a project can be broken up into many pieces, each with their own Git repository. Those separate repositories can then be stitched together into a "superproject" that understands how to handle a collection of repositories. If a change affects multiple modules, it can still be handled in an atomic way:

Users of a development tree have differing needs, which Git supports by not requiring a central repository that all users must interact with. Torvalds believes that the development organization, not the tool, should determine which repositories are central:

For Linux, his kernel Git tree is the center, but for a variety of other users, the "stable" tree or distribution kernel trees for example, their repositories are the source. Those repositories can and do update from time to time from the main tree, but they control when and the users of those trees don't have to care.

On the subject of mapping the current KDE practices to Git, Torvalds is, characteristically, not shy about expressing his opinion:

There is a clash of development models going on and Torvalds is pushing the kernel's model. His reasons are good, though they may not convince everyone, which is why Git tries hard to avoid forcing any particular style. As he did with open source development, Torvalds is trying to lead by example, while not forcing anyone to change.

Reading the full threads including the entire posting by Torvalds will be very interesting to those who follow source code management issues. This culture clash, centralized and somewhat bureaucratic versus decentralized and freewheeling will come up again and again over the next few years. Torvalds seems to think the Git model will work most everywhere and his track record for making smart choices is good. It will be interesting to watch.