LWN.net Weekly Edition for October 14, 2010

Lessons from PostgreSQL's Git transition

The PostgreSQL Project finally switched from CVS to Git in September 2010, and did its first release based on the new Git repository on October 5. Making the switch happen took years and resulted in at least one near-disaster. Other projects that are contemplating, or working on, a transition in their version control system may find useful lessons in how PostgreSQL fared.

A History of CVS

Switching version control systems is a relatively straightforward process for a young project, but not for an old one. From 1996 through mid-September 2010, the PostgreSQL Project was developed using CVS. In fact, the date the CVS server went live — July 8, 1996 — is generally considered the "birthday" of the open-source project, which came out of the ten-year-old university project. Twenty-one major releases and 154 minor releases were committed, branched, and packaged on CVS. A large web and development infrastructure existed around CVS, as well as a multi-step, multi-role release procedure.

In 2004, as Subversion was beginning to become popular, PostgreSQL contributors first started to argue about switching away from CVS. However, Subversion was not seen as sufficiently mature at the time, or as offering enough advantages over CVS. This discussion, and occasional flamewars, continued to crop up on the main "pgsql-hackers" mailing list. Those who wanted to migrate off of CVS split into multiple camps based on the system they preferred, including Subversion, Arch, and Monotone. And, of course, a large group of developers saw no reason to change at all.

As with many projects which operate by rough consensus, where there is no consensus, there is no action. The community verdict was to wait for one version control system to become the clear leader. In retrospect, this turned out to have been a good decision.

First Git Mirror

In 2007, one of the git-cvsimport maintainers from New Zealand decided to set up a persistent and frequently updated Git mirror for the PostgreSQL CVS tree. This mirror was extremely popular and one Google Summer of Code student even did his project using the Git mirror instead of the CVS repository. An increasing number of PostgreSQL contributors started using Git mirrors.

In 2008 a few members of the PostgreSQL web team decided to set up an "official" git.postgresql.org, using FromCVS for conversion, and Gitweb with custom administration scripts for the web version. This was done without obtaining community consensus, and caused some controversy. But once the flames died out, many people started using the repository. However, the mirror became a source of frustration for the developers who depended on it. Synchronization with CVS was often undependable, with changes made in CVS failing to show up in Git.

While one or two Subversion mirrors were created well before the Git mirror, and there was even a Bazaar mirror on Launchpad, none of these were at all popular. By mid-2008, it was clear that a majority of PostgreSQL developers favored an eventual switch to Git.

Evaluation

pgCon is the annual international PostgreSQL conference in Ottawa, Canada. By pgCon 2009, PostgreSQL was one of only a handful of major projects still using CVS. It was time to switch to something, and a long discussion ensued at the developer meeting. With a version of Git available for Windows, a major hurdle had been cleared, but several issues remained to be solved, most of them having to do with project infrastructure.

First, the project has an automated regression testing infrastructure called the PostgreSQL Buildfarm. This network of donated servers and virtual machines does daily or hourly CVS checkouts, builds PostgreSQL, and runs regression tests. The Buildfarm would need to be updated to use Git, which was a challenge because not all of the operating systems represented in the build farm (such as UnixWare and AIX) had Git packages available.

The second challenge was the code review process. Using CVS, PostgreSQL contributors submitted their patches in context diff format to the pgsql-hackers mailing list and the CommitFest application. The committers did not want to change this process.

It was also unclear whether it would be possible to recreate past releases from Git.

Decision to Switch

By pgCon 2010, these issues had been resolved. The Buildfarm code had been patched, and the web team planned to set up a CVS mirror of Git for the build farm members who could not run Git. Developers had tested several back branches, and tweaked the conversion process until it was possible to produce a back-branch release which was identical to the one produced by CVS.

The Buildfarm developers, primarily Andrew Dunstan, worked on transitioning build farm members to using the Git mirror in advance of the switchover. However, the git.postgresql.org mirror proved too unreliable for this purpose, and Andrew had to set up another mirror on Github using some custom scripts. That took longer than the changes to the Buildfarm client code. But it worked, and the build farm servers started to convert to Git.

Accordingly, those at the developer meeting set a date: on August 18th, 2010, PostgreSQL would switch over. The Git repository would become the canonical code base and the CVS repository would become the mirror. This date was based on the assumption that 9.0 would be released by then, and August 18th would be after the first Alpha release for 9.1. Suitably, at pgCon Andrew did a talk on "Git for PostgreSQL developers".

Failure to Switch Over

By August 13th, things were looking somewhat different; 9.0 was not released yet. But everything else was scheduled, so the conversion went ahead. First, the developers froze the CVS tree. Next, Magnus Hagander employed the cvs2git tool to do a final conversion from CVS to Git. Then all of the PostgreSQL developers were asked to test the new repository. Things looked OK.

Then, the day before the Git repository was expected to be opened to new patches, committer Robert Haas noticed a problem:

2010-02-28
        PostgreSQL...
        This commit was manufactured by cvs2svn to create branch REL8_3_STABLE

It seems that, where we had ported patches from later versions to earlier versions, cvs2git had manufactured inappropriate merge commits. Nobody had noticed because it didn't affect the head of any branch, just the history.

We reverted to CVS, and postponed switchover until after 9.0 was released.

The Second Conversion

Over the next few weeks, web team member Magnus Hagander worked with CVS2git developers Max Bowsher and Michael Haggerty to clean up issues with the conversion. In addition to the false merge commits, there were quite a number of other artificial commits and weird history in the converted version, such as branches which didn't exist and reappearing deleted files. A good portion of this was due to the fact that not only were we running an old version of CVS, but repository steward Marc Fournier and others had also done "CVS surgery" a number of times to fix problems over the years.

A second conversion test broke the ability to recreate any release. We discovered the new Debian server we were running it on did not default to ISO date format and had changed all the date strings for the releases. Fixed. A fair amount of "dirty history" in CVS simply could not be converted and needed to be patched in CVS before conversion. That was patched by Tom Lane.

Magnus, Max, Robert, and Tom stuck with it and resolved all of the issues which were considered roadblocks. cvs2git's next release will contain several improvements which are a result of our conversion. Several community members also added documentation to the PostgreSQL wiki on how we would use Git for development, including how to use git.postgresql.org, and how to commit.

On September 20, we released version 9.0 of PostgreSQL. Since Magnus was not heavily involved in the 9.0 release, he was able to spend his time preparing. So, on September 21st, we switched to Git.

Not Done Yet

However, the actual day of conversion was hardly the end. There was still a lot of minor cleanup to do. Contributors had to prune fictitious branches, move tags which appeared in the wrong place, and clean up many other issues.

Because the community wanted to preserve old releases exactly as they had been in CVS, we needed to preserve the file header tags in each file and remove them only from the "tip" of each branch. These are the comments which CVS automatically creates in each file which look like this:

    * $PostgreSQL: pgsql/src/tools/fsync/test_fsync.c,v 1.27 2010/02/26 02:01:39 momjian Exp $

Git doesn't use these tags since they don't work with atomic multi-file commits, so they needed to be removed from current development without removing them from the history.

Administrators also had to fix people's usernames, since contributors had freely used idiosyncratic nicknames and incomplete user information for git.postgresql.org before it was the canonical repository. These were changed to their main e-mail addresses with their real names. And, most of all, hackers who had waited until the conversion to learn Git were asking for command help on the mailing lists.

After the conversion, the project had to resolve policies and contents for .gitignore files, which tell Git what kinds of files to ignore. These are used to prevent developers from accidentally sending in patches containing editor backup files or build artifacts.

One issue which still isn't resolved is the CVS mirror for the Git repository. git-cvsserver turned out to have serious scalability limits, which makes it too limited to support the build farm servers. As a result, Andrew Dunstan required all of the build farm machine owners to migrate their nodes to Git immediately rather than gradually. However, there are still a few machines which are very valuable for testing and cannot run Git and, so far, there is no solution for those.

No Merge Commits

PostgreSQL had introduced a new workflow for reviewing patches in 2008, called "CommitFests" — a bi-monthly process where committers and reviewers clear out the pending patch queue. By 2010, the bugs had been worked out of the CommitFests and everyone thought they were working well. Among other things, the new workflow was helping train new contributors. So nobody wanted to change it. Also, many committers were okay with the migration only if they could still review patches the way they were used to.

The result was the adoption of a policy which will surprise veteran Git users. Per Magnus's blog:

Yes, that's correct. No merge commits. To submit a patch, extract it as a context diff and e-mail it. Committers are to apply the patch under their own names, without branch history. The project has decided, more-or-less, to use Git like it was CVS as far as commits to the main repository are concerned. Rather than adapt the PostgreSQL project's workflow to Git, Git would be adapted to the project's workflow.

It's possible, even likely, that eventually the PostgreSQL project will move towards the "normal Git workflow" where branches and merges would be used for feature work. But it definitely won't happen this year.

The First Release

The conversion wasn't really complete until the project did a release from it. That happened on Tuesday, October 5th with a combined security update. Unsurprisingly, there was a problem.

Security releases usually have fairly tight timing; from the moment someone commits the security patches, the issues fixed are publicly visible. Due to that, and limited familiarity with the Git commands, the source packager missed the final commit on the latest branch (9.0.1). That caused the release to be delayed by a day.

Git can't be blamed for everything, though. The release was delayed for another 2 hours because the Subversion repository which holds the www.postgresql.org web code locked up. But the release did go out.

Benefits of Git

Now that the migration is mostly done, many people are discovering the benefits of working on an up-to-date, distributed, version control tool.

Robert Haas has documented how to get commit summaries and sizes from Git. He wrote a perl script (which Tom Lane improved) that allows you to produce a changelog suitable for release notes from Git. Andrew Dunstan also found new ways to sync his local repository.

The pgAdmin project has also switched to Git and the pgWeb project will hopefully soon follow.

In the future, Git should both allow developers to work on longer-lived forks more easily and to test them more fully; we've already seen this with synchronous replication and SE-Postgres. Hopefully the translation teams will also be able to take advantage of forks on git.postgresql.org to collaborate on translating the docs and messages. Most importantly, Git should help prevent bit-rot in features which take months or years to develop.

Best of all, most PostgreSQL developers were able to continue hacking away without being involved in the switchover at all.

Lessons Learned

Assuming that there are any projects out there who have not yet switched to their distributed version control system of choice, here's a few things to learn from our migration:

• Start with a Git mirror.
• Designate a specific "Git migration team". Make sure they have lots of free time.
• Your first attempt to migrate will probably fail, so you need to be prepared for more than one.
• Changing your infrastructure, workflow, and build tool dependencies is harder than the repository conversion.
• Make friends with the conversion tool authors.
• Write lots of docs about the new tools and workflow.
• The more history you have on your current system, the more work conversion is going to be.
• Things which are broken in your current history are not going to fix themselves when you migrate.
• When testing the conversion, make sure to look at more than HEAD and branch-tips.

The biggest lesson, though, is not to be in a hurry! It was over three years from PostgreSQL's first Git mirror to final conversion, and 16 months of actual preparation. If you take your time and are ready to retry things that don't work the first time, you should be able to have a successful migration to Git.

Comments (64 posted)

UTOSC: Tayler on learning from failure

Web comic author and illustrator Howard Tayler looked at success and failure in business, particularly from the perspective of free content business models, in his keynote talk at the Utah Open Source Conference (UTOSC). Tayler has successfully turned his comic, Schlock Mercenary, into his day job for the past six years, even though the comic is freely available daily on the web. Arriving at a successful business plan is mostly a matter of learning from mistakes, he said, either your own or those of your peers. We live in a time where it is much easier to find out about mistakes, thanks to the internet.

Tayler previously worked at Novell for 11 years, doing everything from tech support to product management. He started Schlock Mercenary in 2000, while he was there, and built a following for the comic. In 2004, he left Novell to do the comic full-time. Though there have been ups and downs, he has succeeded in supporting his family since that time, primarily with book sales of the comic, along with advertising on the web site.

This is not the first time Tayler has spoken to UTOSC. In a presentation at UTOSC 2008, Tayler focused more on his particular business model, and at that time he saw it much differently than he does today. He likened his earlier model of creating a business plan to "making grizzly bear soup", which has a hard part, actually killing the grizzly, and "everything else is just soup recipes". For him, the hard part was to make the comic itself, and the other, moneymaking parts kind of just fell out of that. But, "business plans are not that cut and dried", he said.

We are all storytellers, Tayler said, of one kind or another. His stories are 1000 years in the future, but he also tells himself the same kinds of stories that those of us who aren't comic space opera authors tell. These are the stories that often govern our behavior ("if I work hard I will be able to retire at 65" or "if I always take off my shoes at the airport, I am safe from terrorist attack" and so on). As we keep telling ourselves these stories, some of them become business plans. Unfortunately, those stories are not really reflecting reality.

In The Black Swan, author Nassim Nicholas Taleb attributes success "much more to luck than we would be comfortable with", Tayler said. That book "blows my mind" and he strongly recommended it. One of the examples that Taleb gives is Casanova, who succeeded and failed seven different times in his life, going from prosperity to poverty. To many this is "proof" of the adage "never give up", but Taleb points out that there are likely many more folks who tried just as hard but never achieved that final success, so we don't know anything about them.

The failure stories would be valuable, so that we can learn from them. But people are "disinclined to talk about the failures", so we lose that information. But now, we have lots of ways to find about failures. People love to sweep their failures under the rug, but "there are now spiders under the rug recording those failures", and we can access that information through search engines.

An example that Tayler gave was the logo redesign that Tropicana orange juice did last year. The logo changed from an orange with a straw in it to a glass of orange juice, and within three weeks Tropicana knew they had made a mistake because they were losing millions of dollars. Brands that made "terrible orange juice", like Donald Duck brand, were suddenly outselling Tropicana. Marketing had decided that the brand needed a refresh, but it was a bad decision that looked good at the time it was made. Tropicana went back to the old logo, and in pre-internet days, that controversy would have faded. But you can still find lots of information about that failure on the web today.

On the other hand, people imitate each other's successes all the time. In his talk two years ago, he encouraged people to follow the same path he had as a business plan. But, he said, that kind of plan won't work for everyone as "'web cartoonist' pays better than 'web novelist'". For whatever reason, changing the web comic business model to a product that is strictly words doesn't work.

He has been lucky in that he "hasn't had enough failures to learn from", at least yet. When he left Novell, he set a failure condition: if they had to tap credit cards to pay bills for two consecutive months, it was time for him to go look for a job. Thousands of people don't recognize that their business has crossed a failure boundary, and they keep going hoping that the business is going to work out and pay them back. It is very important to set some failure conditions when starting a business so that failure can be recognized and flexibility can kick in.

Taleb and "dozens of authors" have suggested that business planning needs to be both robust and flexible. It must be robust enough to absorb a few mistakes, while being flexible enough to learn from those failures. Large organizations tend to come to this realization late, he said—if they realize it at all.

Evolution is the ultimate teacher of flexibility, robustness, and learning from failure. Evolution does not move toward a goal, it moves away from things that didn't work. It is, "improving things by making accidental random changes", which results in things like sharks—but also zedonks. Zedonks are a cross between donkeys and zebras that are ornery, sterile, and will "live to a ripe old age on a pony farm biting children". They are one of evolution's mistakes, he said.

While we may not have the time to make changes to business plans the way that evolution does it, we can use the information from various failed plans to guide our choices. You can't try out all of the different possible plans yourself, but new businesses are being started all the time. The ability to see their plans—what works and what doesn't—is an enormous advantage.

There is a flipside to the "law of large numbers", he said. If you roll a six-sided die many times, the average is 3.5, but rolling it long enough will produce long strings of consecutive sixes. By looking at the failures of others, "we can game the system", and make it more likely that our plans will come up sixes. "Lightning wants to hit something", he said, so "put up a lightning rod at the leading edge of a storm".

Tayler then took questions from the audience, most of which centered on the specifics of his particular business model. He brings in roughly 30% of his revenue from web advertising and 50-60% from the sale of several different books. There are some revenue sources he hasn't tapped yet, like speaking engagements, mostly because he likes talking for free. He cautioned against having any single revenue stream that makes up more than 40% of your income. At one point, he was in that position with Google AdSense and was concerned that Google could just "randomly turn that off", which could have been catastrophic.

Community is very important to his business. When he sells something, he wants that customer to "feel like they are part of a club". That requires good customer service, and making a connection with people at conferences and shows. There are certain cities where he sells much more than the population would indicate, like Austin and Detroit, because he has been there multiple times interacting with his fans.

In response to a question about telling good ideas from bad, Tayler pointed to a meeting that he attended in 1997 when he was working for Novell. In that meeting, then-CEO Eric Schmidt said that the internet needed a "directory", which was a great idea, he said, but one that Novell couldn't deliver. So Schmidt went to a startup (Google) and helped make it happen.

It is important to get a lot of feedback on an idea; "the more eyes on your idea, the better". Many in his field make the mistake of believing that their customers are the same as they are, so they design their products and such for themselves. Getting others to review product plans and other business ideas will help avoid that trap.

There may be a big change coming for publishing with the advent of e-books, he said. There is a belief that a "tipping point" exists once e-book sales reach 25% of printed book sales. Unlike some others in the publishing business, though, he is cautiously optimistic about his prospects if that happens. If it doesn't happen, he will be fine because he sells paper books. If it does happen, he has 200,000 readers that already read things online, so he is confident that he can find a way to sell content to them.

He has already done some e-publishing, starting with an app for the iPhone. He chose that market to start with because he heard that "iPhone users have big fat open wallets and low IQs", he joked. The truth is that iPhone users have a reputation for being more willing than Android users, for example, to pay for content. After a year with a subscription-model iPhone app, he made $1200, which he split with the developer. That was not enough income to continue that experiment, so he now has free ad-supported apps for iPhones, Android, and the iPad. The iPad is "the way to read my comic", he said, and believes that it will be a big player in the e-comics world.

Comments (6 posted)

Bradley Kuhn dives in full-time at the Software Freedom Conservancy

The Software Freedom Conservancy (SFC) has been in business for four years, so it seems about time for the organization to have a full-time employee. But is the organization ready to support a full-time employee? What started as a small offshoot of the Software Freedom Law Center (SFLC) is now home to more than 20 open source projects, with others waiting in the wings. It has a solid set of services for its organizations, but fundraising may be a challenge for now full-time Executive Director Bradley Kuhn.

Kuhn, known for his work as Executive Director of the Free Software Foundation and later with the SFLC, has been the driving force behind the Conservancy since its inception. As he writes in his introductory post, Kuhn has been doing the work to keep the Conservancy running in his "spare" time outside of his work with the SFLC. As a part-time volunteer, Kuhn has done a remarkable job of bootstrapping the Conservancy, but additional skills will be needed to make the Conservancy self-sufficient enough to carry one or more employees.

The SFC grew out of the SFLC, and was launched in April 2006. While the SFLC was equipped to provide pro-bono legal services for FOSS projects, there was an obvious need for more than just legal services.

What the SFC Does

The conservancy provides several services that projects need, but that developers are not ordinarily well-equipped or enthused to do themselves. The SFC provides a non-profit umbrella for projects, so that they can receive tax-deductible donations as a 501(c)(3), and provides fundraising assistance. The SFC also helps with contract negotiation, which comes in handy when a project wants to work with a contributor company, hold a conference, or any other activities that require contracts. If desired by the project, the SFC can also hold physical and legal assets, ranging from trademarks to servers donated to a project.

In other words, the SFC performs all manner of legal and financial work on behalf of projects so that contributors can focus on the project and not administrivia.

How do the client projects feel about the SFC? Matt Mackall of the Mercurial project, says that he's pleased: "Things have been good so far. They're fairly responsive and take a load of paperwork off our plate."

Mackall, primary author and project lead for Mercurial, says that the decision to join the SFC was spurred by a threat from BitMover to one of Mercurial's developers. Mackall recently decided to work on Mercurial full-time so, to enable that, donations come in for Mercurial to the SFC. He then receives payment as an independent contractor from the conservancy.

Aside from handling donations, Mackall says that the Conservancy has been helpful in many other ways ranging from making it possible to get T-shirts for contributors, to supporting Mercurial's licensing move from GPLv2 to GPLv2+, and assistance with developer sprints in Paris and Chicago.

The SFC isn't the only organization that does this, nor is it the first. For instance, Software in the Public Interest (SPI) has been serving in a similar role for Debian and other projects for many years. Kuhn sees it as a good thing that several non-profit homes for projects exist:

Next Steps for the Conservancy

Rapid growth is not in the cards for the Conservancy. In its four years, the SFC has signed up an impressive set of projects that run the gamut from Linux distributions to media players. The list includes more than 20 projects, from major and well-established projects like Samba to smaller projects like Libbraille, which provides a shared library for easily developing software for Braille displays.

The current projects aren't the only projects that wish to join the Conservancy. Kuhn says that there's a waiting list to get in and he hopes to have the resources to accept most of those in waiting. The SFC has standardized on letting in new members twice per year, to coincide with the biannual director's meeting.

Likewise, Kuhn isn't planning to expand the services offered by the SFC just yet.

One of the things that the Conservancy does offer is help with license compliance. The SFC has been busy in protecting the rights of BusyBox, which is often shipped in embedded devices without vendors complying with the requirements of the GPLv2. The SFC worked with the SFLC to file suit in December of 2009 against Best Buy, Samsung, Westinghouse, JVC, and several other companies for violating the GPLv2 in shipping BusyBox.

The SFC received a default judgment [PDF] against Westinghouse in July for $90,000 in damages and $50,000 to cover costs of the suit. However, Kuhn says that Westinghouse is trying to avoid paying the judgment, so it may be some time before the Conservancy can recognize the income. When it does, Kuhn says that the any money for damages are earmarked for the member project, while money for costs have gone to the SFLC to help it continue to serve its members.

Expect more of the same. Kuhn says that GPL enforcement is "definitely" an area where the SFC will be more active in 2011:

Funding

Finally, Kuhn says he wants to increase the Conservancy's available operating funds, but that will depend heavily on the generosity of the community and corporate sponsors. Right now, he says that the SFC has "about six months of operating expenses in cash for 2011, so my first goal is to bring that cash fund up to a year's worth of expenses and make sure I can deliver on a sustainable budget, including one employee."

Kuhn is the first full-time employee of the SFC, which raises the question of how funds will be raised to pay Kuhn's salary going forward. How does the Conservancy plan to raise money for that and other necessary services? First off, Kuhn is "bootstrapping," the move by being a full-time volunteer until 2011. He says that the SFC will be paying only for expenses until 2011.

Will the Conservancy be tapping member projects for funds? Kuhn says yes, but only for those that wish to contribute. "As part of Conservancy's fundraising strategy, we also ask our member projects to give a small percentage of funds raised in their earmarked account back to Conservancy generally. We hope to never have to make that mandatory, although we do have to tell very large projects that are in need of a lot of services from Conservancy staff that it will be difficult to assure ongoing services without this component."

Even if all projects choose to contribute, it still won't cover all the funding needs for a full-time salary — much less all of the services offered by the Conservancy. To that end, Kuhn says the Conservancy will seek "a diversified" donor base, and has been encouraging FOSS supporters to give to FOSS-related non-profits and not just the SFC:

Kuhn says that he doesn't want to depend too heavily on corporate donors, and doesn't have a specific plan to target donations from companies. He does talk to potential sponsors at conferences, and expects a contribution for 2011 from Google. So far the Conservancy appears to have done well by its members and on a shoestring budget, but a more aggressive and targeted fundraising plan would be more comforting. This is, perhaps, an area where the Conservancy should actively seek contributors to work on a fundraising plan and specific funding targets.

Funding concerns aside, the SFC has achieved quite a bit in its four years without the benefit of having a full-time employee. Providing the SFC can meet its fundraising goals to sustain Kuhn full-time, 2011 should be a very interesting year for the organization.

Comments (4 posted)