Fedora mining for COPR

Ubuntu's Personal Package Archive (PPA) feature is a popular way for developers—and, in some cases, project teams—to publish binary packages without risking too much upheaval to the end user's local package set. Ubuntu has offered the service since 2007, and while other distributions have rolled out similar services of one form or another over the years, Fedora is now targeting the complete "PPA-like" offering with an effort of its own. That effort is called the COPR project, which has existed in limbo for several years. But COPR recently acquired a new maintainer, who is asking hard questions about how much of the service Fedora should really be building from scratch.

According to the Fedora wiki, COPR stands for "Cool Other Package Repo," though there is little reference to that acronym expansion anywhere else (perhaps understandably, for those who remember the origins of KDE). In any case, the COPR project was initiated in 2010 with an eye toward offering a PPA-like service for Fedora developers, although it appears that no substantial development began until 2012.

Building packages and the case for PPAs

Of critical interest was duplicating the ease-of-use of the Ubuntu PPA system. The Launchpad PPA service automatically builds programs for both the 32- and 64-bit x86 architectures, with any number of Ubuntu releases available as the target environment, and serves up the resulting packages as an Apt repository. Users can add the PPA repository with a variety of Apt tools, then install, update, and remove the PPA-provided packages just like those from the distribution itself. Just as importantly, developers only need to upload source packages to the system—although, of course, they are responsible for hunting down and fixing any build errors.

On the Red Hat/Fedora side, it has long been possible to set up a private RPM repository. Over the years many such repositories were established and proved popular for acquiring packages outside of—or newer than—the official Fedora releases. Dag Wieers, for example, ran his eponymous "DAG" repository for many years using a homebrew build system, and later followed it up with the RepoForge project. But both required packagers to oversee the build process manually.

More recently, the Fedora People Repositories project offered Fedora packagers server space to publish unofficial RPM packages, although again, manual effort was required to set up the build environment and to publish the resulting packages. The wiki also states that the repositories "should only be used for packages that are intended for end-user non-transient use," a use case that excludes putting packages out for public review.

At the same time, Fedora has developed the Koji build system to automatically build RPM packages for the distribution's core components. It would seem, then, that Koji is a natural fit to take the place of the manual package-building process usually required of those wishing to publish a personal RPM repository.

But there is another build system that has seen wider usage than Koji, the Open Build Service (OBS). Originally designed by openSUSE, OBS can be used to build packages, package sets, or even entire distribution releases, targeting a wide variety of distributions and architectures.

COPR returns

As mentioned earlier, the COPR concept has been around since 2010, but it was not until late 2012 that Slavek Kabrda and Seth Vidal began coding on it in earnest. They produced a working prototype (although it has not been open to public testing). But due to the tragic loss of Vidal in July and Kabrda's commitments to other projects, the future of COPR seemed, at the very least, uncertain.

But the project soon gained the attention of Miroslav Suchý, who has since undertaken a reassessment of the community's specific requirements of a PPA-like service, and has started asking hard questions about the architecture best suited to implement them.

In Suchý's first blog post on the subject, he enumerated the higher-level use cases for the service. The existing COPR prototype takes a single source RPM as input, builds the package on a virtual machine, and imports it into a Yum repository. In addition to that basic functionality, Suchý said, COPR needs to be usable for projects built on top of Fedora (such as OpenShift or the various Fedora SIGs' "spins"), it should provide personal—perhaps even private—repositories for users, and it should facilitate automated rebuilds of upstream packages for testing purposes. These missing features are what will make COPR useful for teams of developers, for individuals doing testing or branch development, and for monitoring the build-ability of packages.

Those useful outcomes constitute the real goals of the system, after all: making COPR a service that improves the development process, not just automating a single task. But the problem is that COPR is currently a standalone system, not integrated with the other pieces of the Fedora infrastructure. To meet the community's real needs, Suchý said, COPR should either be rebased on Koji or Fedora should adopt OBS. Koji, of course, is already a Fedora project, but OBS offers more features.

In a second post, Suchý explored the costs and benefits of rebasing COPR on Koji. There would need to be changes to Koji itself, including adding support for multiple packages of the same name scattered among different user accounts (and for multiple branches of a project within one user account) and adapting Koji to build packages on virtual machines instead of in chroots. On the other hand, those changes to Koji would improve it in its own right. In addition, the Fedora project would benefit from improved tooling all around, because other components (like mock and various other existing build scripts) will receive attention. The downside would be that Koji cannot easily be modified to add support for monitoring and automatically rebuilding packages, and its support for collaborative team features is minimal.

Suchý then examined the prospect of rebuilding COPR on top of OBS. In the plus column, OBS already supports many of the features missing from Koji (such as teams, package monitoring, and multiple branches of a package). It is also more mature, has six full-time developers, and has a more full-featured web interface. In fact, OBS already supports virtually the full feature set desired for COPR, including automatically publishing a Yum repository for built packages; the existing COPR prototype code would not really be necessary.

In the minus column, however, OBS uses a significantly different build system than the one used to build Fedora packages elsewhere. The workflow would be different than the one used to build official packages on Koji, and dependency resolution is performed using a different utility. A different workflow would require re-educating packagers, but using a different dependency resolver brings in the possibility that Koji and OBS would actually produce different packages.

The great debate

Suchý does not propose an answer in his posts. There are pros and cons to each path forward, and he estimated six to seven months would be required to implement either solution. Ultimately, he put the question to the community to decide—which, for a community-driven distribution like Fedora, is probably the most appropriate approach.

The options received quite a bit of debate on the fedora-devel list. Some, like Colin Walters, found the prospect of differences between the OBS and Koji build systems to be a major problem (and a point in favor of the Koji-based approach). Toshio Kuratomi said that asking the Fedora system administrators to add OBS to their already-full plates would be too costly, while others pointed out that asking packagers to learn two build systems would increase the packager's workload (in addition to the technical differences between the systems). There were also security questions raised, since OBS uses virtual machines rather than Koji's chroot-based approach.

In the end, Suchý concluded that the consensus leaned toward rolling out the existing version of COPR in the short term, and working on integration with Koji over the coming seven months. But he did not give up on OBS altogether. Although few in the mailing list discussion were excited by it, Suchý said he still thought it offered more than COPR and Koji can together, so he also vowed to "try to get (in spare time) OBS to Fedora anyway and build some community around it. And revisit the decision in two-three years." So Fedora packagers actually have more than one option to look forward to, regardless of which build system they prefer.