LWN Weekly Edition
Front pageSecurity
Kernel development
Distributions
Development
Announcements
->One big page
This page
Previous weekFollowing week
| |||||||||||||
DevelopmentThe State of ConaryOnce hailed as the "next-generation" of package management, Conary was introduced by Eric Troan in 2004 at the Ottawa Linux Symposium (OLS). Though Conary hasn't replaced traditional Linux packaging technologies, it is in wider use than one might think. The next release promises better system management, but is anyone actually using Conary, and where's it going? The answers are yes, and possibly beyond Linux. Conary was meant to solve some deficiencies that exist in standard package formats. For instance, that package versioning as expressed by RPM or Debian packages does not allow for branches, only a linear newer/older model. Conary was developed to make it easier for users to create their own distribution, from a collection of repositories. The idea being that one might pick and choose repositories from which to install GNOME, Firefox, etc., rather than getting all of their software from Fedora or Debian or Ubuntu. This has not quite come to pass, at least for most users. While there are distributions using Conary, the primary usage of Conary these days seems to be building custom distribution appliances for businesses. How is Conary Different?To learn more about Conary and its current state, we interviewed rPath's Michael K. Johnson, founding engineer at rPath and founding technical leader of the Fedora Project. Despite its age, many Linux users have probably never heard of Conary or only have heard of it in passing. Fewer still are likely to be familiar with the details. Though Conary is lumped into discussions of package management, it's a bit more than that. Conary is described as "distributed software management system" for Linux distributions, as opposed to a package management system. Rather than managing software as "specialized archives" (as Johnson calls RPMs and Debian packages), Conary packages are references to files in a database. The packages contain references to components, which are divided by their roles in a package — such as runtime requirements, documentation, libraries, etc. Conary actually works as a sort of distributed source control system. Software comes from specific repositories, and the associations are much more granular than Debian packages or RPMs. For example, it's possible to remove a file from the system and, when the package that owns the file is updated, the individual file is not reinstalled. Files are treated as first class objects in Conary, and can be managed individually if desired. Packages can have branches called shadows, a customized version of the package that references the original plus changes, or for minimal changes it's possible to have a "derived package" that applies changes without rebuilding a package. As the SCM heritage suggests, Conary also has rollback capabilities that are much more elegant than what is allowed by RPM or dpkg. Conary also allows for "groups," something like a metapackage or task, that pulls together the components that make up a collection of software meant to be installed together. GNOME or KDE might be distributed as a group, or a collection of server software that contains all of the libraries, applications, and supporting software that needs to be installed. In short, Conary introduces much more detail and flexibility in managing system software. Conary 2.2 is due out "soon," and Johnson says "near-final" snapshots are already being used in Foresight Linux development. Johnson says that 2.2 introduces a new and more flexible way to manage systems: Conary 2.2 introduces "system models", which you can think of as "groups lite". A system model allows you to describe concisely how a system is different from a group on which it is based. Instead of building a group for each unique software combination, you can build fewer base groups, and then express minor unique variations on a per-system basis where conflict is unlikely. It makes it possible to have a more dynamic building-block approach to configuring systems, without giving up the control that Conary provides. As an example, it is reasonable to have a model that expresses, 'This is one of my web application server systems, built from my group-mywebapp manifest. It is a Dell server, so I will add to it the Dell hardware support packages that my organization uses, which I have bundled together in group-dell-packages. It is being deployed in my Atlanta data center, so I need to add the administrative credentials required for all systems deployed in my Atlanta data center, so also include my atlanta-data-center-credentials package that sets up those credentials appropriately.' Johnson went on to give a detailed example of the process and group definitions behind the changes that would be required to set up the server, in just a few lines. The upshot here is that Conary 2.2 adds features that make it very easy to clone and manage systems in a few commands. System models are the primary new feature in 2.2, but Johnson says it also has memory improvements and uses less bandwidth. Conary and rPath AdoptionFor all its technical advantages, Conary (like rPath) has yet to take the world by storm. None of the major distributions have switched to Conary as their package management system or base for development. But that doesn't mean that it's not in use. Conary and other package management systems are not mutually exclusive. Johnson says there was an "epiphany," about what Conary could do early in 2009.
We realized that we could analyze packages for
different packaging systems (e.g. RPM) and provide the same essential group
build capabilities for sets of packages (e.g. RPMs).
Support for "encapsulating" other package formats was introduced in Conary 2.1. rPath has been offering custom versions of CentOS, Red Hat Enterprise Linux, SUSE Linux Enterprise, and rPath. If you don't see much of Conary in the wild, where is it being used? Johnson says that its largest audience is "enterprises using rPath's product line, based on Conary, to manage diverse systems on a massive scale," in other words "enterprise appliances." Johnson says that ISVs are also a significant audience for Conary, and are using Conary and rPath's tools to deliver software as software, hardware, and virtual appliances. He cites the Department of Energy, EMC, Fujitsu, IBM, and Qualcomm as customers who are using Conary and other rPath tools to build and manage software. In general, Johnson says that there are hundreds of rPath derivatives, but not all are public:
The derivative products we have in mind are
not like the typical 'respin.' Our derivative products will be less
visible, precisely because our products were intentionally "unflavored" so
that the derivatives wouldn't require co-branding.
Some derivatives that use Conary aren't rPath-based at all. Johnson says many customers are using rPath supported versions of SLES, RHEL, and CentOS to create multiple products or "system definitions" from those platforms. Of course there's also Foresight Linux, which is based on rPath and Conary. Foresight has had its ups and downs, and was on the ropes briefly when rPath laid off the developers who were working on the distribution. There's also interest in using Conary with major distributions, albeit in a slightly different way. For instance, there's Boots, a Conary-encapsulated mirror of Fedora. Interest in Boots started when Johnson proposed a change in direction for Foresight Linux. And Conary has been adopted for some derivative versions, like the Openfiler Storage Appliance. Beyond LinuxIn 2005, Johnson suggested that Conary was not limited to Linux, and could be by the BSDs and other operating systems. So far, Johnson says that rPath has "not received significant feedback" from customers saying it'd be worthwhile for the company to package any of the BSDs. It's technically possible, but not in demand. But the company is building support for managing Windows packages. Johnson says that the company is building in support to create MSI installable packages for Windows, and the company specifically is hiring for field engineers that have experience with MSI packaging and other Windows system provisioning. Johnson says that the company is also working on managing other package types, so it may not be long before the rPath rBuilder tools support creating Ubuntu or Debian based appliances as well. Though Conary has not replaced traditional package management for most Linux users or developers, nor has rPath become a household name for Linux users, it's more successful than one might think at first glance. Conary may well be worth a look for developers and ISVs that create software appliances, or for enterprises that wish to have more control over the management of their systems.
Brief items Development quotes of the week
In fact, overall Py3k uptake is very slow. Which isn't very
surprising, as there's very little to gain by switching to it and
more than a little headache.
-- Matt Mackall
At any rate, most of the world appears to still be on Python 2.6 and can blissfully ignore that's there's no 2.8 planned for probably another year or more. Which means the Python devs still have a year to come to their senses about discontinuing the language people actually use.
Recently, a few python packagers from a couple Linux distributions
started thinking about wanting to port more python modules to
python3 to aid in migration and respond to the user demand for
python3 versions of some software. We came to the conclusion that
we're all feeling our way down this path, writing a little patch
here and a little patch there as we try to make a package here or a
package there compatible. This is good organic growth but has some
limitations: different distributions tending to reinvent the same
changes, patches floating around unaccepted by upstreams, and
common porting issues having to be discovered by each person
individually.
-- Toshio Kuratomi working to improve the situation
The Oracle employees who are members of the OpenOffice.org project
and who expressed themselves these past days have displayed a
disturbing lack of understanding of Free and Open Source Software;
LibreOffice is, after all, and until proven otherwise, a downstream
version of OpenOffice.org, and as such deserves inclusion into the
OpenOffice.org community. I can only imagine what it would be like
if Debian was rejecting the Ubuntu employees among its teams,
calling it a fork.
-- Charles
H. Schulz
The sad thing in the bigger picture, is to see the community of
companies grow (and fail) faster than the lessons / experience of
past failures percolate through to their leadership. In my
experience most people totally miss the most difficult piece of
software engineering: which sounds like it should be software - but
is really about people - and more importantly - them working
together collaboratively in a constructive, friendly, and
incremental fashion.
-- Michael
Meeks
There is not one out-and-out success story of a company building a
great high-quality custom user interface on the standard Linux
stack, except Android, which is hardly a model of collaborative
software development.
-- Dave
Neary
Asterisk 1.8.0 releasedVersion 1.8.0 of the Asterisk telephony system has been released; this is a major, long-term-supported release. New features include secure RTP support, IPv6 SIP support, calendaring integration, a new call logging system, and more; see the change summary for more information than you will ever possibly be able to use. (Thanks to Graham Cantin).
DVDAuthor 0.7.0DVDauthor is the tool which does the real work behind every other Linux-based DVD authoring application. Version 0.7.0 has been released; changes include better encoding support, more flexible configuration, and more.
KDevelop 4.1 releasedKDE.News has the KDevelop 4.1 announcement. There are improvements in patch exporting, script integration, PHP support, and a new hex editor, but the headline feature seems to be Git support. "That means that we have support for the basic features for management of a VCS-controlled project, like moving, adding and removing files inside the project. Additionally we integrate the basic VCS features like comparing and reviewing local changes, sending our changes back to the server, updating the local checkout and annotating files."
The Mordor I/O libraryMozy, an online backup provider, has announced the release of some of its code under the BSD license; that includes the Mordor C++ I/O library. "Mordor is a high performance I/O library. One of its main goals is to provide very easy-to-use abstractions and encapsulation of difficult and complex concepts, yet still provide near absolute power in wielding them if necessary."
Mozilla's "Chromeless" projectMozilla Labs has announced the "Chromeless" project, aimed at making it easier for developers to create browser interfaces. "The 'Chromeless' project experiments with the idea of removing the current browser user interface and replacing it with a flexible platform which allows for the creation of new browser UI using standard Web technologies such as HTML, CSS and JavaScript." There is a "pre-alpha prototype" available now.
Shed Skin 0.6 releasedShed skin is a Python-to-C++ compiler; the 0.6 release is now available. It includes some major changes in how program analysis is done, allowing it to scale to larger ("several thousands of lines") programs. See the announcement for more information.
Valgrind 3.6.0 releasedValgrind 3.6.0 is available. New features include support for the ARM architecture, updated distribution support, an understanding of the SSE4.2 instruction set, an improved profiler, and experimental new heap profiler, and more; see the release notes for details.
Newsletters and articles Development newsletters from the last week
Gmail vs. Zimbra Desktop 2.0 (Linux Magazine)Over at Linux Magazine, Joe "Zonker" Brockmeier takes Zimbra Desktop 2.0 for a spin, comparing it to the email interface that Gmail provides. While there is much to like with Zimbra, which is open source, he found Gmail to be easier to use. "Another major feature for Zimbra is that it allows you to use pretty much any mail service. So you can tie Zimbra Desktop to an IMAP server that you control, and all your mail belongs to you. Totally. For some folks, this feature alone is going to make Zimbra (or another email client) far more desirable than Gmail. While I am sometimes uneasy with Googles ever-increasing collection of data, Im not personally concerned that someone at Google is reading my email. And Googles Gmail reliability has improved to the point that I havent had a problem reaching my mail in several months."
Rosegarden - An open source MIDI / audio multi-tracker (The H)The H has a lengthy review of Rosegarden. "If access to computers and networks have given us the means to copy and distribute recorded works, they have also given us the means to create our own music. If computers and networks have made it possible to take the programming art into the home, they have also taken the black arts of the recording studio closer to the back bedroom, and in so doing, have made the making of music more accessible to greater numbers of people - much as free software has made it easier for 'hobbyist' programmers to join in and test their skills and make a difference. Tools for GNU / Linux and free software have played their part in this evolution, and Rosegarden is a significant part of the canon, a well structured MIDI / audio sequencer and musical notation editor with a well thought out user interface which has put usability and ease of learning to the fore."
Yocto Project aims to standardize embedded Linux builds (LinuxDevices.com)LinuxDevices.com has an overview of the Yocto Project just announced by the Linux Foundation. "Unlike build systems based on shell scripts or makefiles, the Yocto Project automates the fetching of sources from upstream sources or local project repositories, says the project. Its customization architecture is said to allow the choice of a wide variety of footprint sizes as well as control over the choice or absence of components such as graphics subsystems, visualization middleware, and services. Yocto is based on the GNOME-derived Poky Linux, a well established platform-independent, cross-compiling build system that uses the same architecture as the OpenEmbedded build system. "
Page editor: Jonathan Corbet |
|||||||||||||