Thunderbird and Fedora: what should a package update change?

The Fedora project recently experienced controversy thanks to an updated Mozilla Thunderbird package that surprised some users by introducing major changes. The resulting debate focused on when changes to a package are big enough that they warrant a new release instead of an update, and on Fedora's processes for testing and packaging updated upstream packages.

The Birds

Jeff Garzik first reported the problem to the fedora-devel-list mailing list on October 11. The update was to the package thunderbird-3.0-2.7.b4.fc11, and introduced two new features: "smart folders" mode and global search functionality via the "Gloda" global database. The smart folders mode is an alternate presentation mode of Thunderbird's folder pane; it combines folders such as the Inbox from multiple accounts into a single, unified folder. Gloda works by creating an index of all of Thunderbird's mail.

The updated package caused Garzik trouble because both new features were turned on by default, resulting in a surprise rearrangement of Thunderbird's Inboxes, and a sudden (and lengthy) freeze when Gloda — on its first run — attempted to build an index of Garzik's extensive email archive. Others on the mailing list reported similar surprise and dissatisfaction, noting that some Thunderbird users are required to keep their home and work email accounts separate for legal reasons, and that with several gigabytes of stored mail, Gloda not only slows the system down to the point of unresponsiveness during indexing, but it also consumes considerable disk space storing its index in the user's home folder.

Furthermore, there was no indication that this update to the package would introduce any substantial changes; it is beta 4 of Thunderbird 3.0, and the previous betas did not introduce either feature. The change-log of the package indicated only that beta 4 was a security update. Finally, also compounding the issue is the fact that Fedora 12 is scheduled to be released in November 2009, and Garzik argued that introducing a major change to a key package should have been postponed until the new release of the distribution rather than be pushed out to the stable release so late in the release cycle.

Fedora's official policy is to follow what the upstream package does, so the decision to turn on "smart folders" mode and Gloda in the update was following correct protocol because the change originated with the official Thunderbird release from Mozilla. Garzik's contention that the user interface changes and new features were too big to be pushed in what appeared to be a minor update received some support, but not consensus. The list did seem to agree, however, that by not mentioning the new features, the terse change-log contributed to the confusion.

Testing 1, 2, 3

Ultimately, the participants in the discussion thread reached consensus that the package update was handled incorrectly but that, more importantly, the project needed to study the event so that it could implement a process for better catching similar problems in the future. Fedora packages are published through the Bodhi system, in which maintainers submit packages for review, the release team approves packages and moves them into a special "testing" repository, and when sufficient testing has been performed, pushes the packages into the "stable" repository.

While in "testing," testers report their experience by voting +1 or -1 "karma" points. By default, when the package achieves a +3 karma score, Bodhi pushes it to "stable" automatically, although the maintainer can disable this behavior if he or she feels more testing is warranted. Several readers decided that the +3 threshold was too low for Thunderbird 3.0 beta 4, given the changes it introduced. Adam Williamson also dug into the test feedback, and noted that two of the +1 votes that eventually contributed to the automatic push actually reported problems in the attached comments, despite their positive vote.

Richard Hughes observed that getting adequate feedback from testers is a challenge with no easy solution — for any distribution. The karma threshold in Bodhi is set at +3 because few packages receive significantly more feedback. Benny Amorsen speculated that Fedora could recruit more users to test packages by notifying them of available test packages at login. Hughes and several other liked the idea, and suggested that a test-package notification system could be hooked into PackageKit, although if implemented it should remain deactivated in upstream PackageKit builds so as to not offend other distributions.

Rollback

In addition to the problem of recruiting more package testers, several on the fedora-devel-list felt like developing a general-purpose rollback or downgrade procedure was in order. The specific harm in the case of this Thunderbird update is not difficult to correct; "smart folders" mode and Gloda can be disabled as defaults in the next update. Fedora could also push a new package of Thunderbird 3.0 beta 4 with the disabled changes the only difference from the previous update.

The fix is not always so simple, however: if an application changes its data storage format in an upgrade, downgrading could cause data loss or unpredictable behavior. The list debated several possible strategies, including using operating system snapshots to allow the user to roll back a system, data included. In the end, though, there is no abstract way to ensure that an upgrade is completely reversible, including data formatting changes. The best Fedora or any distribution can do is keep a close watch for problems and trust its package maintainers to release fixes when an update causes trouble for end users.

Thunderbird 3.0 is expected to enter release candidate status the first week of November, so interested parties may wish to monitor the Fedora package to see whether or not the problems encountered with the beta 4 release reappear. Looking further forward, the possibility of a test-package-recruiting application could be good for not only Fedora, but other Linux distributions as well. Hughes volunteered to mentor student work on such a project for Google's Summer of Code — although that is still many months away.