LWN: Comments on "GUADEC: Work in the pipeline for GNOME"

GUADEC: Work in the pipeline for GNOME

guillemj — Tue, 28 Aug 2012 20:13:36 +0000

> 1) You always have the previous working system to boot into. If NetworkManager breaks, you may set up a pin locally for the old version, but that doesn't do you a lot of good if you don't have networking to download it =)

That's why front-ends like apt store previously downloaded packages and keep them in a cache on disk (I'm guessing yum might behave in a similar way).

> 2) The release manager has the option to push arbitrary versions at any time - the closest analogy I can think of would be if someone made a Debian package containing pins. Say it was like "known-good-versions.deb", that dropped a file into /etc/apt.conf.d/known-good-versions.conf.

In Debian, this role is taken by the testing distribution.

> Basically I think reverting back to older known good versions should be a nearly-constant operation in the development tree. It should be easy for both users and release managers.

Sure, but that does not really require downgrading, one can always prepare a new package that backs out a specific change, in the same way you'd do with a project on a git repository, going always forward (like your mozilla example). It does require someone to bisect the problematic code though, but that needs to be done anyway at some point. Either that or the user can revert to a previous local version.

GUADEC: Work in the pipeline for GNOME

walters — Sat, 25 Aug 2012 16:53:07 +0000

There's bits of documentation on how things work:

https://live.gnome.org/OSTree/VCS
https://live.gnome.org/OSTree/Ostbuild
http://git.gnome.org/browse/ostree/tree/src/libostree/REA...

And these are all derived from my original README file which is here:
http://git.gnome.org/browse/ostree/tree/README.md

I've been trying to move that last one into individual wiki pages, but it's probably the best overview from a "hacker" perspective still.

GUADEC: Work in the pipeline for GNOME

walters — Sat, 25 Aug 2012 16:42:33 +0000

Mmm...there are two fundamental differences versus a user pinning packages on their own machine.

1) You always have the previous working system to boot into. If NetworkManager breaks, you may set up a pin locally for the old version, but that doesn't do you a lot of good if you don't have networking to download it =)

2) The release manager has the option to push arbitrary versions at any time - the closest analogy I can think of would be if someone made a Debian package containing pins. Say it was like "known-good-versions.deb", that dropped a file into /etc/apt.conf.d/known-good-versions.conf.

Basically I think reverting back to older known good versions should be a nearly-constant operation in the development tree. It should be easy for both users and release managers.

If you look at how Mozilla makes Firefox, they "back out" changesets often. Just run "git log --grep=Back" in mozilla-central. Here's an example.

GUADEC: Work in the pipeline for GNOME

scottt — Sat, 25 Aug 2012 06:40:47 +0000

> And Epoch has serious drawbacks - for a package like gtk+, we'd have to go and reflect the Epoch bump in every other package using gtk3-devel.

Why would this be required? Because a bug in the gtk3-devel headers requires a recompile of everything built against it? (but surely that's rare)

GUADEC: Work in the pipeline for GNOME

jengelh — Sat, 25 Aug 2012 00:55:43 +0000

>both of them by default look for upgrades

Of course they will — they have no other means. However, "upgrade" is a misnomer, it should really be called "preferred version".

Developers tag their packages such that a tool can decide which among two is more preferred. If gtk-2.27 somehow is more preferred than gtk-2.28 due to a bug, you, as a {upstream and/or distro-level} developer, have to indicate the preference to the tool, *somehow*. Whether that happens by means of Epoch prefixes, amount of lolcats shipped alongside the package, or manual intervention by the end-user (pinning), the choice is irrelevant.

And ostree, the way you describe it, looks a lot like pinning by the end-user. A local "master" branch in a git clone is similarly pinned — you have to manually run pull/merge to change it.

GUADEC: Work in the pipeline for GNOME

daglwn — Fri, 24 Aug 2012 22:01:55 +0000

This is a really neat idea. I'm not clear on how OSTree uses git, however. Are you updating cloned workareas from many upstream source repositories and then rebuilding into a new root filesystem or are you keeping a global "every file in the filesystem" image. That is, are you storing binaries in git?

GUADEC: Work in the pipeline for GNOME

walters — Fri, 24 Aug 2012 18:38:08 +0000

Do note though the current gnome-ostree build is actually broken for a few reasons =) I don't actually run it myself because I need to finish making /etc work. However, to see it log into GDM and show gnome-shell at least, you can download this revision.

GUADEC: Work in the pipeline for GNOME

walters — Fri, 24 Aug 2012 18:01:04 +0000

Oh, there are many possible workarounds. "yum" does have "distro-sync". Debian's apt has "pinning", and will downgrade. But both of them by default look for upgrades. And the default is very influential on the architecture of the entire system.

For example, in Fedora's Koji, packages that aren't tagged as being part of some "release" are subject to garbage collection. So if GTK+ regressed, you might find that the previous package had already vanished from the build server/archive mirrors, so you can't even download it.

And what's more important - there's no sane mechanism to push a revert on the build server side, unless you resort to Epoch or ~really. And Epoch has serious drawbacks - for a package like gtk+, we'd have to go and reflect the Epoch bump in every other package using gtk3-devel. That's extreme pain.

More realistically of course, what you would do is cherry-pick a "git revert" of the breakage. But at that point - you're lying about the Version field. You're fighting the system.

The combination of OSTree and the gnome-ostree build system fixes this in two separate ways:

1) You always by default have at least the previous OS you booted into before deploying a new update. In fact, on my laptop, I have hundreds of individual GNOME build snapshots stored in /ostree. And each build points to the exact git repositories and revisions that built it. So I could fork off what I have locally at any time.

You'd be amazed at just how much more pleasant it is to follow Free Software development when you have a "go back to what worked" button locally. In fact - you have two separate undo buttons. OSTree doesn't affect at all whatever distribution you have installed. So you can just reboot back into that too.

2) On the gnome-ostree side, I can pick the git revision of any module I want at any time. And I have used this power several times so far to keep things working while we decide what to do.

It's hard to describe just how powerful this is. For example, I could decide one day to switch the entire tree to KDE. You'd do a pull, and it wouldn't affect your running system at all; there's nothing equivalent to RPM/dpkg deleting files underneath GNOME while it's running. You just download the delta, then deploy a new root, and reboot into it. The next day, I could switch back. And you would basically only download 64 bytes of checksum pointing to the original GNOME tree, and switch back.

GUADEC: Work in the pipeline for GNOME

jengelh — Fri, 24 Aug 2012 17:13:37 +0000

>boot into a pre-regression OS until a fix is available. Such an option is impossible with traditional packages like those used in RPM and Debian systems, he said, because they rely heavily on the version numbers assigned to packages — and the definition that version numbers must strictly increase over time

Why do they have to strictly increase? What about rpm --install --oldpackage gtk-2.0...rpm?

GUADEC: Work in the pipeline for GNOME

walters — Thu, 23 Aug 2012 02:11:44 +0000

Thanks for a good article!

I have just a few minor corrections to the OSTree segment:

* It is entirely possible to deliver "asynchronous" security updates; what I more meant is that doing so requires dedicated person-power that I don't believe we have. The entire system is designed to allow forking/merging, and a security update is no different from a slightly modified tree.

NixOS on the other hand really suffers on the security update side because the package system forces a rebuild of *everything* when libc changes. See https://live.gnome.org/OSTree/NixOSComparison

* ostbuild actually monitors over 200 git repositories right now, not just one. Every commit to X.org gets immediately built for example. Some things are fixed at specific tags though - my one server would kind of choke doing continuous integration of WebKitGtk =)

* The marketing/branding thing is really independent of the build system. It's more a generic issue when shipping something more directly consumable than git repositories.

I wish the article had highlighted more what I consider the biggest issue - the inability of both dpkg and rpm to go backwards. I'd revert Fedora rawhide for example back to a state where it could start GDM this very second if it were possible. Reverting is not always the right answer of course - it's important to debug things. But it's a lot more important to have a constantly usable development tree.