Not logged in
Log in now
Create an account
Subscribe to LWN
LWN.net Weekly Edition for May 23, 2013
An "enum" for Python 3
An unexpected perf feature
LWN.net Weekly Edition for May 16, 2013
A look at the PyPy 2.0 release
Day: GNOME OS
Posted Aug 8, 2012 6:45 UTC (Wed) by drag (subscriber, #31333)
Example of trying to deal with a problems related to updating Linux software:
Posted Aug 8, 2012 8:07 UTC (Wed) by krake (subscriber, #55996)
I am not sure I understand all details, but it sounds like the Windows or OS X app stores do not overwrite the files programs when they apply updates.
Do they transition between versions on next boot/login?
Or do they shut applications down when they update them?
Posted Aug 8, 2012 8:40 UTC (Wed) by drag (subscriber, #31333)
Not sure how it works for either platform. I am very much a Linux guy and have ignored other platforms.
With Windows you have locks on the files you have open. So at least you can know if files you are using have been changed when you fork+exec stuff. With the application handling updates, like Windows versions of Firefox and Chrome do, then that gives you a lot of control. You just launch a update process and tell the user to stop the application before the application is applied. With Linux not allowing regular users to do updates and plus having the update process entirely outside of the control of the application this sort of functionality is probably impossible to replicate.
It is probably also worth examining closely how Microsoft has gotten to mitigate the problems surrounding 'DLL Hell' for the majority of use cases. Versioning issues can and do certainly occur in Windows, but for the majority of users it isn't going to a problem unless their system has been around for a very long time or they are dealing with very old and creaky software.
Going back to 'Gnome OS'...
It seems like the 'Gnome OS' approach that is currently being explored is to avoid the use of traditional Linux package management and utilize some modern Linux file system tricks to make it possible to install applications and much of their dependencies in unique directories while minimizing the amount of duplicated files between application directories.
Here is one of their initiatives right now:
> Fundamentally, OSTree is designed with the idea that having multiple "roots" installed is a normal condition, and is optimized for it.
On the surface it sounds similar to what I, and I believe many others, typically do to deal with installing video games on Linux. Since most of the time most 'native games' I want to play are not packaged directly for my distribution I very much prefer to download tarballs of binaries then to build them. Even if I build them I still want them separate from my 'OS'. So typically I just install everything in my ~/.local/ directory and launch from from scripts via the command line. Other people may have ~/apps or ~/games, but it's similar.
Also this is the approach taken by things like the old Cedega or the newer 'Playonlinux' stuff for installing non-native games. Each game gets their own tailored wine versions and environment since best compatibility with the distro-installed wine version is unlikely. They have many 'multiple roots' for each game's specific windows environment. This seems to work out really well for what is normally a very unpleasant/difficult activity (installing windows games on Linux)
I haven't heard about this before the above mentioned blog post so what I said in regards to Gnome's approach is likely largely incorrect.
Posted Aug 8, 2012 8:58 UTC (Wed) by krake (subscriber, #55996)
Of course. But the article doesn't cover privately controlled installations but how to cope with app store based updates.
Since they ran into a problem on Linux, it would have been interesting to know how the Windows or OS X app stores handle this.
Or maybe the article was misleading in that it is actually not about Linux being more difficult to handle but app store based application updates vs. privatly controlled ones.
In this case it would be strange that the two different approaches were not described for the same platform.
One has to wonder if the author's intent was to simply bash Linux by using a tightly vendor controlled installation on the other platforms and a store controlled installation on Linux, making the latter more complex to handle.
Of course this would be absurd but I've seen similar things way to often to completely rule it out.
Posted Aug 8, 2012 10:10 UTC (Wed) by drag (subscriber, #31333)
Another way to put it, more or less:
Due to Windows habit of locking files it has forced people to use a model of installing that avoids file clobbering issues present in Linux. Something else magically happens on OS X.
It seems to do the same thing in Linux with a system-wide package manager you would need a package manager that is aware of the processes that users are running, installs the software update to a alternate directory, sends a notification to users to shut down the application, and when they do that then rename the new directory over the old one.
Or something like that.
I am very interested in how iOS/OS X handles application updates through the app store. The blog posted hinted at versioning directories in the app bundle, but I have no clue what happens with iOS and such.
Posted Aug 8, 2012 11:36 UTC (Wed) by krake (subscriber, #55996)
Well, he clearly failed on the second part, since the posting does not contain enough information why that is not a problem on Windows and OS X.
For example it is missing the information krakensden added, i.e. that Windows can schedule a rename to the next reboot, allowing the app store to schedule replacement of files but on the downside also requiring system restart.
As I said it I don't want to assume the worst but it smelled a lot like comparing different types of upgrade mechanisms
Posted Aug 8, 2012 14:19 UTC (Wed) by drag (subscriber, #31333)
And yes it is comparing update methods. They need to do what they do in order to deal with some of the bad aspects of how Linux installs packages.
They could ignore that completely and have each user install Chrome in their home directory, but apparently it is easier to write a bunch of code to work around rpm/deb-style package management then to try to convince Linux users to not depend on rpm/deb for installing and updating Chrome/Chromium.
He also seems to indicate that it's not going to be a issue with them on Chromium OS.
Posted Aug 8, 2012 14:40 UTC (Wed) by krake (subscriber, #55996)
Well, than the blog posting is without merrit.
If you specifically chose a technique for something you should understand its properties. And then you deal with those properties. Doesn't make any sense to complain about having to deal with them when comparing with a technique that does not have them.
Like complaining that parallel processing with child processes needs extra work when accessing shared data and stating that multi threading would make that way easier.
"They could ignore that completely and have each user install Chrome in their home directory..."
"...but apparently it is easier to write a bunch of code to work around rpm/deb-style package management then to try to convince Linux users to not depend on rpm/deb for installing and updating Chrome/Chromium."
Well, if it is easier to deal with app store limitations than not to be in the app store than that is a conscient choice. Complaining on how easy it would have been not to be in the app store is just whining.
So marketing (or managment) decided that being in the app store outweights extra engineering effort. Deal with it!
Posted Aug 8, 2012 16:19 UTC (Wed) by cmccabe (guest, #60281)
Posted Aug 8, 2012 16:24 UTC (Wed) by drag (subscriber, #31333)
Posted Aug 9, 2012 8:31 UTC (Thu) by krake (subscriber, #55996)
They could have avoided the issues by going the route of a stand-alone installer but obviously found it more viable to do it the other way.
Posted Aug 8, 2012 12:40 UTC (Wed) by Mog (subscriber, #29529)
Posted Aug 8, 2012 13:17 UTC (Wed) by nix (subscriber, #2304)
Posted Aug 8, 2012 14:10 UTC (Wed) by drag (subscriber, #31333)
Well the application or user is given no notification that a update was performed for most (if not all) Linux systems unless the user is the one that did the update itself.
If it's a single user environment then it's not that big of deal, but I suppose it can cause headaches unless the administrator makes sure to upgrade the system while everybody is logged out.
Posted Aug 8, 2012 14:43 UTC (Wed) by krake (subscriber, #55996)
Posted Aug 8, 2012 16:22 UTC (Wed) by drag (subscriber, #31333)
The code used to do such things is suppose to be rather ugly, but unfortunately Google has terminated it's code search stuff and thus the links from his post showing what Chrome does is broken. I don't have time to hunt down the details myself right now.
Posted Aug 8, 2012 20:08 UTC (Wed) by nix (subscriber, #2304)
The thing is, a forking architecture like this is a good architecture for other reasons, compared to a plain exec() architecture. Among other things, it means that relocation happens only once, so it saves memory because pages dirtied by relocations are dirtied only once, and it saves time because relocation happens just once, at Chrome startup (an important consideration when the binary is 110Mb). KDE 3 used exactly this architecture just to minimize relocation overhead.
Again, this thing is *five hundred lines*. It's *trivial*. I've written similar things in under a day. And people are proposing massive invasive changes to every program storing state in /etc, and massive invasive changes to the way updates are carried out, to save on the terrible overhead of writing something tiny like this. Why? I am baffled.
Posted Aug 8, 2012 22:26 UTC (Wed) by walters (subscriber, #7396)
<p>What's even better - it's <b>still racy</b>. If the user launches an application while an update is happening, the app can get half of the old files, and half of the new. There is no way to solve this without coordination at every layer of the stack.</p>
Posted Aug 8, 2012 23:00 UTC (Wed) by nix (subscriber, #2304)
In fact with only a few exceptions (e.g. Perl if you want to use CPAN), you can mark such subtrees immutable and nothing whatsoever breaks. (I've been doing this for years.)
Yet again, this seems like a massive invasive change for very nearly undetectable benefit -- and it still doesn't solve the only instance of this problem that I can ever recall encountering, the 'updated application state in your home directory breaking downgrade' problem.
It's a shame I'll not be at LPC or we could have a good argument over this :)))
Posted Aug 9, 2012 15:41 UTC (Thu) by arafel (subscriber, #18557)
I think you might have misread the blog post:
Posted Aug 10, 2012 8:16 UTC (Fri) by nix (subscriber, #2304)
(Obviously this model will not apply to everything, but it does in fact apply to a lot. You need an abstraction around file I/O to do it, but Chrome already *has* that, and thanks to libio and open_memstream() anything can get it quite easily, without major rewrites.)
Posted Aug 8, 2012 16:48 UTC (Wed) by endecotp (guest, #36428)
On iOS, if you install an update to an app from the store, then any currently-running instance of the app is killed first.
Only one app is "foreground" at any time on iOS, so when the App Store app is in the foreground, the app being upgraded cannot be. When an app is backgrounded, it is told that it is going in to the background and it is expected to save its state. It might eventually be resumed but it could be terminated without any further notice due to power-off, low memory, etc. So the case of an update is not special.
I don't know how it works on the Mac, where presumably the app being upgraded might still have windows visible. My guess is that it is killed with some notice first.
Posted Aug 9, 2012 8:39 UTC (Thu) by krake (subscriber, #55996)
I wouldn't be so sure. The blog indicated that Chrome wants to be running during and after the upgrade and it said Linux makes that difficult and Windows and OS X don't.
Therefore we msut assume that the Mac store update process allows applications to continue to run. There is just no confirmation on that yet and not details how it does it.
Posted Aug 9, 2012 9:30 UTC (Thu) by hummassa (subscriber, #307)
Posted Aug 8, 2012 9:00 UTC (Wed) by krakensden (subscriber, #72039)
Indeed. The Windows equivalent of rename(2) actually takes a flag that schedules the move after the system restarts, which makes lots of install/upgrade activities kind of trivial.
Posted Aug 8, 2012 10:23 UTC (Wed) by slashdot (guest, #22014)
This is pretty much the worst feature of Windows.
It's absolutely horrible to not be able to replace files in use, when such an operation is clearly conceptually trivial as shown by Linux.
Don't copy it please.
The RIGHT way of solving the problem is not that, but it is:
1. Install applications in their own versioned directories instead of scattering their files all over
2. Add kernel support to unlink a directory, which once the last reference drops (or after reboot) causes recursive unlink of all its contents (or alternatively, use a daemon+initscript to do the same)
3. Make applications keep the their directory open as an fd (and by keeping a connection to the daemon if using that solution), and use openat() on that fd to open their files
4. Package removal works using the new kernel unlink feature, but the directory is still accessible via the application's fd and is physically deleted only when application is closed or reboot (due to #2 and #3)
5. Package upgrade works by removing the previous version as before, then creating a new directory with a different version number with the new one, then redirecting /usr/bin/APP to that directory.
Posted Aug 8, 2012 10:39 UTC (Wed) by hummassa (subscriber, #307)
Just move it to the an application-recyclebin on uninstall/upgrade. (3) becomes unneeded, (4) and (5) become moot. And you can rollback upgrades and uninstall easily. Empty/tidy the application-recyclebin on boot.
Posted Aug 8, 2012 11:36 UTC (Wed) by slashdot (guest, #22014)
So, you need either the kernel or a daemon that can notice the application files are no longer in use and can zap then quickly.
Not totally sure on the best mechanism for that, but it's definitely possible.
Posted Aug 8, 2012 13:16 UTC (Wed) by nix (subscriber, #2304)
(Personally, I keep my versioned per-app trees around for a month to allow for later downgrading. I have never had a single problem with the script that deletes them after that much time.)
Posted Aug 8, 2012 16:41 UTC (Wed) by rahvin (subscriber, #16953)
Posted Aug 8, 2012 17:11 UTC (Wed) by drag (subscriber, #31333)
It wouldn't be difficult to put a logic into the cron job to check to see if it's on battery and delay until it's on the charger or above 80% charge or something of that nature.
Posted Aug 9, 2012 13:53 UTC (Thu) by etienne (subscriber, #25256)
Posted Aug 10, 2012 8:18 UTC (Fri) by nix (subscriber, #2304)
Far better to symlink the new things into place in some shared tree. (Of course, symlinks have slight annoying semantic changes, so a few pieces of software need adaptation to this, though maniac stow and graft users like me have already done it for most stuff. If this was really Plan 9, we could just bind-mount everything into place in your per-user /bin, and have a command that refreshed all your /bin mounts when you wanted them refreshed, flash-updating you to the latest goodies but only when you wanted it.)
Posted Aug 10, 2012 9:20 UTC (Fri) by etienne (subscriber, #25256)
Posted Aug 8, 2012 14:27 UTC (Wed) by nye (guest, #51576)
Honestly, 99% of the solution would be simply to use a flag that performs the rename not on next reboot, but when all current locks on the file have been dropped.
The kernel needs to know that anyway, so it should be possible without some nasty polling daemon, and it would turn 'reboot system' updates into 'restart application' updates, which solves the problem for anything other than updating files belonging to system processes, for which it doesn't feel so unreasonable to have to reboot.
Each version of Windows requires fewer reboots than the last, so maybe there actually is a flag that does this and it's "simply" a matter of updating the entire world to use it(?)
(Ultimately that would provide similar update semantics to Linux, which might mean the same problem described up-thread unless some thought is put into it, but it *would* mean that MS get to keep the semi-mandatory locking they love so much)
Posted Aug 8, 2012 10:36 UTC (Wed) by nix (subscriber, #2304)
> Fundamentally, OSTree is designed with the idea that having multiple "roots" installed is a normal condition, and is optimized for it.
That's gonna fly.
Posted Aug 8, 2012 11:44 UTC (Wed) by walters (subscriber, #7396)
Posted Aug 8, 2012 13:13 UTC (Wed) by nix (subscriber, #2304)
This took me less than five seconds to think of, but *someone* thought that the problem was nasty enough to propose on the OSTree wiki the rewriting of all existing applications to not rely on /etc. Because that will be so much simpler.
Posted Aug 8, 2012 13:28 UTC (Wed) by walters (subscriber, #7396)
The page you're referring to is the *long term* plan. It's doable over time.
There are a variety of shorter term approaches, and I have an OSTree branch that in fact started on a "stupid merge" strategy like what you're describing.
Anyways, if you are interested in coding instead of commenting in a textarea, you can also join firstname.lastname@example.org - thanks!
Posted Aug 8, 2012 14:27 UTC (Wed) by walters (subscriber, #7396)
Posted Aug 8, 2012 16:33 UTC (Wed) by cmccabe (guest, #60281)
It seems like the feature you want (atomic updates) could be retrofitted to the existing systems. btrfs supports the concept of transactions at the filesystem level, which is exactly what you need here.
I'm pretty sure that the RPM guys would be overjoyed to have someone sane take over as the upstream maintainer. Please, please, please, think before you reinvent the wheel.
Posted Aug 8, 2012 16:53 UTC (Wed) by walters (subscriber, #7396)
Basically however you slice it, you need a way to have two operating systems installed. Whether they're squirreled away in a BTRFS snapshot or unpacked as hard links is really an implementation detail.
The *real* challenges are around configuration management, how you present this to the system administrator, etc. For example, it's clearly desirable if the system software management tool can tell you stuff like "you have N OS images installed, the additional snapshots are taking up XX megabytes of disk space compared to the current".
To do that kind of thing, you need something aware of *both* the rpm/deb state and the BTRFS state.
Furthermore, one huge advantage of the current OSTree design (that I need to promote more on the wiki) is that it parallel installs inside your existing distribution - you don't need to switch disk formats. All you sacrifice is disk space.
You don't have access to that package of bioinformatics software in the latest GNOME OS? No problem, reboot into your regular host distribution and use it. Switch back when you want to see the latest GNOME.
Posted Aug 8, 2012 21:10 UTC (Wed) by cmccabe (guest, #60281)
#1 can (probably) be solved just by batching everything up that rpm or deb does into a single btrfs transaction.
#2 is a little more subtle. Most update systems that I'm aware of don't even try to solve this problem. For example, Android forces you to restart your apps after you've upgraded them. (Application restarts are less of a big deal in Android, though, because of how they designed process lifecycle.)
In general, it seems sufficient to restart most applications. Maybe a flag could be introduced indicating whether the application needs to be auto-restarted, or whether it chooses to handle updates at the application level (by using inotfy, or by keeping a bunch of fds open like Chrome does).
Please don't bother re-inventing the chroot wheel for GNOME. I know how to set up chroots and virtual machines, thanks very much.
Posted Aug 8, 2012 17:01 UTC (Wed) by walters (subscriber, #7396)
(It's now on the wiki too)
Posted Aug 8, 2012 19:54 UTC (Wed) by nix (subscriber, #2304)
As far as I can see, the 'stupid merge' strategy -- or a smarter one with custom merge drivers -- is the only sane approach. What's the alternative? As far as I can see, there is none other than some bizarre intent to avoid all systemwide configuration forever, which, as I said, is just not going to fly with virtually any upstream you don't already control.
Any system that retains configuration state in any way will retain the problem of merging it -- and systemwide state isn't the largest problem in any case, it's per-user state, which is far more mutable and often contains references to data that must not be lost (be that emails or financial info). What are you going to do, wipe everyone's dotfiles whenever a downgrade takes place? Keep the config with the earlier trees, so if you make a change after upgrading it's *lost* on downgrade? Neither seems even slightly sane to me.
What OSTree is proposing is somewhat unclear, but appears to require rebooting on *every single package upgrade* so as to switch into a new chroot containing that package. That means several times a day for me. Not a bloody chance am I letting something like *that* near any of my systems outside a VM: it is transparently ridiculous and optimizing for a very few programs that might need to take extra measures to avoid being broken by updates happening underneath them, like Chromium, yeah, already *did*, so it is clearly not impossible: the zygote is also not exactly rocket science, weighing in at well under a thousand lines of code).
Now maybe laptops don't mind being rebooted this often, but a server with NFS-mounted home and scads of daemons and remote users relying on it, no way. (And that's where I do most of my work, upgrading and downgrading everything but the kernel when necessary with nary a hiccup, and no reboot-associated state loss. Yes, I know this setup is 'obsolete' now in the brave new Web 3.0 world where everyone uses horrible proprietary web services rather than anything like shared filesystems, but it meant I and some friends could band together and buy one big RAIDed box with ECCRAM that none of us could otherwise afford, and it means lower power costs than each of us owning a medium-big box.)
What's bizarre is that you reference nixos, which does most of this right (and which I am not associated with in any way, despite the coincidence of names: nixos *does* do some things wrong, including, last time I looked, paying no attention to the fact that most shared library upgrades do not break compatibility) -- and then you go on to propose a system which is as far as I can tell worse in every way in which it differs from nixos. Why?!
Posted Aug 8, 2012 20:05 UTC (Wed) by walters (subscriber, #7396)
On the "live updates" part: Absolutely nothing precludes having a mechanism which attempts to apply "live" updates from the new root to the currently running system. In fact, one could trivally reproduce the semantics that dpkg/RPM provide by just doing a "rsync --delete /ostree/trees/new/ /ostree/trees/current/" underneath the bind mount.
But the point is that comes *afterwards*. Make operations safe by default, and optimize later. But remember - there's lots of evil race conditions that happen on "live" updates as package managers do today.
Also remember that I fully intend to allow having a hybrid OSTree+package model, where you have a base system (basically ostree+dependencies) that must be updated atomically, but everything else can be mutated at runtime, if you like.
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds