LWN: Comments on "Debian considering automated upgrades"

Debian considering automated upgrades

pabs — Sun, 18 Dec 2016 22:50:11 +0000

needrestart has more features and is generally better.

Debian considering automated upgrades

satbyy — Sun, 18 Dec 2016 20:58:00 +0000

I have been using checkrestart all these days. Is needrestart any different ? Both seem to do the same thing.

Debian considering automated upgrades

pabs — Sun, 18 Dec 2016 14:32:42 +0000

needrestart can be configured to automatically restart system services with upgraded libraries, so it should work quite well with unattended upgrades. What issues did you find with it?

Debian considering automated upgrades

kooky — Sun, 18 Dec 2016 12:33:43 +0000

I've used the cron-apt package for maybe 10 years now. With the -d flag removed so that it installed upgrades rather than just downloading. On lots of servers and desktop machines.

Mix together with https://packages.debian.org/jessie-backports/reboot-notifier and I know when I need to reboot. (makesure cron can send email)

[ I've always meant to write a script to do overnight reboots based on reboot-notifier, but I just haven't done it yet ]

It just works. I'd rather be secure and have a 30 second overnight outage every few months.

For a few absolutely critical servers, I do updates manually.

The needrestart package described is good, but it really doesn't work well with unattended upgrades.

Debian considering automated upgrades

Cato — Sat, 17 Dec 2016 09:34:38 +0000

A fairly nice Ansible role that installs and configures unattended-upgrades for Debian and Ubuntu: https://github.com/jnv/ansible-role-unattended-upgrades.git

Ansible is lightweight compared to Puppet/Chef/Salt etc - only requires SSH and Python on target server (and you can bootstrap to install Python if needed), and is easy to learn since its scripts are executed sequentially. See http://docs.ansible.com/

Debian considering automated upgrades

Cato — Sat, 17 Dec 2016 09:31:19 +0000

Snapshotting can help, but btrfs still seems not ready for production, and LVM snapshots are quite buggy - see http://serverfault.com/questions/279571/lvm-dangers-and-c...

Debian considering automated upgrades

davidstrauss — Fri, 16 Dec 2016 23:01:17 +0000

> The ideal solution here, of course, is reboot-less kernel upgrades

There is no such thing as "reboot-less kernel upgrades." There is live kernel patching, where you inject a modified implementation or filtering routine (usually to mitigate a vulnerability), but you're still running the old kernel. Eventually, such live patches will stack up and be too unwieldy to keep adding -- and the box will need to reboot. These live patches also have to be specifically implemented as live patches; you can't just take a kernel diff and naively make one. (For example, the upstream fix may alter a key data structure in the kernel, while a live patch usually can't do that and has to take a different approach.)

I'm no aware of any vendor, even the enterprise ones, that promises the ability to perpetually avoid reboots.

Debian considering automated upgrades

maniax — Fri, 16 Dec 2016 10:30:53 +0000

My 2c:

For the cattle farms and the pets I run and had run, I've seen very few problems with automated updates in the last 7 years, mostly on servers which were co-administated with people without enough experience. Otherwise, cron-apt does it job pretty well, and if you read debian-security-announce, you know what to expect and if there's something that will affect your code or services.
(and you need to remember to set to hold mysql and whatever db server you use, to skip the downtime on that and because that upgrade is one of those you need to watch)

I've tried to do the path of manual update (first on one server of the cluster, then on the rest), but with the quality of debian security updates, it's a waste of time, the automated stuff just works and you can check it in your email in the morning.

Automated release upgrades are a different matter, but the existing tools make that pretty easy to do them manually and seem to work fine (which has been one of the reasons for my dislike of ubuntu, where those were creating a lot more problems)

Debian considering automated upgrades

zlynx — Fri, 16 Dec 2016 04:10:08 +0000

This could be done somewhat safely with BTRFS or LVM. Make /usr read-only. Create a read-write snapshot of /usr, apply the updates to that snapshot and then somehow atomically replace /usr with the new snapshot. Just mount over it? Is there a way to unmount the old /usr after, or would this plan just result in an eventual pile of 100 copies of /usr?

Debian considering automated upgrades

zlynx — Fri, 16 Dec 2016 04:04:17 +0000

Library updates have, in the past, inconvenienced me. Of course, I do blame it on bad developers.

For example, some binary data file that is used by a set of applications. It is accessed through a library. The main application keeps on running, using the old library and the old data file.

But wait! What happens when some innocent DBUS event launches one of these small utility programs? It spins up, the new library determines that it should upgrade the binary data file, it locks it, updates it IN PLACE, and continues on. Everyone should be happy right?

No! Suddenly the old app that is still running has a new format to deal with and it has no idea what just happened.

Sometimes I think certain developers never actually use their own stuff. And if I recall correctly this was "fixed" by having the install script just kill everything that could have the file open.

Debian considering automated upgrades

JanC_ — Fri, 16 Dec 2016 01:48:40 +0000

Also see: https://github.com/mvo5/unattended-upgrades (the default configuration files are under /data/ )

Debian considering automated upgrades

JanC_ — Fri, 16 Dec 2016 01:42:16 +0000

That shouldn't really be a problem with 'unattended-upgrades':

* by default it only upgrades for security issues; you could also configure it to only upgrade from your own private tested repository, if you think that's necessary on certain critical systems
* it allows for a configurable blacklist of packages that will never be upgraded
* it can try to fix interrupted upgrades automatically (or you can disable this)
* you can configure it to delay upgrades until shutdown/reboot
* automatic reboot is possible but optional, can be scheduled to happen immediately or at a particular time, and when disabled it's easy for other programs or scripts to check if a reboot is wanted at a more opportune moment

etc.

And in any case, removing or disabling it is easy if that's what you really want/need.

Debian considering automated upgrades

pabs — Fri, 16 Dec 2016 01:00:18 +0000

The discussion was mostly about cloud stuff.

Debian considering automated upgrades

pabs — Fri, 16 Dec 2016 00:57:53 +0000

> There are no major changes in software in debian stable, that's the point of debian stable.

The browsers are updated to new major versions in Debian stable, since there is no sane way to provide normal security updates for browsers in 2016. Luckily Firefox has ESR, but still, that will eventually lead to having to backport Rust to be able to compile Firefox ESR.

Debian considering automated upgrades

st — Thu, 15 Dec 2016 23:58:22 +0000

I would expect that either pinning the necessary libs to specific versions using apt pinning or building the custom code into a .deb package that depends on specific versions of the libs would solve this problem while still allowing the system in question to benefit from automated updates of the rest of the packages.

Debian considering automated upgrades

mcatanzaro — Thu, 15 Dec 2016 23:38:19 +0000

Fedora tried this years ago and the result was bricked desktops when users powered off or rebooted during an update that they didn't know was even in progress. Some very careful attention should be paid to user-interface requirements in order to avoid rehashing this problem (not that I expect this to actually happen...). At least for the GNOME, which is dead-set on supporting only offline updates to improve reliability, that's all going to have to happen downstream.

I'm surprised that Debian is going the complete opposite direction of Fedora here on update reliability. It's one thing to automatically *prompt* users to install updates, or to do so automatically during a reboot... but completely unattended seems to make more sense for servers that are actually unattended.

Debian considering automated upgrades

Seegras — Thu, 15 Dec 2016 15:22:05 +0000

> What if this is a compute node that runs some self-compiled code
> which requires certain libraries.

If this really is your problem, the automated upgrades are not your problem.

I usually pummel those who deploy "self-compiled code which requires certain libraries." with some kind of heavy implement, in order to hammer in some sanity into their brains.

Debian considering automated upgrades

itvirta — Thu, 15 Dec 2016 13:44:16 +0000

> Changing these libraries while the process is running,

Library upgrades are usually done by unlinking the old version and creating a new one, instead of overwriting the file,
which would corrupt the binary image in memory.

> or upon a new start of the software (triggered by scripts or events), will eventually lead to problems that are unpredictable and difficult to recover from.

If a stable update contains more than bugfixes or the like, thus breaking any sensible application, I would count that as a bug with the updated package itself.
Of course, if you're running something very sensitive, disable the automatic updates.

> do we want reboots during presentations

Of course not. But unattended-upgrades doesn't reboot, it merely sends an email suggesting nicely that you might want to.

Debian considering automated upgrades

mikapfl — Thu, 15 Dec 2016 13:15:03 +0000

Well, on a compute node that runs some brittle self-compiled code, you just opt out of the automated upgrades. We are only talking about the default value changing, debian will certainly not make it impossible to change the configuration.
And for your fear of desktop usage breaking due to "major changes in software": We are talking about debian stable updates here. There are no major changes in software in debian stable, that's the point of debian stable. By the way: On the desktop, all the big desktop environments already have their own automated or semi-automated updating systems, so for the desktop this is not even new, it is the reality already.

Debian considering automated upgrades

tnoo — Thu, 15 Dec 2016 06:20:00 +0000

This looks nice, but is a really bad idea, especially on servers. What if this is a compute node that runs some self-compiled code which requires certain libraries. Changing these libraries while the process is running, or upon a new start of the software (triggered by scripts or events), will eventually lead to problems that are unpredictable and difficult to recover from.

And on desktops: do we want reboots during presentations, or major changes in software before an extended trip without reasonable internet connection? Don't follow the path of commercial dummy-user systems, we're all adults here.