LWN: Comments on "Cuthbertson: NixOS and Stateless Deployment"

Cuthbertson: NixOS and Stateless Deployment

hitmark — Fri, 16 Jan 2015 12:26:21 +0000

To me it seems the NixOS approach is easier to bail out if shit hits the fan and you are left with basic tools.

But that is coming from having used a distro similar to NixOS for a few years, and had to fix my own idiocy during that time.

Cuthbertson: NixOS and Stateless Deployment

mmarq — Sun, 11 Jan 2015 19:23:57 +0000

I haven't test it yet... i think is 3) (manipulating vfs)... i came across it, and point it out because while Lennart work is yet to came, an not so soon, and seems quite complex envisioning changes to most filesystems and more functionality for systemd (as if needed more push back!), Bedrock you can already test it... simply 'there is genius in searching for simple solutions' ;)

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Thu, 08 Jan 2015 17:57:14 +0000

You can also use user namespaces to map uids/gids inside an application to the global system IDs. That's how Lennart's proposal plans to do this.

Cuthbertson: NixOS and Stateless Deployment

Limdi — Thu, 08 Jan 2015 16:21:03 +0000

As I think of it: uid/gid problem

When installing a package adduser/useradd does some stuff and adds a username with corresponding uid/gid to /etc/shadow.

When I have an apache2 server (debian) which serves sites from /var/www/html and install a package on another os (arch) dokuwiki. It installs for example some files into /var/www/html/dokuwiki(it does not do that, just as an example). Both have the user www-data:www-data as owner of these files.

But when merging both layers, when forcing the arch-dokuwiki to be served from apache2 of another os, they have probably not the same uid/gid.

Apache2 can most likely read as XX4, but it cannot write to it(changing configuration, ..), since its mounting the files with the wrong uid/gid.

There are several ways of handling this:
1. I dont care.
2. Ignore permissions when accessing files from other os'es
3. Merge /etc/shadow's -> same users are allowed to access their uid/gid counterparts in other os'es
4. Replace uid/gid's with names in the filesystems. Instead of storing uid/gids one can store the owner-name and owner-group in the filesystem-entry, or one might have a transformation table, which owner-name/owner-group corresponds to which uid/gid on the current filesystem.

To 1: Some function will not work as expected, see dokuwiki example.
To 2: Security implications?
To 3: I dont know how uid/gid access is implemented. But I suppose a kernel module like vfs would be able to allow such access. It would be necessary to access /etc/shadow, possibly.
To 4: Not doable. Not enough cared before, not enough will care in the future I suppose.

Cuthbertson: NixOS and Stateless Deployment

Limdi — Thu, 08 Jan 2015 14:21:43 +0000

What does Lennart address?
* Hard-reset
* Runtimes/frameworks -> upstream can package against it
* Multiple os'es beside each other
* Easy/possibly automatic downgrade(on fail)

What bedrock addresses
* Multiple os'es beside each other
* Automatic resolution of dependencies which are not installed on the os being satisfied by another os (only reproducable if all client's are reproduced, cause dependencies get satisfied automatically. or document access to other os'es)
* Force usage of files of other os'es
* Easy downgrade(on fail)

Did I forget something?

I like the idea to just be able to use packages/files from other os'es, which might not have the same bugs as the os im on right now.

Is this possible with Lennart's proposal too? I mean when the packages are very similiar. Otherwise bedrock's approach wouldn't work anyway I think.

Cuthbertson: NixOS and Stateless Deployment

fandingo — Thu, 08 Jan 2015 01:36:55 +0000

> i don't quite follow ?... Opensuse 13.2 as example ships with btrfs as the production root fs... it uses the snapshot feature as a backup feature with possibilities of rolling back (like in windows)... and if i understand correctly what Lennart proposes with systemd + btrfs snapshotting ability - and extending it to other fs - is more like a versioning system for system images and packages.

The root FS has nothing to do with how Lennart's proposal would function. Each application and layer thereunder would be a separate Btrfs subvol that is read-only.

Cuthbertson: NixOS and Stateless Deployment

walters — Wed, 07 Jan 2015 22:48:08 +0000

rpm-ostree does solve that specific problem of no longer used packages: https://github.com/projectatomic/rpm-ostree/blob/master/d...

Here's a commit which adds a package: https://git.fedorahosted.org/cgit/fedora-atomic.git/commi...

And here's a commit which performs an atomic swap (adding one, removing another): https://git.fedorahosted.org/cgit/fedora-atomic.git/commi...

Removing a package from that json would also drop it out of client systems.

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Wed, 07 Jan 2015 22:20:15 +0000

> If your script doesn't work on the new servers that are "correctly" specified, then you didn't test the script on the right servers :-)
Yet this happens quite often. If you do not enforce a policy that you MUST test every change on clean servers then errors WILL eventually creep in.

I've seen this leading to vicious loops when people don't want to test changes on clean servers because they are afraid to be bitten by other people's bugs.

> It seems ridiculous to say that there's a fundamental flaw in Puppet because someone might *accidentally* delete a line of config they need.
Why not?

It's like with Python vs. Haskell - it's not a Python's flaw that you can write a program that fails at runtime because of incorrect types or race conditions in the mutable data. You must test your code thoroughly, after all.

However, there are other approaches that eliminate these problems entirely.

Cuthbertson: NixOS and Stateless Deployment

PaulWay — Wed, 07 Jan 2015 22:08:11 +0000

If your script doesn't work on the new servers that are "correctly" specified, then you didn't test the script on the right servers :-)

As I said, this is not about "what happens if we rely on a package that isn't there", this is "what happens if we don't rely on a package that is there". And my corollary to that is "and why are we doing that anyway?" It seems ridiculous to say that there's a fundamental flaw in Puppet because someone might *accidentally* delete a line of config they need.

Have fun,

Paul

Cuthbertson: NixOS and Stateless Deployment

mmarq — Wed, 07 Jan 2015 21:07:32 +0000

i don't quite follow ?... Opensuse 13.2 as example ships with btrfs as the production root fs... it uses the snapshot feature as a backup feature with possibilities of rolling back (like in windows)... and if i understand correctly what Lennart proposes with systemd + btrfs snapshotting ability - and extending it to other fs - is more like a versioning system for system images and packages.

I like the 'container ideas', but for more simple systems it makes simple upgradings quite tedious and cumbersome.

... and it doesn't solve the upstream problems. For it, all dependencies should be in the same package (that is, includes runtime + framework + config ) and it will be a single package or meta package... source code package... meaning the developers only have one main version (scratch) and the package systems will be clever enough do deal with 'deltas' on installing, and the systems and filesystems clever enough to deal with the different versions of binaries installed.

So IMHO the best solution doesn't pass so much by systemd and or snapshotting filesystems, but as example giving to the "git versioning system" its 'Nix like' package management ... lets call it upstream enforced, don't pull downstream up, force upstream down ... and the systems and filesystem only have to have 'Bedrock like' ability by simply manipulating the virtual filesystem http://bedrocklinux.org/introduction.html

Now upstream only does it once plus upgrades plus versions choosing their own runtimes and frameworks, and the distro packagers can repackage it to RPM DED whatever they like ( the in git versioning system package manager should help).

Then i believe the tendency for the the binary only distros will be to re-factory everything to their traditional approach ( perhaps the cutting edge devel versions could be source based), but the source code distros will tend with time to mount the git repositories directly into end users systems, and no matter what automated build systems, and end user package systems they came up with, there could be direct upstream to the very downstream...

Cuthbertson: NixOS and Stateless Deployment

mmarq — Wed, 07 Jan 2015 19:56:14 +0000

Funny some (perhaps most) of the 'features' that Lennart Poettering talks about achieving with a possible/probable next systemd developments and btrfs...

http://0pointer.net/blog/revisiting-how-we-put-together-l...

seems already addressed in here

http://bedrocklinux.org/introduction.html

manipulating the virtual filesystem ! ... interesting...

Cuthbertson: NixOS and Stateless Deployment

raven667 — Wed, 07 Jan 2015 19:45:55 +0000

> I mean btrfs is far from a race horse, mater of fact it loses badly for ext4 and xfs in many tests,

Just to point out that btrfs would only be used in a read-only context so a huge class of possible problems, such as (re)write performance and fragmentation over time causing slowdowns or any advanced feature bugs, can't be a concern.

Cuthbertson: NixOS and Stateless Deployment

mmarq — Wed, 07 Jan 2015 19:26:04 +0000

> The base system is guaranteed to be immutable and have a stable ABI, perhaps only incorporating minor security fixes.

There will be the day!.. if you talk of all the distros the same.

The major business of a distro, i mean a major distro not re-spins, is differentiation... i'm afraid this will only intensify the "need" for each distro to having their one runtime(s), their own framework(s), their own made little differences...

I like the idea of "empowering upstream" and i like most of the ideas of systemd, but i don't know if the idea of it being so much 'filesystem centric' is the best approach... isn't it going to impact performance ? ... i mean from a development point of view is no problem, but for all the performance suckers linux die-hard users in us, is a regression, and all are serious.

I mean btrfs is far from a race horse, mater of fact it loses badly for ext4 and xfs in many tests, is it because this subvolume + snapshot ability ? ... and now they want to give it to ext4 and xfs ?? (talkin very bad ideas perhaps).

If the idea empowering upstream is to resolve the 'dependency hell' why not develop a package system with advanced "chroot" abilities ... Nix Package System is a very good start, it only misses this "chroot" ability (IMHO) for 'in package dependency resolve' instead of scanning zillions of different package keys, and for making it really usable as updating system where a simple trivial bug fix doesn't have to be another package version ( the same with systemd snapshots)

Cuthbertson: NixOS and Stateless Deployment

sjj — Wed, 07 Jan 2015 16:11:33 +0000

It also makes perfect sense if you happen to run on imperfect hardware (by my estimation this covers 100.0% of available hardware).

And yes, imperfect sysadmins. This also covers 100.0% of available stock. I have been tempted to set root's $EDITOR to "poweroff" on occasion. Setting chattr +i on key files is, sadly, a surprisingly effective first measure.

Cuthbertson: NixOS and Stateless Deployment

niner — Wed, 07 Jan 2015 10:53:40 +0000

"It makes little sense to use any configuration management for one server."

Oh but it does as soon as you have more than one admin. Or even with one admin, it might help. Not so much for reproducing the configuration to other machines but for documentation of needed changes. It helps answer questions like "Why did we change this setting?", "Why did we install this package?" or "Is this workaround needed anymore?".

Cuthbertson: NixOS and Stateless Deployment

fandingo — Wed, 07 Jan 2015 04:44:22 +0000

> Maybe I don't understand the idea, but what qualifies for having existence here? Config files? Documentation? Logs? Temporary files? Tests? My own notes? Backup copies? Do I have to specify all of these things explicitly or they get deleted? What if I forget something?

The key part is that most top-level directories (besides /var, /tmp, /home, and /run from the article) are mounted read-only. So if you *can't* create, say /etc/httpd/httpd.conf, outside the nix configuration tool. The read-only limitation for those "stateless" directories is what makes the goals of Nix work.

I have a ton of experience with Puppet and Chef. After years of working with them in different situations and environments, I've soured on their benefits. Traditional configuration management utilities are just too easy to work around, especially when there are many admins involved, that the consistency and reproducibility never gets there. I've only read about Nix, but it seems like the enforcement for configuration is a major benefit.

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Wed, 07 Jan 2015 03:13:47 +0000

> I agree with his idea of the 'snowflake server', and I agree that we should try to see servers as "cattle not pets" to use the current term of art. But in his example, is there a *problem* if the 'foo' package is installed on a server that doesn't need it?
Yes, there is a problem. You might in future make a change (for example, a maintenance script) that uses the deleted package 'foo'. It will work on old installations that still have this package but will break on new clean installs.

And that happens with an awful regularity in practice.

Cuthbertson: NixOS and Stateless Deployment

rahulsundaram — Wed, 07 Jan 2015 00:58:46 +0000

> It makes little sense to use any configuration management for one server.

I use Ansible for such purposes and it works really well. Agentless and can be run against a local system or over ssh.

Cuthbertson: NixOS and Stateless Deployment

dlang — Wed, 07 Jan 2015 00:40:19 +0000

> it makes little sense to use any configuration management for one server.

I disagree with this, you need some sort of configuration management for that one server so that when it dies, or when you need to upgrade to a new OS version, or you need to move to newer hardware, you can recreate the server with confidence.

Now, for one server, this can be backups and notes, not a complex thing like Puppet. Pick a tool that's appropriate to your needs. But you do need to have something.

The "don't put anything on there that's not absolutely needed" falls from the "if it's not installed, it can't cause problems" mindset (in part from security), but it does need to be tempered with the complexity of the number of different configurations that you need to test. It's frequently worth having some packages installed that aren't absolutely needed in order to reduce the number of different builds that you need to maintain.

Cuthbertson: NixOS and Stateless Deployment

PaulWay — Wed, 07 Jan 2015 00:26:49 +0000

Strange that "everything not specified is left alone" is treated as a bug. To me this is an excellent feature of Puppet - it allows one to learn how to use it without having to place ones entire system in the hands of ones CMS and face potential obliteration.

Maybe I don't understand the idea, but what qualifies for having existence here? Config files? Documentation? Logs? Temporary files? Tests? My own notes? Backup copies? Do I have to specify all of these things explicitly or they get deleted? What if I forget something?

I agree with his idea of the 'snowflake server', and I agree that we should try to see servers as "cattle not pets" to use the current term of art. But in his example, is there a *problem* if the 'foo' package is installed on a server that doesn't need it? Why, in this hypothetical example, has the sysadmin deleted this requirement to have 'foo' on the servers? If it's so important to remove, why haven't they instructed Puppet to remove package foo on all but the servers that require it? Puppet can do that. It sounds like this is a straw man.

Likewise, this 'everything not allowed is denied' approach sounds needlessly strict. Do you have to list every package in your system there? Every dependency? What if that list changes? How do you try out new things? How do you reverse an upgrade - it's certainly not by reverting to the previous configuration in your configuration management. Do you specify ${editor} on each server? Should you even need to?

And fundamentally the 'servers are cattle' argument is flawed, to my mind, for most environments. It's a privileged position taken by people who have massive data centres to look down on that long tail of sysadmins who manage one, ten, fifty machines but do not need any more than that. It makes little sense to use any configuration management for one server.

I generally agree with configuration management, but NixOS fails to convince me.

Have fun,

Paul

Cuthbertson: NixOS and Stateless Deployment

lynxlynxlynx — Tue, 06 Jan 2015 22:47:17 +0000

... labeled as "alpha preview" on their own site. I'm sure nix has its own problems (something about password storage comes to mind), but it's beyond alphas and betas.

Cuthbertson: NixOS and Stateless Deployment

dlang — Tue, 06 Jan 2015 18:58:14 +0000

and "all of them" really is the right answer for a distro like Debian.

Packages (with _very_ few exceptions) should not care if they are being run on a bare metal install, in a VM, in a docker container, or any of the other variations (most of them should not be able to tell, if they can, it's a flaw in the container/vm that they are running in)

and each option has advantages and disadvantages, so Distros should support all of them.

Cuthbertson: NixOS and Stateless Deployment

SEJeff — Tue, 06 Jan 2015 18:43:06 +0000

Oh I totally agree. There is no reason Debian legitimately can't do all of these. I was just moaning the recent init system that shall not be named debacle that caused so much A-grade talent to leave or dial back their effort in the project. I don't blame it on said init system, but on poor politics and governance of the project.

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Tue, 06 Jan 2015 18:38:35 +0000

Why not, in this case? Debian can perfectly well have Snappy and Lennart's packaging system with Rocket containers running along.

Cuthbertson: NixOS and Stateless Deployment

SEJeff — Tue, 06 Jan 2015 17:53:34 +0000

Clearly with Debian the answer will be, "ALL OF THEM!". If anyone disagrees, a GR will be proposed. It is quite sad really.

Cuthbertson: NixOS and Stateless Deployment

SEJeff — Tue, 06 Jan 2015 17:52:20 +0000

That isn't 100% true if you want to be pedantic. Project Atomic uses rpm-ostree[1], which they document[2] how to make your own updates with. Looks pretty straightforward to me.

[1] http://www.projectatomic.io/docs/os-updates/

[2] http://www.projectatomic.io/blog/2014/11/build-your-own-a...

Cuthbertson: NixOS and Stateless Deployment

njd27 — Tue, 06 Jan 2015 13:44:31 +0000

Looking forward to the debates when Debian decides which of these to adopt!

Cuthbertson: NixOS and Stateless Deployment

drag — Tue, 06 Jan 2015 13:40:04 +0000

Probably would want to throw in there the 'Rocket' containers and the App Container Specification, which you covered a bit by linking CoreOS.

https://github.com/appc/spec/blob/master/SPEC.md

https://coreos.com/blog/rocket/

http://containerops.org/2014/12/19/docker-vs-rocket-gimme...

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Tue, 06 Jan 2015 10:47:04 +0000

Ubuntu Snappy very much exists. I don't know anything more about Lennart's proposed system.

Cuthbertson: NixOS and Stateless Deployment

lynxlynxlynx — Tue, 06 Jan 2015 10:44:52 +0000

But do these two exist in a practical state yet? Last I heard the systemd one was just a blog post with no code, while I'm not familiar with the other.

Cuthbertson: NixOS and Stateless Deployment

wmf — Tue, 06 Jan 2015 01:19:31 +0000

AFAIK ChromeOS/CoreOS/Atomic only solve the problem for the OS itself; they don't really support library or app packages (this isn't much of a problem for CoreOS/Atomic since you're supposed to only run Docker containers on them). Nix/Lennart/Snappy let you compose a system out of multiple immutable components.

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Tue, 06 Jan 2015 00:50:16 +0000

Well, we've tried practically all other approaches. What's the saying, "sysadmins will always do the right thing, after trying everything else"?

Classic stateful server management doesn't really scale because it's too labor intensive. So distributed systems people had to deal with reproducible configurations earlier...

Cuthbertson: NixOS and Stateless Deployment

SEJeff — Mon, 05 Jan 2015 23:57:59 +0000

3 more actually...

1. ChromeOS - http://www.chromium.org/developers/design-documents/softw...
2. https://coreos.com/using-coreos/updates/
3 Redhat's Atomic Host (which has RHEL/CentOS/Fedora variants): http://projectatomic.io/

As far as I'm aware, all 3 of those ship updates as entire operating system images and write them out to a separate partition. So you reboot to upgrade or fall back to the previous operating system.

It is kind of funny to think that the "new" thing to do is Blue / Green deployment at an individual host level for OS upgrades. The distributed systems camp (much of what I do) has been doing this for some time.

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Mon, 05 Jan 2015 22:31:13 +0000

They are built on a similar idea - you take some base system (which is semi-immutable) and install immutable packages (frameworks, apps) on top of it.

The base system is guaranteed to be immutable and have a stable ABI, perhaps only incorporating minor security fixes.

Then when an application is started, the OS manager creates a new container and bind-mounts all the dependent packages on top of the base system. There is no mutable package state - if you change some of the packages then you need to create a new system starting from the base.

This is very similar to NixOS, except that they use absolute paths for dependencies and they build everything from source to use these paths.

Cuthbertson: NixOS and Stateless Deployment

deepfire — Mon, 05 Jan 2015 22:07:41 +0000

Cyberax, now, having read up some on the system proposed by Lennart[1], I'd like to hear how you think it's similar to NixOS.

To me, the similarity appears.. surfacial, at best.

--
1. http://0pointer.net/blog/revisiting-how-we-put-together-l...

Cuthbertson: NixOS and Stateless Deployment

Cyberax — Mon, 05 Jan 2015 21:50:58 +0000

There are another two system that closely mirror the way NixOS works.

1) Lennart's proposed "package management" system.
2) Ubuntu's new application packaging system.

Cuthbertson: NixOS and Stateless Deployment

SEJeff — Mon, 05 Jan 2015 21:13:08 +0000

Also worth mentioning is GNU's guix, which is basically nix written in Guile.

http://www.gnu.org/software/guix/