LWN: Comments on "Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems"

Single Lennart of Failure?

zdzichu — Fri, 27 Mar 2015 12:42:10 +0000

I did some analysis last year, using LWN's who-wrote script. The results are at http://enotty.pipebreaker.pl/2014/12/30/who-wrote-systemd/

Single Lennart of Failure?

dsommers — Fri, 27 Mar 2015 11:13:19 +0000

drag++

This might give you an idea of how many contributes to systemd, as well as the development pace:
http://thread.gmane.org/gmane.comp.sysutils.systemd.devel

I'm quite sure someone will be able to pick up the pieces if Lennart or Kay moves on to other projects.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Lennie — Sat, 21 Jun 2014 19:28:06 +0000

Here is one method:

Use DHCP to assign a domain to the system so the machine can use DNS to look up the services you need to configure the system.

The service can supply machine specific information.

An other method is what is already used which is cloudinit.

It is used with virtual machines a lot, but can also be used with bare metal. Cloudinit is automatically started at startup and talks to a well-know local-link IP-address on the network. Usually using HTTP on 169.254.169.254.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

drag — Thu, 19 Jun 2014 13:21:41 +0000

You could choose to provide the configuration with file system labels.

Like LABEL=/var or whatever. Then configure they OS to check for those at boot up. That way you don't have to care about the underling hardware.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

zlynx — Thu, 19 Jun 2014 13:04:30 +0000

If you have a system with local disks, which is what I think you're talking about, then one solution would be to write a udev rule. Detect a RAID controller, launch a script, probe the controller and its drives and create a mount unit for it at /raid or /var/data or whatever you like. That way systems with locally attached storage get it set up, but it doesn't interfere with other systems that just have a boot drive.

Or just do it with Puppet or whatever configuration tool you're using. A stateless system isn't very useful until it has grabbed itself some state after all. The configuration doesn't need to be in /etc, it can be on a network service.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Thu, 19 Jun 2014 12:23:09 +0000

But that is a statement of fact. /etc/fstab is completely unnecessary on a systemd system; useful and convenient, yes, but not necessary.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

rwmj — Thu, 19 Jun 2014 12:19:05 +0000

My original reply was about your assertion "/etc/fstab is unnecessary with systemd, so that's not a problem... Systemd merely generates native mount units from it for backwards compatibility."

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Thu, 19 Jun 2014 11:59:24 +0000

I imagine this to be out of scope for what systemd needs to do; there are too many ways to solve the problem for it to make sense for systemd to provide the one true way.

I imagine with a stateless system a computer would start up, get an IP via DHCP or IPv6 autoconfiguration, then connect to a fleet cluster eg. via anycast or DNS, and start providing some service. No need for a persistent /etc at all.

Another approach could be a systemd generator included in the base image that does much the same, but dynamically generates systemd units under /run depending on its environment.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Thu, 19 Jun 2014 11:50:07 +0000

Why would you need /etc? The question was about stateless systems, which by definition do not have local configuration. Also, again by definition, a factory reset wipes out any local configuration.

Still, even in the cases where you do want /etc, you can use native mount units instead of /etc/fstab. It is strictly unnecessary on systemd systems. Just put your mount units in /etc/systemd/system/.

Of course, using /etc/fstab with systemd is perfectly fine... I do so myself. The whole point is that systemd doesn't care about it at all. The helper tool "systemd-fstab-generator" reads it if it exists and creates mount units for you during early boot (under /run/systemd/generator/), but the actual mounting logic in systemd doesn't even know that the file exists.

The same approach is used for other "legacy" configuration files such as /etc/crypttab

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

joib — Thu, 19 Jun 2014 11:41:17 +0000

My point is that in many cases some amount of state is necessary, and if one can solve how to "import" this necessary state into an otherwise "stateless system", one could then take advantage of the benefits of stateless systems without having to let go of the concept entirely.

I'm not saying systemd developers are responsible for solving this problem; they're doing what they can and "factory reset" capability is certainly useful for many usecases.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

rwmj — Thu, 19 Jun 2014 09:47:35 +0000

I'm thoroughly aware of how systemd works, thanks. However my local configuration is still going to need to go in /etc somewhere (and best of all in /etc/fstab), so creating .mount units is no solution.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

davidstrauss — Thu, 19 Jun 2014 09:44:32 +0000

> Systemd doesn't magically autodetect your mounts.

Actually, it supports that too, but for only specific mount points. The initial feature landed last year. "A logic to automatically discover and enable home and swap partitions on GPT disks has been added." [1]

[1] http://lists.freedesktop.org/archives/systemd-devel/2013-...

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Thu, 19 Jun 2014 09:32:53 +0000

You misunderstand how systemd works. *every* mount has a corresponding mount unit. Those can be defined natively as systemd unit files. However, for backwards compatibility, systemd generates mount units from fstab -- that's all. You can simply define all mounts as native units and delete fstab and nothing will break.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

rwmj — Thu, 19 Jun 2014 09:23:12 +0000

Unless you use NFS mounts, or want a permanently mounted second disk, or want to bind-mount a fast disk over /home or other cases where /etc/fstab is not redundant ...

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Thu, 19 Jun 2014 06:39:15 +0000

I don't quite understand your point. You would simply put all information needed to get the base system running outside /etc; mounts, services (like database servers, if it even makes sense to run those in a stateless environment) etc. would be configured via unit files under /usr.

Networking would have to be done via DHCP or some other means, but that or dynamically populating /etc is out of the scope here.

The issue before has been that /var and /etc contained important information that couldn't just be thrown away on each boot without hosing the system, making it next to impossible to implement a reliable "factory reset". Given the way systemd works, a factory reset would just rm /etc and /var and on the next boot all the default settings under /usr would be in effect until overriding configuration is put in place under /etc.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

joib — Thu, 19 Jun 2014 06:23:45 +0000

Right. So then you're back in square 1 (well, lets be generous and say 1.5) in that you need some configuration to tell the system where to mount the DB tables from (be it fstab or whatever systemd replaces it with). Hard to do if you don't have some kind of persistent /etc.

So how is stuff like that supposed to work in the brave new stateless world?

1. Look up stuff from DNS SRV records, etcd, or some network service like that? But that goes back to the previous problem, how do you know e.g. the IP's of the etcd servers, look them up in SRV records for your domain (which you got from dhcp)? Plus most services don't know how to do this, so until the future arrives one has to write scripts that generates config files e.g. from etcd data?

2. Custom rpm's?

3. Custom initramfs that populates /etc with your specific stuff?

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

sjj — Wed, 18 Jun 2014 22:40:21 +0000

Yeah, I think you're probably right - it all just feels adhoccy from the sidelines since there is no Central Linux Planning Office with five-year plans, rather than diverse groups and individuals exploring solutions. Also, future history might value the invention and adoption of git as high as linux itself. No closed source company can match a loose collection of geeks with github accounts, filled with laziness, impatience, and hubris, bent on reinventing the wheel.

And some kind of tipping point is almost here. It's fascinating how we seem to be converging on a Linux From The Future (Project Atomic, CoreOS) with containers and service discovery tools.

If there are any anthropology grad students around, a dissertation on the design evolution of this part of the FLOSS world would be interesting to read!

/blowiating

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Wed, 18 Jun 2014 22:00:09 +0000

Nothing.

Systemd doesn't magically autodetect your mounts. Systemd supports vendor-supplied unit files under /usr, so a stateless system would likely just use native unit files residing under /usr/lib/systemd/system/

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Wol — Wed, 18 Jun 2014 21:21:32 +0000

So what happens if (as I have done in the past) you have multiple /var or /boot or whatever partitions on the same machine? (Because lilo said "would you like SuSE or Slack or Gentoo?")

Cheers,
Wol

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

jspaleta — Wed, 18 Jun 2014 18:51:24 +0000

So now that there is infrastructure to make declarative system users possible, work has to be done to get packaging policy updated to first allow this to be used and then to make it the preferred way to do things.

As the blog says, distro policy in both rpm and deb camps might stand in the way of using this feature right now in official distro packaging. So there's probably going to be a little bit of heartburn around updating the packaging workflows and best practices. Hopefully not too much heartburn as I think the benefits are pretty obvious.

-jef

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

drag — Wed, 18 Jun 2014 17:44:31 +0000

'Serious DB installs' can be taken to mean installations done by people that actually care about their databases and put effort into configuring things correctly rather then just accepting the defaults; regardless of what particular type of DB they are using.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Chousuke — Wed, 18 Jun 2014 17:13:49 +0000

/etc/fstab is unnecessary with systemd, so that's not a problem... Systemd merely generates native mount units from it for backwards compatibility.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

jubal — Wed, 18 Jun 2014 16:52:52 +0000

Whatever it is you're implying about MySQL seriousness (and if you're not aware of it, MySQL IS serious enough for many industry players), that is a question of how you configure the mysql daemon. Most distributions, unfortunately, will put both replication log files along the mysql error log in /var/log/mysql.

Single Lennart of Failure?

drag — Wed, 18 Jun 2014 16:52:24 +0000

How many systemd contributers are there now? Hundreds?

Might as well speculate what will happen if all the GCC developers decided to drop off the edge of the world.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

drag — Wed, 18 Jun 2014 16:50:32 +0000

> Will it be useful in the real world? Hope so, I guess we'll see.

There is little doubt in my mind that it will be useful.

The traditional Linux OS is rapidly turning into little more then just a temporary runtime environment for kicking off applications. The more effortless you can make it to get a Linux OS bootstrapped the better off it'll be for everybody.

We are in a era were previously it took nearly a week to get a new OS environment deployed in a typical 'enterprise' environment (need to budget new hardware, justify it, order it, pick it up, install it, etc etc) down to a few minutes (RHEV or VMware 'clone virtualmachine') and eventually it's going to be less a second.

For the purposes of sandboxing and running things like Docker you will want the ability to launch new Linux OSes and destroy them several times a hour. On demand Linux deployments.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Darkmere — Wed, 18 Jun 2014 12:54:00 +0000

Most "Serious" DB installs already keep databases on one partition (and transaction-logs on another, to avoid the fsync issues) so wiping /var on boot is completely viable, as long as /etc is maintained so etc/fstab isn't disappearing....

Single Lennart of Failure?

clump — Wed, 18 Jun 2014 12:10:08 +0000

Suppose the systemd maintainers abandon their work. Recall devfs: https://lwn.net/Articles/139595/

It wouldn't be ideal if key maintainers disappeared, however it wouldn't stop Linux.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

cyperpunks — Wed, 18 Jun 2014 12:08:36 +0000

Clean /var at boot? Please setup replication of MySQL data in /var/lib/mysql before using :-)

Single Lennart of Failure?

rahulsundaram — Wed, 18 Jun 2014 10:50:17 +0000

It's still unclear. RHEL decision making is largely which patches to cherry pick or backport from upstream. There already are other people involved in that.

Single Lennart of Failure?

gvy — Wed, 18 Jun 2014 10:07:06 +0000

Rather meant decision making than just expertise.

Single Lennart of Failure?

tialaramex — Wed, 18 Jun 2014 10:06:29 +0000

Yes, this starts to remind me of the late-1990s UNIX® vendors saying well, you can't guarantee this Linux thing is going to stay around, and you can't be sure these x86 PC vendors will stay in business. Well gee, I know how they're going to stay in business, they'll use all this cash I'm going to give them instead of you.

And sure enough, seen from 2014 those points look ridiculous. Everybody wants to assure you that you're safe with them, but the reality is that it's the economics, if nobody else is using systemd in five years it'll take a lot of Red Hat effort to keep RHEL 7's systemd on life support. But most likely everybody will be using systemd in five years, and so it'll be no different from supporting X or GCC or dozens of other important complicated moving parts that everybody shares.

Single Lennart of Failure?

danpb — Wed, 18 Jun 2014 09:35:20 +0000

If you look at the systemd dev mailing list you'll see a large pool of active contributors and maintainers who are capable of sustaining the systemd community long term - systemd is far more than just a 1 or 2 person show these days. Red Hat too has more than adequate resources to be able to support RHEL systemd, and any other RHEL package for matter, even if individuals were to move on to other areas of interest. Such an important piece of RHEL system infrastructure is maintained by a group of people precisely to spread the expertise around and avoid a low bus-factor. IOW there's nothing to fear about long term systemd supportability.

Single Lennart of Failure?

dakas — Wed, 18 Jun 2014 08:56:00 +0000

In a worst case scenario, the source code is still there for others to fix or replace.

That's like putting your food store in a cave with hibernating bears. In a worst case scenario, there will be an animal trainer in the next travelling circus.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

burdi01 — Wed, 18 Jun 2014 08:34:20 +0000

If one regards stateless as just a(n initial) state then e.g. live CDs have done this for ages ... Also it is easier to freeze an actual state (including an initial one) than to create one from scratch. I still have a simple script and config file to do this for the PartedMagic distribution laying around.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

epa — Wed, 18 Jun 2014 05:50:38 +0000

Indeed, any kind of preinstall/postinstall script in an rpm or deb package should be considered a code smell. There should be a way to declare whatever is needed and have the package manager do it intelligently, rather than running arbitrary shell commands. (The shell commands will still be needed as a last resort when you want to do something nobody has thought of yet!)

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

jspaleta — Wed, 18 Jun 2014 04:23:47 +0000

I actually don't think this is ad-hoc. I think historically 2004 was sort of a watershed year for really really thinking hard about the state of things and picking a direction and a set of goals to shoot for.

We have Havoc's whitepaper on stateless _and_ 2004 was also the year we saw people band together and form up Project Utopia and start really addressing the need for dynamic hardware management.

I think systemd (and even upstart may it rest in peace) is actually part of an... intelligent design (NOT natural evolution! HA!)... of code implementations around some fundamental goals generated in the 2004 timeframe. No not ad-hoc at all... but a systematic and cyclical process implementation, refinement and eventually replacement with better implementations to better meet the goals as laid out way back in 2004.

I really don't see the container or visualization as being anything really new as a use case that wasn't provided for by the goals of Stateless Linux and Project Utopia together in 2004. Containers are an interesting use case only because its clearly becoming a dominant use case now, very quickly, and a usecase that benefits greatly from stateless. Whereas previously the uses cases that could really benefit from stateless were more niche.... well manicured thin client islands among a wild sea of fat clients in 2004.

But it does beg the question, is work towards "Stateless Utopia" a positive feedback loop? Sorry here comes the run on sentence that will bake your noodle...

As the people working towards a Utopic Stateless Utopia, got better at meeting the goals with newer code implementations, did their work make it easier for admins to move their workloads to configurations that depended on aspects of Stateless Utopia, in turn raising demand (and therefore the value) for even better Stateless Uptopia implementations?

I think perhaps that's what's happening. We've sort of hit the tipping point now in the last couple of years in terms of real world installs that benefit. And we've gotten here only because of the persistent development in the intervening years to just part of the way, dragging some of us kicking and screaming by our beards into the future.

-jef

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

kjp — Wed, 18 Jun 2014 03:44:14 +0000

amazing post. love the declarative system users.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

bronson — Wed, 18 Jun 2014 02:54:36 +0000

Well I can say this: stateless would make a few of my raspberry pi one-offs much MUCH nicer to use.

Will it be useful in the real world? Hope so, I guess we'll see.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

sjj — Wed, 18 Jun 2014 01:42:18 +0000

Thanks for that historical link. I for one welcome our new stateless overlords!

We've certainly moved a lot closer to stateless, in a kind of ad hoc way. Only some greybeards remember manual mknod in /dev, and even hand editing stuff in /etc on a modern linux laptop is becoming a lost art due to sensible defaults and autoconfiguration.

User data is scattered around the world, in islands in the cloud. Enterprises haven't bothered backing up user machines in years - they give you a network share or a Box account and it's your responsibility to copy stuff you want to keep. I would love a distributed filesystem I could use on a laptop though - connect to network and let it sync, detach and go.

Sysadmins who ssh to servers and edit files locally are going to find themselves unemployed. If it ain't in git, it might as well not exist. Server applications need to start looking for some service discovery mechanism for configuration.

Exciting times, and I'm glad there are people willing to push the envelope against the always-somewhat-surprising conservatism of some techies.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

clump — Tue, 17 Jun 2014 23:34:45 +0000

Stateless, still enjoying use in mainframe and HPC worlds, is hard to do properly. Enterprises have moved to virtualization and templates. There is far more overhead with a virtualized approach, but cpu, disk, and provisioning time are cheap enough. Easier, but not necessarily better.

Large scale deployments and cloud do make the stateless use case more interesting. Overhead is more expensive.

Docker has an interesting approach. Rather than messing with startup on the operating system, Docker discards the entire instance unless it's committed or kept running. Docker processes are "stateless" unless you'd like to commit them.

systemd and Docker have very different responsibilities, of course.