Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems [LWN.net]

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 12:08 UTC (Wed) by cyperpunks (subscriber, #39406)
Parent article: Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Clean /var at boot? Please setup replication of MySQL data in /var/lib/mysql before using :-)

to post comments

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 12:54 UTC (Wed) by Darkmere (subscriber, #53695) [Link] (19 responses)

Most "Serious" DB installs already keep databases on one partition (and transaction-logs on another, to avoid the fsync issues) so wiping /var on boot is completely viable, as long as /etc is maintained so etc/fstab isn't disappearing....

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 16:52 UTC (Wed) by jubal (subscriber, #67202) [Link] (1 responses)

Whatever it is you're implying about MySQL seriousness (and if you're not aware of it, MySQL IS serious enough for many industry players), that is a question of how you configure the mysql daemon. Most distributions, unfortunately, will put both replication log files along the mysql error log in /var/log/mysql.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 17:44 UTC (Wed) by drag (guest, #31333) [Link]

'Serious DB installs' can be taken to mean installations done by people that actually care about their databases and put effort into configuring things correctly rather then just accepting the defaults; regardless of what particular type of DB they are using.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 17:13 UTC (Wed) by Chousuke (subscriber, #54562) [Link] (16 responses)

/etc/fstab is unnecessary with systemd, so that's not a problem... Systemd merely generates native mount units from it for backwards compatibility.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 21:21 UTC (Wed) by Wol (subscriber, #4433) [Link] (8 responses)

So what happens if (as I have done in the past) you have multiple /var or /boot or whatever partitions on the same machine? (Because lilo said "would you like SuSE or Slack or Gentoo?")

Cheers,
Wol

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 18, 2014 22:00 UTC (Wed) by Chousuke (subscriber, #54562) [Link] (7 responses)

Nothing.

Systemd doesn't magically autodetect your mounts. Systemd supports vendor-supplied unit files under /usr, so a stateless system would likely just use native unit files residing under /usr/lib/systemd/system/

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 6:23 UTC (Thu) by joib (subscriber, #8541) [Link] (5 responses)

Right. So then you're back in square 1 (well, lets be generous and say 1.5) in that you need some configuration to tell the system where to mount the DB tables from (be it fstab or whatever systemd replaces it with). Hard to do if you don't have some kind of persistent /etc.

So how is stuff like that supposed to work in the brave new stateless world?

1. Look up stuff from DNS SRV records, etcd, or some network service like that? But that goes back to the previous problem, how do you know e.g. the IP's of the etcd servers, look them up in SRV records for your domain (which you got from dhcp)? Plus most services don't know how to do this, so until the future arrives one has to write scripts that generates config files e.g. from etcd data?

2. Custom rpm's?

3. Custom initramfs that populates /etc with your specific stuff?

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 6:39 UTC (Thu) by Chousuke (subscriber, #54562) [Link] (3 responses)

I don't quite understand your point. You would simply put all information needed to get the base system running outside /etc; mounts, services (like database servers, if it even makes sense to run those in a stateless environment) etc. would be configured via unit files under /usr.

Networking would have to be done via DHCP or some other means, but that or dynamically populating /etc is out of the scope here.

The issue before has been that /var and /etc contained important information that couldn't just be thrown away on each boot without hosing the system, making it next to impossible to implement a reliable "factory reset". Given the way systemd works, a factory reset would just rm /etc and /var and on the next boot all the default settings under /usr would be in effect until overriding configuration is put in place under /etc.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 11:41 UTC (Thu) by joib (subscriber, #8541) [Link] (2 responses)

My point is that in many cases some amount of state is necessary, and if one can solve how to "import" this necessary state into an otherwise "stateless system", one could then take advantage of the benefits of stateless systems without having to let go of the concept entirely.

I'm not saying systemd developers are responsible for solving this problem; they're doing what they can and "factory reset" capability is certainly useful for many usecases.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 11:59 UTC (Thu) by Chousuke (subscriber, #54562) [Link]

I imagine this to be out of scope for what systemd needs to do; there are too many ways to solve the problem for it to make sense for systemd to provide the one true way.

I imagine with a stateless system a computer would start up, get an IP via DHCP or IPv6 autoconfiguration, then connect to a fleet cluster eg. via anycast or DNS, and start providing some service. No need for a persistent /etc at all.

Another approach could be a systemd generator included in the base image that does much the same, but dynamically generates systemd units under /run depending on its environment.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 13:21 UTC (Thu) by drag (guest, #31333) [Link]

You could choose to provide the configuration with file system labels.

Like LABEL=/var or whatever. Then configure they OS to check for those at boot up. That way you don't have to care about the underling hardware.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 21, 2014 19:28 UTC (Sat) by Lennie (subscriber, #49641) [Link]

Here is one method:

Use DHCP to assign a domain to the system so the machine can use DNS to look up the services you need to configure the system.

The service can supply machine specific information.

An other method is what is already used which is cloudinit.

It is used with virtual machines a lot, but can also be used with bare metal. Cloudinit is automatically started at startup and talks to a well-know local-link IP-address on the network. Usually using HTTP on 169.254.169.254.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 9:44 UTC (Thu) by davidstrauss (guest, #85867) [Link]

> Systemd doesn't magically autodetect your mounts.

Actually, it supports that too, but for only specific mount points. The initial feature landed last year. "A logic to automatically discover and enable home and swap partitions on GPT disks has been added." [1]

[1] http://lists.freedesktop.org/archives/systemd-devel/2013-...

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 9:23 UTC (Thu) by rwmj (subscriber, #5474) [Link] (6 responses)

Unless you use NFS mounts, or want a permanently mounted second disk, or want to bind-mount a fast disk over /home or other cases where /etc/fstab is not redundant ...

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 9:32 UTC (Thu) by Chousuke (subscriber, #54562) [Link] (5 responses)

You misunderstand how systemd works. *every* mount has a corresponding mount unit. Those can be defined natively as systemd unit files. However, for backwards compatibility, systemd generates mount units from fstab -- that's all. You can simply define all mounts as native units and delete fstab and nothing will break.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 9:47 UTC (Thu) by rwmj (subscriber, #5474) [Link] (4 responses)

I'm thoroughly aware of how systemd works, thanks. However my local configuration is still going to need to go in /etc somewhere (and best of all in /etc/fstab), so creating .mount units is no solution.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 11:50 UTC (Thu) by Chousuke (subscriber, #54562) [Link] (2 responses)

Why would you need /etc? The question was about stateless systems, which by definition do not have local configuration. Also, again by definition, a factory reset wipes out any local configuration.

Still, even in the cases where you do want /etc, you can use native mount units instead of /etc/fstab. It is strictly unnecessary on systemd systems. Just put your mount units in /etc/systemd/system/.

Of course, using /etc/fstab with systemd is perfectly fine... I do so myself. The whole point is that systemd doesn't care about it at all. The helper tool "systemd-fstab-generator" reads it if it exists and creates mount units for you during early boot (under /run/systemd/generator/), but the actual mounting logic in systemd doesn't even know that the file exists.

The same approach is used for other "legacy" configuration files such as /etc/crypttab

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 12:19 UTC (Thu) by rwmj (subscriber, #5474) [Link] (1 responses)

My original reply was about your assertion "/etc/fstab is unnecessary with systemd, so that's not a problem... Systemd merely generates native mount units from it for backwards compatibility."

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 12:23 UTC (Thu) by Chousuke (subscriber, #54562) [Link]

But that is a statement of fact. /etc/fstab is completely unnecessary on a systemd system; useful and convenient, yes, but not necessary.

Poettering: Factory Reset, Stateless Systems, Reproducible Systems & Verifiable Systems

Posted Jun 19, 2014 13:04 UTC (Thu) by zlynx (guest, #2285) [Link]

If you have a system with local disks, which is what I think you're talking about, then one solution would be to write a udev rule. Detect a RAID controller, launch a script, probe the controller and its drives and create a mount unit for it at /raid or /var/data or whatever you like. That way systems with locally attached storage get it set up, but it doesn't interfere with other systems that just have a boot drive.

Or just do it with Puppet or whatever configuration tool you're using. A stateless system isn't very useful until it has grabbed itself some state after all. The configuration doesn't need to be in /etc, it can be on a network service.