>> You don't have to use cgroups; you do have to support the features systemd currently uses cgroups to support, such as supporting daemons that fork or double-fork or have child processes, while still respawning daemons that exit or crash.
> Why does the init system have to do this job?
Its in the best position to do this as it's what starts the daemons and has a parent/child relationship with them. Process respawning is also available in SysV init as configured by /etc/inittab but that feature is incomplete.
> If a daemon exits or crashes, is it really the right thing to just start a new copy? what damage did the failed instance do that may need to be cleaned up?
I would think that in the general case the answer is very much YES. If you aren't worried about service outages I suppose you could make failure permanent, maybe wrap the daemon startup in a script which disables the service after the daemon exits. Seems kind of silly though.
> If a daemon has a complex set of dependancies (or deliberately does something like a double-fork to distance itself),
Double forking is more an implementation detail needed by other init systems, you don't need or want to double fork with systemd, although things will still work if you do.
> is it really enough to only consider it 'failed' if they all exit?
I'm not sure what happens if you challenge the init system in this way, what's the Right Thing(tm) to do in this case? If the processes maintain a parent/child relationship then you can rely on signals otherwise if the processes take input you can manage their sockets to detect failures.
> or should there be something more application aware (i.e. a daemon specific watchdog) that does this?
I suppose you could do that too, to handle cases where a process is wedged and needs to be restarted, but you solve 99% of the problem with just basic process monitoring using pretty standard techniques.