upstart as cron replacement [LWN.net]

upstart as cron replacement

Posted Aug 31, 2006 19:41 UTC (Thu) by vmole (guest, #111) [Link]

You've given two good reasons why cron needs to be replaced. I'm still not convinced that it makes sense to combine it with init. Yes, init is conceptually the right place to do stuff on startup and shutdown, and providing users a good way to do this would be good. But for periodic jobs, I don't see why init is a better fit.

Now I'll actually go read about Upstart and learn why I'm wrong :-)

upstart as cron replacement

Posted Aug 31, 2006 21:05 UTC (Thu) by smurf (subscriber, #17840) [Link]

init has the unique property that it gets SIGCHLD for all processes which happen to die after their parent. Thus it is able to keep track of exactly when a job is finished, even if that job daemonizes or forks itself. (For the most part.)

This means that you don't have to futz around with either putting your jobs into the background (standard /etc/init.d/* scripts) or preventing them from doing so (if you use start-stop-daemon, or some other way to catch their PID). init can keep track of the job's state and do the Right Thing, which is presumably what you've told it to do in your config file(s).

This also applies to cron jobs and the locking one might want to apply to their execution.

I don't know if you have looked at the cron code. The thing is ugly, and not just because 19-year-old code shows its age. The twenty #ifdef DEBIANs it has acquired don't exactly help either; the Debian patch, when you run it through wc -l, is larger than the original sources.

In principle, hacking support for the features Upstart offers into cron -- it could monitor the events Upstart generates, and extending the crontab syntax doesn't look to difficult -- is certainly possible, but (not only IMHO) a rewrite would be a lot cleaner. The task is too important to do it as a dirty hack.

And if you rewrite it anyway, you might as well put the task into init where it can profit from init's aforementioned special status.

upstart as cron replacement

Posted Aug 31, 2006 22:48 UTC (Thu) by vmole (guest, #111) [Link]

I don't know if you have looked at the cron code. The thing is ugly, and not just because 19-year-old code shows its age.

Have I looked at the cron code? Heh. Far too long, far too much. I was the Debian cron maintainer for ~10 years (Javier Fernandez-Sanguino took over last year, bless his soul). You don't have to convince me the code is not pretty. However, a package maintainer can only do so much before the codebase is no longer compatible with patches from others.

The twenty #ifdef DEBIANs it has acquired don't exactly help either; the Debian patch, when you run it through wc -l, is larger than the original sources.

Well, I though keeping the Debian specific functionality (as opposed to general fixes) seperated was worth the small amount of visual noise; opinions may vary. If I'd realized how big it was going to grow, maybe I would have chosen differently. Regarding the patch size, you can't compare the linecount to the original - your counting both lines removed and lines added (one each for changed lines), plus all the new stuff in the ./debian directory. And there is *a lot* of functionality in the diff: support for cron.d, some support for DST and clock changes, big security fixes.

Defensiveness aside, I certainly wouldn't argue that you should hack Vixie cron - it needs to be re-written from the ground up, designed around whatever features are desired. I'm just not convinced that combining init, cron/at/etc., and inetd is the correct choice, despite (IMO) superficial simularities. In particular, I think maintaining appropriate security domains may be more complicated than one would like.

upstart as cron replacement

Posted Aug 31, 2006 23:13 UTC (Thu) by smurf (subscriber, #17840) [Link]

I'm just not convinced that combining init, cron/at/etc., and inetd is the correct choice, despite (IMO) superficial simularities. In particular, I think maintaining appropriate security domains may be more complicated than one would like.

I'm not entirely sure about that myself; certainly the requirements for a safe /sbin/init replacement are a lot higher than those for anything else in the system, and there's something to be said for keeping the thing as simple as possible.

Personally, I wouldn't be entirely happy writing something that complex, but that's simply because I do like to run my code under strace or gdb in order to figure out why it misbehaves, and I'd have to adopt a couple of different debugging techniques. ;-)
You can't trace the process with PID one (for good reasons, too).

upstart as cron replacement

Posted Sep 3, 2006 8:20 UTC (Sun) by bockman (guest, #3650) [Link]

init has the unique property that it gets SIGCHLD for all processes which happen to die after their parent. Thus it is able to keep track of exactly when a job is finished, even if that job daemonizes or forks itself. (For the most part.)

It could export this feature: when an orphan child dies, upstart could notify other interested processes of the event. You would need a registration API and a way to transmit the information. Then an independent cron replacement could use this service.

Maybe. Just an idea. :-)

Ciao
------
FB

upstart as cron replacement

Posted Sep 12, 2006 12:02 UTC (Tue) by robbe (guest, #16131) [Link]

> init has the unique property that it gets SIGCHLD for all processes which
> happen to die after their parent. Thus it is able to keep track of exactly
> when a job is finished, even if that job daemonizes or forks itself. (For
> the most part.)

Let's say init starts a daemon (e.g. apache as pid 100) directly and it daemonizes itself (i.e. forks to pid 105, and pid 100 terminates). The init will then get a SIGCHLD from pid 100 and supposes the daemon job as "finished". It doesn't know that pid 100 and pid 105 are connected. If the daemon exits or crashes later init will notice that pid 105 has terminated, but does not connect this fact with the apache job it started sometime in the past. (One can usually suppress daemonization via a commandline argument, though.)

Right now most daemons are started via scripts that setup the environment etc. If init runs this script it knows its pid and when it finishes. Again, it does not know the pid of the real worker process that the script forks off.

Some of your other comments to this article rely on part on this assumption which I argue to be wrong...

upstart as cron replacement

Posted Sep 12, 2006 12:28 UTC (Tue) by smurf (subscriber, #17840) [Link]

It's true that per se, init doesn't associate job 105 with the process 100 that it started. But that is easily fixed.

The simplest way is to actually tell init which pid the daemon has -- ideally in the same call which tells it that the daemon in question is now up and running, instead of merely starting up.

upstart as cron replacement

Posted Aug 31, 2006 20:50 UTC (Thu) by kh (subscriber, #19413) [Link]

it doesn't know not to run my database sync job until *after* mysql has started up

This seems to me to be a perfect example of what not to put in a cron replacement daemon (because of feature creep). Currently, I believe you could have the database sync job test if mysql is running (and either start it or exit on error status if not - but I at least would want to control that in my script, not have cron decide). I don't want a super daemon constantly monitoring all the other daemons and deciding what cron jobs to run and which not to.

upstart as cron replacement

Posted Aug 31, 2006 22:59 UTC (Thu) by smurf (subscriber, #17840) [Link]

Currently, I believe you could have the database sync job test if mysql is running (and either start it or exit on error status if not - but I at least would want to control that in my script, not have cron decide). I don't want a super daemon constantly monitoring all the other daemons and deciding what cron jobs to run and which not to.

?? The decision which jobs to run would lie with the author of the job, not with upstart: you decide on the rules for your script, not cron or anybody else.

Besides: What do you mean by "constantly monitoring"? If it's an init replacement, it doesn't need to monitor anything, because it knows: it gets the information from the kernel for free, no need to poll anything, much less /proc.

The idea behind integrating all of this in some way is that you can just add the "requires foobar" idea to your cron or at job (remember this will be started after Edgy, so nobody has any concrete ideas about syntax yet).
Then, Upstart will either auto-start the mysql / foobar / whatever service for Joe User when necessary, or delay the cron job.

Joe can't do the former himself, he doesn't have the privileges; the latter is a waste of resources -- if you know beforehand that /usr/sbin/foobard is not running, there's no point in starting Joe's job.

upstart as cron replacement

Posted Sep 1, 2006 1:27 UTC (Fri) by kh (subscriber, #19413) [Link]

?? The decision which jobs to run would lie with the author of the job, not with upstart: you decide on the rules for your script, not cron or anybody else.

You did not understand what I was trying to say, I am not comfortable with the idea of moving the logic for tests (such as "is the database server running?") from the sync (or whatever) script, where is currently resides, to the cron replacement (i.e. the update config (crontab?) file). And I do not understand how it is possible to do so with a terse language for the config file, and yet have update be aware of every different possible daemon, their states, the system states, & etc. That seems like a LOT of new features for cron.

No service-specific tests

Posted Sep 1, 2006 7:28 UTC (Fri) by smurf (subscriber, #17840) [Link]

The test upstart will do, using the mysql example, is not "can I connect to port 3306 and get a sensible reply", but "I have forked off /usr/sbin/mysqld and it hasn't died yet".

Testing for actual services isn't upstart's job; I agree that that makes no sense whatsoever.

NB: I don't know how upstart is going to decide that a service has finished starting up. I assume it'll be something along the lines of "the startup script has exited but some of its children are still running" by default, but I haven't checked.