5 Answers
5

Use a configuration management system, like Chef or Puppet. Have the configuration management server push out the appropriate cron/Upstart/monit/whatever configurations to the various nodes, depending on their roles. Yes, it's probably more of an investment to set this up than spewing crontab files all over the place manually, but you will wind up with a centralized point of control over all the servers you're trying to manage.

I tried reading about Chef and Puppet on their respective sites. These definitely solve part of my problem which is distributed configuration. It is not clear though whether these tools can manage the lifecycle of distributed processes too. Any input regarding this please?
–
Prashanth EllinaMay 20 '12 at 5:14

I would use, say, Chef to push a monit or Upstart configuration to all your nodes to start up the process and keep it running. Those are the right tools to manage process lifecycle, with Chef helping to get that configuration out to the machines that you want to use it on. If you want to stop a process (which is managed through Monit, for example) on a set of nodes, you can run the Chef command "knife ssh role:appserver sudo monit stop application" or something like that. Chef will do essentially a cluster ssh to all appropriate nodes and run that command.
–
cjcMay 20 '12 at 10:45

For cron updates you could place a copy of your cron files on a web server and have your machines update their cron files based on the contents of the files stored on the web server. This would allow you to do cron updates in one place. The down side is that all of your computers would only be as secure as the web server since anyone that can manipulate the contents of the web server could then run arbitrary commands on any of your systems.

Rather than a web-server, why not give them SSH keys for a restricted account on another machine (to enable automatic login), and use rsync? That would avoid man-in-the-middle attacks on the updated crontab, and also enable the various servers to download only the changes in the crontab.
–
DaraelMay 17 '12 at 12:18

If managed by inittab -- edit inittab,
If managed by system startup (/etc/init.d & /etc/rc[0-6].d) -- use chkconfig or service

From man inittab

respawn
The process will be restarted whenever it terminates (e.g.
getty).

To maintain these in a multi-system replicated system, I can only think of something like rsync. I don't know if something like NIS/YP would be usable. You may be looking for a more enterprise-level solution.