script to failover nagios servers if and when their down - automtically done via bash script that will require some ground work to be done in order to achieve the end result of each nagios server looking out for one another.

I have also done a few other things

cd /etc/;

ln -s /usr/local/nagios/etc ./nagios

cd /etc/nagios;

ln -s /usr/local/nagios/libexec ./plugins

Requirements.

Actual nagios cfg files -

CMS (configuration Management System - puppet,chef or cfengine)

I have used puppet and used the recursive folder plugin to push all the actual cfg files to nagios nodes:

If you do not have cms in place you could configure this script to use existing unison to synchronise configuration across the nodes. simply by adding company folder as a space seperated list to sync_folder variable.

Unison configuration :

You could use the unison protocol and add /usr/local/nagios/etc/objects/$company as a folder to also be synchronised and bypass all this CMS requirements. You would need to move :

Nagios will then read all the files within the sub folders of each datacentre recursivly on each nagios host... (this simplify definition of each config file etc and makes it a lot easier to script this solution.

So once you have puppet pushing out the configuration which in short all nagios hosts have all the configrations but only load up the relevant datacentre folder for what it is supposed to monitor:

3. to change nagios configuration so that it is generated via CMS and within nagios.cfg there is preferably only one line which is the datacentre name that loads up all relevant configuration for that site

4. # AUTOMATION ADD HERE to be added to nagios.cfg under the main config line that is pointed out in point 3.

5. /opt/nagios-sync as a folder that is synchronised amongst all nagios server nodes using unison (all done within script)