Using nco_postmsg to monitor the TEMS server

Monitoring the Tivoli Infrastructure has greatly improved over the last few years with IBM’s MoSWoS programme however the monitoring of the actual Tivoli Enterprise Monitoring Server (TEMS) itself has always been an issue as if this server stops working all the rest of the monitors will fail.

IBM wrote a document in 2006 in which they suggested using a script on the TEMS server to monitor processes and this is probably still the best solution. This tip updates that example to use the new OMNIbus command nco_postmsg which allows you to send an alert directly to an ObjectServer. This utility accepts name-value pairs for the alert data and constructs an SQL INSERT statement, which is used to insert a new row of data into a specified database table in the ObjectServer.

You can run nco_postmsg from the command line, or you can develop scripts or automations that use the nco_postmsg command to send alerts to the ObjectServer. Multiple instances of the nco_postmsg utility can also run simultaneously.

Installation

The nco_postmsg utility is installed with the Probe Support feature of Tivoli Netcool/OMNIbus, and can therefore be deployed separately from the other Tivoli Netcool/OMNIbus features, on one or more hosts. To do this do a custom install and then deselect every option except for Probe Support as shown below.

Setup

Once this is done edit the $OMNIHOME/etc/nco_postmsg.props file and add the following lines changing the values as appropriate:

MessageLevel: ‘warn’ # Message reporting level

Name: ‘nco_postmsg’ # Name of client

UserName: ‘root’ # User to connect as

Password: ‘ZZ’ # Password for user

Server: ‘ORBD_DEMO’ # Server to connect to

Table: ‘alerts.status’ # Table to insert event

Version: FALSE # Display version information

Edit $NCHOME/etc/omni.dat and change the values as appropriate to refect your ObjectServer’s name, hostname and port.

e.g.

#

# omni.dat file as prototype for interfaces file

#

# Ident: $Id: omni.dat 1.5 1999/07/13 09:34:20 chris Development $

#

[ORBD_DEMO]

{

Primary: mgmtserver3 4100

}

[NCO_GATE]

{

Primary: mgmtserver3 4300

}

[NCO_PA]

{

Primary: mgmtserver3 4200

}

[NCO_PROXY]

{

Primary: mgmtserver3 4400

}

Then you will need to create an interfaces file by running the $NCHOME/bin/nco_igen.

Test

Once this is done you should be able to test the command and see the event arrive on the OMNIbus console.

The Monitoring Script

For this tip I have used a slightly modified version from the original IBM document. This script could be improved but for now it works quite well. The script below is a shell script. For a batch version that works on Windows, see below.

Running the Script

On UNIX a good way to run the script is to use inittab as this will respawn the script if the monitor itself falls over.

To do this edit /etc/inittab and add in the following like:

yt:2345:respawn:/opt/yourtivoli/bin/tems_monitor.sh #TEMS Monitor

And respawn the init process.

kill -HUP 1

This will start the script and the monitoring process.

ps -ef | grep tems

And hopefully if the TEMS should ever fall over we will get an erro posted to the OMNIbus server.

ITM installed on Windows

For those with ITM installed on Windows, below is the a version of the monitoring script as a batch file. Configured with a custom task schedule to run the script this mirrors the aim of the shell script above.