Bartosz Baranowski - blog

Tuesday, November 8, 2011

Sometimes ago we had brainstorm session during regular SLEE core chat. Community users shed some light on real use cases and their expectations with regard to management of live system.

It came out that simple JMX is sometimes not enough. Especially for deployment where some existing third party management tool/application is running.

Community users directed us in direction of SNMP. It came out that some of them tried to enable our containers (or even JBoss) to allow SNMP management, without much luck. So we decided to step in.

Sometime ago Jean Deruelle and Tom Hauser made an effort to bring back JBoss SNMP adaptor. Main goal of their work was to:

add SNMP V3 support

add MIB files generator

update adaptor code to AS5

update adaptor code to use netsnmp4j library

Fruit of their work has been embeded in last MSS release, in MSS-1.6. The 1.6 is first SNMP enabled container from Mobicents family. Documentation for binary adaptor included in MSS can be found here.

Note that this documentation will change in near future.

Source

While first draft (Alpha1) of adaptor and MIB generator were fine for test, it turned out that releasing its source code as binary maven artifact proven to be a problem. So we decided to rework it a bit to be more logical in both structure and deploy process.

SNMP identifies devices by using something called M.I.B. . In short MIB is a way of organizing resources. MIB has tree like structure, where each root has non-negative index. Resources in MIB are identified by concatenation of indexes which lead to particular node. Example URI looks as follows: 1.0.12.4124.66.8365

This URI, has a specific name, which is object identifiers or , in short the OID.

OIDs are managed by IANA. The IANA OID registry contains a list of registered entities that use OIDs to reference internal structures (e.g. directory attributes). Each entity is assigned an initial referential value (e.g. 1.3.6.1.4.1.2312 for Red Hat, seehttp://www.oid-info.com/get/1.3.6.1.4.1.2312) and thereafter the internal numbering plan is up to the entity to define (e.g. 1.3.6.1.4.1.2312.100.1.1 might refer to the "ActiveThreadCount" attribute of the JBoss Application Sever). This allows for precise referencing of data attributes in a company or other entity.

This is very important. It means that OIDs, inside of company/entity leaf of MIB are managed solely by that company/entity. In other words, resources from the same company, have the same referential value and rest can be freely assigned - depening on whim of administrator.

He can for instance decide that following OIDs have :

RedHat : 1.3.6.1.4.1.2312

RedHat US : 1.3.6.1.4.1.2312.0

RedHat US Office 4 : 1.3.6.1.4.1.2312.0.4

Office 4 JSLEE : 1.3.6.1.4.1.2312.0.4.1

Office 4 JSLEE Server 1 : 1.3.6.1.4.1.2312.0.4.1.1

Office 4 JSLEE Server 2 : 1.3.6.1.4.1.2312.0.4.1.2

Office 4 MSS Server 1 : 1.3.6.1.4.1.2312.0.4.2.1

Office 4 MSS Server 2 : 1.3.6.1.4.1.2312.0.4.2.2

SNMP managed attributes are identified by MIB entries. Each MIB entry has type associated to it. For instance, lets go through the "ActiveThreadCount":

1.3.6.1.4.1.2312 identifies RedHat(OID type)

1.3.6.1.4.1.2312.100 identifies JBoss appliaction server(OID type)

1.3.6.1.4.1.2312.100.1 identifies MBean(OID type)

1.3.6.1.4.1.2312.100.1.1 identifies MBean property, which can be managed with SNMP(Integer32 type)

Detailed information about Management Information Base can be found here.

The playground

For this exercise I used single Ubuntu machine. It is possible to use windows, however it has to be 32bit. This limitation exists due to SNMP manager does not support 64bit.

Lets advance step by step with proper setup.

First thing that needs to be addressed are the dependencies for SNMP implementation. Since I had desktop version of Ubuntu (10+) , only dependency I had to install was lib-perldev

baranowb@port3:~/tmp$ sudo apt-get install libperl-dev

Once dependency is there, we can install SNMP program/daemon. Since there is no precompiled package(and version 5.7 seems to be broken) I went for net-snmp v 5.6.1.1. To get it simply download source(unless there is package now or there is package for your OS):

This indicates that source can be build. To do so, issue following command:

baranowb@port3:~/tmp/net-snmp-5.6.1.1$ make

Once compilation sequence is done, SNMP binary is ready to be installed:

baranowb@port3:~/tmp/net-snmp-5.6.1.1$ sudo make install

Now we are almost ready to have SNMP run. Last step before SNMP configuration magic is to expose *.so objects. Either link libraries to proper place, or add /usr/local/lib to LD_LIBRARY_PATH. However, if you do the latter, remember, that SNMP programs must run with SU privileges(or specific group).

Now what we need is JBoss, SLEE, adaptor and some specific setup. To get JBoss go here and download proper version. To get SLEE up and running, follow instructions found here(or simply download binary release from here.)

Once SLEE is up and you ensure it can run, it's time to brew some notifications from SLEE. By default SLEE emits notifications on following:

SLEE container state changes

alarm is beeing emited

Obviously, the first one is useful kind, however lets add some complexity to this exercise and brew some alarm. To emit alarm, I would advise to create simple Service with single SBB, which on ServiceStartedEvent and possibly on ActivityEndEvent(service activity) will raise alarm. In my test I processed only ServiceStartedEvent. My SBB looks as follows:

However to kickstart everything as fast as possible, you could simply edit one of existing examples, like sip-uas.

SNMP and MIB magic

Above concludes steps on SLEE side, atleast for simple setup. Now SNMP adaptor has to be configured. For this I did not use current trunk, nor I did use binary version. This is because trunk version had some errors(fixed already) in poms . On top of that there have been some MSS modifications in configuration files. Which rendered that source a bit useless outside MSS. After sometime I just decided to go highway and hack around/fix that.

Reason against binary version available from jboss wiki is that it.... was wrong, for some reason it had bad version linked(should be fixed shortly).

Ok, to get it running, simply download source from our repository(links at the beginning of this article) or get it from patches. This source is pure JBoss SNMP adaptor, it has no setup ready for SLEE. This is our task now, to bring this up to speed.

For sake of beeing complete, in step by step setup, I will explain how to modify this source to enable SLEE. In general case, you would use preset that we have for SLEE. I will write simple article about setup in following days.

Now, first thing we need to do, is to identify resources we want to expose, resources that can be exposed to SNMP management. In JSLEE we have few sets of MBeans present for management purposes.
That is:

javax.slee.management.* - all standard MBeans

org.mobicents - mobicents specific MBeans

Now, we need to identify one weakness in SNMP - it allows to manage configuration, that is access and set value of MBean attributes. It does not allow to invoke methods like "clearAlarms(String type);". So by default, without some hack, we will be able to manage limited subset of SLEE configuration. However it should be enough for a jump start.

First thing that has to be done before we can create config files for adaptor is to come up with MIB scheme.

Obviously we should reuse RedHat referential value: "1.3.6.1.4.1.2312". Our MIBs will start with this prefix. For sake of simplicity, lets assume we dont have to divide MIB to provide distinction between countries, offices, buildings etc. In other word we have flat MIB space. Now lets assume following:

oid - oid suffix which will be appended to oid-prefix. In case of array/collection types, indended value has to be appended with '.1'. For instance SleeState can have oid='.1', but Subsystems will have oid='.5.1'.

mode and max-access (reduntant, I know, this should be cleared in next source iteration) - determine type of attribute access mode=[ro|rw] and max-access=[read-only, read-write]

Above code snippet defines two MBeans as MIB accessible resources. For SleeManagement MBean it resrves OID='.1.3.6.1.4.1.2312.101.1.100.1' and for Alarm it reserves OID='.1.3.6.1.4.1.2312.101.1.100.2'. There is nothing special about this setup, so lets move to notifications.

One very important limitation, is that notification can be mapped only to resource already existing in MIB. In short, there has to be definition of resource in 'attributes.xml' in order to have definition of notification.

Lets assume we want to receive SNMP trap/info for all SLEE basic notifications. So we want to map notification from SleeManagement and Alarm MBeans. This can be achieved with following:

Notification mapping file is not hard to understand. Each mapping has following attributes:

name - name identifying this entry, in general it should match 'definition-name'

inform - optional boolean attribute, if present and set to true, notification is converted to SNMP inform, not trap

security-name - name of user to match entry in 'users.xml'

What is interesting is defined in mbean child elements. The var-bind-list declares actual binding of notification data to SNMP world. The var-bind-list has single attribute, the wrapper-class. This attribute must be set to class name of a class which implements org.jboss.jmx.adaptor.snmp.agent.NotificationWrapper ( by default its org.jboss.jmx.adaptor.snmp.agent.NotificationWrapperSupport ). Default class handles conversion of common notification data. It binds following data to tag( tag is a logical name, its similar in meanig to association-name):

Tag

Description

n:message

javax.management.Notification.getMessage()

n:sequenceNumber

javax.management.Notification.getSequenceNumber()

n:timeStamp

javax.management.Notification.getTimestamp()

n:type

javax.management.Notification.getType()

n:all

javax.management.Notification.toString()

n:class

javax.management.Notification.getClass().getName()

a:startTime

snmp agent/adaptor start time

a:uptime

snmp agent/adaptor up time

a:trapCount

snmp agent/adaptor trap count

u:

looked-up in n.getUserData()

Non standard binding can be defined as child elements of var-bind-list as var-bind. Each var-bind element creates binding for some attribute. The binding has following data which describes it:

oid - full OID identifying this variable. (NOTE: compare OID of bindings, to enterprise value and value of oid-prefix of MBean definition in attributes.xml)

tag - logical name of binding. It follows pattern [a|u|n]:beanAttributeName

type - ASN.1/SNMP data type

The tag value (beanAttributeName) is in general name of attribute(follows java bean convention).

Now depending on tag prefix, name has different meaning:

a - following name must be a name of MBean attribute. The oid MUST match one defined in attributes.xml.

u - the prefix:name is looked up in Notification.getUserData() (only in case it contains java.util.Map instance) and copied to payload. For this type of tag, the oid must not have to match existing attribute, MIB entry with this OID will be created( by generator )

n - name following this prefix must be a valid java bean attribute of notification class. As in case of u the OID must not match existing attribute.

Now what is interesting in above example, is that it does not use standard wrapper-class and defines some non standard var-binds. Lets see how custom wrapper handles conversion:

Having our magic classes and configuration files ready, we can start thinking on proper deployment.
First thing is to build mib-gen( the dev version has artifacts version set to 5.1.1-SNAPSHOT).
Change directory to snmp-mibgen(or whatever it is called localy) and simply issue maven order:

Once maven is done. Switch to snmp-adaptor directory and edit notifications.xml and attributes.xml files. Put there snippets outlined above. Those files can be found in snmp-adaptor/adaptor-service/src/main/resources.

Last thing you need to do, is to edit jboss-service.xml to instruct adaptor how to behave. Edit this file, search for following snippet:

Edit it and append following declaration:

This will instruct Adapter to do following:

try to register to declared MBeans as NotificationListener

if DynamicSubscriptions attribute is set to true, it will will try to register as NotificationListener once declared MBeans are available

One important remark here, note that there is notification element. It has attribute type. This attribute value is used as filter for notification. Important thing here is that its not matched as regular expression. The pseudo code for this operation looks as:

Having done above, we can generate MIB, build adaptor deployment unit and start to play SNMP.
To do so, simply issue maven order in snmp-adaptor directory (note that JBOSS_HOME) has to be defined. Output should look similar to one below(it deploys deployer, but we dont care for it now):

Once above completes we are ready to play with SNMP. That is if assumption that default users/managers are fine with setup, but since we will use single machine, it should. To receive traps, we have to configure SNMP tool(net-snmp).

The hex value in above snippet is ENGINEID. For me it did not change. If you need to double check, see what is written to /var/net-snmp/snmptrapd.conf file. (Note: for some reason, snmp tool did not use config files from /etc directory. I had to edit /root/.snmp/snmptrapd.conf) .

Having done above ,there is one final step that needs to be taken. We need to copy MIB file. It is located in $JBOSS_HOME/server/default/deploy/snmp-adaptor.sar directory.
First check where MIB should be stored, by issuing following command:

net-snmp-config --default-mibdirs

I had to copy MIB files to /root/.snmp/mibs, so my directory structure looked as follows:

root@port3:~/.snmp# ll -R

.:

total 20

drwxr-xr-x 3 root root 4096 2011-10-27 09:54 ./

drwx------ 12 root root 4096 2011-11-04 16:54 ../

drwxr-xr-x 2 root root 4096 2011-10-27 15:53 mibs/

-rw-r--r-- 1 root root 216 2011-10-27 09:54 snmp.conf

-rw-r--r-- 1 root root 544 2011-10-27 09:54 snmptrapd.conf

./mibs:

total 16

drwxr-xr-x 2 root root 4096 2011-10-27 15:53 ./

drwxr-xr-x 3 root root 4096 2011-10-27 09:54 ../

-rw-r--r-- 1 root root 4217 2011-11-02 13:07 mobicents-slee.mib

Now we are ready to fire up the daemon. If everything is setup properly we can execute following(and see similar output):