EPAgent RESTful API: Java Example

This article describes a fully functional java program which utilizes the EPAgent RESTful interface to submit metrics to CA APM Enterprise Manager. It will post a single metric to the EPAgent. (provided by PoetryFan David Martin).

So I was able to get my little java program to report metrics successfully using this method, but after the program executes, the Introscope Investigator still reports those as live metrics. Shouldn't they be graying out because they are not reporting any longer? Is there another way to test the metrics I'm getting without restarting the EPAgent process everytime I want to start fresh?

The metrics aren't graying out before they're not arriving attached to a Java, .NET, PHP, or node.js agent. They're being reported by you. The EPAgent has no way of knowing that your program has stopped reporting, so the metrics stay live.

However, the example program doesn't include a full example of the JSON that should RESTfully POST correctly:

{"host" : "<agent hostname>", (Optional. If set, it must match that in IntroscopeEPAgent.properties)

"process" : "<agent process name>", (Optional. If set, it must match that in IntroscopeEPAgent.properties)

"agent" : "<agent name>", (Optional. If set, it must match that in IntroscopeEPAgent.properties)

You'll notice that you can optionally supply the agent hostname, process name, and agent name itself. If you provide these values, then with subsequent runs of your program you could just change the agent name slightly. Then you would get a new agent in your EPAgent tree, and not have to bounce the EPAgent to test out the new metrics.

So I tried to do this in order to get a new agent node and it did not work. Trying to modify the agent name returns a code 400 complaining that it's an invalid request. The only way I can get it to work is if I supply the host/process/agent that match in the properties file. I've tried multiple combinations like including all or just some of those optional parameters and it chokes every time there's a difference between what I provide and what's in the properties file. Below is the JSON I've been sending. Modifying any of the host/process/agent flags fails. When they match, it works fine.

The normal agents are not using the REST API to report their data. Those have a pre-defined timeout interval, because you know when the agent will report.. With REST the same is not true. You may only have new values every few minutes, OR send data multiple times per interval for example. I'm not aware of a way to shut off metrics with the REST API at present, but this is an excellent idea. Can you put an idea / enhancement request on the APM board?

################################# Agent Metric Aging# ==============================# Detects metrics that are not being updated consistently with new data and removes these metrics.# By removing these metrics you can avoid metric explosion. # Metrics that are in a group will be removed only if all metrics under this group are considered candidates for removal.# BlamePointTracer metrics are considered a group. ## Enable/disable the metric agent aging feature. # Changes to this property take effect immediately and do not require the managed application to be restarted.introscope.agent.metricAging.turnOn=true## You can choose to ignore metrics from removal by adding the metric name or metric filter to the list below. # Changes to this property take effect immediately and do not require the managed application to be restarted.introscope.agent.metricAging.metricExclude.ignore.0=Threads*

But I decompiled the EPAgent.jar and couldn't find any mention of them anywhere. Maybe they were deprecated, but not removed from the properties.

Still, there seems to be a hidden property that defaults to a very large value by default (86400). Perhaps you could try setting this property:

introscope.agent.metricAging.heartbeatInterval.hidden.base=15000

Since the units aren't clear from the property name, I can't tell you if that means fifteen seconds or not. Try that first, then lower it it to like 4, which if the units are 15s intervals would put you in the one minute timeout range.

I found these guys too:

introscope.agent.metricAging.turnOn

introscope.agent.metricAging.heartbeatInterval

introscope.agent.metricAging.dataChunk

introscope.agent.metricAging.metricExclude.ignore

introscope.agent.metricAging.numberTimeslices

I think turnOn and heatbeatInterval are likely to be helpful. Maybe numberTimeslices. I'd steer clear of dataChuck and exclusions.

Maybe, just maybe this is more relevant section of configuration related to what happens when there is no data/zero.

It did occur to me it would be good to control this in the HTTP server, the difference being you don't generally define a plugin for the HTTP server, you just point things at it / metric feed.

Now, if that HTTP server could create multiple listener ports and you changed the 'metricNotReportedAction' per listener port, that might work..

# Stateless plugins report 0 to many metrics on each invocation. The metricNotReportedAction property# determines action taken for any numeric metric which is not reported by one invocation but was reported on# the previous invocation as follows:# Value Meaning# same continue to report same value (until metric is aged out) - this is default if metricNotReportedAction omitted# zero report metric with value 0 (if metric type is stringevent or timestamp, action 'same' is performed)# stop stop reporting metric value# The MATCHEDPROCS example below uses processAvailability plugin to return value of 1 for each running process# whose name matches the regular expression specified by the match parameter (i.e. processes whose name starts with# 'f' in this case). The metricNotReportedAction parameter value 'zero' is used to reset metric value to 0 when a process# is no longer running.#introscope.epagent.stateless.MATCHEDPROCS.command=perl processAvailability.pl -match ^f -distinctmatch#introscope.epagent.stateless.MATCHEDPROCS.delayInSeconds=900#introscope.epagent.stateless.MATCHEDPROCS.metricNotReportedAction=zero