Polling a JSON REST API with an Apache Camel component

If you have been playing around with integration frameworks, you are probably aware of Apache Camel, as it is one of the two big open source players together with the Spring Integration framework. While benchmarking Talend ESB and Redhat Fuse ESB products, which both include Apache Camel in their stack, I had the opportunity to dive a bit more into the concepts of the framework and figured there was some interesting material for a blog post !

The purpose of this post is to provide some guidelines to build a custom Camel component for your enterprise integration needs. Our example will show how to poll an external JSON-based REST API – in this case, the Feedly API – to periodically retrieve and process information – in this case, to log retrieved RSS feed titles.

Step 2: Discovering the Feedly API and setting the properties file

For this example we will be using the Feedly Search API, as it is simple to use and does not require user authorization. It provides suggestions of feeds related to a topic; hereafter an example of a GET call to the API:

GET http://cloud.feedly.com/v3/search/feeds?query=iot

There are two parameters related to this API call, which are the “query” parameter and the request frequency (“delay”). Isolate these two parameters in a dedicated /src/main/resources/feedly.properties file:

#Feedly API call properties
delay=10000
query=iot

Step 3: Setting up the custom component, endpoint and consumer

Camel custom components are responsible for creating Camel endpoints, which in turn create consumers and producers. The RedHat Fuse ESB documentation explains those concepts quite clearly. For our example, we will be therefore creating 5 classes:

FeedlyComponent: our custom Camel component

FeedlyEndpoint: our custom Camel endpoint

FeedlyConfiguration: a configuration POJO, which makes it easier to configure our custom endpoint

Note that we do not create a Producer implementation, as we only do read-only GET calls to the API.

FeedlyComponent class

The class inherits UriEndpointComponent in order to allow better documentation generation, as we shall see later. The createEndpoint method is responsible for instanciating new endpoints and populating the endpoint configuration from the parameters map. Note that the parameters map MUST be emptied afterwards, which can be achieved automatically by using the inherited setProperties method; otherwise Camel shall throw an ResolveEndpointFailedException.

FeedlyConfiguration class

Again, @UriParams and @UriParam annotations shall be useful later for generating the documentation of our component.

@UriParams
public class FeedlyConfiguration {

@UriParam
private String query;

@UriParam(defaultValue = “60000”)
private int delay = 60000;

public int getDelay() {
return delay;
}

public void setDelay(int delay) {
this.delay = delay;
}

public String getQuery() {
return query;
}

public void setQuery(String query) {
this.query = query;
}
}

FeedlyConsumer class

Our custom consumer extends ScheduledPollConsumer in order to automatically gain polling capabilities. The poll method is responsible for choosing the Feedly API method to call, based on the operationPath stored by the endpoint. The HTTP client does the call while GSON converts the JSON response into a list of Feed items. Finally the list is stored in an Exchange object to be passed to the next Camel processor.

Step 4: Enable auto-discovery of the custom component

Create the /src/main/resources/META-INF/services/org/apache/camel/component/feedly file and indicate the fully qualified name of your custom component class. In our example it is com.paloit.examples.camel.custom.feedly.FeedlyComponent.

class=com.paloit.examples.camel.custom.feedly.FeedlyComponent

By doing so we allow Camel to automatically register our custom component so that we can start building routes using it!

Step 5: Run the example

Finally create the Main class below to build a route using our custom component. We use the Camel PropertiesComponent to load our previous feedly.properties file and get the “delay” and “query” parameter values. Also note that, since our custom consumer produces a list of Feed items, we need to use the Camel split EIP to process each Feed individually. Our example just logs the Feed, but it could be stored or forwarded to another endpoint, depending on your needs.

public class Main {

public static void main(String[] args) throws Exception {

System.out.println(“Starting example, press CTRL+C to terminate”);

org.apache.camel.main.Main main = new org.apache.camel.main.Main();
main.enableHangupSupport();