Monday, February 13, 2012

Advanced Features of Messaging Products I

Introduction
Enterprise messaging software came into the market in late 1980s. With the evolving of Java / “Java Messaging Service” (JMS) and “Service Oriented Architecture” (SOA) platforms in 2000s, nowadays enterprise messaging is not only about communications between applications, but also it becomes a pattern or style for integration in enterprise application landscape. Today we call messaging software “Message Oriented Middleware” (MOM), which is defined as “software or hardware infrastructure supporting sending and receiving messages between distributed systems” [1].
The motivation for adopting MOM software in a lot of projects stems from the need for guaranteed message delivery, flexible routing and message transformation. Another fact is that sometimes quite a lot of useful features of these products are overlooked, although those features would bring brilliant ideas for integration. We call them “advanced” features in the sense that they are not directly related to core features (message routing, delivery) but they strengthen those core features. This article briefly explains some features of MOM products by using ActiveMQ as the example. Hopefully it ignites the sparkle for easy and innovative integration solutions in some projects.
MOM versus JMS
One major problem with MOM software in early phases was lack of standards, until Java EE gave the JMS specification, which gained popularity and support of most MOM products. As a result, people often consider or call MOM software “JMS products”. However, MOM and JMS are different concepts.
Most MOM products implements JMS (Microsoft MSMQ does not), and they can do much more than that. JMS specification focuses on JMS elements and message structure. As a contrast, a protocol called “Advanced Message Queuing Protocol”, AMQP, is emerging that defines features regarding “message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security”. Some MOM vendors already claimed the support of AMQP, such as OpenAMQ, StormMQ and RabbitMQ.
The features described in this article are not part of standard JMS specification. However, they are supported by various MOM products; the terminology could be different, though. In this article, several features offered by ActiveMQ are selected to explain alternative solutions to address common business problems. ActiveMQ has much more features than those described here. To see the complete list, please check [3].
Case study: AccNet
An imaginary company called AccNet is introduced to help understand features explained here. AccNet is a telecom provider that offers internet connections and related services to customers across various countries. Applications from various business units such as order department, dispatch centers, billing department and credit department communicate with each other via ActiveMQ .
AccNet has the following functional requirements related to their order fulfillment and billing logic:
1. Order fulfillment probably involves engineer service or hardware dispatch service. AccNet has several service teams and dispatch centers within each country. Depending on the work load of each team, those service requests are processed and they need to be processed by only one of the service teams.
2. After orders are received and fulfilled, the order/account information should be sent to both billing department where customer bills are to be decided and credit department which is responsible for processing the actual payment from customers.
3. A complete order might consists of several order messages, for example, orderMsg1, orderMsg2, and so on. Those messages are produced sequentially and they need to be consumed in the same way because orderMsg2 cannot be processed before orderMsg1. This order of processing is guaranteed in a single-consumer environment. However, application producers and consumers normally are deployed in clustered environments. The correct order of message consumption, based on the order the messages are produced, is required in the whole clustered environment.
Solutions with Features Explained
The features to be explained fall into 4 categories: destination features, consumer features, dispatch features, and message features, as defined in [3]. This article is the first part of two SOA focus articles and it focuses on some destination features from ActiveMQ.
1. Destination Features1.1 Composite destination
Composite destination is a virtual destination that represents a list of physical destinations. Utilizing this feature could effectively address the 2nd business requirement.
Suppose the billing department for AccNet Benelux has the queue called NL.Order.Billing; and the credit department has queues called NL.Order.Credit and BG.Order.Credit respectively. (These queues are not necessarily geographically distributed in different locations. If they are, features such as message forwarding should be used as well to address the issue.) A plain solution can be implemented from the queue producer side, as illustrated below:

By applying composite destination, the solution can be simplified.

From message producer perspective, it doesn’t take care of sending messages to different queues any more. The task of delivering messages to queues is shifted to message broker. It’s more efficient because the message is sent only once, which has less network impact and less resource impact in message producer side.
Another point worth mentioning is that the destinations are not necessarily queues. The composite destination can consist of one queue and one topic. In that case, the prefix indicating the destination type should be added. For example, a composite destination can be “queue://NL.Order.Credit, topic://NL.Order.Audit”.
1.2 Wildcard destination
Wildcard adds more flexibility when defining destinations. “*” is used to match any name string; “>” is used to match any following string and combination of strings.
For example, the destination string “NL.Order.*” matches any queue names such as NL.Order.Credit, NL.Order.Billing, and etc. “BG.>” matches any queue names that starts with “BG.”, such as “BG.Order.Credit”, “BG.Customer”, and so on.
1.3 Virtual topic
Virtual topic feature offers an alternative to composite destination. It’s quite useful when you want to scale up the number of consumers. It’s the solution from message consumer perspective.
Requirement 2 needs the same message being delivered to multiple applications/message clients. Instead of using the “queue” concept as described in [1.1 Composite destination] and [1.2 Wildcard destination], we can use the “topic” concept to meet the requirement.
Suppose the order messages are sent to a topic called “NL.Order”; both applications from credit department and billing department subscribe to this topic. However, the downside of the plain publish-subscribe solution is:
• Normally subscriber is not durable, which means there could be message loss. This is not acceptable by the business.
• If we make subscribers durable, the solution is not scalable, because only one consumer can be active for one subscriber and one ClientId.

By using virtual topic, we can have multiple consumers for each application, which is usually required by businesses. Here’s the illustration of the solution:

This solution look a bit complex, however in practice to achieve it is quite easy. Only 2 steps are required:
• Configure NL.Order is as a virtual topic, not a normal JMS topic. By default, virtual topics has “VirtualTopic” in the name (The name is customizable).
• The message clients are actually queue consumers, instead of normal topic subscribers. The queues they consume are created on the fly if clients are built to pick up messages from queue with names that end with “.VirtualTopic.NL.Order”.
With this solution, the 2 drawbacks of native JMS topic are overcome. Queue consumers are durable and by nature they compete for messages that are in the queue. So this is a scalable solution with load-balancing capability.
1.4 Message forwarding
Message forwarding feature effectively addresses the 1st requirement, which demands a load-balancing mechanism among remote consumers. In reality, it can be implemented in multiple ways, such as master-slave configuration, customized routing mechanism, queue bridges and so on. Here’s the illustration of how ActiveMQ resolves it:

The challenging part of the requirement is that the broker for the order queue is probably not in the same network or same geographical location as brokers for the service teams, while the messages from the order queue should be delivered to remote applications in a load-balancing manner.
To make this happen in ActiveMQ is really simple, which is to configure message forwarding from the Order Fulfillment MessageBroker to a list of other brokers. Message forwarding can be configured for either the broker or some specific queues. After that, service request message consumers are able to consume messages from corresponding message brokers without the need to create the queue in remote brokers. Message consumption behaves the same as if those consumers pick up messages from the Order Fulfillment Broker, and load balancing is also achieved between 2 remote consumers.
Summary
This article focuses on destination features of MOM products, using ActiveMQ as an example. A follow-up article will further explain consumer features, dispatching features and message features and how they address business problems. A table listing similar features/terms from different MOM products will also be provided.
References
1. http://en.wikipedia.org/wiki/Message-oriented_middleware
2. http://en.wikipedia.org/wiki/AMQP
3. http://activemq.apache.org/features.html