BizTalk Server 2009 : Building a Resequencing Aggregator

So now that you know how to
solve the problem of receiving messages out of order, you can properly
receive messages, resequence them, and send them in the correct order to
their destination. But what about batch scenarios—what happens in the
case where you need to combine the received messages, order them, and
then aggregate them into a batch? This need is quite common in the EDI
world where batching has existed for years. Also, in many legacy system
integration projects, large aggregated data files are needed to load
downstream systems with data. For example, let's look at the company ABC
example. Assume that orders received from the FTP bulk upload are
compressed and need to be sent as a batch to the fulfillment system,
which only understands flat text files. Luckily you have all the pieces
you need to solve this except one. To recap, you can

Use the Unzip component to decompress the files.

Use the resequencing pipeline and pipeline component from the resequencing example to add the sequence information.

Use the resequencing orchestration to order them properly.

However, you still have no way to a) combine the
messages into one message, and b) turn the sorted list of the messages
received into a flat file. Luckily, with BizTalk 2009 this becomes a
trivial task. There are, however, two possible solutions to this problem
depending on how the messages are received. If the messages are not
received as described, but are received as independent Interchanges, you
need to use the Resequencer pattern described previously. If they are
indeed received as a batch, a more elegant solution is available.

If the messages are received independently, a simple
change is required to the resequencing orchestration to allow the
pattern to work. All you need to do is not
send the messages as they arrive and wait to receive all messages
before sending them. Once all the messages in the batch are received,
you can use the new Call Pipeline from Orchestration code that ships
with BizTalk 2009 to send the SortedList to a send pipeline
that contains a flat-file Assembler component. So long as the messages
are defined with a schema that uses the flat-file extensions of BizTalk,
the pipeline will return a text message with all the messages inside
aggregated in their proper order. The orchestration will look something
like Figure 1.

The code for the Call Send Pipeline Expression shape will look like this:

You will need to use a Construct shape, since the shape is creating new messages via the component. The SendPipeMsg is of type Microsoft.XLANGs.Pipeline.SendPipelineInput Messages. The reason why you need to add these messages to the SendPipeMsg
via a component is because within the component you will need to loop
through each of the messages stored in the sorted list, and add them one
at a time to the SendPipeMsg to ensure that the order is
properly maintained. Alternatively, you could use a Loop shape in the
orchestration, but this is much easier done in code. Once this is done,
the send pipeline is called, the messages are aggregated, and the
complete message is sent to the send port.

Figure 1. Aggregating resequencing orchestration

2. Solution 2: Not So Status Quo—Messages Received as a Batch

In the resequencing aggregator outlined earlier, the
messages are received as a complete Interchange. They may be out of
order when stored inside the ZIP file, but they are completely batched
as a whole. In this case, you actually don't need a resequencing
orchestration and can implement this solution entirely within a pipeline
and use content-based routing.

Unzip component: The Unzip component extracts the ZIP file contents and creates a multipart message based on the files included.

Resequencing Decoder: The component examines a message and determines where it fits within a unique sequence.

What if you modified the Resequencing Decoder to
handle multipart messages? In this scenario, the component would have
additional smarts to know that multipart messages can also be
resequenced. In this case, the component would simply loop through each
of the message parts, examine its data, determine what number it is in
the sequence, and promote that property along with the unique sequence
ID and the last message in the sequence property. Also in this case, you
need to use another message part feature—they have a property bag. So
instead of promoting the values to the message context of the message,
you simply write these values as key/value pairs to the parts property
bag. Once all the message parts are resequenced properly, the message
can flow through BizTalk like any other message. The send port hosting
the FlatFileAssembling component would be subscribed for messages of
this type, parse the incoming messages, and output one aggregated
message, which is sent to the downstream system.

While the preceding example
is technically easier to implement, it has some drawbacks. First, as
stated earlier, since pipeline execution is stateful, should the
pipeline fail for whatever reason, processing stops. You will manually
need to have some error-handling logic and subscribe to the failed
message for processing. This is explored in the next section. Also, care
should be taken to properly performance test the solution. If the ZIP
file is large and contains a high number of individual transactions, the
CPU on this machine could be affected. Make sure that you test this
scenario properly before implementing it.