Quoc D. Bui&#39;s BlogBusiness Platform Division, Customer Advisory (Technology) Team
http://blogs.msdn.com/b/quocbui/atom.aspxTelligent Evolution Platform Developer Build (Build: 5.6.50428.7875)2009-10-16T19:56:00ZBizTalk Server–distinguished fieldhttp://blogs.msdn.com/b/quocbui/archive/2010/11/30/biztalk-server-distinguished-field.aspx2010-11-30T12:04:52Z2010-11-30T12:04:52ZThe distinguished field is used to expose the value of a node of a message instance to BizTalk Server system. However, if there is a record that has a parent node that has multiple instances, then that record won’t be able to be highlighted as distinguished. An error message of “ The node can occur potentially multiple times in the instance. Only nodes which are guaranteed to be unique can be promoted .“ will appear. This constraint was purposely designed to remove any ambiguity in accessing the...(<a href="http://blogs.msdn.com/b/quocbui/archive/2010/11/30/biztalk-server-distinguished-field.aspx">read more</a>)<img src="http://blogs.msdn.com/aggbug.aspx?PostID=10098271" width="1" height="1">Quoc Buihttp://blogs.msdn.com/quocbui/ProfileUrlRedirect.ashxBizTalk Patterns–part 2http://blogs.msdn.com/b/quocbui/archive/2010/11/26/biztalk-patterns-part-2.aspx2010-11-26T01:44:09Z2010-11-26T01:44:09ZI recently posted another common design pattern. It is called the Sync-Async pattern. This pattern consists of a client making a request to a server service for information, whereby the client will provide an endpoint for the server to send that information to, in an asynchronous response. However, the client also requires an immediate technical acknowledgement from the server service, within the same synchronous connection that the client has established, to be assured that the request has been...(<a href="http://blogs.msdn.com/b/quocbui/archive/2010/11/26/biztalk-patterns-part-2.aspx">read more</a>)<img src="http://blogs.msdn.com/aggbug.aspx?PostID=10096839" width="1" height="1">Quoc Buihttp://blogs.msdn.com/quocbui/ProfileUrlRedirect.ashxBizTalk Patternshttp://blogs.msdn.com/b/quocbui/archive/2010/09/30/biztalk-patterns.aspx2010-09-30T02:05:26Z2010-09-30T02:05:26ZI recently presented at the ACSUG September Meeting: BizTalk Solution Patterns with Quoc Bui from the BizTalk Customer Advisory Team in Auckland, New Zealand.The main thrust of the presentation was on BizTalk Patterns. These patterns, and much more, can be obtained on Windows Server AppFabric Customer Advisory Team site, where I have entry titled BizTalk Patterns part 1 – Asynchronous Aggregation &#160; that explains one of the patterns that I communicated in Auckland. Technorati Tags: BizTalk Server...(<a href="http://blogs.msdn.com/b/quocbui/archive/2010/09/30/biztalk-patterns.aspx">read more</a>)<img src="http://blogs.msdn.com/aggbug.aspx?PostID=10069620" width="1" height="1">Quoc Buihttp://blogs.msdn.com/quocbui/ProfileUrlRedirect.ashxBizTalk patterns – the parallel shapehttp://blogs.msdn.com/b/quocbui/archive/2009/10/16/biztalk-patterns-the-parallel-shape.aspx2009-10-16T10:56:00Z2009-10-16T10:56:00Z<P align=left>The parallel shape in BizTalk Server (v2004 to v2009) is widely misinterpreted to be multi-threaded.</P>
<TABLE border=0 cellSpacing=0 cellPadding=2 width=512>
<TBODY>
<TR>
<TD vAlign=top width=288><A href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_2.png" mce_href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_2.png"><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: inline; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; BORDER-LEFT-WIDTH: 0px" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb.png" width=244 height=179 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb.png"></A></TD>
<TD vAlign=top width=222>
<P align=left>What the parallel shape does well is that it ensures that all branches complete their execution before the next process takes place. The parallel shape does a context switch on a thread wait (busy waiting, such as waiting for a message to come in from a receive port) between branches.</P></TD></TR>
<TR>
<TD vAlign=top width=288><BR>The reason why the parallel shape is single-threaded, is because the orchestration workflow itself, is single-threaded. However, it is still possible to conduct “multi-threaded” execution within BizTalk Server. <BR><A href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_18.png" mce_href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_18.png"><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: inline; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; BORDER-LEFT-WIDTH: 0px" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_8.png" width=269 height=311 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_8.png"></A> <BR>&nbsp; </TD>
<TD vAlign=top width=222><A href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_6.png" mce_href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_6.png"><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: block; FLOAT: none; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; MARGIN-LEFT: auto; BORDER-LEFT-WIDTH: 0px; MARGIN-RIGHT: auto" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_2.png" width=143 height=244 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_2.png"></A><FONT size=1>*Image courtesy of Werner Van Huffel <BR></FONT><BR>A common practice to “multi-thread” the parallel shape is to use the “<A href="http://msdn.microsoft.com/en-us/library/aa559372(BTS.10).aspx" mce_href="http://msdn.microsoft.com/en-us/library/aa559372(BTS.10).aspx">Start Orchestration</A>” shape, which will instantiate a separate orchestration workflow, with its own thread, and resources.<A href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_20.png" mce_href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_20.png"><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: block; FLOAT: none; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; MARGIN-LEFT: auto; BORDER-LEFT-WIDTH: 0px; MARGIN-RIGHT: auto" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_9.png" width=121 height=60 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_9.png"></A> <BR>Hence, all the logic would reside in the child orchestrations that would be called from the parent orchestration.</TD></TR></TBODY></TABLE>
<P>However – suppose that the number of branches of executions are not known at design time, then the parallel shape would not be suitable to use. A different approach will be required.</P>
<P>A classic example of parallel execution is de-batching a large message file into smaller files, and then aggregating all the results into a single large result. Each subset (small file) represents an independent unit of work that does not rely or relate to other subsets, except that it belongs to the same batch (large message). Eventually each child process (the independent unit of work) will return a result message that will be collected with other results, into a collated response message to be leveraged for further processing.</P>
<P><A href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_24.png" mce_href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_24.png"><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: inline; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; BORDER-LEFT-WIDTH: 0px" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_11.png" width=502 height=118 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_11.png"></A> </P>
<TABLE border=0 cellSpacing=0 cellPadding=2 width=498>
<TBODY>
<TR>
<TD vAlign=top width=137><A href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_26.png" mce_href="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_26.png"><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: inline; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; BORDER-LEFT-WIDTH: 0px" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_12.png" width=122 height=188 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_12.png"></A> <BR><FONT size=1>*Image courtesy of Werner Van Huffel</FONT>&nbsp;</TD>
<TD vAlign=top width=359>
<P>0. The initial step consists of de-batching the large message in the receive pipeline into smaller child messages. </P>
<P>1. The receive pipeline publishes the child messages to the message box database.</P>
<P>2. Child business process orchestrations (or 2-way send ports) subscribe against the child messages to accommodate business requirements. Note that individual business processing does not explicitly require orchestrations.</P>
<P>3. Responses from Line-of-Business (LOB) applications are submitted back to the child orchestrations (or to 2-way send ports).</P>
<P>4. These responses are aggregated back by an aggregator (can be accommodated by an orchestration as depicted, or by a SQL stored procedure).</P>
<P>5. The aggregator will submit the final aggregated result.</P>
<P>6. The next process (which may or may not use the final aggregated result) will proceed.</P></TD></TR></TBODY></TABLE>
<P>De-batching is conducted within the receive pipeline. To learn how to split a large file into smaller files, please refer to the <A href="http://msdn.microsoft.com/en-us/biztalk/bb608375.aspx" mce_href="http://msdn.microsoft.com/en-us/biztalk/bb608375.aspx">BizTalk Server 2006 samples</A> – <A href="http://go.microsoft.com/fwlink/?LinkId=99739" mce_href="http://go.microsoft.com/fwlink/?LinkId=99739">Split File Pipeline</A> example. Even though the sample is based on BizTalk Server 2006 (R1) – it remains relevant for BizTalk Server 2006 R2, and BizTalk Server 2009.</P>
<UL>
<LI>Note: There are several examples on how to de-batch messages, such as de-batching from SQL Server, as communicated by <A href="http://blogs.msdn.com/rgarg" mce_href="http://blogs.msdn.com/rgarg">Rahul Garg’s blog</A> on <A href="http://blogs.msdn.com/rgarg/archive/2007/11/12/biztalk-sql-receive-adapter-debatching-messages.aspx" mce_href="http://blogs.msdn.com/rgarg/archive/2007/11/12/biztalk-sql-receive-adapter-debatching-messages.aspx">de-batching</A>. </LI></UL>
<P>The next step is a bit trickier. The smaller messages need to be correlated somehow, so that their results can be aggregated into a singular large response message (for steps 4 and 5 to be successful). Correlation is typically done in orchestration, but the proposed method here avoids the use of orchestration altogether. In order to have correlation in messaging, please refer to <A href="http://blogs.msdn.com/paolos/default.aspx" mce_href="http://blogs.msdn.com/paolos/default.aspx">Paolo Salvatori’s blog</A> on <A href="http://blogs.msdn.com/paolos/archive/2008/07/21/msdn-blogs.aspx" mce_href="http://blogs.msdn.com/paolos/archive/2008/07/21/msdn-blogs.aspx">Synchronous To Asynchronous Flows Without An Orchestration</A>.</P>
<P>In it, he describes a method that requires a 2-way receive port (such as a web service) , because he leverages a couple of message context properties that are unique to a 2-way receive. These are</P>
<UL>
<LI>ReqRespTransmitPipelineID </LI>
<LI>IsRequestResponse </LI></UL>
<P>In addition to</P>
<UL>
<LI>EpmRRCorrelationToken </LI>
<LI>Correlation Token </LI></UL>
<P>It is strongly suggested to learn how correlation can be achieved in messaging (without orchestration) as described by Paolo Salvatori before proceeding to the next step.</P>
<P>Once that method has been mastered, a slight adjustment is needed. Since the de-batching method is leveraged in the pipeline, correlation in messaging can be achieved with a 1-way receive port (such as picking up the large message from a file directory), by using message context properties that are unique to the de-batch method.</P>
<P>These are</P>
<UL>
<LI>InterchangeID </LI>
<LI>InterchangeSequenceNumber </LI>
<LI>LastInterchangeMessage </LI></UL>
<P>Each child message (which is now in the messagebox database) should instantiate a business process (either with an orchestration workflow or a 2-way send port) that will eventually receive a result message.</P>
<P>Finally an aggregator is required to collect and collate the results from the child business processes. This aggregator can be an orchestration, or the preferred way would be from a SQL stored procedure.</P>
<P align=center><IMG style="BORDER-RIGHT-WIDTH: 0px; DISPLAY: block; FLOAT: none; BORDER-TOP-WIDTH: 0px; BORDER-BOTTOM-WIDTH: 0px; MARGIN-LEFT: auto; BORDER-LEFT-WIDTH: 0px; MARGIN-RIGHT: auto" title=image border=0 alt=image src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_13.png" width=473 height=192 mce_src="http://blogs.msdn.com/blogfiles/quocbui/WindowsLiveWriter/Biztalkpatterns_1310C/image_thumb_13.png">&nbsp;<FONT size=1>*Image courtesy of Paolo Salvatori</FONT></P>
<UL>
<LI>A message is received by a FILE Receive Port </LI>
<LI>The message is routed to a SQL Send Port where the original message is wrapped into a sort of envelope by a map which uses a custom XSLT. The outbound message is an Updategram which calls a stored procedure to insert the message into the table. </LI>
<LI>A SQL receive port calls a stored procedure to retrieve the records of a certain type from the custom table. This operation generates a single document. </LI>
<LI>This document is routed to a File Send Port where the message is eventually transformed into another format. </LI></UL>
<P>This method was also communicated in <A href="http://geekswithblogs.net/asmith/archive/2005/06/06/42281.aspx" mce_href="http://geekswithblogs.net/asmith/archive/2005/06/06/42281.aspx">http://geekswithblogs.net/asmith/archive/2005/06/06/42281.aspx</A></P>
<P>Note that the XSLT can be eventually replaced by a custom pipeline component or directly by an envelope (in this case the send pipeline needs to contain the Xml Assembler component). </P>
<TABLE border=0 cellSpacing=0 cellPadding=2 width=564>
<TBODY>
<TR>
<TD vAlign=top width=352><BR>Sample code is provided courtesy of Ji Young Lee, of Microsoft (South) Korea. Uncompress the ZIP file with the directory structure intact on the C:\ root directory (i.e.:<STRONG> c:\QuocProj</STRONG>). </TD>
<TD vAlign=top width=210>
<P><FONT size=1></FONT>
<CENTER><IFRAME style="PADDING-BOTTOM: 0px; BACKGROUND-COLOR: #fcfcfc; PADDING-LEFT: 0px; WIDTH: 98px; PADDING-RIGHT: 0px; HEIGHT: 115px; PADDING-TOP: 0px" title=Preview marginHeight=0 src="http://cid-742ca4b5187233f7.skydrive.live.com/embedicon.aspx/BlogFiles/QuocProj.zip" frameBorder=0 marginWidth=0 scrolling=no mce_src="http://cid-742ca4b5187233f7.skydrive.live.com/embedicon.aspx/BlogFiles/QuocProj.zip"></IFRAME></CENTER>
<P mce_keep="true">&nbsp;</P></TD></TR></TBODY></TABLE>
<P>Please note the standard disclaimer that abdicates any form of responsibility: <BR></P>
<P><FONT size=1>The sample code is the property of Microsoft Corporation ®.</FONT></P>
<P><FONT size=1>THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED&nbsp; OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR&nbsp; FITNESS FOR A PARTICULAR PURPOSE.</FONT></P>
<P><FONT size=1>The sample code is not supported under any Microsoft standard support program or service. These samples are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the samples and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the samples, code, and documentation, be liable for&nbsp; any damages whatsoever (including, without limitation, damages for loss of business profits,&nbsp; business interruption, loss of business information, or other pecuniary loss) arising out of&nbsp; the use of or inability to use the code, samples, or documentation, even if Microsoft has been advised&nbsp; of the possibility of such damages.</FONT></P>
<P>For more information concerning the parallel shape, see “<I>How to Configure the Parallel Actions Shape</I>” topic in the <A href="http://msdn.microsoft.com/en-us/library/aa559632.aspx" mce_href="http://msdn.microsoft.com/en-us/library/aa559632.aspx">BizTalk Server</A> documentation. </P>
<P>Here are some related blogs regarding the parallel shape:</P>
<UL>
<LI><A href="http://www.arquitecturadesoftware.org/blogs/joaomartins/archive/2006/06/06/352.aspx" mce_href="http://www.arquitecturadesoftware.org/blogs/joaomartins/archive/2006/06/06/352.aspx">http://www.arquitecturadesoftware.org/blogs/joaomartins/archive/2006/06/06/352.aspx</A> </LI>
<LI><A href="http://blogs.neudesic.com/blogs/daniel_sepp/archive/2007/09/21/Daniel_Sepp.aspx" mce_href="http://blogs.neudesic.com/blogs/daniel_sepp/archive/2007/09/21/Daniel_Sepp.aspx">http://blogs.neudesic.com/blogs/daniel_sepp/archive/2007/09/21/Daniel_Sepp.aspx</A> </LI>
<LI><A href="http://www.codeproject.com/KB/biztalk/BizTalkParallelBranching.aspx" mce_href="http://www.codeproject.com/KB/biztalk/BizTalkParallelBranching.aspx">http://www.codeproject.com/KB/biztalk/BizTalkParallelBranching.aspx</A> </LI>
<LI><A href="http://www.traceofthought.net/permalink,guid,e2e45c5b-e8f3-4c05-9c60-d6e5e1ec29b3.aspx" mce_href="http://www.traceofthought.net/permalink,guid,e2e45c5b-e8f3-4c05-9c60-d6e5e1ec29b3.aspx">http://www.traceofthought.net/permalink,guid,e2e45c5b-e8f3-4c05-9c60-d6e5e1ec29b3.aspx</A> </LI></UL>
<P mce_keep="true">&nbsp;</P>
<DIV style="PADDING-BOTTOM: 0px; MARGIN: 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; DISPLAY: inline; FLOAT: none; PADDING-TOP: 0px" id=scid:0767317B-992E-4b12-91E0-4F059A8CECA8:3fbe64ef-23d3-4608-a1c4-1356e4b07f7c class=wlWriterEditableSmartContent>Technorati Tags: <A href="http://technorati.com/tags/BizTalk+Server" rel=tag mce_href="http://technorati.com/tags/BizTalk+Server">BizTalk Server</A>,<A href="http://technorati.com/tags/parallel" rel=tag mce_href="http://technorati.com/tags/parallel">parallel</A>,<A href="http://technorati.com/tags/de-batching" rel=tag mce_href="http://technorati.com/tags/de-batching">de-batching</A>,<A href="http://technorati.com/tags/aggregate" rel=tag mce_href="http://technorati.com/tags/aggregate">aggregate</A>,<A href="http://technorati.com/tags/aggregator" rel=tag mce_href="http://technorati.com/tags/aggregator">aggregator</A>,<A href="http://technorati.com/tags/debatch" rel=tag mce_href="http://technorati.com/tags/debatch">debatch</A>,<A href="http://technorati.com/tags/large+message+processing" rel=tag mce_href="http://technorati.com/tags/large+message+processing">large message processing</A></DIV><img src="http://blogs.msdn.com/aggbug.aspx?PostID=9908096" width="1" height="1">Quoc Buihttp://blogs.msdn.com/quocbui/ProfileUrlRedirect.ashx