Oracle Blog

A site for SOA thought and discussion.

Service Aggregation

A reader of my book asked about using OSB to aggregate services. His scenario was fairly involved, but the crux of
the issue came down to the following question:

How do I get a list of information from one low level service, and then iterate over that list, using the list
information to query other low level services?

This pattern is called "service aggregation" and it a fundamental capability of the Oracle Service Bus.
In this blog entry I will describe the overall integration pattern and then walk through the solution code, which you
can download at the end of this blog entry.

The scenario I used is as follows: the have a single, high-level proxy service that takes in a postal code and returns
a list of customers and the products that they have purchased from our fictitious company. The high-levle proxy service
has to call two different services to accomplish its goal. The first is a lower-level service that takes the postal code
and returns a list of customers. The second lower-level service take the ID of a single customer and returns a list of
products that that specific customer owns. The higher-level proxy service then aggregates these two different lists of
information into the return value for the caller.

Service aggregation is an extremely common use for the service bus. The Oracle Service Bus can act as a facade over the
lower level services, hiding their technical details and their interrelationship from the service consumer. Service
aggregation adds alot of value to the overall system by providing a higher level of abstraction. Ideally, you can even
hide the very existence of your applications from your service consumers, greatly increasing your overall business and
IT agility.

Let's begin with the WSDL for the high level proxy service. This WSDL is going to be focused on the business need, and
not have any information about how the work is performed within the proxy service. The proxy service takes only apostalCode as the argument for the service, and then returns as aggregated list of customers and their products
that live in the given postal code. The WSDL for the proxy service can be found in the WSDLs folder of the sample source code
with the file name of CustomerProduct.wsdl. return type for the service call looks like the following:

The two low-level services are fairly straight-forward in their implementation. The Customer.wsdl defines the
service interface for the low-level service of the same name, as does Product.wsdl. One interesting thing to
note is that the structure of the Product in Product.wsdl is different from the structure used in the top-level
proxy service. I did this deliberately to make the problem more realistic. You will commonly be transforming message
between application specific formats, and the more abstract formats used by your higher level services. This is a
common pattern simply because you don't want to have to change all of your services and recompile all of your
service clients just because you upgraded or changed one of the applications in your application landscape.

Lets take a quick look at how a Product is defined in the Product.wsdl file and compare it to the Product
element defined in the CustomerProduct.wsdl file shown in Listing 1.

<xsd:complexType name="ProductType">

<xsd:sequence/>

<xsd:attribute name="id" type="xsd:int" />

<xsd:attribute name="name" type="xsd:string" />

</xsd:complexType>Listing 2

All we have really done is changed the id and name elements from the Product.wsdl definition of a product into
attributes in the CustomerProduct.wsdl definition of the Product business concept. It may seem minor, but it is
the key to loosely coupling systems. When you hear peope talk about "canonical models", this is what
they mean:; defining your own model for information that is independent of any specific application. This also
gives you the flexibility to add meta data to your canonical models that may not be available in the application
specific versions of these concepts/structures.

Figure 1 shows the basic structure of the message flow for the top-level proxy service, XQueryLoopingExample.proxy.
In the request pipeline we have a service callout to the CustomerService.proxy service. In the request pipeline of the
service callout we see an assign statement. This assign statement simply prepares the customerListRequest variable with the
correct data for calling the service.

Figure 1. The overall message flow of the XQueryLoopingExample.proxy

Immediately below the service callout we see another assign statement where we create a new variable calledcustomerProductList. Ultimately, this variable will contain all of the customer and product information
that we need to pass back to the caller of the proxy service. For now, it just contains the proper elements that the
CustomerProduct.wsdl requires:

Figure 2 shows the details of the For Each action. The purpose of the Foro Each action is to iterate over
lists of information. In our example, we need to iterate over the list of customers returned from our first service
callout.

Figure 2. The For Each section of the message flow

Figure 3 shows the properties for the For Each action. The Index Variable and Count Variable fields
are optional for our purposes here, but I like to fill them in just so I can get some idea of the number of records
being selected. This can be handy when you are first configuring the For Each action and you are still trying to get
the XPath statement correct.

Figure 3. The For Each properties

As you can see in Figure 3, we are using the XPath statement of //Customer to retrieve every customer in the customerListResponse variable (which was populated by the service callout) and to place the individual customer
information into the customer variable.

Now that we can iterate over the list of customers, returned by the first service callout, we are able to execute
the second service callout to the ProductService.proxy to retrieve the list of products for each customer. This service
callout operates in the same manner as the first one did. We have an assign statement in the request pipeline of the
service callout that prepares the request document. The response from the service invocation is stored in the productResponse variable.

Here is where things start to get interesting and a little fun. At this point we have two variables that contain the
information that we need to merge into the final response document. Those two variables are $productResponse and $customer. Now we could write a series of assign statements to build up the response document, but that would be
very cluttered, reduce the readability of the message flow and actually reduce performance a bit. Instead we will use a single
assign statement that will use an XQuery transform to do the real work. Figure 4 shows the graphical view of the XQuery
transform.

Figure 4. The XQuery transform

When we create the transform, we simply select the two data types that represent the input variables. In our case
the input types are a getCustomerByPostalCodeResponse/CustomerList/Customer and a getCustomerProductResponse,
respectively. We also need to specify the return type, which is a getCustomerProductByPostalCodeResponse/Customer.
Once this is done, the Workshop development environment brings up a graphical view of the source and return types and we simply
drag and drop from the source to the return type to create the transformation. The only part that may not be obvious is
the dashed arrow on figure 4, which connects the source and destination Product elements. This is done to tell the XQuery
engine that there are possibly multiple Products on each side and that we want the XQuery engine to create a Product in
the return variable for each Product it finds in the source variable.

At this point in the loop, we now have a single customer record with all of the products owned by that customer. The
insert action is used to insert the $customerProductRecord information into the existing $customerProductList.
The rest of the proxy service is very straight-forward. There is a replace action in the response pipeline of the pipeline
pair that places the $customerProductList into the $body variable.

Finally, there is a validate action that validates that the data
we are returning conforms to the expected data format defined by the web service. This is definitely a best practice when
you are developing your services. Validation does take additional time, so it is often turned off in production, but while you
are developing your services, it can save you alot of time and headache when debugging.

Conclusion

That's it for this entry. Hopefully I have shown that service aggregation is not only easy to do, but that it also has
the potential to really help your overall architecture by loosening the couplings between your service consumers and low level
service providers. It gives you an opportunity to create your own layers of abstraction over the low level services and
applications, which in turn puts you in control of your architecture.

I have also received a question about using a converter class to convert EJB information into. XML. That will be my next
entry. Until then, get your hands dirty and keep your code clean!