IT Knowledge from developers for developers

In times of business process engines, ESBs and SOAs you might think that the good old batch processing is forgotten. But insurance companies and banks have large amounts of data to be moved and here the batch is still the first option to choose. For some customers we have implemented Java-based batch processes and have made good experiences. In many projects we found complex and individual implementations. So, let’s have a look at some standard technology. It’s worth it.

Our partner springsource offers an Open Source Batch Framework in addition to the already very popular components such as the springframework, Spring MVC and Spring WebFlow. Unfortunately, there are many outdated web articles from the early days of the framework.

Technically, the framework is well documented. At this point I would like to show a practical example and not mentioning individual details. Using Spring Batch makes it possible to implement a working solution in a short time and only a little effort.

Example from a Car-Insurance-Company

A customer switches to another insurance company and tells his current claim categories to the new insurer. To check this, the new insurer contacts the GDV (association of german insurers). The GDV provides the so called VWB-service, the system to verify this specified categories. The basis for the communication are text files with a fixed-length record structure.

Basic Configuration

The configuration for the processing of incoming VWB-messages with Spring Batch Version 1.x looks like this:

A Spring Batch job consists in most cases of 1-n steps. In this example a special SkipLimitStep is used, where you can configure exactly which types of exceptions will be accepted or will cancel the job directly. This is usually very helpful because not all records can be interpreted correctly and a restart is not necessary, when only a few items are wrong.

The configuration also shows that the individual resources (e.g. input file, log files) are injected in the step as a listener. The purpose here is to use a Spring Batch component, which is responsible for clean creation and processing of files. In addition it is also possible to use wildcards for file names.

This process can be configured exclusively in the XML file. The FlatFileItemReader receives a reference to the input file and delivers each line to a LineTokenizer. The standard implementation of PrefixMatchingCompositeLineTokenizer transforms the data into FieldSets, comparable to an array or a database ResultSet, where each field can be accessed by an index. The GDV provides each record with a record type as a prefix, so that the LineTokenizer always knows exactly which fields to be mapped. Different implementations for example using dynamic record lengths are available. The FieldSetMapper is the only place where you have to implement some code. The implementation of the method public Object mapLine(FieldSet fieldSet) creates from a FieldSet the target object. This example is using a generic implementation that creates a Java object, which is later transformed using XStream into an XML document.

2. ItemWriter: Processing and persistence of the items in the target system

From the perspective of Spring Batch not much is happening here and is not supposed to! The goal should always be to delegate to a business service which is responsible for the processing. This design rule leads to the advantage of better testability and reusability by other components – for example in online processing. In the first stage the document is attached only to the target contract. So manual processing is required after the batch.

3. Tasklet: E-Mail the log files

Of course, as with any clean implemented software component there must be a monitoring. Conceivable are different approaches. Spring Batch offers a listener interface for almost any place in the job. The VWB sample log entries are written per item, which provides information about the success/failure type of processing. In the last step the MailTasklet sends the appropriate log files to the responsible persons.

Testing

As expected from Spring the testability of components is very easy. The job configuration can be tested with all needed dependencies using the well-known testing framework components of Spring. Here is an example providing a basis for a test:

Conclusion

The example shown is mostly based on the old version of the framework. Currently 2.1 is released and offers useful innovations, including a simplification of the configuration. In one of my next blog entries I will discuss the differences in detail. Another interesting topic in this context is the use of Spring Integration, where we would be back in the ESB world 😉 I would be grateful for any feedback and suggestions for topics relating to Spring Batch 🙂