This forum is now a read-only archive. All commenting, posting, registration services have been turned off. Those needing community support and/or wanting to ask questions should refer to the Tag/Forum map, and to http://spring.io/questions for a curated list of stackoverflow tags that Pivotal engineers, and the community, monitor.

AnnouncementAnnouncement Module

Collapse

No announcement yet.

Writing a Spring Batch FileReader with a Header and FooterPage Title Module

Writing a Spring Batch FileReader with a Header and Footer

Jun 15th, 2009, 11:21 AM

OK, newbie question here. I'm trying to write a simple FlatFileItemReader where each input file has a header line, several body lines, and a footer line. I configured a PatternMatchingCompositeLineMapper for the reader, since the header, footer, and body lines begin with a different sequence of characters, and I couldn't find a way for the reader to "skip" the first and last line of the input to process the body of the file. There may be another way to do this, but I haven't found an example of it yet.

Assuming that this is the way to do it, my first question is this: If the step has a reader, processor, and writer like the following:

If you're still with me, that would mean that the PatternMatchingCompositeLineMapper.mapLine() method would return different objects in the reader. can you infer from this that the processor and writer (themselves composites) delegate which processor/writer to call based on the type that mapLine() returns?

I would really appreciate it if someone could tell me if this approach is sound. This is my first time around with Batch, and the learning curve (and lack of starting up docs) is a bit of a problem.

As you will see in the source code, the CompositeItemProcessor works by passing the item into a series of delegate ItemProcessors. By contrast, you need an ItemProcessor that passes the item into *one* of the delegate ItemProcessors. To do this, you will need to write a custom composite ItemProcessor like this:

Comment

Thanks for the help here. It didn't really occur to me that the Composite processes a succession of things on what is being passed to it, but now it makes total sense. In the case where I am processing a file with a header line and footer line (which are really being ignored for the most part) is a CompositeLineMapper really the right approach? For this particular app (just a POC at this point) something simpler would be great.

Is there a better pattern for this?

Thanks again!

Brian

Comment

The FlatFileItemReader has a "linesToSkip" property and a "skippedLinesCallback" property that when used in combination, processes the first n lines using the callback instead of the regular process. You could use this for your header, but it wouldn't help you with the footer.

Comment

What you could do is write your own FileReader (and use FlatFileReader as a template) - the only thing you would need to change is readLine() to cater for the the footer row to be omitted - the FlatFileReader already has an option to skip the header. (lineCount attribute)

Comment

Actually, the footer has something useful that I can least log (the number of entries and the dollar amount total in the file). I implemented the processor as you (Dan) indicated, and now I come to the writer. Since the write() method takes a List, which (depending on the chunk) could contain header, body, or footer elements, it seems I might have to iterate the list to see what kind of item I have here, like:

Comment

To whom it may concern,
I have a similar problem with readers but do not have any pattern to distinguish headers and footers from main body. Exactly, I know the numbers of lines at the first of the source (headers) and end of source (footers).
I could utilize "linesToSkip" and "skippedLinesCallback" to handle headers. However, in this manner, for handling the footers, I have to write alternative "FlatFileItemReader" contains "footerLinesToSkip" and "footerLinesCallback.”
Exactly I add a FIFO (First In First Out) buffer with size of footer to class. Here is the detailed information about mentioned buffer:

Then I fill buffer in "doOpen" method without increasing "lineCount”, remove an object from buffer for each request of reading in “doRead” method, and return it as read object. Then nextLine will be added to buffer.
Therefore, we always can see next "footerLinesToSkip" of resource. Whenever the nextLine would be null, removing items from buffer in doRead Method would be stopped and null would be returned to indicate end of input body and "handleFooter" method would be called.