Featured Post

This example uses the same controller as in a previous post but adds a use case to support email validation. A Commons Validator object is ...

Tuesday, May 10, 2011

Comparing 2 File Lists in Talend Open Studio

This post uses the add() and remove() Java Collections calls to the set difference of two lists of files. Files from 2 directories are listed and one directory's files are removed from the other. The result is a difference used in later processing.

java.util.List.add() and List.remove() can gather the results of a tFileList for later processing. Pulling in a library like Commons Collections, these lists can be easily searched, sorted, transformed, and operated on with a variety of set-based operations (intersection, subtraction, etc.).

tFileList produces an iteration that can be converted to a flow using the tIterateToFlow. The flow can then drive a tJavaRow. The tIterateToFlow has the benefit of conveniently defining a flow schema. The schema in this example is a single field (the full file path), but could be extended with additional columns for the other variables produced by tFileList.

Job Comparing 2 Directories' Files

tSetGlobalVar

The job starts with a tSetGlobalVar, listed as "Init Builder List". You may have seen examples of setting variables of simple types like "FileName" set to "C:\data\file1.txt". This job sets a variable that is a Java object which will be unpacked in the other components.

Use the following as the Key and Value of the tSetGlobalVar component. There are no quotes around the 'new' statement.

"builderList" / new ArrayList<String>()

tFileList

Both tFileLists -- "List csv files" and "List txt files" -- are standard. They are searching for files of pattern "*.csv" and "*.txt". (In a production job, these components should use context variables that parameterize the directory.)

tIterateToFlow

The tIterateToFlow components connect the tFileList with custom Java code (tJavaRow). The tIterateToFlow defined a single-field schema. If additional file list information were needed, the schema could be expanded to include other tFileList variables.

tFlowToIterate Configuration

Both tIterateToFlow components are similar except for the tFileList source.

The job ends with the execution of a tJava. This tJava can deal with the build-up lists or could be replaced with another set of iteration components.

You can always integrate large blocks of Java code into Talend for some functionality that's missing off-the-shelf. However, it's usually more maintainable to keep the jobs as standard as possible. If you do choose to write Java code, try packaging the Java code in a routine or component rather than embedding chunks throughout the job.

This job leaves off at the tJava with a java.util.List of files that you can use for later processing. I added a new post here: http://bekwam.blogspot.com/2012/08/iterating-over-java-collection-with.html. This post explains how to work with the List.