In this new version of Article (Talend) Stress Test 3: Lookups & Filters, we will measure the use of resources and the availability of the results, obtained by joining data between the flat file and MySql, filtering and writing the results on other flat file, everything on the new hardware architecture.

The "JVM Arguments" (After several changes looking for the best finish times) the parameters went from: [Xms:256M - Xmx:1024 ] to [Xms:2048 - Xmx:6144M] clearly impacting on the final results: 112s a 30s. This reduction of almost 3/4 of the time is a direct consequence of the increased rate of rows/sec [53.000 r/s to almost 200.000 r/s].

Leaving a minute this particular test, in real life, it is necessary to access the same dimension on different occasions, for example, Time Lks are common in different models. Generate a new connection to access these common tables for each model (generally in low volume) is redundant. It would be advisable to centralize the downloads, implementing in this case HashMap. What is a HashMap? is a Java data structure maintained in RAM, therefore the data access is much faster.