First stress test: Transfering 6 millions of records from a Flat File, to CSV file.

It was a pleasure doing tests with tools like this, Talend and transitivity Jasper, I was amazed at key points of development. Connectivity is one of them. The processing speed with so few resources is another. Things like the logs, I did not like much, but everything can not be perfect.

Referring to this case, I played a lot with Xmx and Xms as variables, and record the changes so that they could see as impacted at runtime.

As in most of the Tests, there are several cases using different techniques, setting of variables, with the same objective, that satisfy the requirement and improve the performance. Always the first case is the best, and the rest of the cases, are jobs without tuning, or an incomplete set of techniques (but helped on the way).

From the first test we had optimal results, but try to find the point of equilibrium between the saturation of the resources and benefits.

To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk.

Rows:

6.024.000 M

Columns:

37Columns

Resources:

Virtual machine with: 2 GB RAM, Talend like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources)

Structure:

(Metadata)

* flexibility in the management of metadata

Design & Run

Elapsed time (s)

183 Secs.

Rows per sec (avg)

32.796 rows/sec

How to Improve Perform

- Adjust the parameters: Xms -Xmx (as shown in the figure above)

CASE 2: -Xms512M, -Xmx1280M

Objective:

To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk.

Rows:

6.024.000 M

Columns:

37Columns

Resources:

Virtual machine with: 2 GB RAM, Talend like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources)

Structure:

(Metadata)

Design & Run

Elapsed time (s)

203 Secs.

Rows per sec (avg)

29.651 rows/sec

How to Improve Perform

- Adjust the parameters: Xms -Xmx (as shown in the figure above)

CASE 3: -Xms256M, -Xmx1024M

Objective:

To measure elapsed time reading and writing 6 million rows, from Flat file, to .CSV file, working on local disk.

Rows:

6.024.000 M

Columns:

37Columns

Resources:

Virtual machine with: 2 GB RAM, Talend like main process over the virtual plataform. The resources used are anecdotal, today, Any production environment has enough processing power for current and future requirements. The objective here, is to build, to execute and to measure with the same environment (regardless of the limited resources)