The Evolution of ETL

In todays world, data is generated and collected faster than ever before. In the past, most data was generated by humans, typing data into application forms, ringing up purchases at point of sale machines and so on. Now, the majority of the data created is machine generated, collected in application logs and produced by sensors. The verbosity and sampling rate of these sources has exploded as computing capacity has expanded, storage has become cheaper and the business value of this data has increased. To meet these extreme challenges, a new breed of platforms has been developed including Hadoop, a wide range of NoSQL stores and cloud-enabled infrastructure.

As experience tells us, just putting the data somewhere is not the goal. Extracting, transforming and loading this data to an analytic system is what brings true life and enablement to the data we have collected. For this purpose, we have our trusted workhorse, the ETL platform. But it, too, must evolve in order to serve in this challenging new world.