Big Data – Then and Now

When I first started my career in IT in the mid 90’s I was a Data Analyst. I absolutely loved it. I felt like Sherlock Holmes trying to find the one key ‘clue’ that would explain everything. It was a natural fit for me as I am very inquisitive and love to seek answers. I believe that no matter how great business is, it can always be better if we can just find the answer in the data.

Specifically, as a data analyst I worked with Point of Sales (POS) data from retail giants Wal-Mart and Target.

Wal-Mart was at the forefront of big data long before it even had a name. In the mid 1980’s they decided to build their own Consumer Behavior software that reported on this new technology called Barcodes. This was no easy feat; according to Frontline they started on development in 1985 and invested 4 billion dollars building it over the next seven years. They officially launched “Retail Link” in 1992. It was truly revolutionary. All of a sudden, suppliers had access to all of their inventory and sales data by SKU, by hour, by store. It was a massive amount of data and it was powerful.

As an analyst, I combined the sales and inventory data with weather patterns, major events, holidays and trends in the news. I also performed market basket analysis (aka Affinity Analysis) to see what products sold well together and when. I analyzed pricing, packaging and shelving strategies to see what trends and what worked and what didn’t.

It’s crazy when I think back to all I accomplished with so much data and ‘primitive’ analytical tools like[...]