As Stokes describes (and which he also mentions really comes from a presentation by Twitter’s Nathan Marz), there are “fast” big data and “slow” big data problems. For the “fast” problems, you apply a set of pre-developed algorithms and tools to the incoming datastream, looking for events that match certain patterns so that your platform can react in real-time. However, sometimes you need to ask questions of the data, and then analyze the results, which can’t be done in real-time effectively. This describes the “slow” problems, or as Stokes puts it, where you gather information and test hypotheses by running queries against a vast backlog of historical data.

It turns out that the natural evolution of analytics is to go from “slow” problems to “fast” problems, turning the inquisitive understanding of the data, requiring analysis, into faster number-crunching analytics. Knowing the right way to generate these “fast” analytics requires an solid analytics engineering discipline, especially when the problems being answered get harder and harder.

Related Posts:

I currently serve as Vice President of Decision Science at CenturyLink. I've previously served as a leader in the Advanced Risk & Compliance Analytics (ARCA) practice at PwC and as Director of Data Science & Analytics Engineering at Areté Associates. I've served the public as Chair of the Thousand Oaks, CA Planning Commission. I have been married to my wife Stephanie since 1993, and we have a wonderful daughter Monroe. Learn more about me »

Disclosure of Material Connection: Some of the links in the post above are “affiliate links.” This means if you click on the link and purchase the item, I will receive an affiliate commission. Regardless, I only recommend products or services I use personally and believe will add value to my readers. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.”