The cloud and big data

When the cloud came into
being, it brought with it immense storage power at cheaper rates. It ushered in
the era of Big data. As a result, it also raised expectation levels in the minds
of statisticians - and decision makers who depended on them - that this would do
wonders to their decision-making processes.

Boon or bane?

The sample space had drastically
increased due to social media and IoT, leading to more data being made available
now. Applying statistical models to this huge data would improve the probability
of a predicted event occurring (or not occurring) or improve the reliability of
the forecast by pushing the R squared value to near unity. Right? Wrong. The data
deluge only added to more noise than dependable signals.

Illusion or disillusion?

As time went by, people became
disillusioned by the failure of the system to aid them with reliable
information in decision-making. So as their hyper-expectations were not met,
they just drop off quickly without pursuing further this journey.

The signal and the noise

It was now the turn of the
experts to come with their reasons as to why such huge data could not help them
decide better. One significant reason is that while there is enough data - and
more – for the model, it requires a great deal of cleaning – removing the noise
in the data that could distort the results and predictions before this data can
be put to any use at all.

Persistence pays!

Early adapters of technology gained over the long run. Microsoft and
Amazon are examples of winners who persisted in their vision to make this big data
the fuel to their decision-making engine. They soon gathered themselves up from
the trough of disillusionment to the slope of enlightenment by applying scientific
methods to the data gathered and adopting newer methods to remove noise and
false signals from the data. This way, they could arrive at real signals that
aided in building reliable data models. They have climbed to the plateau of
productivity now with their data models helping them in better decision making
based on information.

Here are a few points to ponder:

People expect a lot from technology today, but the problem is while we have a lot of data, there are not enough people who possess skills to make this big data useful and not enough training and skill building efforts being put to make data scientists out of this huge population of technology experts.

Cleaning up data is the first big problem in predictive analysis – there are many external factors that might tend to distort the data that has been collected.

If we are considering a correlation between two variables and don’t know what causes this correlation, it is better not to consider this correlation at all. (Star fish predicting the FIFA world-cup winner or a baseball team’s win or lose determining the movement of the share-market).

Seeking for signals desperately, people end up with more noise than signals – so they make decisions with their instinct / gut feeling / experience playing an 80% part and statistics playing the last 20%. Instead, we should be guided by statistics 80% and leave the rest to our instincts and that too only if there is a drastically negative indicator in the statistical model.

Here are some suggestions to reduce the noise
and arrive at signals:

Start with a hypothesis / instinct and then keep refining it as you go
ahead with analysis – this might sometimes lead to reverse your hypothesis.

Think probabilistically

When predicting, consider
the margin of error (uncertainty) of the historic data and then include that in
the prediction to make a decision. The person that discloses the greatest
uncertainty is doing a better job than the one who conceals this uncertainty to
his prediction. Three things to carry with while predicting: Data models,
Scientific theories that influence the situation and experience (learn from the
number of forecasts made and the feedback about the forecast)

Know where you come from

Consider the background and
the existing biases of the possible forecaster / decision maker and the
situation the data is being collected / considered

Try and err

Companies need to focus on
the 80% effort for the last 20% results to retain the competitive advantage –
real statistics of a few customers would be better than hypothetical data of a
huge number of customers.

Notes:

Large and smart companies especially Technology firms should dare to take risks in the competitive advantage area. Most of the risk-taking will pay off. As they are big, they can bear failures unlike small firms and individuals in which case this might be termed as gambling.

People make better inferences from visuals than just data presented as raw data. Charts must show simple essential info. Unless required to bring greater clarity, we must avoid showing more information that crowd together on the charts to create more noise.

People must become Bias detectors – raise business questions and be apprehensive about magic bullet solutions.