Nate Silver’s model had given Trump a 29% chance of victory and other models tracked by The New York Times had put Trump’s chances at 15%, 8%, 2%, and 1%. Silver pointed to the uncertainty associated with his prediction due to polls not being perfect, “correlated errors” among polls across important states, and larger-than-usual undecided voters. Indeed, small errors in estimates in a few states can have a significant impact on the electoral vote.

But let us not get distracted by the details surrounding the statistics and ask ourselves a simpler question: why were ALL data-driven models wrong in terms of expectation? Considering that policy makers and managers must occasionally make decisions based on expectations despite the uncertainty surrounding them, what can we learn about data-driven prediction for such instances?

Systematic bias in data

The first lesson is about questioning sources of systematic bias in the data itself. Data rarely come packaged in a form that is useful for making predictions about the outcomes you care about. This can be especially true for infrequent decisions. The poll data were no exception. My initial response to the media question was that the data must have been biased, considering that all data-driven models were consistently wrong in their expectation prior to Election Day.

I conjectured to the reporter that the bias could have been due to the social stigma associated with supporting Trump openly. There appears to be growing evidence supporting this hypothesis. People didn’t want to be viewed as racist and sexist, especially if there is something else about his message that was appealing, such as renegotiating trade agreements or lowering corporate and individual taxes.

On election night, David Brooks of The New York Times called this the “leaner” phenomenon: people at a diner leaning over to a friend and whispering “I just might vote for Trump.” Indeed, three days before the election an article in the New York Daily News referred to data from a Pew Research survey and ABC polls suggesting fractures among spouses and friends about the election. Articles in the New York Times and the Wall Street Journal reported a similar theme, of Trump supporters not wanting to talk about it. The bias in the data could have arisen due to other reasons, but the lessons to be learned are the same

Biased data can have the effect of making a point prediction seem more certain than it should be. In turn, this causes overconfidence about how you interpret the outputs of a model. For example, poll data showing a steady 65% to 35% Clinton vs Trump distribution in the critical swing states would make prediction easy — assuming the weekly distributions represented the truth.

But, what if due to factors such as social stigma, a fifth of the 65 Clintonites were actually supporting Trump but were unwilling to reveal their real preferences? In this case, the true distribution would be much more even, with a larger degree of uncertainty associated with predicting the winner, especially going into the final week when preferences can be volatile. Unless the sources of bias that might be shielding the true distribution are identified and quantified correctly, no amount of modeling sophistication will yield the correct point prediction.

Getting the data right is hard work and it isn’t an exact science, but it can a worthwhile endeavor especially when the stakes are high.

Human bias in interpreting data

The second lesson is to be aware of human bias in interpreting data, specifically, in corroborating prior belief instead of questioning it. This can lead to a herd mentality, commonly observed in markets and in groups of individuals. It is well known that professionals across a range of professions have a tendency to use data to confirm their biases. They are too eager to ignore pesky data. Initial hypotheses often crowd out dissenting evidence.

It is likely that this phenomenon fed on itself to the point where the media, with a clear preference for Clinton, interpreted most of the data in her favor. Most political commentators opined that she handily won the debates based on Trump’s ignorance of facts and his misstatements. In reality, people may not have to put as much weight on such criteria as the media. Indeed, as one commentator remarked, “voters took Mr. Trump seriously but not literally, even as his critics took him literally but not seriously.” Were the media’s assumptions about what people care about inaccurate?

Infrequent events are harder to predict

Perhaps the most important lesson is the recognition that “big data” and predictive analytics tend to work best when there are large numbers of independent instances of the phenomenon of interest, but can be challenging in predicting infrequent events such as elections, earthquakes or financial meltdowns. In the latter case, there are not enough occurrences on which to build a robust predictive model and we must rely on other proxies that must be questioned carefully.

In general, the rarer the occurrences of the phenomenon being modeled, the more careful we need to be in integrating data-driven solutions with human intelligence in building predictive models. In this particular case, the human intelligence part should have guided us to a more sophisticated formulation of the prediction problem by recognizing and accounting for the bias in the data that we were being alerted to by the media but chose to ignore.

This is a key “prediction 101” lesson for managers and policy makers: do I have enough of the right data to make predictions about things I care about? If not, is it available or can it be created?

Vasant Dhar is a professor at the Stern School of Business and the Center for Data Science at New York University. He is chief editor of Big Data, a journal.

Intraday Data provided by SIX Financial Information and subject to terms of use. Historical and current end-of-day data provided by SIX Financial Information. All quotes are in local exchange time. Real-time last sale data for U.S. stock quotes reflect trades reported through Nasdaq only. Intraday data delayed at least 15 minutes or per exchange requirements.