How could professionals who are paid so much money, have so many resources and the backing of such outstanding large and qualified institutions be so wrong?

An over-reliance on quantitative analysis, combined with using the wrong assumptions.

The conventional approach to Wall Street forecasting is to use computers to amass enormously complex spreadsheets combining reams of numbers. Computer models are built with thousands of inputs, and tens of millions of data points. Eventually analysts start believing that the sheer size of the models gives them validity. In the analytical equivalent of "mine is bigger than yours" the forecasters rely on their model's complexity and size to self-validate their output and forecasts.

In the end these analysts come up with specific forecast numbers for interest rates, earnings, momentum indicators and multiples (price/earnings being key.) Their faith that the economy and market can be reduced to numbers on spreadsheets leads them to similar faith in their forecasts.

But, numbers are often the route to failure.

In the late 1990s a team of Wall Street traders and Nobel economists became convinced their ability to model the economy and markets gave them a distinct investing advantage. They raised $1billion and formed Long Term Capital (LTC) to invest that money using their complex models. Things worked well for 3 years, and faith in their models grew as they kept investing greater amounts.

But then in 1998 downdrafts in Asian and Russian markets led to a domino impact which cost Long Term Capital $4.6B in losses in just 4 months. LTC lost every dime it ever raised, or made. But worse, the losses were so staggering that LTC's failure threatened the viability of America's financial system. The banks, and economy, were saved only after the Federal Reserve led a bailout financed by 14 of the leading financial institutions of the time.

Assumptions - right or wrong - underly all forecasts

Incorrect assumptions played a major part in how Wall Street missed the market prediction for 2013. All models are based on assumptions. And, as Peter Drucker famously said, "if you get the assumptions wrong everything you do thereafter will be wrong as well" --- regardless how complex and vast the models.

Conventional wisdom holds that conservative economic policies underpin market growth, and that liberal Democratic fiscal policies combined with a liberal federal reserve monetary program would be inflationary and bode poorly for investors, and the economy, in 2013. These deeply held assumptions were, of course, reinforced by a slew of conservative commentators that supported the notion that America was on the brink of runaway inflation and economic collapse from rising debt. The BIAS (Beliefs, Interpretations, Assumptions and Strategies) of the forecasters found reinforcement almost daily from the rhetoric on CNBC, Bloomberg, Fox News and other programs widely watched by businesspeople from Wall Street to Main Street.

From an unconventional source - an unconventional, but accurate, forecast

Interestingly, when Obama was re-elected in 2012 a not-so-well-known investment firm in Columbus, OH - far from Wall Street - took an alternative look at the data when forecasting 2013.

Polaris Financial Partners took a deep dive into the history of how markets perform when led by conservative vs. liberal policies and reached the startling conclusion that Obama's programs, including the Affordable Care Act, would actually spur investment, market growth, jobs and real estate! They had forecast a double digit increase in all major averages for 2012 and extended that same double digit forecast into 2013 - far more optimistic than anyone on Wall Street.

CEO Bob Deitrick and partner Steven Morgan concluded that the millenium's first decade had been lost. Despite pro-business Republican leadership, the eqity markets were, at best, sideways. There were fewer people actually working in 2008 than in 2000; a net decrease in jobs. After a near-collapse in the banking system, due to deregulated computer-model based trading in complex derivatives, real estate and equity prices had collapsed.

"Fourteen years of stock market gains were wiped out in 17 months from October, 2007 to March, 2009" lamented Deitrick.

Re-analyzing history looking closely at context as well as actions

Polaris Partners concluded the situation was eerily similar to the 1920s at the end of Hoover's administration. A situation which was eventually resolved via Keynesian policies of debt-financed increased fiscal spending while interest rates were low, and federal reserve intervention to both expand the money supply and increase the velocity of money under Republican Fed chief Marriner Eccles.

Further, while most people conventionally think that tax cuts led to economic growth during the Reagan administration, Polaris Financial turned that assumption upside down and put the biggest positive economic impact on the roll-back of tax cuts a year after being pushed by Reagan and passing Congress. Their analysis of the 1980s recovery focused on higher defense and infrastructure spending (fiscal policy,) a massive increase in debt (the largest peacetime debt increase ever) coupled with a more balanced tax code post-TEFRA.

Different assumptions, and process, improves forecasting

Thus, eschewing complex econometric models, elaborately detailed spreadsheets of earnings and rolling momentum indicators, Polaris Financial focused instead on identifying the assumptions they believed would most likely drive the economy and markets in 2013. They focused on the continuation of Chairman Bernanke's easy monetary policy, and long-term fiscal policies designed to funnel money into investments which would incent job creation and GDP growth leading to an improvement in house values, and consumer spending, while keeping interest rates at historically low levels. All of which would bode extremely well for thriving equity markets.

The vitriol has been high amongst those who support, and those who oppose, the economic policies of Obama's administration since 2008. But vitriol does not support, nor replace, good forecasting. Too often forecasters predict what they want to happen, what they hope will happen, based upon their view of history, their traing and background, and their embedded assumptions. They forecast from their long-held assumption base. But, ideology is not helpful when forecasting, and can be a big negative.

As Polaris Financial pointed out in beating every major Wall Street firm over the last 2 years, good forecasting relies on looking carefully at historical outcomes and understanding the context in which those results happened. Rather than relying on an interpretation of the outcome, forecasters must look instead at the facts and the situation; the actions and the outcomes in context.

There are no universal economic, or market, truths

In an economy, everything is relative to the context. There are no absolute programs that are universally the right thing to do. Every policy action, and every monetary action, is dependent upon initial conditions as well as the action itself.

Too few forecasters take into account both the context as well as the action. And far too few do enough analysis of assumptions, preferring instead to rely on reams of numerical analysis which may, or may not, relate to the current situation. And are often linked to assumptions underlying the model's construction - assumptions which could be out of date or simply wrong.

The folks at Polaris Financial Partners remain optimistic about the economy and markets for the next two years. They point out that unemployment has dropped faster under Obama, and from a much higher level, than during the Reagan administration. They see the Affordable Care Act opening more flexibility for health care, creating a rise in entrepreneurship and innovation (especially biotechnology) that will spur economic growth. Deitrick and Morgan see tax programs, and rising minimum wage trends, working toward better income balancing, and greater monetary velocity aiding GDP growth. Their projection is for improving real estate values, jobs growth, and minimal inflation leading to higher indexes - such as 20,000 on the DJIA and 2150 on the S&P.

Update 21 January, 2014. I just discovered this NYT article from October, 2012 in which the journalists points out that those forecasting a weak 2013 (or recession) seem to have strong ties to the Republican party. Well worth a read, now 16 months later, as the author seems to have been a better forecaster, by recognizing the forecasters' assumptions, than the forecasters themselves.

I'm an expert on business growth and overcoming organizational obstacles to success. I do keynote speaking at conferences and management meetings, and lead workshops for companies wanting to find their next growth engine. I am the author of "Create Marketplace Disruption: H...