Hedge Funds and money managers have come a long way from the days of handwritten financial models. In fact, stock and commodity exchanges report that more than 80 percent of today’s trade decisions are computer driven. But a broader change is underway as Wall Street begins to embrace artificial intelligence. By combining AI with big data, researchers are taking advantage of unusual trading opportunities that, until recently, were too complex or expensive to implement. Some of the largest hedge funds are actively pouring resources into AI-driven analysis and competing to hire the best and brightest talent in the field.

Traditional Investment Strategies Come With Serious Limitations

Since the 1980s, two competing schools of thought have co-existed in the investing world.

Fundamental analysis, by far the more prevalent, seeks to value an asset and create price predictions from financial factors and economic data related to a specific company, commodity or currency. While this approach provides good theoretical boundaries on prices, it suffers when abrupt shifts in market sentiment cannot be explained by the metrics under consideration. Ultimately, a security’s value depends not only on measurable metrics but also highly subjective views of the future and, to further complicate matters, the views of everyone else in the market.

The second approach is quantitative investing, which largely eschews the subjective forecasts of fundamental analysis in favor of statistical models applied to historical data. Quants, as practitioners are known, develop models that are back-tested on years of data to optimize trading performance. Despite sometimes-heavy reliance on complex math, many of these models are fairly simple at their core. This is by design: as models grow in complexity they become harder to work with and explain. Artificial Intelligence allows some of this complexity to be offloaded to an automated system, allowing for richer quantitative models built directly from data.

Artificial Intelligence Builds Models that are Both Objective and Flexible

Artificial intelligence — and, more specifically, machine learning — is changing the way quants build trading models. This approach, borrowed from the scientific community, uses computer programs that “learn” from data and adjust their behavior as information changes. Traders can answer questions about financial markets using the same basic algorithms Google uses to build self-driving cars and Amazon uses to recommend products.

Machine learning is most useful when there is too much data for human analysts to process efficiently, or when the nature of the data makes human-driven analysis difficult. Financial markets provide numerous instances of both challenges.

For example, some quant algorithms comb social media for clues as to which brands are growing or shrinking in prominence. The idea of processing such large volumes of text might sound impressive, but text is just the beginning — these algorithms can even recognize that blurry bottle of Coke in the background of a vacation photo. Similar algorithms count cars in the parking lots of big box stores by analyzing satellite photos. What was once a sea of unstructured data becomes a stream of usable metrics, all without human intervention.

On the less exotic side, machine learning provides new ways to look at simple data streams such as stock prices. The human eye can recognize patterns with enviable speed and reliability, but it also has a tendency to see patterns that aren’t actually there, and it has almost no ability to handle multidimensional data: consider the last time you tried to cram a dozen metrics onto the same chart. In contrast, machine learning systems can find patterns that are subtle, dependent on numerous factors and verifiably real.

All of this has the effect of providing information that, while technically available to the general public, is not easily integrated into the average trader’s decisions. This can give a significant edge to the hedge fund willing to reach for it and, perhaps, launch its own satellite. That said, the stock market is embarrassingly hard to predict even with such an edge. All those cars in the parking lot may not have bought anything. And if they did, maybe the company’s CEO expressed some worry on the conference call and thousands of hours of computation went towards predicting a factor that the market ignored. Successful application of machine learning still requires an artistic touch and, in many cases, a little luck.

Still, AI introduces some impressive new tools to the quantitative trading arsenal. Traditional investment models rely on assumptions that certain conditions — that may never have been real in the first place — will persist indefinitely. AI systems can dig deeper, incorporate more data and completely change decision-making, when conditions justify it.

Despite Critics, AI is Finding its Place in Finance

Skeptics point out all this data mining has produced nonsensical market indicators that may be useless going forward. The temptation to build large models on high-dimensional data can lead to a problem known as overfitting: the equivalent of memorizing the answers on a test without understanding the material. And when it comes to financial markets, there is the risk that many AI techniques have already become commodified. Once enough traders start to use a particular technology, the opportunity tends to be arbitraged away.

But none of this is stopping the biggest and best managers from committing to this technology — and successfully. Brand-name hedge funds such as such as Renaissance Technologies, Two Sigma Investments and Bridgewater are already adopters, as are countless startups. Flush with investor cash and unrestricted investment mandates, these firms are constantly searching for new, more efficient sources of returns.

So, are these machine-driven strategies a good thing? If successfully applied in science, why not investing? AI critics like Elon Musk claim all we’re creating is a dangerous world of robot dominance, and some fear that AI-driven investing will only increase the risk of another Flash Crash like we saw in 2010. But proponents will point out that, when properly managed, these technologies can improve liquidity and add value to any form of investment analysis. After all, while human intelligence and processing power remain static, machines are becoming more powerful and data more abundant. Regardless, we are in the midst of a rush to explore this new frontier and, once it’s been unleashed, it’s hard to picture AI moving backward.