AI News, Why is machine learning in finance so hard?

Why is machine learning in finance so hard?

Financial markets have been one of the earliest adopters of machine learning (ML).

Even though ML has had enormous successes in predicting the market outcomes in the past, the recent advances in deep learning haven’t helped financial market predictions much.

Even though there are a number of papers claiming the successful application of deep learning models, I view those results with skepticism.

The issue of data distribution is crucial - almost all research papers doing financial predictions miss this point.

We expect the distribution of pixel weights in the training set for the dog class to be similar to the distribution in the test set for the dog class.

In addition to making sure the test and train sets have similar distributions, you also have to make sure the trained model is used in production only when the future data adheres to the train/validation distribution.

While most researchers have been mindful not to incorporate look-ahead bias into their research, almost everyone fails to acknowledge the issue of evolving data distributions.

For example, even if we have a complete understanding of what happened during the great depression of the 1930s, it’s hard to convert it to a form that makes it usable for an automated learning process.

(Please note that mixture of experts is a very common technique to combine the models from the same scale - almost all quant asset management firms employ this technique.) I

If there is one thing you take away from this post, let it be this: Financial time-series is a partial information game (POMDP) that’s really hard even for humans - we shouldn’t expect machines and algorithms to suddenly surpass human ability there.

What these algorithms are good at is the ability to unemotionally spot a hardcoded pattern and act on it - this unemotionality is a double-edged sword though - sometimes it helps and other times it doesn’t.