Looking for your opinion on the matter of Kalman filters. Have you successfully used the technique for stock return forecasting? If not, what are the usual applications of Kalman filters within finance? For me to be able to implement it, I will need to invest some serious time to understand it first. This is the reason I am asking whether you guys see any value in it for this purpose.

Should be in basics.Kalman filter is a way to estimate the path of the hidden variables in a state space model.Then the idea is that you need to understand the state space framework and then you apply the kf. Iteratively you can use the kalman filter the compute the conditional expectation of you hidden variable (markovian) the compute the likelihood and so if you have coefficients to estimate you can use ML or similar.You can look at it from the bayesian perspective and you have priors there, that opens up the game to more complex MCMC in case your model is not in closed form.

Stock trading? Why not SS models have applications all over finance and are a good one to keep in your toolbox.

I know there are a lot of applications for trend following where things can get complicated with this stuff.

Keep it simple and robust, bad starting values or priors can cause instability

The Kalman filter itself is the easy part. The hard part is formulating what goes in to your state space model (what hidden states you have, how they evolve over time, and how they give rise to the observed returns). Given you have a good (i.e. closely resembling reality) state space model, then applying the model with the Kalman filter will give you reasonably good forecasts. But again the big if is whether your state space model is good to begin with.

Yes that is indeed what I have been able to gather. But that also begs the question, if I knew the state space model, or I guess the state space variables, wouldn't that be very similar to knowing the x-variables in a normal linear regression? Wouldn't the only benefit of the Kalman filter in that case be to somewhat improve the accuracy of the model when forecasting?

Or am I missing something? My question is as follows: is there something fundamentally different in trying to determine state space variables compared to determining independent variables in linear regression? Of course this is all in the context of financial time series, I know that these models have been developed for a different purpose from the beginning.

Missing a lot...For reference I would go through this book from durbin and koopmanhttps://www.amazon.com/Time-Analysis-State-Space-Methods/dp/019964117X

It will also be easy to implement following their algorithms.A lot of the existing packages do not allow easy definitions of models so it is probably easier to follow those instructions to filter and estimate parameters.

The main applications are time varying parameter models, dynamic factor models and autoregressive models. To name a few.

I am a bit puzzled by this thread. It makes Kalman filters look trivial.

Trivial? Perhaps in an ideal world in which God grants one a linear state space model and where all noise is additive, white and Gaussian. Who lives in that world?

Even in engineering, where masses and moments of inertia can be measured very accurately, tuning a Kalman filter is a difficult art. Of course, in engineering failure is quite obvious. In econometrics, however...

And the folks at NASA's Jet Propulsion Lab do not use the standard Kalman filters when sending things into space. Not sufficiently robust. They use more robust algorithms one can only find in exotic books that only a few graduate students have even heard of.

There were exactly two occasions outside of finance (once profesionally with GPS data and once as a state estimation component of a toy project implementing PMC stuff) where I actually used Kalman Filters and new beforehand that it was the way to go.....but still it wasn't easy (or I am dumb, who knows)... and I had to dig deep and came up with some two layer Kalman Filter I stole from a paper.

I now have a very hard time envisioning scenario in finance where I would use Kalman trust the results...

"I now have a very hard time envisioning scenario in finance where I would use Kalman trust the results... "

there is example of rather simple scenario in the cross-sectional equity approach:

if you have exposures of the stocks to "price" and then you force overall portfolio exposure to be 1 for the current moment and zero portfolio exposure for 1 year ago then when trying to solve this linear system of equations you can apply Kalman filter to get the weights. And the resulting signal will obviously resemble cross-sectional momentum. And you can generalize this approach easily.

I've used KFs for about a decade in HFT; more often than not they've provided marginal benefit over simpler algorithms like EWMAs and such, but I've appreciated the structure. I've used both EKFs and UKFs in my space days; can't comment on how, but the idea that KFs don't get used in space is crazy -- often they are literally the optimal solution.

Where things get challenging is that kalman filters lack some very often desired properties (hard bounds, guaranteed numerical stability), there exist some pretty fancy solutions that target this, but in finance, I'd never bother, just have a meta component that ensures things aren't going wonky, and disable/reset in case you end up in unsafe gradient hell.

There are no surprising facts, only models that are surprised by facts

FWIIW my first consulting job was putting a Kalman filter in an early G3 phone with accelerometers. It wasn't so bad, though I had to band-aide it with some tricks to get it to do what I wanted it to (aka ignore gravity).

I know a guy who was successful with Kalman-like approaches in futures markets. I think he was doing something like what Zoho says. Or, like this:

EMAs are recoverable basis functions for any linear model. And that extends to any continuous non-linear model, if you feed them into a universal approximation like a neural network or kernel machine.

Therefore, I'd say your default should just be OLE on a whole bunch of EMAs spanning the full range of potentially relevant time horizons. OLE is still an unbiased linear estimator in the time-series domain. It's true you need time-series specific regressions if you want to rely on the p-values for coefficient signifigance or point estimation. But nowadays you're probably cross-validating anyways, and the computation cost of regression is basically free.

IMO Kalman filters, or any other state-space model, should probably only be used if you have very strong inductive priors or a small/high-variance dataset. Otherwise just get a shit-ton of data and throw it at a regression of EMAs.

KF is not trivial at all but not even that difficult to implement for basic simple models.Certain is that it is highly unstable if mispecified, which is often the case.There is also a case of sensitivity to starting values, which in the bayesian world would be sensitivity to priors.A sausage machine basically.MCMC works best for state space more complex models, like the ones you can work on in trend following.There too you need to be careful and model instability is often linked to almost non stationary mc samples.

On a more intuitive way you can figure this out easily, you are trying to estimate a hidden state that drives the observable Y, and this state is markov and has some behaviour given by the model of choice. Still the state is not observable, so unless you are denoising, you need to know exactly what to expect from this behaviour or you are just guessing and coming out with a mispecified model.

Bit late, but just to weigh in on state space models - I've used them a lot in HFT (for example pressure in related assets). There are some areas where non-linear dynamics come up and particle filtering is very useful (stochastic-vol is the most prominant example I can think of where noise is multiplicative). Some people will do EKF/unscented (I think this is hard to make work in practise). I like Bayesian stats so I use sequential MC for this kind of thing (to make it work in practise you need a resampling step or your importance weights will be crap, but this can be scary with non-stationarity). The hardest part as already mentioned by others are the classic noisy signal processing issues. Choice of prior, GIGO and all that. Filter, filter and then filter again (hint: one can combine the above two approaches via Bayesian anomaly detection).