In the last course of our specialization, Overview of Advanced Methods of Reinforcement Learning in Finance, we will take a deeper look into topics discussed in our third course, Reinforcement Learning in Finance.
In particular, we will talk about links between Reinforcement Learning, option pricing and physics, implications of Inverse Reinforcement Learning for modeling market impact and price dynamics, and perception-action cycles in Reinforcement Learning. Finally, we will overview trending and potential applications of Reinforcement Learning for high-frequency trading, cryptocurrencies, peer-to-peer lending, and more.
After taking this course, students will be able to
- explain fundamental concepts of finance such as market equilibrium, no arbitrage, predictability,
- discuss market modeling,
- Apply the methods of Reinforcement Learning to high-frequency trading, credit risk peer-to-peer lending, and cryptocurrencies trading.

Taught By

Igor Halperin

Transcript

Now, let's talk about the full time-dependent solution of the vagus model. It's also so easy to solve, it's dynamics and I leave it for you as a homework exercise and give you just the final solution, which is shown on the equation 15 here. I show here two equivalent forms of the solution that both help to clarify different aspects of the solution. Let's first consider a normal case when both kappa and theta are positive. If you now look at the solution in its first form, you can see what happens as time goes to infinity. The second term in the denominator dominates and then the exponent cancels out in the numerator and the denominator. And therefore, in the limit when time goes to infinity and the solution approaches the value theta over kappa. The second form of the solution probably shows this even more clear. The time dependence of typical solution is shown in this graph. So what we can see is that we can absorb a variety of slightly different behaviors if we vary kappa withing some bounds. In this graph, we see that varying kappa by a factor of 5 between the smallest and the largest of the values shown here, we obtain a similar order of magnitude of differences in the output, so it's reasonable. So we can see that if copper is positive then the model behaves quite reasonably. For example, if kappa is 5 times 10 to negative 3 shown as the red line on the bottom of this graph, the stock will grow only a little bit in the long run. And on the other hand, if kappa is five times smaller as in the blue line here in 30 years from now, such talk will grow about five times more than the first one. So we can get quite flexible chorus by varying kappa. So far so good as long as we look at positive kappas. Now, let's take a look at the solution for negative kappas. Here I take the same values of copper that I used in the previous example and simply flipped their signs, and now the picture looks totally weird. What we see here is that at a certain point in time the process explodes to a positive infinity and then re-emerges from a negative infinity a bit later. And after that it continues a smooth evolution without any further jumps, but stay in actual strictly negative this whole time. So in a sense, this is a non-physical solution if we read that xt is an asset price that can only stay non-negative. And also, please note that the position in time of such jump to infinity that we can call the singularity location depends on the value of kappa. The larger the absolute value of kappa, the sooner the explosion happens. for example, for the red line here correspond into the largest absolute value of kappas in our set, we get a singularity in about 7 years from now while for the blue line, singularity is in about 20 years. And I call this singularity partly because it reminds me famous book called, The Singularity is Near, by Ray Kurzwell whom we mentioned before in this specialization. So depending on your tastes, these behaviors suggests sonic either very pathological or very interesting that goes on with the model when kappa is smaller than 0,. In either financial biology, usage of model is sometimes constraint in per meter space. When a model exhibits some strange or wrong behavior for certain values of parameters. For example, we could simply say that the vagus model is only well-defined for positive values of kappa and forget the whole problem. Or perhaps, alternatively, we could say that the model can be used for negative values of kappa, but only for very short times until it explodes to a positive infinity. But in physics, a strange or pathological model behavior for certain parameter values is often a key to something deep. So let's try to look at things at the level of formulas. If we look back at the solution we wrote, we can quickly see the reason it exploded at finite times when we consider it negative values of kappa. In this case the denominator in the solution touches 0 at some point in time, and this is exactly when the explosion occurs. This happens at the time moment t infinity that is computed as shown in equation 16 here. So if kappa is negative in this formula, then the argument of the logarithm is larger than 1. Hence the value of the logarithm and of the whole expression is positive and gives us the location of the singularity point for negative kappas. And thus we will see later this infinity suggests that the more those should be regularized to smooth out such behavior. But perhaps even more interesting point here is that the singularity is also present if kappa is larger then 0 and the difference is that in this case, the singularity is actually, is in the past. There is t infinity is in fact, negative. For positive values of kappa, the process emerges from a negative infinity at time t infinity which is negative then it becomes positive for some later time. And finally, emerges as a non negative or positive initial value x naught times 0. And the time to infinity here is such that once the system emerges from a negative infinity at this moment, and time t equal 0, it becomes exactly equal to x naught. And even more strange, you can check that if we go even deeper in the past, then we'll find out that immediately before this negative time at infinity, the system was actually close to a positive infinity. So what we are seeing was either positive or negative variance of kappa is that we cannot escape the singularity with the vagus model. If kappa is positive, the singularity is in the past, if the kappa is negative, the singularity is in the future. So here we have our first example of a case when we need, in a sense, to go beyond reinforcement drawing, or beyond the simple vagus model and invoke some other considerations that would be quite different from everything we did so far in this course. Normally in models, we just start with some initial conditions and take them as given and do not look back in the past. But here we do and find something entirely weird to our entirely facing 18, again, depending on your taste. Now again, we can either take what we found or leave it. And I actually believe that the first option is much better, so we will continue with it in the next video.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.