Posts: 22

Topic: Is it possible to see used param values while doing Monte Carlo?

Hi,I want to try to make experiment for choosing more robust indicator values. Idea is to test some set of different strategy indicator values and choose not the best, but take mean/average (not sure what is difference). Now somewhere in the forum discussion is going about "anti-optimization", and it would be my variation to try check it out if this way would show some better(stable) results with unseen data chunk.

Monte Carlo test now randomize indicator values perfectly, but I can not see what values was used and of course get average of them. Maybe there is some temporary file where I can see them?

Re: Is it possible to see used param values while doing Monte Carlo?

Monte Carlo doesn't remember the tests parameters. It only collects and arranges the results in the Confidence Table.

If we decide to implement Anti Optimizer, it will randomize the parameters and will remember all variants. The traders will be able to select one of the calculated results as a fitness rate. For example 0 % - will be the least profitable and 100 % - the most profitable (or other fitness metric). It will be possible to take the one that is at 80% of the dispersion (with 80% profit).

Re: Is it possible to see used param values while doing Monte Carlo?

I guess we need to make sure if that thing has positive effect. Will make couple experiments "by hand" with this idea few variations on dozens strategies. Will let you know if results will be better than if chosen best variables results.

By the way, I am planning to upgrade my system to AMD Ryzen 7, 8 cores, 16 threads. I hope there is no any limit with FSB about using max cores/threads?

Re: Is it possible to see used param values while doing Monte Carlo?

Re: Is it possible to see used param values while doing Monte Carlo?

Hi,Just finished experiment. Maybe someone will find results useful Planned to do it with at least 20 strategies, but stopped with 5, because it is very time consuming and I can see that strategies results getting worse, so for me it is enough proof not to use it. Used different strategies, pairs and time frames.

Question was if I choose not the best backtest parameters will it trade better than optimized results on unseen data chunk? Does choosing not the best parameter set makes sense? Answer what I found is no. Best sum of unseen data profits is fully optimized strategies. Also it made profits in all 5 strategies, when with other methods some strategies was in loss, for me it is indication for more robust results.

However it can be that choosing not the best parameter set is effective in other conditions. I mean that FSB chose best from ~300-600 optimization runs, but if I chose best from thousand runs from other backtesting platform, probably there is a lot bigger chance to get overfitted strategies, and that "Anti optimizer" would make sense and better end results Also I am aware that using bigger sample size can change things, but I am OK with what I did to stay away from this idea.

Re: Is it possible to see used param values while doing Monte Carlo?

I like this topic -- and have a different perspective...

Consider OOS. Many people feel that a strategy that performs well for both IS and OOS is more "robust" or less curve-fitted and will, therefore, perform better in a live account on future data. But isn't there another explanation? If the OOS portion is similar to the IS portion (i.e. the IS portion did a good job training the strategy for the OOS portion), then it is not surprising the strategy does well in both. That is, the OOS portion really didn't exercise the strategy in a way that would reveal whether or not it truly is more robust.

footon wrote:

But with a little bit of sadness we have to admit it's another way found, which doesn't work. What will we do next?

In a recent post you mentioned something along the lines we have all the data, we just need to figure-out what the right question is to ask -- which is great insight. So, in this particular experiment, what exactly was the question? The experiment yielded interesting results that can help going forward. I'm curious to hear what people think the right questions are.

For example -- here are a few questions:

Does it make a difference which data set is used when generating, optimizing and back testing a strategy?

Should you always use the latest (and greatest data set) when back testing, or is there a particular chunk of historical data that routinely does a good job of training strategies even though it might be "old"?

What is the definition of robust? And if it is a good thing, then does a metric currently exist that measures "robustness"? Or could a new metric be computed?

Re: Is it possible to see used param values while doing Monte Carlo?

rantampla -- thanks for the good word. It is appreciated.

rantampla wrote:

After few months test live EAStudio and FSB, the FSB Strategies are little better then EAStudio...

I also spent a lot of time with EA Studio and recently came to the same conclusion. It did create some nice strategies that I continue to use, but I've now returned to FSB. The main thing I miss is the ability to create an EA that contains a portfolio of strategies. That is a very powerful feature.

I'm currently working on a tool that will allow one to merge multiple FSB strategies into a single EA -- like EA Studio. Popov's code is very clean and organized, so I think it is possible. I hope he doesn't mind...

Re: Is it possible to see used param values while doing Monte Carlo?

sleytus wrote:

rantampla -- thanks for the good word. It is appreciated.

rantampla wrote:

After few months test live EAStudio and FSB, the FSB Strategies are little better then EAStudio...

I also spent a lot of time with EA Studio and recently came to the same conclusion. It did create some nice strategies that I continue to use, but I've now returned to FSB. The main thing I miss is the ability to create an EA that contains a portfolio of strategies. That is a very powerful feature.

I'm currently working on a tool that will allow one to merge multiple FSB strategies into a single EA -- like EA Studio. Popov's code is very clean and organized, so I think it is possible. I hope he doesn't mind...

No Problem i feel it, and then i must say thanks, it is much big helping for me

I have read, that Mr. Popov from 01.10.17 EA-Studio make Pause and he will work on FSB, and i hope he can make Portfolio like EA-Studio.

Re: Is it possible to see used param values while doing Monte Carlo?

Hi Sleytus,Your questions is very good ones. And it points me to look for the answer to Regime analysis (maybe even combination with Markov chains). But I know nothing about it, just read some on the net and in Market Wizards book 30 year old (I can look for exact name and paragraph), that one system trader is using Markov chains to decide when to turn off and turn on different systems.Logically if you train EAs in more trendy data, you expect them to work great in trends live trading. And opposite with more ranging conditions. Maybe there is some indicators pointing out when it is time or there is edge to make these on/offs...There is some link: https://www.quantstart.com/articles/hid … on-using-r . I will take it deeper read to this too ....

Robust means working consistently on different pairs and timeframes. It is subjective in my opinion.

Re: Is it possible to see used param values while doing Monte Carlo?

Irmantas wrote:

Maybe there is some indicators pointing out when it is time or there is edge to make these on/offs..

Yes -- an indicator or strategy that is not used for trading, but only to provide a clue as to the trend or pattern of the incoming data. And this could then be used to turn on / off different EA portfolios -- or something like that.

Re: Is it possible to see used param values while doing Monte Carlo?

Re: Is it possible to see used param values while doing Monte Carlo?

Thanks for replays guys! I see a need for experiment which will use 3 OOS to test out if double OOS would make money on third OOS Good thing that trading is not boring, because you will never run out of ideas to test. Until you become profitable then you will need to stick to what works...

Re: Is it possible to see used param values while doing Monte Carlo?

Beru wrote:

it is not a guarantee of good strategies but they work better than others

Thanks for attaching a picture -- it helps a lot.

When I look at the picture then I would not necessarily interpret this as a good strategy. For example, in the "First OOS" section there are only 3 winning trades over the course of 2-3 months (20% OOS 1.1.2016.- 1.5.2017). That means winning trades are a very *rare event*. And that is confirmed in the "Second OOS" -- a winning trade is a very rare event. If you were to place this in a live account with new incoming data then what is the probability you'll have a winning trade within the next month, or 6 months or next year? It's hard to say.

I don't know the optimal number of trades when performing back testing, but I've read elsewhere that it should be at least 100 - 200 trades per year.

Re: Is it possible to see used param values while doing Monte Carlo?

Re: Is it possible to see used param values while doing Monte Carlo?

When your strategy gives good results for the OOS section it is not because it has magic properties and can see into the future. The reason why it does well in the OOS section is because the OOS section is simply a continuation of the IS section.

The best way to take advantage of OOS is to begin with a strategy that *fails* OOS. In this situation you at least know the IS and OOS sections are different. You then treat the entire data set as IS and attempt to optimize your strategy until you get a balance chart that is nearly linear throughout the IS and OOS sections. If you can do that, then you know that strategy is more robust -- since it has been trained against two chunks of different data.

OOS does no harm -- except it does cause people to think they have a strategy that can see into the future.