As described this requires that i have a large number of IloOplModel objects existing at the same time, which gives me a speed up, but it really eats memory. The flow for creating the objects is as:
var subOplArrays = new Array(numVessels);
var subCplexArrays = new Array(numVessels);

Originally i did as you suggested, calling generate for each subproblem in each iteration.

But with this approach the "generate()" call ends up using 90 % or more of the computational time, as each of my subproblems are usually easily solved. So the whole idea of this different approach is to avoid the generate calls, by simply updating the changed dual values in the subproblems, as all other values are the same from iteration to iteration.

The problem is now that all these subproblems eat all the mem, so it does scale to sufficiently large instances.

Re: Mem issue, each IloOplModel takes 13 MB

Originally i did as you suggested, calling generate for each subproblem in each iteration.

But with this approach the "generate()" call ends up using 90 % or more of the computational time, as each of my subproblems are usually easily solved. So the whole idea of this different approach is to avoid the generate calls, by simply updating the changed dual values in the subproblems, as all other values are the same from iteration to iteration.

The problem is now that all these subproblems eat all the mem, so it does scale to sufficiently large instances.

you can try to reduce the number of thread of CPLEX. It will reduce the memory usage.

the second point is to isolate a subproblem in .mod and .dat. After you can run it and analyze it in the console. You would be able to find where the memory is used in the subproblem and reduce it size by modifying the model. Tuning example are:

use tuples instead of array

use Range instead of Set

use internal data instead of external data

use integer id key instead of string

..... find other hint in CPLEX_OPL training book from Academic Initiative. see previous post

after modification you can replicate the changes into your original model.

I am already running with 1 thread and i've set the reduce memoty switch (in .opl file, i guess thats sufficient?). These helped a bit.

But its not my perception that there is a massive memory usage in cplex itself. When stepping through the code, its the "generate()" calls of the ilog model that eats the majority of the mem, so i think the solution should be found in opl somehow.

Re: Mem issue, each IloOplModel takes 13 MB

I am already running with 1 thread and i've set the reduce memoty switch (in .opl file, i guess thats sufficient?). These helped a bit.

But its not my perception that there is a massive memory usage in cplex itself. When stepping through the code, its the "generate()" calls of the ilog model that eats the majority of the mem, so i think the solution should be found in opl somehow.

I've added a profiler dump of the full problem (master and subproblem), but it lacks details to really unxderstand the issue.

So you're right that I need to: "isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)", but how to ? I've looked through the help and documentation, but hasnt been able to find any ?

Re: Mem issue, each IloOplModel takes 13 MB

I've added a profiler dump of the full problem (master and subproblem), but it lacks details to really unxderstand the issue.

So you're right that I need to: "isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)", but how to ? I've looked through the help and documentation, but hasnt been able to find any ?

Thanks, It worked. I succesfully exported the internal and external data to .Dat files. Following i've been able to run the subproblems alone and review the profiling, pls refer to attached for details.

But im not able to get much value out of this, the Mem usage is reported at 6.7 MB, a bit less than my experience from the whole program execution, but comparable. Going through the high mem users of these 6.7 i see: "CPLEX MIP Optimization", EXTRACT and LOAD_MODEL posts. But none of these seem to indicate that my model is mem-inefficiently constructed or I'm putting too much (unnneeded) data into it ? So am i looking at it the right way ? I cant find the insight that should help me lower the mem usage.

I still think it should be possible to reduce mem usage, if you look at the tab "Engine Log", you'll see that the model is quite small, only 3151 non-zeros, i've cant see why this should take up 6.7 MB ?

Re: Mem issue, each IloOplModel takes 13 MB

Thanks, It worked. I succesfully exported the internal and external data to .Dat files. Following i've been able to run the subproblems alone and review the profiling, pls refer to attached for details.

But im not able to get much value out of this, the Mem usage is reported at 6.7 MB, a bit less than my experience from the whole program execution, but comparable. Going through the high mem users of these 6.7 i see: "CPLEX MIP Optimization", EXTRACT and LOAD_MODEL posts. But none of these seem to indicate that my model is mem-inefficiently constructed or I'm putting too much (unnneeded) data into it ? So am i looking at it the right way ? I cant find the insight that should help me lower the mem usage.

I still think it should be possible to reduce mem usage, if you look at the tab "Engine Log", you'll see that the model is quite small, only 3151 non-zeros, i've cant see why this should take up 6.7 MB ?

I've tried this. Did very little to change mem consumption. Or actually nothing ;) I can scale to the same problem sizes with and without the profiling, although the mem consumption is a little bit less

Having one subproblem at a time, creating the models when needed, solving and destroying afterwards. With a lean model and using the cmd version as "oplrun -deploy" the generate() calls are fast and each iteration runs smoothly without taking up much mem.
Thanks for the help.