Then I remove and reload my model (In order to move it to it's starting position). After this I initiate all get variable calls with simx_opmode_streaming again.

For set variable calls I'm using the simx_opmode_oneshot or simx_opmode_oneshot_wait operators.

I thought doing like this would prevent the simulation from slowing down but it still does. How do I know it's getting slower? I'm timing my simxSynchronousTrigger call which are getting longer and longer during simulation.

Is there something else to have in mind in order to not get any memory leaks or whatever is causing this issue?

I can show more code if necessary but I tried to extract the important parts and thought it would be more helpful than just printing all my code.

I tried to just restart the simulation instead of removing and reloading my robot model, this made the problem disappear. Restarting the simulation is slower than reloading the model and I would like to use the fastest option. So I still wonder why this problem occur, do I reload the model in the wrong way maybe?

I am not sure due to what reason the simulation gets slower. Is it the simulation itself, or the remote API?
My first guess was that it could be the remote API, where streaming data is not properly discontinued. But it seems you are discontinuing streaming commands, so that is not the reason. Can you notice a memory usage increase that is more than what you expect?
Maybe when remove your model, not everything is removed (e.g. a collision checking object) and you end-up with many duplicates? (normally this is automatically handled when removing the model, but specific collision objects are persistent)