I just verified this also happens with setting linear velocities as well. The loop isn't infinite so much so it takes a very long time to complete and doesn't produce desirable results, it's almost guaranteed to happen if I try this on all the actors at
once, like so:

The problem might be somewhere else then. The TOI loop might just be the place it uses the most time. Would it be possible for you to send me the project? I've seen this problem once before while working with Box2D and it was due to an object being added
to the world multiple times. I would have to take a look at the stack trace to figure out where the problem comes from.