Motion tracking out of focus footage

I'm having difficulties accurately motion tracking a clip that starts in focus and goes out of focus.
Few tutorials on youtube exist on the matter, and the one I found that I thought would help here:

doesn't give the same result when I try.

The beginning of the clip looks like this with the green solid in the correct position

And in the end it the footage looks like this (I took out the motion tracking data and just let the green solid remain static after a few attempts)

Obviously in the end I'm make the solid appear to be out of focus too, but my problem is tracking the solid without having to do it frame by frame, because it never gives the smooth result auto-tracking does (as we all know).

I used mocha to try and track some points near the tool box, thinking that would be a good spot to track, and it did alright until the clip goes completely out of focus.

Does anyone have a technique or suggestions on how to solve this problem? There has to be a way.

As much as I'd hate to say it, Mike's right. Reshoots are your best bet. However, you can maybe try to use a contrast adjustment to get a high contrast scenario, and do your track on that. You also might be able to try the same type of a thing with a cartoon effect black edge or a threshold. Basically anything that can give you a high contrast point to look at will help. Just pre-comp them before you attempt it. Just note that this will still probably create a subpar track. In the future, if you are wanting to focus on a specific point and rack focus, you can use the lens blur effect and create a focus map. You can then rack back in forth. Otherwise, if you are just blurring the entire image, any of the other blur effects will do what you need with a few key frames.

This footage wasn't shot by me, nor was the the camera operator instructed to change focus.
I'm a post-production guy first and foremost, so the cinematography/direction wasn't my responsibility.
I understand your suggestions to re-shoot, but that sadly isn't an option. Thank you for the contrast suggestion, I'll try that, or worst case scenario track it frame by frame.

Reshooting in these situations is always the best solution, but seldom is it a viable option for a variety of reasons. Clients now often expect a lot for little, and unfortunately they are often told they can get it. Pre-production goes out the window and it falls on the post-production to fix everything.

mocha can actually handle out of focus footage if the shape is large enough and you crank up the % of pixels used. However, like the other responses say - at some point you may need to resort to manual tracking.