How to track motion for position AND scaling

I have an image of a tv with a green screen inside of it. The tv starts off small and increases in size. So I would like to track the scale and position of it as it increases in size. I'm starting with it at the end at its largest size because it's easier that way. However, when I place the two track points on either side of the tv as soon as I hit the analyze frame back button...the two trackers jump to the middle of the tv image. In other words, they don't stay where I'd placed them. And I don't get any useful tracking information. I'm sure this is a simple fix. And hope I don't have to go Mocha.
Thanks!

Mocha might not be a bad idea, because it allows you to export your data to the corner pin effect, which would be ideal if you are placing an image into a television. You might want to check out,
Mocha Corner Pin Tutorial

"I have not failed 700 times. I have succeeded in proving that those 700 ways will not work. When I have eliminated the ways that will not work, I will find the way that will work."
---THOMAS EDISON on inventing the light bulb.