What "AF" setting are you using..? AF-C would cause that... adjusting focus as you move back... which would be necessary if you are indeed moving backwards after pre-focusing at a different focal length..!
Maybe compose the shot first and select the appropriate AF point instead..?