B.L.E., it is the time of flight between two points in the bullets trajectory that determines how far from the line of sight the bullet drifts sideways.

If that was strictly true, then the .300 Whisper (essentially a .223 necked up to .30 caliber) shooting that 220 grain Sierra hollow point boat tail match bullet with a BC of .608 with a muzzle velocity of 1040 fps should have drifted way more than the .17HMR shooting that 17 grain bullet with a BC of .125 at 2550 fps.

Lag time, the amount of extra time added to time of flight due to slowing down increases exponentially even if the velocity decay is linear.
To understand this, visualize driving 60 mph and someone going 80 mph passes you. Even though you don't decelerate, he keeps getting farther and farther ahead of you and the lag time between the two cars increases.

The only way for a bullet to stop the increase in lag time after being slowed down would be to somehow re-accelerate back to the original muzzle velocity and stay at that velocity, then the lag time would be constant from that point onward.