It's basic vector math. Usually, you think about these positions as points in space in this context, but you can also see them as a vector from the origin to that point. Just draw this scene in 2D in paper and you'll see, why the difference vector is the vector from the camera's position to the center of the object. If you normalize it, you have your needed direction vector.