As you pan around, the transition value is from where you first began the gesture.

Look at how you move the button. You just keep adding the larger and larger transition to each updated center. What you need to do is add the latest transition to the center as it was when the pan gesture was first recognized.

So either reset the translation (like in your posted code), or save the original center of the button when the gesture's state is .Began and apply the translation to the original center value.