So I figured that following the intuition of tensor2tensor and other transfer learning schemes of training TrackNet on other sports or other projectile videos and images where the x,y coordinates of the object being tracked on the 2D video frame are provided.

I am probably going to use VGG annotator to create more tennis specific data and post in on Github but in the meantime if anyone knows of any public datasets out there, please let me know.