You may not need to be excellent in maths, but if you want to program, then it is necessary to understand how to handle objects onscreen. Like yaustar said, for physics programming, object interaction is dictated by physics and math formulas that are applied to the collisions and movements. The best thing you could do is to make small games or game demos in your spare time to familiarize yourself with that sort of programming, and from there, you can determine yourself whether or not you have enough math knowledge to be an effective game programmer.