No it's not true. I know a few mathematics students at my university. I kept contact with some of them since we were in the same class during the freshman year. They do not have a physics knowledge required in order to be considered as "good at physics". Of course if they suddenly study physics, it may be easier for them than say for people not involved in sciences.
I also had a very "mathy" professor who has very few knowledge in physics but had a very mathematical abstract mind. So the answer is clearly no. Thus being good or even excellent in mathematics doesn't make you good at physics, much less a physicist. So it's not an implication, although you might meet good mathematicians that are good at physics.

This doesn't apply to everyone, of course. But, this is interesting. When I took my Integral Calculus class before, we were doing a kinematics problem and one student asked what was the difference between a Kilogram and a Newton. My professor had no idea, and one of my classmates who was also in my physics class tried to explain it.

I think that the best example is girls in general. There are a lot of girls interested in maths but a large majority of them seem to be totally horrid at physics. To me it seems like girls usually takes a much more mathematical approach to physics, I guess that they have worse physical intuition or something like that.

But it could also be that girls with physical intuition don't go to physics, but that would be a bit strange in my opinion since physics is a lot more fun then.

I think that the best example is girls in general. There are a lot of girls interested in maths but a large majority of them seem to be totally horrid at physics. To me it seems like girls usually takes a much more mathematical approach to physics, I guess that they have worse physical intuition or something like that.

But it could also be that girls with physical intuition don't go to physics, but that would be a bit strange in my opinion since physics is a lot more fun then.

You have to keep in mind that female and male brains work differently. In general women have much better rote memory and linguistic skills than men, but their brain isn't focused around spatial-mathematical thinking.

Anyway, as far as I can tell, a lot of mathematics is just knowing your algorithms, but in a sense this could apply to physics as well(just not as concretely).

Most Physics courses are a few (usually one or two) levels behind your regular Math courses, so to some extent yes. If you are like three levels beyond, let's say you are taking Calculus IV, differentials, or Calc III and you are doing first-year physics, you might have more advantage over everyone who is taking Calc II at the same as Physics

In my experience (3rd year Undergraduate physics major, up to Calculus II and Differential equations), math can be harder to do, but conceptually a problem is not hard to understand (ie, you get a, say, homogeneous differential equation you need to solve, the concept of a DE is the same as a simple seperable-in-variables DE, the process to get the solution is merely more involved), whereas physics is generally the opposite -- the mechanics of it, the integrals and algebra and whatnot, are generally more numerical and simpler, but the abstract thinking that leads you to what process to use can be a big stumbling block.

Obviously this is just at a basic level and there are exceptions to every rule, i haven't studied either in depth enough to know concretely

Also, I am female and I have struggled with conceptual material, especially visualization of 3D environments. I also struggle with pure logic (for instance, computer science algorithms). Ironically, my summer job is in computer programming, so I will have to pony up and work through my problems. Of course, my problems in these areas may be due to my personal weaknesses-- but blaming it on gender is nice sometimes.

For me, I feel like Physics (introductory Physics at least), involves more actual rote calculations than the math I'm currently doing (algebra, analysis), which requires more thinking and little to no actual calculations (elementary algebra).

In a way, math and physics are quite alike. Both math and physics majors usually acquire a greater ability to solve problems, to analyze, and to logically prove things. These skills make both types of majors equally valuable in most advanced fields, where being able to think for yourself is of great importance.

The discrepancies (math majors not being good at physics and vice versa) could be attributed to the fact that some majors simply do not like physics or math. However, there is no general rule, and we cannot generalize. Some math majors simply cannot grasp complex ideas applied to real life (Think quantum mechanics or string theory), whereas the converse also holds for physics majors.

To sum up, I think there is a correlation between math/physics majors and their proficiencies at physics/math, respectively.

For me, I feel like Physics (introductory Physics at least), involves more actual rote calculations than the math I'm currently doing (algebra, analysis), which requires more thinking and little to no actual calculations (elementary algebra).

I consider the act of moving letters that you do in elementary algebra to be calculation. You usually stop using numbers in both physics and maths courses very quickly except as coefficients for variables.