Maybe this is an unpopular view, but I think there is something unnatural about mathematics for most people. It tends to be much more abstract than other subjects. I say this as someone who has always enjoyed math and majored in it.

And it does make sense to have "math for mathematicians" or "physics for physicists" classes. You see this a lot in college (often intro classes are divided into different "levels"). This allows some students to accelerate and others to still learn relevant materials but at a more appropriate pace.