Yes. Lots of physical activity, where you burn more calories than you take in, can lead to a catabolic state where it may be difficult to increase muscle mass. To increase muscle mass requires additional work to promote a more anabolic state, with resistance training and increased caloric/protein intake.