Conceptual Momentum Question

1. The problem statement, all variables and given/known data
When a ball hits a wall and comes back, it originally had a momentum P. However, when the ball comes back, it has a momentum of -P (as in going in the opposite direction.) According to the law of conservation of momentum, since the ball had an impulse of -2P, the wall must have gained 2P. How do you explain what happened to the 2P since the wall doesn't move?

2. Relevant equations

3. The attempt at a solution
Just saying, this isn't a homework question or anything so put this question on low priority XD.

I was thinking that the wall does move but because of how large the mass of the wall is compared to the mass of the ball, we just don't perceive a movement. For example, in a building, when a ball hits the wall, the wall gains a momentum of 2P, but since the wall is attached to the building, the building as a whole gains a momentum of 2P, but since the building is attached to the ground, the entire earth gains a momentum of 2P, etc. and since Earth has an infinitely large mass compared to the ball, there is no change relative to the actual velocity.