The size of your penis is very dependent on blood flow. When erect, it's engorged with blood.

Your body redirects blood flow to where it is most needed. If you're doing a lot of physical exercise, blood flow to your muscles increases massively. As you've only got so much blood to go around, other bits which are less active get less.