Family and twin studies suggest that up to 50% of individual differences in human fertility within a population might be heritable. However, it remains unclear whether the genes associated with fertility outcomes such as number of children ever born (NEB) or age at first birth (AFB) are the same across geographical and historical environments. By not taking this into account, previous genetic studies implicitly assumed that the genetic effects are constant across time and space. We conduct a mega-analysis applying whole genome methods on 31,396 unrelated men and women from six Western countries. Across all individuals and environments, common single-nucleotide polymorphisms (SNPs) explained only ~4% of the variance in NEB and AFB. We then extend these models to test whether genetic effects are shared across different environments or unique to them. For individuals belonging to the same population and demographic cohort (born before or after the 20th century fertility decline), SNP-based heritability was almost five times higher at 22% for NEB and 19% for AFB. We also found no evidence suggesting that genetic effects on fertility are shared across time and space. Our findings imply that the environment strongly modifies genetic effects on the tempo and quantum of fertility, that currently ongoing natural selection is heterogeneous across environments, and that gene-environment interactions may partly account for missing heritability in fertility. Future research needs to combine efforts from genetic research and from the social sciences to better understand human fertility.