Testosterone concentrations in men are associated with cardiovascular morbidity, osteoporosis, and mortality and are affected by age, smoking, and obesity. Because of serum testosterone's high heritability, we performed a meta-analysis of genome-wide association data in 8,938 men from seven cohorts and followed up the genome-wide significant findings in one in silico (n = 871) and two de novo replication cohorts (n = 4,620) to identify genetic loci significantly associated with serum testosterone concentration in men. All these loci were also associated with low serum testosterone concentration defined as <300 ng/dl. Two single-nucleotide polymorphisms at the sex hormone-binding globulin (SHBG) locus (17p13-p12) were identified as independently associated with serum testosterone concentration (rs12150660, p = 1.2×10−41 and rs6258, p = 2.3×10−22). Subjects with ≥3 risk alleles of these variants had 6.5-fold higher risk of having low serum testosterone than subjects with no risk allele. The rs5934505 polymorphism near FAM9B on the X chromosome was also associated with testosterone concentrations (p = 5.6×10−16). The rs6258 polymorphism in exon 4 of SHBG affected SHBG's affinity for binding testosterone and the measured free testosterone fraction (p<0.01). Genetic variants in the SHBG locus and on the X chromosome are associated with a substantial variation in testosterone concentrations and increased risk of low testosterone. rs6258 is the first reported SHBG polymorphism, which affects testosterone binding to SHBG and the free testosterone fraction and could therefore influence the calculation of free testosterone using law-of-mass-action equation.