Background: Although erectile dysfunction (ED) is common, little is known about the impact of ED on the quality of life (QoL) among African men. Materials and Methods: We used the International Index of Erectile Function (IIEF) to evaluate ED and the WONCA charts to assess QoL among employees of a university. Results: A total of 508 men with a mean age of 43 ± 10 years were studied. IIEF5 scores of <22 were present in 406 participants (77.9%). Mild ED, mild-to-moderate ED, moderate ED, and severe ED were present in 34.6%, 26.6%, 10.4%, and 5.7%, respectively. Systolic and diastolic blood pressure were significantly lower in those with ED. Marital status, alcohol, cigarette, physical inactivity, obesity, hypertension, diabetes mellitus, and antihypertensive drug use were not associated with ED. ED was associated with poor QoL in the domains of social activities (odds ratio [OR] = 4.35; 95% confidence interval [CI]: 1.01–18.7), and overall health (OR = 2.27; 95% CI: 1.07–4.82). However, there was no association of ED with poor QoL in the domains of physical fitness (OR = 1.46; 95% CI: 0.82–2.59), feelings (OR = 1.43; 95% CI: 0.75–2.72), daily activities (OR = 4.72; 95% CI: 0.61–36.4), and change in health (OR = 1.75; 95% CI: 0.58–5.26). Conclusion: ED negatively impacts QoL in men.