The present study was designed to investigate the effect of a genetic risk factor for Alzheimer's disease (AD), ApolipoproteinE ε4 (APOEε4), on the ability of the brain to modulate activation in response to cognitive challenge in a lifespan sample of healthy human adults. A community-based sample of 181 cognitively intact, healthy adults were recruited from the Dallas-Fort Worth metroplex. Thirty-oneAPOEε4+ individuals (48% women), derived from the parent sample, were matched based on sex, age, and years of education to 31 individuals who were APOEε4-negative (APOEε4-). Ages ranged from 20 to 86 years of age. Blood oxygen level-dependent functional magnetic resonance imaging was collected during the performance of a visuospatial distance judgment task with three parametric levels of difficulty. Multiple regression was used in a whole-brain analysis with age, APOE group, and their interaction predicting functional brain modulation in response to difficulty. Results revealed an interaction between age and APOE in a large cluster localized primarily to the bilateral precuneus. APOEε4- individuals exhibited age-invariant modulation in response to task difficulty, whereas APOEε4- individuals showed age-related reduction of modulation in response to increasing task difficulty compared with ε4- individuals. Decreased modulation in response to cognitive challenge was associated with reduced task accuracy as well as poorer name-face associative memory performance. Findings suggest that APOEε4 is associated with a reduction in the ability of the brain to dynamically modulate in response to cognitive challenge. Coupled with a significant genetic risk factor for AD, changes in modulation may provide additional information toward identifying individuals potentially at risk for cognitive decline associated with preclinical AD.