Cognitive attention and perceptual saliency jointly govern our interaction with the environment. Yet, we still lack a universally accepted account of the interplay between attention and luminance contrast - a fundamental dimension of saliency. We measured the attentional modulation of V4 neurons' Contrast Response Functions (CRFs) in awake, behaving macaque monkeys and applied a new approach which emphasizes the temporal dynamics of cell responses. We found that attention modulates CRFs via different gain mechanisms during subsequent epochs of visually driven activity: an early contrast-gain - strongly dependent on pre-stimulus activity changes (baseline shift), a time-limited stimulus-dependent multiplicative modulation, reaching its maximal expression around 150 ms after stimulus onset, and a late resurgence of contrast-gain modulation. Attention produced comparable time-dependent attentional gain changes on cells heterogeneously coding contrast, supporting the notion that the same circuits mediate attention mechanisms in V4 regardless of the form of contrast selectivity expressed by the given neuron. Surprisingly, attention was also sometimes capable of inducing radical transformations in the shape of CRFs. These findings offer important insights into the mechanisms that underlie contrast coding and attention in primate visual cortex and a new perspective on their interplay, one in which time becomes a fundamental factor.