An emerging theory in visual attention is that it operates in parallel at two distinct timescales – a shorter one associated with moment-to-moment orienting of selective visual spatial attention, and a longer one (>10s) associated with more global aspects of attention-to-task. Our question is whether this slower fluctuation in task-related attention biases the same mechanism of early attentional selection as selective attention. Given that past studies have consistently revealed visual field asymmetries in selective visual attention, the objective of the present study was to determine whether sensory processing in the two visual fields is differentially modulated by whether or not one is paying attention to the current task. Participants performed a simple target detection task at fixation while event-related potentials (ERPs) to task-irrelevant visual probes presented in the left and right visual fields were recorded. At random intervals, participants were asked to report whether they were “on-task” or “mind wandering”. Our results demonstrated that sensory attenuation during periods of “mind wandering” relative to “on-task”, as measured by the visual P1 ERP component, was only observed in the left visual field. Alternatively, the magnitude of sensory responses in the right visual field was insensitive to the two different attentional states. Taken together, our results point to a visual field difference in task-related attention, one that mirrors asymmetry found in selective visual attention.