Abstract: Paying attention to the right thing at the right time underlies the ability of humans and other animals to learn, perceive, and interact with their environment. What is the role of memory in guiding attention? According to ''the world as an outside memory'' theory, humans exploit the stability of the world to access external information on demand, leading to conscious perceptions that are seemingly rich and continuous without requiring detailed and persistent internal representations. An alternative theory postulates that attention deployment relies on detailed memory traces (lasting about 1s in real world conditions) of recent attention targets. Here we resolve this apparent discrepancy by showing that the impact of memory on attention deployment depends on the availability of semantically persistent context. We asked human observers to visually explore MTV-style video clips, in which unpredictable scene changes occur every 1-3 seconds, and quantified the ongoing ability of a memory-free model of attention deployment to predict rapid gaze shifts (saccades). Scene changes triggered memory-free influences on attention deployment that overwhelmed previous influences within less than 250 ms. These initial sharp increases in the impact of memory-free influences were followed by gradual decreases, reflecting slower increases in competing memory-dependent influences, and final increases to an average level, demonstrating that the overall impact of scene changes on attention deployment subsides within 2.5 seconds. Our results indicate that when semantically persistent context is available, recently triggered memories play an important role in guiding attention to specific locations in the visual field, well beyond the time span of visual persistence.