Körner and Gilchrist (2007) had subjects search the same 10-letter display twice, consecutively for different targets. Their participants found the target faster in the second search when they had recently inspected it while it was a distractor in the first search. In Experiment 1 we tested whether search for the second target is faster even if the target had not been inspected previously. We presented a target letter in the second search which had either been inspected during the first search or not. Additionally, when the target was absent in the second search, we measured the time until a specific distractor (so-called secret target) was fixated. Likewise, this secret target could have been either fixated during the first search or not. This allowed us to study memory not only for distractors which became targets but also for distractors which stayed distractors throughout. Results showed that the target was found faster even if it had not been fixated previously. In addition, the secret target was fixated later on when it had been fixated as compared with when it had not been fixated. Some of these effects may be due to the variable amount of time needed to complete the first search. Therefore, in Experiment 2 we used a gaze-contingent technique to control for the duration of the first search. We also varied display size (5, 11, and 17 items, respectively). Replicating the findings of Experiment 1, there was a benefit for both inspected and non-inspected targets independent of display size and a preference for fixating non-inspected distractors earlier during the second search. These results provide evidence for a flexible memory mechanism which generally guides search away from old items and towards new items. If, however, a recently inspected (old) item becomes a target memory can also guide search back to it.