Presenting a distractor that is similar to a pre-defined target in a rapid serial visual presentation (RSVP) task deteriorates the detection performance of the target. This is most likely because observers prepare for top-down attentional sets (or templates) to reject distractors and accept the target. In this study, we investigated whether the rejecting process of distractors is influenced by pre-knowledge about what to reject: category or individual item. Observers identified an alphabet letter embedded in the RSVP stream of nonsense line patterns. Before the target onset, a digit appeared as the distractor to be rejected at various distractor-target timings. In category-cueing condition, we presented “#” as the cue indicating that any digit distractor would appear. In item-cueing condition, we presented a specific digit (e.g., “5”) as the cue indicating the specific digit would appear. In both the cueing conditions, we found substantial deficits in target letter identification. Evidently, the distractor was processed automatically, which in tern hindered the performance. Interestingly, the time-course of performance deterioration implied that the rejecting process can be commenced only after the expected distractor is registered; the performance reduction started later in the item-cueing condition than in the category-cueing condition. The present results suggest that the visual system can arrange an attentional set for a stimulus category and reject a distractor without item individuation. Nevertheless, it appears that the visual system cannot help using a template for individual items when an informative cue indicates which specific distractor item would come.