Many ERP studies have reported face responses at various latencies. So far, it has proven difficult to tease apart the contribution of low-level factors from face processing per se. VanRullen and Thorpe (J. Cog. Neurosci., 2001) used a method to isolate task-related differences by comparing ERPs recorded on the same images seen as targets and non-targets in different tasks. Here, we used a similar strategy to isolate task-related differences linked to face processing. We present results from two experiments in which subjects had to categorize briefly presented photographs of natural scenes. In the first experiment, subjects decided whether images contained animals or human faces presented at different scales. In the second experiment, subjects responded to close-up views of animal faces or human faces. Except for animals in the first experiment, all task-dependent differences were surprisingly weak and of relatively long latencies, an effect that might be related to the “by default” processing of faces up to a high-level. This contrasts strongly with the remarkably accurate behavioral responses of the subjects and their very short behavioral reaction times (Rousselet et al., J. Vis., 2003), implying that strong and early task-dependent ERP differences are not required for performing such high level visual tasks. In addition, we show that meaningful ERP differences are not necessarily correlated with reaction times. Instead, we argue that some strong differential effects occurring from 135 ms between physically different sets of stimuli almost certainly reflect processing that is nevertheless intimately related to the identification and recognition processes.