In a standard demonstration of holistic face processing, differences in an irrelevant feature (e.g., mouth) change the perceptual representation of a relevant feature (e.g., eyes). Unfortunately, this simple demonstration cannot reveal the nature of the perceptual representation derived from holistic processing. We made use of a set of face stimuli constructed so that incremental differences in a single feature led to systematic changes in performance when pairs of faces were presented for same-different judgments. We then assessed the nature of the perceptual representation in holistic face processing by incrementally varying the degree of difference in relevant and irrelevant features. Participants saw two faces and judged if the relevant features (i.e., eyes) were identical while ignoring differences in the irrelevant features (i.e., mouth). Performance improved with each increment in degree of difference in the relevant feature if the difference between the irrelevant features was small. However, this was contextualized by an interaction in which incremental differences did not influence performance systematically. Rather, performance depended on a striking interaction between the perceptual difficulty of the relevant feature and the degree of difference in the irrelevant feature. When the relevant features were easy to differentiate, discrimination was good regardless of the irrelevant features. However, when the relevant features were difficult to differentiate, discrimination was especially poor if the irrelevant features were also hard to differentiate, but improved significantly if the irrelevant features were easy to differentiate. The implication of these findings will be discussed in reference to categorical perception.