A fundamental aim in the quest to understand face recognition is to reveal which features determine the identity of a face. Previously, we discovered a subset of facial features that determine the degree of perceptual similarity between unfamiliar faces. We did that by constructing a multi-dimensional facial-feature space, and correlated face space distances, based on rating of each feature, with perceptual similarity judgements. We found that features for which subjects have high perceptual sensitivity (high-PS), including the hair, the eyes - shape and color, eyebrow thickness, and lip-thickness, were strongly correlated with face similarity judgements. In contrast, features for which subjects have low perceptual sensitivity (low-PS), including face-proportion, eye-distance or skin-color, did not affect face similarity judgements. To determine whether these features are critical for face recognition, in the current study we replaced these features in familiar faces and asked subjects if they recognize the modified face. In particular, we examined recognition as a function of the type of change, high or low-PS features, and the number of features that were changed. Subjects were either presented with low or high-PS changes and first saw faces that underwent maximal number of changes followed by faces that underwent smaller number of changes. Our findings show that changing five low-PS features had little influence on face recognition, and was similar to the effect of changing only one or two high-PS features. Changing four or five high-PS features made the famous face unrecognizable. In contrast, a face recognition software was similarly sensitive to both types of changes. We conclude that human face recognition depends on a subset of features for which we are highly sensitive. These features may be invariant under different views, illuminations or expressions. These findings challenge prevalent claims, which primarily emphasize the role of configural information in face recognition.