Contrasting traditional face recognition models previous research has revealed that the
recognition of facial expressions and familiarity may not be independent. This dissertation attempts to localize
this interaction within the information processing system by means of performance data and event-related potentials.
Part I elucidated upon the question of whether there is an interaction between facial familiarity and the discrimination
of facial expression. Participants had to discriminate two expressions which were displayed on
familiar and unfamiliar faces. The discrimination was faster and less error prone for personally familiar faces
displaying happiness. Results revealed a shorter peak latency for the P300 component (trend), reflecting stimulus
categorization time, and for the onset of the lateralized readiness potential (S-LRP), reflecting the duration of pre-motor processes.
A facilitation of perceptual stimulus categotization for personally familiar faces displaying happiness is suggested. The discrimination of expressions was not facilitated in further experiments using famous or experimentally familiarized, and unfamiliar faces. Part II raises the question of whether there is an interaction between facial expression and the discrimination of facial familiarity. In this task a facilitation was only observable for personally familiar faces displaying a neutral or happy expression, but not for experimentally familiarized, or unfamiliar faces. Event-related potentials reveal a shorter S-LRP interval for personally familiar faces, hence, suggesting a facilitated response selection stage. In summary, the results suggest that an interaction of facial familiarity and facial expression might be possible under some circumstances. Finally, the results are discussed in the context of possible interpretations, previous results, and face recognition models.

Previous research has revealed that the recognition of facial expressions and familarity may not be independent as postulated by traditional face recognition models. This dissertation attempts to localize this interaction within the information processing system by means of performance data and event-related potentials (ERPs). A simple paradigm was used in all experiments asking participants to perform a two-choice reaction time (RT) task either to discriminate facial expression or to discriminate facial familiarity. The respective task irrelevant dimension was varied independently of the task-relevant dimension (e.g. half of the presented faces in the expression discrimination task belonged to familiar faces).

Part I elucidates upon the question of whether there is an interaction between facial familiarity and the discrimination of facial expression. In Experiment 1, portraits of personally familiar and unfamiliar faces were categorized according to the emotional expressions happiness and disgust. Categorization was faster for portraits of personally familiar persons when compared to unfamiliar persons. This was especially pronounced for portraits displaying happiness. This advantage for familiar faces was not due to differential expressiveness of the portraits because it disappeared in participants for whom all portraits where unfamiliar. In Experiment 2 the same stimulus set was used as in the previous experiment. In addition, ERPs were recorded for 16 participants during the same expression categorization task. Although the weaker performance data of Experiment 1 were replicated, the peak latency of the N170 component of the ERP, reflecting structural encoding of the face, was not affected by familiarity. Also, the latency for the interval between the onset of the lateralized readiness potential (LRP) and the response (LRP-R), reflecting the duration of motor processes, was unaffected by familiarity. In contrast, the latency of the P300 component of the ERP, reflecting stimulus categorization time, and the interval between stimulus and LRP-onset (S-LRP), reflecting the duration of pre-motor processes, were shorter for happy familiar faces when compared to happy unfamiliar faces. Together the results suggest a facilitation of perceptual stimulus categotization for personally familiar faces displaying happiness. In order to elucidate upon the reduced effect in the RT of Experiment 2, another experiment was conducted with a slightly changed design. In addition, the skin conductance response was recorded to personally familiar and unfamiliar faces. This time, the facilitative effect of familiarity in performance data increased whereas it was not reflected by ERPs. Therefore, Experiment 4 and 5 used experimentally familiarized and unfamiliar faces in order to have a better control over the stimulus set. By discriminating happy from angry faces (Experiment 4) or neutral from angry faces (Experiment 5) no facilitation was observed for experimentally familiarized faces. Hence, Experiment 6 used a set of stimuli consisting of famous and unfamiliar faces because semantic knowledge may be necessary for an interaction between facial familiarity and facial expression to emerge. Contrary to the hypothesis, no facilitation was observed for famous faces when discriminating neutral from happy faces. Together, the results of Part I imply a late perceptual but pre-motoric locus of the facilitative effect of familiarity on the discrimination of facial expression. Thus, the degree of familiarity may influence such an interaction since it was not observed for famous and experimentally familiarized faces. Different interpretations are discussed.

In Part II the question was raised whether there is also an interaction in the opposite direction. It was hypothesized that it would be possible to find an interaction between facial expression and the discrimination of familiar faces. Experiment 7 used the same personally familiar and unfamiliar faces as the first three experiments. Participants performed a familiarity discrimination task where they were shown portraits displaying happiness, disgust, or a neutral expression. Personally familiar faces were categorized faster as familiar if they displayed a happy or neutral expression. This advantage for happy and neutral familiar faces appears to be localized in the response selection stage as was suggested by an earlier onset of the S-LRP. In a final experiment participants performed a familiarity discrimination task on experimentally familiarized and unfamiliar faces. Again, it was hypothesized that facial expression has a facilitative effect on the discrimination of facial familiarity. However, no interaction was observed between facial expression and the discrimination of familiarity.

In summary, the results suggest that an interaction of facial familiarity and facial expression might be possible under some circumstances. Contrary to previous results this interinteraction is symmetrical because it was observed in the expression discrimination task as well as in the familiarity discrimination task. However, the degree of familiarity and the type of facial expression may be important for an interaction as it only emerged for personally familiar faces displaying happiness. Finally, the results are discussed in the context of possible interpretations, previous results, and face recognition models.

Introduction
Topic and overview

Faces are very important stimuli in social communication. They convey relevant information like identity, age or gender. Admittedly, there are also many other cues like speech, voice, and intonation or body postures that are relevant for recognizing a person or for interaction with that person. Nevertheless, faces are still the most important cues for communication. The act of encountering a familiar person’s face triggers both a process of recognizing facial familiarity as well as the processing of emotional or semantic information which in return is triggered by the multitude of information a face conveyes. Despite the seemingly complicated processes that are involved, we are all face experts using the skill of face recognition effortlessly and easily beginning at birth. Therefore, it is not surprising that face-like schematic visual patterns are more effective in capturing the attention of newborns when compared to non-face-like configurations (Johnson, Dziurawiec, Ellis, & Morton, 1991; Maurer & Young, 1983). It seems that already in infants the skill of recognizing individual faces has begun to develop, as demonstrated by their preference to their mother’s face (Bushnell, 2001). This may have a high ecological value in order to receive attention and help from the caring person – it is essential for survival.

Neuropsychological impairments can also provide insight into the importance of the ability to recognize faces in normal life. Prosopagnosia, a selective inability to recognize the identity of familiar faces despite intact visual recognition of other objects is a striking example. This impairment, which is mostly based on widespread bilateral lesions of the occipitotemporal cortex, is socially disabeling although patients can develop other strategies to compensate the problem. In addition, the selective loss of specific aspects of face recognition hints to the many processes that are involved in this skill. Firstly, a face has to be recognized as a face before many other aspects can be extracted. One can recognize the identity of a face and recall semantic information about a person. On unfamiliar faces one can still recognize the gender or possible age of a person. Facial expression as another aspect of face recognition is very important in social communication. All these various kinds of information can be used for further cognitive processing.

This dissertation will focus on the recognition of facial expression and identity of familiar and unfamiliar faces. Based on common observation, the recognition of facial expression and of facial identity seems to function easily in everyday life. We can recognize a familiar face out of innumerable faces and extract information about facial expressions in just a moment. Moreover, the recognition of facial expressions appears to be independent of identity for we can recognize a facial expression independent of the familiarity of a person. In return, we do not need the facial expression for recognizing a persons identity. On the other hand, situations can occur in which one has to look twice to recognize a familiar person because we have never seen her with that particular expression. In addition, most people would contemplate whether an unfamiliar person is familiar or not when this person smiles at us in the street. These two examples show that there is also reason to assume an interaction between the processing of facial expressions and identity.

The present dissertation raises as its main question whether there is a facilitative interaction between the perception of facial expression and facial familiarity on a cognitive and functional basis. Although most of the data suggest an independence of facial familiarity and facial expression (e.g. Young, McWeeny, Hay, & Ellis, 1986; Young, Newcombe, de Haan et al.,1993; Bobes, Martin, Olivares, and Valdés-Sosa, 2000) recent data lead to the conclusion that there exists a facilitative interaction of both processes in one or the other direction. Recent studies suggest an interaction between facial familiarity and the discrimination of facial expression (Baudouin, Stansone & Tiberghien, 2000; Boudouin, Gilibert, Sansone & Tiberghien, 2000a; Schweinberger & Soukup, 1998), as well as between facial expressions and the discrimination of familiar faces (Endo, Endo, Kirita & Maruyama, 1992). According to these findings it is hypothesized that facial familiarity may facilitate the discrimination of facial expression. On the other hand, facial expression may also influence the decision that a face is familiar.

Now, I want to give a short overview of the present dissertation. In the introduction two models of face recognition are outlined and just a brief insight is given into empirical controversies concerning the main question of an interaction of facial expressions and facial familiarity. In addition, all basic principles, the paradigm, and the methods used are explained in order to comprehend the two subsequent experimental parts. The introduction is followed by two experimental parts. Part I introduces experiments which raise the question as to whether there is a facilitative interaction between facial familiarity and the discrimination of facial expressions. Part II investigates whether facial expressions can act as a facilitative on the decision of whether a face is familiar or not.

Detailed empirical evidence is cited in the introduction of the Part I leading to the main hypothesis of a facilitative interaction between facial familiarity and the discrimination of facial expression. Six experiments are reported which applied an expression discrimination task and various dependent measures. Experiment 1 to 3 used personally familiar and unfamiliar faces and tried to elucidate the main question by means of performance data, event-related potentials (ERPs), and skin conductance response (SCR). A possible interaction of the processes in question should be reflected in reaction times (RTs) and error rates. The ERP components served as time markers that are assumed to be linked to particular functional processing stages. The behavioural data and ERP components in combination with their overall picture can give a hint to the functional cognitive processes that are facilitated – that is, the functional locus of a possibly observed interaction. The performance data of Experiments 1 to 3 suggest a facilitative interaction beween facial familiarity and the discrimination of facial expression. Although not clear-cut, a facilitative interaction for late perceptual processing stages was observed as was reflected by the ERP data. Therefore, it was intended in the subsequent Experiments 4 and 5 to improve the control over the stimulus material by using a stimulus set with unfamiliar faces. A learning procedure was applied in order to familiarize one half of the faces. Participants had to perform an expression discrimintion task in a consecutive test phase. In contrast to the behavioural data of the previous experiments no interaction was found between perceptual familiarity and the discrimination of facial expression. This was rather unexpected and may be due to the lack of semantic information of the experimentally familiarized faces. Therefore, the following Experiment 6 used famous faces as a stimulus set because semantic knowledge can be presumed. Again, behavioural data and ERPs were used as dependent measures. Contrary to the predicted effect no facilitation due to facial familiarity was observed for the expression discrimination task. At the end of Part I a conclusion is drawn concerning the main hypothesis of an interaction between facial familiarity and the discrimination of facial expressions. The results suggest that a facilitative interaction of facial familiarity and the discrimination of facial expressions is observable under certain circumstances. Based on the results of Experiments 1 to 3 the interaction was pronounced for personally familiar faces displaying happiness. ERP results, although not clear cut, suggested late perceptual processing stages as the functional locus of interaction as indexed by the P300 component. An effect was only observed for personally familiar faces displaying happiness. This suggests that also the degree of familiarity as well as the kind of emotional expression may play an important role for an interaction of facial familiarity and facial expression discrimination.

In Part II the question is raised as to whether there is also an interaction of facial familiarity and facial expression in the opposite direction. Results are cited leading to the hypothesized interaction between facial expression and the discrimination of facial familiarity. Hence, Experiment 7 employed a familiarity discrimination task using personally familiar and unfamiliar faces. Again, behavioural data and ERPs were recorded. In line with the results of the previous part the data suggested an interaction between facial expression and the discrimination of facial familiarity only for personally familiar faces. This time, clear-cut ERP results pointed to response selection as the facilitated functional processing stage. In the last experiment (Experiment 8) participants had to perform a learning procedure and a subsequent familiarity discrimination task on experimentally familiarized and unfamiliar faces. No evidence for an interaction between facial expression and the discrimination of facial familiarity was found for familiarized and unfamiliar faces. Again, the degree of familiarity might be important for an interaction between both processes as it was only observed for personally familiar faces.

Finally, all collected results are discussed thoroughly and disproportioned to each other in the general discussion. A conclusion is drawn in the attempt to answer the raised hypotheses and questions.

In the last decades face perception has become an important field in cognitive science and the body of literature addressing the issue of face recognition has grown to reflect its importance. Yet, as always, there are still many unclear points concerning the involved functional architecture, as well as the neural correlates of the already mentioned cognitive, emotional, and also automatic processes that are triggered by the multitude of facial information. Although the various processes involved in face recognition work easily in everyday life, it is complicated to explain them on a cognitive and functional basis. Many attempts have been made to model the involved processes and different aspects of face recognition (Bruce & Young, 1986; Burton & Bruce, 1993; Haxby, Hoffman, & Gobbini, 2000). However, the framework of a dissertation is narrow and can only mention the models that are relevant to the broader question. Only a short introduction is given about the empirical literature concerning the topic of face recognition.

One of the most influencial models of face recognition was introduced by Bruce & Young (1986; Fig. 1). It is a funtional model and based on empirical results as well as on data derived from clinical observations of patients who suffer from the selective loss of different aspects of face recognition. The model assumes several specialized modules which subserve the functional processes. The hierarchically ordered modules are thought to work in parallel and independently. Seven distinct codes are proposed as output information of the functional modules which can be derived from faces. An expression-independent description (structural code) is extracted from the first view-centered pictorial description (pictorial code) in an initial structural encoding process. This output information is matched with a face recognition unit (FRU) which is thought to exist uniquely for familiar faces. If the face is familiar, a following person identity node (PIN) contains semantic information about the recognized person (identity-specific semantic code). Finally, for familiar faces the name can be recalled (name code). The other processes of expression analysis (expression code), directed visual processing (visually derived semantic code), and facial speech analysis (speech code) are based on the earlier pictorial code as they can be performed on both unfamiliar and familiar faces.

Figure 1. The functional model of face recognition by Bruce and Young (1986)

The model makes also assumptions about the recognition of facial expression which is important for the topic of the present dissertation whereas other functional models do not (Burton, Bruce, & Johnston, 1990; Breen, Caine, & Coldheart, 2000). It is assumed that the recognition of facial familiarity and of facial expressions function in parallel and also independently. Both processes rely on different codes, the pictorial code and the view-independent structural code, respectively. Although the model of Bruce and Young (1986) can explain a wide range of empirical results (Young, 1998) there are also findings which contradict the assumptions of their model (e.g. Rossion, 2002; Baudouin et al., 2000; Schweinberger & Soukup, 1998; Endo et al., 1992). Especially the proposed independence and serial succession of the functional modules is questioned by these results and imply further research. The model has been complemented (Burton et al., 1990) in the last years and modifications have been suggested (Abdel Rahman, Sommer, & Schweinberger, 2002). Nonetheless, the original model by Bruce and Young (1986) and the refined version (Burton et al., 1990) has proven to be convenient to generate hypotheses as it can explain a wide range of results from studies that have been published since then (Le Gal & Bruce, 2002; Eimer, 2000; Bentin & Deouell, 2000; Ellis, 1989). However, it is a functional model which is all-embracing but only explains the gross processes. It is also important to understand the specific information which is processed by the proposed functional modules.

The recognition of faces can rely on different features or cues which supply information. Differentiations can be made between “first order” features such as eyes, nose, hair, mouth or shape information and “second order” features mainly including the configuration or spatial arrangement of the first order features. In addition, pigmentation and texture of the skin can also convey relevant information for facial recognition. Another differentiation can be made between external or „cardinal“ features (Ellis, 1986) like hair, hairline or face shape and internal facial features (eyes, nose or mouth as well as their spacial arrangement). Ellis, Shepherd, and Davies (1979) found facilitated recognition from internal, when compared to external, features only for famous faces. It is suggested that unfamiliar face recognition and face matching relies more on external facial features (Bruce, Henderson, Greenwood, et al., 1999). In contrast, internal features gain importance the more familiar a face becomes (Young, Hay, & Ellis, 1985) as they can vary less than external features (Bruce & Young, 1998). It is obvious that the recognition of facial expression relies mainly on internal features. One could suppose that overlapping information which is used for the recognition of facial familiarity and facial expression may cause an interaction between both processes. However, Calder, Young, Keane et al. (2000) examined this question by applying the composite effect to analyze the configural features that are used to discriminate facial expression or identity. The composite paradigm (see Bruce, 1988) shows that the recognition of configural features for facial identity or expression is disturbed when the top and bottom half of two different individuals or facial expressions are aligned (composite effect). For an expression discrimination task results of Calder et al. (2000) revealed composite effects in RT which were independent of the identities represented by the facial top and bottom half. For an identity discrimination task the composite effect was also independent of the expressions displayed by the two facial halfs. In another study Calder, Burton, Miller et al. (2001) applied a principle component analysis (PCA) of the pixel intensity information to the Ekman and Friesen (1976) expressive faces. The resulting factors from the PCA were analyzed further with a linear discrimination procedure in order to identify the factors that are most important to recognize facial expressions, facial identity or the gender of a face. The computational procedure revealed that the coding of facial expression relies largely on different components when compared to the coding of identity. Both studies of Calder et al. (2000; 2001) suggest that the configural information which is used to recognize either facial identity or facial expression is different and only overlapping partly.

Many studies have implied face recognition proceedes in a series of separable stages or functional processes (Campbell, Brooks, de Haan & Roberts, 1996; Nachson, Moscovitch, & Umilta, 1995). These processes can be selectively impaired as is evident from lesion studies. Just one example is prosopagnosia (see above), the inability to recognize previously familiar faces. In these patients the ability to recognize and match unfamiliar faces is still intact. Hence, different processes can be assumed which subserve the recognition of familiar and unfamiliar faces. Individuals who suffer from prosopagnosia do not recover from their impairment and are only able to compensate for it with nonfacial cues like the voice or clothing of a familiar person. This suggests that face recognition might rely on a specialized neuronal system in the brain (Kanwisher, McDermott, & Chun, 1997). In the last decade there has been an unsolved controversy concerning the topic of whether this system is independent of the general visual object recognition system (Gauthier & Tarr, 1997; Gauthier & Logothetis, 2000).

An important question, within face recognition research, attempts to isolate the underlying neuronal substrate which is involved. The loss of the ability to recognize previously familiar faces after damage of certain brain regions in patients with prosopagnosia may hint to the importance of these brain regions for familiar face recognition. This impairment recognisable mainly after bilateral damage of the inferior temporal and occipitotemporal cortex (Tranel, Damasio & Damasio, 1988; Damasio et al., 1986) although cases were observed after unilateral damage in the right hemisphere (Uttner, Bliem, Danek, 2002). These findings points to a right hemisphere advantage for face recognition which also has been suggested in other studies (Ellis, 1989; Schweinberger & Sommer, 1991; Rossion, Schiltz, & Crommelinck, 2003). Most of the studies, with heathy participants, addressing this question use functional magnetic resonance imaging (fMRI) or positron emission tomography (PET). Both methods have a high spatial resolution. Studies using ERP data,with the advantage of a high temporal resolution, give a nice complement in order to explicate the temporal interplay of the involved structures and processes. The model of the distributed human neural system for face perception (Figure 2) proposed by Haxby, Hoffman, and Gobbini (2000) brings together the most relevant results of fMRI, PET and also ERP studies from the last years. Hence, it is a good summary of the knowledge concering the neuronal substrate underlying face recognition. Haxby et al. (2000) identified a core sytem in the occipitotemporal visual extrastriate cortex. The inferior occipital gyri is important for the initial visual analysis of faces. Projections to the lateral fusiform gyrus and to the superior temporal sulcus subserve the analysis of invariant aspects (identity) as well as of changeable aspects of faces (facial expression, eye gaze, or lip movement). The core system is supported by an extended system including brain regions that are important for several aspects of face perception and processing but also for other cognitive tasks. It acts in concert with the core system and includes processes that facilitate spacially directed attention to faces, speech or expression perception as well as the processing of semantic mediated information. Although the model shares some elements of the afore mentioned model from Bruce and Young (1986) it is much more related to today’s knowledge about the neural system. Therefore, it may provide more plausible predictions for experiments that are concerned with questions of face perception and face processing.

Figure 2. A model of the distributed human neural system for face perception by Haxby, Hoffman, and Gobbini (2000).

Despite the model of Haxby et al. (2000) relying on a strong body of experimental results, there are still many unclear points and open questions. For example, the functional separation of the different regions, the temporal properties of the processes and interactions among the regions via back projections or links, or the role played by the lateral fusiform gyrus in expression perception because of possible characteristic expressions between individuals (Haxby et al., 2000). In addition, the distributed system may allow interactions of the functional processes that are ascribed to the different regions through the rich interlinking of the brain regions. Interactions between regions via neural linking and the temporal sequence of processing might be important prerequisites of an interaction between facial expressions and facial identity. The model allows this possibility, although it is unspecific about an interaction between both functional processes.

In summary, the research on face recognition has gained importance over last decades. The functional model of face recognition by Bruce and Young (1986) has itself proven to be influential and it is useful to explain a range of experimental results. Nonetheless, modifications based on conflicting results have been suggested. In general, the processing and recognition of faces relies on various featural, spatial, and configural information. An important question concernes the underlying neuronal system that subserves face recognition. The proposed model of Haxby et al. (2000) integrates the results of many studies examining this question.

Facial expression

Since the publication of the functional model of face recognition by Bruce and Young (1986) there exists a strong body of research which takes as its main question the recognition of facial identity. (There is a brief overview of this research above.) In contrast, fewer studies have been concerned with the recognition of facial expressions. Even though the research on facial expressions has a long tradition (e.g. Darwin, 1965). This imbalance only started to change in the last decade and still pertains (Calder, Lawrence, & Young, 2001a). In the paragraphs following, I will attempt to give a short introduction into the research of the recognition of facial expressions. The difference between the production of emotions and facial expressions within the sender and the detection or recognition of the facial expressions by the perceiver has to be pointed out. Although emotions can be perceived via different modalities (e.g. voice) or cues (facial cues, bodily gestures), the face seems to be the most important cue to perceive an emotional state. Thus, many studies focus exclusively on the perception of facial expressions.The production of emotions within the sender and also the expression of emotions via other non-facial gestures is not the topic of this dissertation. Hence, only a short overview will be given about research on the recognition and perception of facial expressions as well as the underlying functional and cognitive processes within the brain.

In his 1872 published book “The expression of the emotions in man and animal” Charles Darwin described in detail how humans and animals express facial emotions. Based on his thorough observations, he claimed that facial expressions are universal throughout all cultures and races, and they are not learned. They have their origions in the facial expressions of animals. After Darwins groundbraking work it almost took a century until his observations were confirmed by systematic studies of Ekman and Friesen (1968; 1971). They portrayed several expressions such as happiness, fear, surprise, and disgust and displayed them to participants from different cultures (e.g. USA, Brazil, Chile, Japan). Independent of the cultural background the participants were able to identify the facial expressions correctly. In 1971 Ekman and Friesen even presented the photographs to people in New Guinea who had no contact to western or eastern literate cultures. When describing antecedent situations for a certain expression these people picked the correct pictures in almost all cases. Until today it is widely accepted that expressions are innate and not learned. They are complex patterns of facial muscular, and neuronal actions controlled by the central nervous system and triggered by specific stimuli (Ekman, 1984). However, it has to be mentioned that there are also gestures that are learned and culture specific. In addition, this might also hold true for the situational conceptualization of facial expressions or the learned suppression of expressed emotions in several situations.

The universality of facial expressions has led to the proposal of so-called basic emotions. Although the number varies between studies, the most popular categorization is reflected in the set of emotional expressive faces by Ekman and Friesen (1976). Displayed by several male and female individuals it containes facial expressions of anger, disgust, fear, happiness, sadness, and surprise. Previously it has been an issue whether the perception of facial expressions is categorical or dimensional. The proposed basic emotions imply a categorical perception of emotions. Observations of categorization errors of facial expressions led Woodworth (1938) to the assumption that emotional expressions are conceptualized along the continuums pleasantness-unpleasantness, and attention-rejection. One of the most influencial dimensional so-called circumplex models comes from Russell (1980). He introduced two bipolar dimensions of pleasure-displeasure and degree of arousal. Nonetheless, recent results using morphed faces and expressions support the notion that facial expression perception is categorical (de Gelder, Teunisse, & Benson, 1997; Calder, Young, Benson, & Perret, 1996).

It is also still up for debate, as whether facial expressions are perceived as parts or as a whole. Configural information of the whole face may play an important role for expression recognition because face inversion makes it harder to recognize facial expressions (de Gelder et al., 1997). It is already known from identity recognition that face inversion disturbes the configural perception and therefore impairs face recognition (Bentin, Allision, Puce et al., 1996; Rossion, Delvenne, Debatisse et al., 1999; Eimer, 2000a). Results of Puce, Allison, Asgari et al. (1996) suggest that the eye region plays an important role in facial expression perception. Possibly, the relative importance of single features depends on the kind of expression because of different patterns of facial and muscular activation. It is conceivable that the eyes are more important for the perception of fear and anger. In contrast, the distinct feature for identifying happiness might be the mouth.

Recently, many studies have examined the neuronal system which subserves the recognition of facial expressions. As for face identity, recognition impairments of neuropsychological patients hint to specific brain regions which may be involved. It is strongly suggested that the amygdala plays a prominant role in the perception and recognition of fear (Adolps, Tranel, Damasio, & Damasio, 1994; Phillips, Young, Scott et al., 1998; Calder et al., 1996). In line with these clinical observations are also studies using fMRI or PET (Morris, Frith, Perret et al., 1996; Vuilleumiere, Armony, Clarke et al., 2002). Although fearful faces have often been used as expressive stimuli, recent studies also examine other facial expressions. Seemingly, a widely distributed system of brain structures subserves the recognition of facial expressions (Adolphs, 2002), and partly overlaps with structures which subserve face recognition in general (see Haxby et al., 2000). Many studies suggest different involvement of brain regions for particular expressions (Kesler/West, Andersen, Smith et al., 2001; Blair, Morris, Frith et al., 1999; Sprengelmeyer, Rausch, Eysel, & Przuntek, 1998; Whalen, Rauch, Etcoff et al., 1998). Structures like the limbic system, the occipitotemporal neocortex (Adolphs, 2002), or the sulcus temporalis superior (Haxby et al., 2000) are reported as important. In addition, the insula and basal ganglia proved to be relevant for the recognition of disgust (Calder et al., 2001a). This is underlined by selective impairments of disgust recognition in patients with Morbus Parkinson (Sprengelmeyer, Young, Mahn et al., 2003), a disease which is caused by the loss of dopaminergic neurons in the basal ganglia.

It was outlined in paragraph 1.2.2., that most researchers agree on a set of basic emotions. Accounts were given for the universality of these emotions because they are to a strong extent independent of cultural background or learning. Basic emotions seem to be categorically perceived. They seem to be recognized on part based facial information although configural information may also play a role. Neurophysiological results suggest a distributed system of brain structures which subserve the recognition of facial expressions. These structures are also important for face recognition in general. The involved brain regions might differ partly between several expressions.

Approach to the topic

According to the functional model of face recognition by Bruce & Young (1986), the processes in question, namely the recognition of facial expressions and of identity, are assumed to be independent. It is apparent, that we do not need the information about ones facial expression in order to identify somebody. On the other hand, we can perceive the expression easily from familiar and unfamiliar people. The claim of independence was tested in a study by Young et al. (1986). Participants had to match simultaneously presented faces that were familiar or unfamiliar and had to react to identity or to facial expressions. Results revealed faster RTs for familiar than for unfamiliar faces in the identity matching task, but not in the expression matching task. According to the hypothesis of the latter task, familiarity is assessed by face recognition units that do not affect the structural encoding nor the expression analysis stage. Hence no difference in RT between unfamiliar and familiar faces is expected. Similar RT results were obtained from Bobes et al. (2000) in an identity and expression matching task. Simultaneously recorded ERPs revealed different topographical distributions of scalp potentials for both tasks and therefore provide evidence for the idea of distinct neural subsystems subserving the recognition of facial identity and facial expression recognition. Results from studies using fMRI point to the same conclusion of distinct neural correlates of facial recognition memory and the perception of facial expressions (Phillips, Bullmore, Howard et al., 1998). In addition, evidence supporting independence of the systems comes from the double dissociation of both processes in patients suffering from brain injury. Tranel et al. (1988) report three patients with Prosopagnosia, an inability to detect facial identity, whose ability to recognize facial expressions was preserved. In another patient study from Young et al. (1993) the authors found a selective deficit in the processing of facial expressions which was completely unrelated to the recognition of familiar and unfamiliar faces. The same conclusion is derived from results of Alzheimer’s Disease patients who were impaired in discriminating facial identities and in naming and pointing to different expressions while the discrimination of facial expressions was preserved (Roudier, Marcie, Grancher et al., 1998). Another line of evidene for the independence of facial familiarity and facial expressions comes from the N170 component (Bentin et al., 1996), an ERP component which has been linked to the stuctural encoding of faces (Eimer, 2000a). It has been shown that this component is insensitive to facial familiarity and facial expressions (Eimer & Holmes, 2002; Herrmann, Aranda, Ellgring et al. 2002). Hence, an interaction between facial expression processing and facial familiarity can be denied at least on early strucural encoding stages.

However, it might be of benefit, from an evolutionary perspective, to perceive the facial expression especially from familiar fellows in order to get reward or to prevent punishment. Furthermore, certain facial expressions like happiness or even sadness are more likely to be expressed to familiar people. Therefore, finding an interaction between the perception of facial expressions and facial familiarity might be possible.

In addition, the computer analogy that the brain is organized in independent modules, which work serially and independently is not presentable anymore (Grossberg, 2000). With its rich interlinking the brain is easily capable of parallel and unifying information processing. The different neuroanatomical areas that are involved in the various aspects of face recognition are interconnected through many efferent and afferent links as can be drawn from the model of Haxby et al. (2000). Thus, interactions of the processes might be possible depending on the temporal properties and availability of different aspects of processed information. There is evidence that the processing of facial expressions starts as early as 80 ms (Pizzagalli, Regard, & Lehmann, 1999) to 120 ms after stimulus onset (Eimer & Holmes, 2002; ) in the human brain. Therefore, it stands to reason that the information extracted from expressive faces may modulate early structural face encoding processes (Pizzagalli, Lehmann, Hendrick et al., 2002; Sato, Kochiyama, Yoshikawa, & Matsumura, 2001).

Relevant to the topic of the present dissertation, the lateral fusiform gyrus, which is involved in the processing of invariant aspects of faces and identity (Haxby et al., 2000), is interlinked with the sulcus temporalis superior and also with the amygdala. Both areas are crucial for the processing of facial expressions. If information of the expressed emotions of a face is availlable early on, it may be used to boost attention or arousal. In return, following perceptual processes might work more efficiently. Krolak-Salmon, Fischer, Vighetto, & Mauguiére (2001) reported differential ERP activity between 250 and 750 ms in occipital and occipito-temporal areas that was related to emotional expression in a gender or expression counting task. They took this as support for top-down modulations of limbic (including amygdala) and frontal areas influencing extra-striate visual areas. It is also well proven that emotional stimuli, including expressive faces, can be processed more easily outside the focus of attention when compared to neutral stimuli (Fox, Russo, & Dutton 2002). Using fMRI, Viulleumier, Armony, Clarke et al. (2002) found increased activation in the amygdala for emotional expressive faces, on task irrelevant locations and independent of spatial attention. Emotional stimuli can guide focal attention to the relevant location because the amygdala is part of the attentional system (Eastwood, Smilek, & Merikle, 2001). Such mechanisms may have been important in the evolutionary development of many organisms to detect threats in the environment. Therefore, it is possible that under certain circumstances the recognition of facial expressions and identity may interact. Fast recognition of facial expressions, especially from conspecifics could have been relevant for survival in evolution. Furthermore, even if identity and expression analysis use different functional and neuroanatomical components (Bruce & Young, 1986; Haxby et al., 2000) they are linked through the cognitive system and an interaction is not necessarily excluded.

Recently, there have been studies that suggest an interaction between facial familiarity and the perception of facial expressions and vice versa. Schweinberger and Soukup (1998) used a selective attention paradigm by Garner (1976) to address the question of an asymetric relationship between facial identity and facial expressions. Four different stimuli varied along the two dimensions identity (person A vs. B) and expression (happy vs. sad). Participants had to perform a speeded discrimination task on either one dimension which is called as relevant. Three different experimental conditions were applied. In the control condition the relevant dimension is varied between stimuli, whereas the irrelevant dimension is kept constant (e.g. only person A displays a happy or sad expression in case of an expression discrimination task). In the orthogonal condition both dimensions varied orthogonally (person A and B displayed both facial expressions, respectively). For the correlated condition both stimulus dimensions are correlated (e.g. person A displayed only the happy expression whereas person B displayed only the sad expression). An increase in RT would be expected for the orthogonal condition when compared to the correlated one in case of an influence of the irrelevant dimension on the relevant one. Accordingly, Schweinberger and Soukup (1998) found increased RTs for the orthogonal condition of the expression discrimination task when compared to the correlated condition. This did not hold true for the identity discrimination task. The results point to an asymmetric interaction between facial identity and the discrimination of facial expressions but not vice versa. There is evidence for an interaction in both directions for famous versus unfamiliar faces. The familiarity of a face can facilitate the discrimination of expression. In a study by Baudouin et al. (2000) participants had to discriminate neutral from happy facial expressions. It was expected that an interaction between facial familiarity and the discrimination task would only emerge if expression discrimination is slowed down by a hard condition. Therefore, faces were displayed either with a shortened presentation time of 15 ms (vs. 400 ms) or with a concealed mouth (vs. the whole face). Results revealed a facilitation of the expression discrimination for famous faces when compared to unfamiliar faces only in the hard conditions. The authors concluded that facial familiarity increases the “perceptual fluency” and therewith the recognition of facial expressions under hard conditions.

Conversely, there is also evidence that facial expressions have an influence on the perception and recognition of familiarity (Baudouin et al., 2000a; Nagayama, 1999). In addition, differential effects were found for personally familiar and famous versus unfamiliar faces when performing a familiarity discrimination task (Endo et al., 1992). The recognition of personal familiarity was facilitated when faces displayed a neutral expression when compared to happy and angry expressions. In contrast, famous faces were recognized faster with a happy expression. It was argued by the authors that a neutral expression is more frequently seen on personally familiar faces whereas famous faces are more often seen with a happy expression.

Although all of these studies suggest an interaction of the perception of facial expressions and facial familiarity in one or the other direction some of them suffer from methodological insufficiencies. In the study of Schweinberger and Soukup (1998) a small stimulus set was used displaying only two different individuals. The paradigm of selective attention as introduced by Garner (1976) was originally designed to explore the perception of simple stimulus dimensions such as shape or colour. Therefore, it is not designed to handle the processing of such complex facial information with overlapping features. A detailed and critical review about the implementation of the Garner-paradigm on facial perception is given by Kaufmann (2002). There is a main problem that arises from using a small set of facial stimuli. Facial expressions and facial identity share at least some overlapping features (Calder et al., 2000), therefore it is not possible to increase the variability of the irrelevat dimension in the orthogonal condition without also affecting the variability of the relevant stimulus dimension. This latter increase of variability can lead to stimulus based differences between the orthogonal and the control condition that are not based on interactions between the two processes. Another important objection, especially when questioning the interaction of facial expressions and facial identity, are different picture based strategies that can be used within both tasks (Kaufmann, 2002). In the identity decision task pictorial strategies might be used to discriminate the individuals based on non-facial cues like overall contrast of the pictures. Such effective strategies may not be possible within the expression decision task because information about expressiveness relies only on internal facial features. Accordingly, this will lead to increased variability of the relevant stimulus dimension within the orthogonal condition when compared to the irrelevant dimension which should exclusively be increased in variability. Kaufmann (2002) was not able to replicate the results of Schweinberger and Soukup (1998) by trying to consider the problems of the Garner-paradigm when using faces as stimuli. However, when using a different paradigm an interaction may emerge.

It can be annotated, critically, to the study of Baudouin et al. (2000), that the perceptual variation (concealed mouth or short presentation time) in order to aggravate facial expression discrimination, may have affected the two particular expressions differently. Possibly, the mouth region is more important for the recognition of happiness when compared to the neutral expression. On the other hand, the perception of identity for familiar people relies more on internal facial features when compared to unfamiliar faces. This implies that the variability in the hard condition was not evenly distributed over the critical dimensions of facial familiarity and facial expression. Hence, an interaction of the hard/easy condition and familiarity in the expression discrimination task may arrise because of the differential effect of the perceptual manipulation on the variability of the familiarity dimension. Evidently, more research is needed to clearly speak for or against an interaction of facial familiarity and facial expressions.

To briefly summarize, experimental data were cited which favour the independence of facial expressions and facial familiarity. This is underlined by clinical observations from patients which suffer from selective impairments of one or the other process. Electrophysiological and functional imaging studies point to separable neuronal subsystems underlying both processes. However, an interaction of facial expressions and facial familiarity is reasonable from an evolutionary point of view and when considering the rich afferent and efferent linking within the brain. Temporal properties of the involved processes may also be important for a possible interaction. There is a strong body of evidence showing that facial expression information is processed early in the brain. Thus, other processes that are involved in face recognition might be affected by a top-down influence from this information. Most importantly, recent data suggest an interaction between the perception of facial expressions and facial familiarity. Thus, some studies suffer from methodological problems. Therefore, more research is needed to clarify the raised controversy.

The present dissertation is an attempt to elucidate this question with a 2-choice RT-paradigm. By means of ERPs a closer look is taken into the temporal properties of the involved processes and the functional locus of interaction. All principles and the basis of the experimental paradigm will be explained in the following section 1.2.4. In addition, an overview is given about the ERP components that are important for the hypotheses of the experimental parts.

Mental Chronometry and Cognitive Psychophysiology

The experimental logic of the present dissertation is mainly based on the overall paradigm of mental chronometry together with cognitive psychophysiology. Therefore, it is important for the reader to get an introduction to this methodology in order to comprehend the experimental design, the working model, and the derived hypotheses (in the experimental part). Due to the narrow framework of the dissertation this introductory overview is far from complete and will mention only the most important and relevant milestones in the history of mental chronometry and cognitive psychophysiology. Several important models can just be sketched roughly.

A major theme in the study of mental chronometry is the question of whether there are separable processing stages witin the cognitive information processing system and how these stages communicate with each other. The assumption is, that mental processes are time-consuming. The sum of all processing is reflected by behavioural data like the RT or error rate. The general experimental paradigm consists of a series of imperative stimuli (auditory, visual, or somatosensory) and a required response mostly as fast and correct as possible. Sometimes a warning stimulus is included ahead of the imperative stimulus either with or without provided information about the upcomming stimulus. Depending on the proposed model of cognitive processing, different hypotheses can be drawn concerning the dependent measures.

Mental chronometry (Posner, 1978) has a long history in the study of human information processing (Meyer, Osman, Irwin, & Yantis, 1988). Already the astronomers in the 19th century searched for ways to measure the speed of mental processes because of individual differences in the subjective measurement of the movement of stars. Bessel (1823, cited after Meyer et al., 1988) for instance introduced a personal equation to measure the difference between these estimates for two different observers.

In 1850 Hermann von Helmholtz (cited after Meyer et al., 1988) introduced the simple RT procedure making him the most important forerunner of modern mental chronometry and cognitive psychophysiology. With his procedure he was able to estimate the rate of neutral conduction by considering the difference in RT between a simple reaction after a tactile lower limb stimulation compared to an upper limb stimulation. The mean RT of the latter task is somewhat shorter when compared to the mean RT of the affore mentioned. The RT difference being caused by the longer distance that the sensory nervous singal has to pass from the lower limbs.

Another major development was the subtraction method and the introduction of the choice RT procedure by Donders (1868, cited after Meyer et al., 1988). His technique used three types of RT-procedures in combination to calculate the duration of putative stimulus discrimination and response selection stages and the simple motor response. By subtracting the RT in the simple RT-task (which is supposed to consist just of the motor process) from the choice RT-task (including all three stages in question) and a go/nogo RT task, which does not include the response selection stage, the duration from either of the three stages can be estimated. Some assumptions are necessary to apply this method. First, the stimulus discrimination and the response selection stage are in strict succession and combine additively. Second, any processing stages may be inserted or deleted in a pure fashion.The procedure’s assumptions have major shortcomings which were pointed out by Külpe (1893, cited after Meyer et al., 1988) and caused a fall in favour for this method. It became quiet in the field of mental chronometry after the criticism by Külpe, although there were still some important develpoments like the discovery of the psychological refractory period by Telford (1931, cited after Meyer et al., 1988), the research on perceptual and response competition by Stroop (1935, cited after Meyer et al., 1988), or the calculation of speed-accuracy tradeoff curves from movement control tasks by Woodworth (1899, cited after Meyer et al., 1988). A speed-accuracy tradeoff curve is a function of error rate versus RT in a given task. In general it reveales a tradeoff between accuracy and movement speed, and shows that faster RTs lead to increased error rates and vice versa.

Stimulated by new developments in computer and communication science around the 1950’s the chronometric paradigm regained importance (Meyer et al., 1988). This was mainly a result of the use of new tools to collect and process data as well as to test models of human information processing. Picking up the ideas of Donders (1868, cited after Meyer et al., 1988) scientists searched for alternative methods to study the durations of human information processing stages. In 1969 Sternberg developed the additive-factor method (AFM) to analyze RT data. The AFM overcomes the problematic assumption of pure insertion of processing stages. At least two experimental factors should be manipulated in order to draw conclusions from their RT effects on the involved stages. Additivity in RT is observed when both factors are manipulated simultaneously and the first factor affects the RT independently of the level of the second one. In contrast, an interaction is observed when one factor shows different effects on RT depending on the level of the second factor. If, for instance, two factors show additive effects on the RT they may influence two different stages in the information processing chain. On the other hand, if two factors show an interaction on RT they act on at least one processing stage in common. The AFM still has some problematic assumptions. It assumes serial stages without temporal overlap and a discrete output of the stages. Keeping in mind the rich interlinking of structures in the human brain these assumptions seem implausable. Anyhow, the AFM is a powerful tool to analyze RT data. Since its publication innummerable studies have relied on this method. Sanders (1998) gives a nice summary of the research about human performance including the AFM. At least six independent and physiologically plausible processing stages have been identified: preprocessing, feature extraction, feature identification, response selection, motor programming, and motor adjustment (Sanders, 1980).

Besides the AFM there are also other models concerning human information processing. In contrast to the assumption of serial processing their stages may overlap in time and work in parallel. The cascade model by McClelland (1979) assumes continuous information transmission between the distinct functional processing ‘levels’. The continuous information output from one processing level can be used by another processing level. This may enable processes to work in parallel. If a certain activation threshold is exceeded, a response is executed at the end. Until today several results point to the existance of parallel processing stages (Abdel Rahman et al., 2000; Miller, 1982, 1983; Eriksen, & Schultz, 1979). Another important issue is whether the information transmission between parallel processing stages consists of continuously increasing activation (McClelland, 1979) or of discrete chunks of information (Miller, 1982).

Although the methods for analyzing RT data described above are helpful to draw hypotheses and conclusions about the underlying processing stages, they are yet only based on the final output of many processing stages in common. In addition, the methods suffer from their more or less physiologically plausible presumptions. Psychophysiological measures were used within chronometric paradigms in the last quarter of the 20th century in the hope of getting more direct measures of the underlaying processing stages. This so called ‘marriage between psychophysiology and cognitive psychology’ (Coles, 1989) brought benefits by using ERPs that are regarded as markers of physiological processes. ERPs are electrical potentials that are time locked to a specific event caused by the simultaneous activity of populations of neurons in the brain (Coles, Gratton, & Fabiani, 1989). If neurons have an optimal allignment they form an electrical dipole and the signal can be recorded at scalp electrodes. However, the recorded electroencephalogram (EEG) reflects all electrical activity within the brain that may not be related to a specific event. Usually, the event related signal is not visable within the noisy EEG. Therefore, ERPs are derived from the EEG by averaging samples of the EEG around this particular event. Hence, the ERP signal is discriminated from the randomly varying backround noise of the EEG. The result is a voltage by time function with positive and negative voltage peaks. Each peak or component can be described in numerous ways – according to polarity, latency, experimental / psychological precondition or according to topographical distribution on the head surface. In general, offline the ERP data are treated further by setting a baseline or applying a low-pass filter in order to increase the signal-to-noise ratio.

There are different possibilities of the quantification of components as they can be used as dependent measures in an experimental design. One possible latency measure is the onset latency of a component which represents the time in ms from the onset of the imperative stimulus to the onset of the component. The onset is defined in a certain way, e.g. the point in time when the signal exceeds a threshold. As another latency measure the peak latency is used, the time in ms from the onset of the imperative stimulus to the highest peak of a component. A third latency measure will be the time in ms from the onset of a particular component to the overt response. In addition to the latency measures, components can be described by using the baseline-to-peak amplitude (in µV) or an area measure by calculating the mean amplitude (in µV) in a defined time range.

In the present dissertation I will use three different ERP components. They are briefly outlined, in order to understand the experimental design and hypotheses of this dissertation.

The first ERP-component to be considered is the N170, most prominent at posterior-temporal and occipital sites. This component is selective for faces (Bentin et al., 1996; Eimer & McCarthy, 1999), although also emerging for single face parts like eyes (Eimer, 1998) and other face-like and well learned stimuli that are distinguishable on an item level (Carmel, & Bentin, 2002; Gauthier, & Tarr, 1997). In addition, it is delayed and enhanced in amplitude for inverted (Eimer, & Holmes, 2002; Rossion et al., 1999; Rossion, Gauthier, Tarr et al., 2000) and contrast reversed faces (Itier, & Taylor, 2002). The mentioned results indicate that the N170 is associated with the formation of a visual representation of a face like stimulus and may reflect the functional process of structural face encoding (Eimer, 2000; Bentin et al., 1996) as denoted by Bruce and Young (1986). Another frequent finding concerning the N170 is the insensitivity to facial familiarity and facial expressions (Eimer, & Holmes, 2002; Bentin & Deouell, 2000). Owing to these properties I propose the N170 to be the first functional marker in the working model outlined below which is supposed to be linked to the initial structural encoding process of a face (see Bruce & Young, 1986).

As another functional marker I use the P300 component, a positive centroparietal deflection peaking not earlier than 300 ms poststimulus. Its amplitude and latency are meaningful for the nature and timing of a participant’s cognitive response to a stimulus (Johnson, 1986). Usually it is elicted in an oddball task where the amplitude increases for infrequent stimuli of the auditory (Duncan-Jahnson, & Donchin, 1977) or visual modality. At this point it is noted that novel non-target stimuli sometimes elicit a more frontally distributed earlier positive deflection that has been referred to as the ‘P3a’ (Snyder, & Hillyard, 1976; Harmony, Bernal, Fernández et al., 2000). Distinction has been made to the ‘P3b’ or P300 as described here, which usually occures after target stimuli. Properties of the P300, like an increased amplitude depending on stimulus probability, on attention (Dujardin, Derambure,Bourriez et al., 1993), on stimulus complexity (Verbaten, 1983), or on subjective stimulus value (Begleiter, Porjesz, Chou, & Aunon, 1983) met in different functional assumptions and models. Frequently adopted is the view that the P300 reflects the online-updating of the working memory (Donchin, & Coles, 1988; Sommer, & Matt, 1990). It is undisputable that the P300 also reflects attentional function (Holdstock, & Rugg, 1995; Dujardin et al., 1993). Most importantly, it is elicited after the categorization of task-relevant stimuli. Therefore, its peak latency is highly correlated to the task relevant dimension. Findings that the P300 latency is affected by stimulus discriminability and degradation (Kutas, McCarthy, and Donchin, 1977; McCarthy & Donchin, 1983) indicate its contingency to perceptual stimulus evaluation. These last two mentioned properties of the P300 component will be most crucial for the working model proposed below.

The third event-related component, the LRP (Coles, 1989), is used to indicate the preparation of the response hand (De Jong, Wierda, Mulder, & Mulder, 1988). It is derived from the readiness potential (Kornhuber, & Deecke, 1965) which is recorded at the electrode sites C3’/C4’. They are placed over the left and right primary motor cortices (M1). In case of preparation of one or the other response hand the negativity at the contralateral recording site is increased because M1 controls the contralateral body side respectively. This asymmetry in the recorded scalp potential at the electrode sites C3’ and C4’ is isolated by a specific calculation procedure (as described below; or see Coles, 1989). The result is a negative going LRP for correct movements as well as a positive deflection for incorrectly prepared movements. The generation of the LRP is at least partly ascribable to M1 (Eimer, 1998). This is supported by the observation of a positive LRP after correctly prepared foot movements (Brunia, & Vingerhoets, 1980). Activated neurons in the M1 controlling the lower extremities are embedded in the central sulcus. They generate an electrical dipole which emanates to the contralateral hemisphere. This results in increased negativity over the ipsilateral movement side. Hence, according to the calculation procedure a positive LRP is derived.

Osman, Moore, and Ulrich (1995) proposed to separate the information processing from stimulus to overt response into two intervals with the LRP. The first interval from stimulus presentation until the beginning of central response activation – that is, the LRP – is calculated synchronized to the stimulus (S-LRP). It is informative about the time demand of processes running before or during the response activation (e.g. Leuthold, Sommer, & Ulrich, 1996). The second interval from the onset of the LRP until the overt response is averaged synchronized to the response (LRP-R). It indicates the time demand of motor processes beyond central response activation. Depending on the processes that are affected by an experimental manipulation latency differences can be expected in one or the other interval. If it acts on processes before or during central response preparation, the S-LRP interval will be affected. On the other hand, effects can be seen within the LRP-R interval if motor processes beyond hand activation are affected by the manipulation.

Experimental design, model and hypotheses
Working Model

As outlined in section 1.2.3. there is reason to believe that the processing of facial expressions and facial identity may be interdependent under some circumstances. There are still many unsolved aspects concerning such interdependency as well as the prerequisits and conditions under which an interaction might be observed. In the present dissertation I elucidate upon the question of whether the recognition of facial expressions and of facial identity can interact under certain conditions. Most importantly, the functional locus of this interdependency comes into question. Performance data and ERP components can help to pinpoint the functional locus within the cognitive system. Because of the used paradigm and tasks participants had to perform, it is possible to take a closer look at the functional chronometry of the perception of facial expressions and identity independent of each other.

In order to clearly outline the hypotheses and draw predictions for all following experiments a simple working model will be introduced. It is mainly based on the functional model of face recognition by Bruce and Young (1986) which has already been described above. The working model tries to integrate the two different experimental tasks which will be used in the conducted experiments with dependent measures. In the experimental parts, participants had to perform a two-choice RT task either to discriminate between two different equiprobable facial expressions on separately presented portraits or to discriminate facial familiarity. In the task formerly mentioned, facial familiarity was varied indepenently of expression – that is, half of the presented portraits belonged either to familiar or unfamiliar faces. In the latter task the other dimension – facial expression – was varied independently. Each person was presented with either two or three different facial expressions. Within the working model different functional processes are assumed to possibly be affected (Figure 3).

Figure 3. Proposal for a functional working model on which the following experiments are based, illustrated by an expression discrimination task.

Firstly, the presented face has to be perceived and categorized as a face in the so called structural encoding stage. As outlined above, the peak latency of the N170 component reflects the temporal properties for this processing stage. Then the relevant stimulus information has to be extracted in order to perform the discrimination task. Here, the P300 component serves as a functional marker since it is related to the perceptual evaluation of task relevant information. It is worth noting that according to the model by Bruce and Young (1986) the expression analysis and the recognition of facial familiarity are thought to be independent processes functioning in parallel. Thus, depending on the experimental task the other irrelevant dimension (e.g. recognition of familiarity within an expression discrimination task) may still be recognized automatically and in parallel. In addition, it has to be mentioned that the recognition of facial expressions is in most cases somewhat faster than the recognition of facial identity. If the familiarity may affect the expression discrimination in an expression discrimination task this may only be possible when this task is slowed down (cf. Baudouin et al., 2000). In a third assumed stage the response hand has to be selected. The LRP reflects the preparation of a specific response side. After motor preparation and adjustment the overt response is executed.

Hypotheses

The main hypothesis of this dissertation is to find a facilitative interaction between the recognition of facial expressions and of facial familiarity. This would contrast the assumptions of the functional model of face recognition (Bruce & Young, 1986). The interaction should manifest itself in the behavioural data. Admittedly, in Bruce and Young’s model both processes are interlinked through the cognitive system and an interaction is not excluded per se. Thus, the temporal properties of the two involved processes are an important prerequisite for an interaction. The present dissertation also questions, under which circumstances the predicted facilitative interaction can emerge. In addition, it is attempted to localize the functional processing stage which is connected to the expected facilitative interaction.

To understand the chronometric approach of the experimental parts this logic will be outlined here in more detail. Several possibilities are left by the attempt to detect the functional processing stage on which a facilitative interaction may occur. If faster RTs are present within a specific condition this facilitation should be reflected in the ERPs by shortened peak or onset latencies. An earlier peak of the N170 component would point to the facilitation of early perceptual processing – the structural encoding stage. If task relevant late perceptual processing stages depict the locus of confluence, an earlier P300 peak latency should be present in the facilitated condition. In addition, the peak latencies of all earlier components – in this case the N170 - have to be the same in this condition. Otherwise the facilitative effect of the earlier processing stage may just have propagated from this stage to the next one. Possibly, the facilitative effect due to an interaction between one process and the other may act on the response selection stage. In this case, shorter onset latencies of the S-LRP should be present within the facilitated condition. Again, this interpretation is only valid if all earlier components do not differ between the facilitated and non facilitated condition. The last functional locus of interaction to be proposed is motor preparation beyond hand selection. If this process is facilitated the interval between LRP onset and overt response (LRP-R) should be shortened.

Since the empirical backround, the main hypothesis, and the principle logic of the experimental paradigm are now outlined, a short overview is given about the following experimental parts. The experimental Part I will elucidate the question whether there is an interaction between facial familiarity and the discrimination of facial expressions. Experimental Part II reserves the question and asks whether there is an interaction between facial expressions and the discrimination of facial familiarity. After the experimental parts all results will be discussed thoroughly in a final general discussion and a perspective for further research will be given.

Part I: Does facial familiarity affect the discrimination of facial expression?

There is reason to believe that facial familiarity and facial expressions can interact under some circumstances. Although not assumed by the model of Bruce and Young (1986), an interaction is suggested by the model of the distributed neutral system for face perception (Haxby et al., 2000) via the interlinking of brain regions which subserve both functional processes. Throughout the research concerning this question, empirical evidence is given for the interaction of identity on the recognition of facial expressions. Schweinberger and Soukup (1998) used the Garner paradigm (Garner, 1976; see section 1.2.3.) to study the relationship between the perception of facial identity, facial expressions, and facial speech. They found the processing of facial identity to be uninfluenced by facial expressions and facial speech. In contrast, the discrimination of facial expressions was affected by identity. This hints to an asymmetric interaction between facial identity and the discrimination of facial expressions. By using the same paradigm, similar results were obtained in a study with schizophrenic patients by Baudouin, Martin, Tiberghien et al. (2002). Again, variations in the identity dimension influenced the discrimination of facial expressions in patients and in the control group. In addition, for schizophrenic patients the interference of identity on the expression discrimination task covaried with the severity of their negative symptoms. This indicates that their malfunction in processing facial affect is based on a deficit in selective attention towards facial expressions. In a behavioral study of Baudouin et al. (2000) participants had to decide if a presented unfamiliar or famous face showed either a happy or neutral expression. Hard and easy recognition of the expression was obtained by an uncovered or covered mouth in Experiment 1 or by varying the presentation times being either 15 ms or 400 ms in Experiment 2. In the covered-mouth condition, participants could more accurately discriminate expressions when the face belonged to a famous person when compared to an unfamiliar one. No difference between famous and unfamiliar faces was found for the easy, uncovered mouth condition. In the second Experiment faster RTs were found for famous faces when compared to unfamiliar ones. In addition, there was an interaction of familiarity and presentation time in such a way that differences between both types of face were only found for the short presentation time. The authors concluded that even if the processing of facial expressions is faster when compared to identity, familiarity improves the recognition of expression because it increases the “fluency” of the processing of a face, including the processing of its expression. Therefore, an effect of familiarity would only be found under conditions where the processing of facial expressions is slowed down. Hence, facial familiarity can act facilitatively on the expression discrimination task.

Given that the above mentioned studies suffer from methodological problems (see section 1.2.3.) it was reasoned that more research is needed to clarify the controversy of whether there is an interaction between the perception of facial expressions and facial familiarity. Nonetheless, the results lead to the main hypothesis of the present experimental Part I. It is expected that facial familiarity facilitates the discrimination of facial expressions. A simple experimental design is used asking participants to discriminate between two different facial expressions whereas facial familiarity is varied independently. For the expression discrimination task it is expected that the discrimination of facial expressions is faster and more accurate for familiar when compared to unfamiliar faces. The discrimination of expression is a relatively fast process, therefore an effect of familiarity on this task might only emerge when the recognition of facial expressions is slow (e.g. slowed down by introducing a condition with low expressive intensity of the displayed expression). Lower error rates are expected for familiar when compared to unfamiliar faces. Error rates should also point to a facilitation of the task which is due to facial familiarity. In addition, parallel recorded ERPs should reflect the facilitation of familiarity within the expression discrimination task. By measuring peak- and onset-latencies it should be possible to localize the functional processing stage which is facilitated for familiar faces when compared to unfamiliar ones.

The following six experiments employ the expression discrimination task. Different stimulus sets were used in order to allow a differential degree of control of facial familiarity. Personally familiar and unfamiliar faces were used as stimuli in experiments 1 to 3. In the experiments 4 and 5 participants had to undergo a learning procedure in order to become familiarized with a set of unfamilir faces. In the subsequent test phase the same number of unfamiliar faces was added to the experimentally familiarized faces. In the final Experiment 6 only celebrities were used within the same experimental design as in experiments 1 to 3. To enable the comparison between familiar and unfamiliar faces one part of the stimulus set – british celebrities – were completely unfamiliar to the participants. The experimental Part I is closed by a final discussion of the results from the six experiments.

Experiment 1
Rationale

In Experiment 1 participants had to perform an expression discrimination task. They were asked to discriminate the facial expressions happiness and disgust on personally familiar and unfamiliar faces as fast and accurately as possible. In addition, the displayed expressions were either weak or strong in intensity in order to vary the duration of expression processing. Thus, the expression discrimination was either performed under an easy (strong expressive intensity) or a hard condition (weak expressive intensity). Two groups participated in Experiment 1. Participants in the experimental group were personally familiar with half of the portrayed people. For the control group all portraits were unfamiliar. The main goal of this experiment was to show an interaction between personal familiarity and facial expression discrimination in the experimental group. In the same way it was intended to test this particular created stimulus set consisting of personally familiar and unfamiliar faces which were matched in age and gender, respectively. If an interaction between the two processes in question is found in the experimental group, it has to be proven that this effect is not based on the stimulus set per se because of mere increased expressiveness of the personally familiar faces. In order to safely ascribe a facilitation in the experimental group to personal familiarity a facilitation should be absent in the control group if the same task is used and stimuli are divided into ‘familiar’ and unfamiliar according to the experimental group.

It is expected that the variation of expressive intensity has a strong effect on RT in both experimental groups. Increased RTs should emerge for faces with weak expressiveness. For the experimental group it is hypothesized that personal familiarity facilitates the discrimination of facial expressions. If familiarity only has an effect on this discrimination when expression analysis is slowed down (Baudouin et al., 2000), faster RTs for personally familiar faces versus unfamiliar faces are only expected for faces with weak expressive intensitiy. Error percentage should follow the same pattern, being decreased for personally familiar faces. In the control group no effect of ‘familiarity’ nor an interaction with expressive intensity is expected when discriminating these faces into ‘familiar’ and unfamiliar ones according to the experimental group. Accordingly, no effect of ‘familiarity’ nor an interaction should emerge for error rates in this group as well.

Method

Participants. The experimental group consisted of twelve participants (8 women and 4 men, mean age = 26,7 years, aged between 20 and 35). Half of the presented portraits displayed personally familiar faces to them. Another twelve participants (7 women and 5 men, mean age = 25,6 years, aged between 19 and 33 years) served as a control group. All presented portraits were unfamiliar for them. Participant in the experimental group received either course credit or payment. In the control group all participants received payment. Participants of both groups had normal, or corrected to normal, vision.

Stimuli and Apparatus. Colour pictures were taken from 16 staff members of the psychology department as personally familiar facial stimuli. Sixteen unfamiliar people were matched in age and gender to the personally familiar people. Each person (Figure 4) was photographed with three different expressions (happiness, disgust, and neutral) in three slightly different positions (frontal view and 10 degrees to the left and right, respectively) with two expressive intensities (weak and strong facial expressions). The eyes looked always into the camera. The mouth was always closed. All pictures were edited in Adobe Photoshop ® to 8-bit pictures with 256 colours and a horizontal and vertical resolution of 125 x 166 pixels. They were presented on a 17-inch screen with a size of 5.0 x 6.6 cm which equals a visual angle of 2.9o horizontal and 3.8o vertical at a viewing distance of 1 m. ERTS® served as experimental software for stimulus presentation and response recording.

Figure 4. Examples of portraits displaying one person with the expressions disgust (top row) and happiness (bottom row), three different head positions, and two different expressive intensities.

Design and Procedure. In a two-choice reaction time-task participants had to discriminate the two facial expressions happiness and disgust. In four consecutive blocks different portraits of 32 people were presented in a randomized order. As mentioned before, each person was presented on different pictures displaying happiness or disgust with weak or strong expressive intensity, as well as in three slightly different head positions. After each quarter of trials, the possibility of a participant-determined break was given. Each experimental trial (Figure 5) started with a blank screen followed by a fixation cross for 500 ms. A facial stimulus appeared for a maximum of 2000 ms on the screen. It was aborted by the reaction made upon choice by the participant. Only in case of early (under 100 ms; “Zu früh”) or late responses (over 2000 ms; “Zu langsam”) was feedback provided immediately. A feedback balance about mean RT, error count and error rate was provided after each block of trials.

The key-to-expression assignment was changed after one half of trials for each participant. The start assignment was counterbalanced across participants. At the beginning and before changing the key assignment participants had to perform 40 practice trials in order to learn the correct assignment. They had to press the correct key according to the verbal description of the expressions (“Freude” or happiness vs. “Ekel” or disgust).

Figure 5. Trial scheme of Experiment 1 and all following experiments.

Data analysis. Statistical analyses of RT and error percentage was performed by means of Huyhn-Feldt corrected repeated measures ANOVA (as in all following experiments). In order to compare the results of the experimental and the control group a between-subject factor group was applied. The within-subject factors familiarity (familiar vs. unfamiliar), facial expression (happiness vs. disgust), and expressive intensity (weak vs. strong) were used. Hence, four different conditions arise for the weak expressive intensity : happiness and disgust for familiar and unfamiliar portraits, and for the strong expressive intensity respectively. For post-hoc comparisons of conditions t-tests were calculated. If more than one comparision was calculated Bonferroni-corrected significance levels were applied (Bortz, 1993).

Results

Reaction time. Figure 6 summarizes the mean RTs and error rate for Experiment 1. In both groups RT was considerably faster for strong when compared to weak expressions (F(1,22) = 220.4; p < .01; M = 832 vs. 936 ms). Most important, the expression discrimination was faster for familiar when compared to unfamiliar faces (F(1,22) = 17.9, p < .01). This effect of familiarity depended on the between-subject factor group (F(1,22) = 4.7; p < .05). It was only present in the experimental group (t(11) = -4.3, p = .001) but absent in the control group (t(11) = -1.3, p > .05). In both groups a three-way interaction with the factors familiarity, expression and intensity was found. In the experimental group, post-hoc tests revealed an advantage for familiar faces when compared to unfamiliar faces only for portraits with strong happiness (754 vs. 841; t(11)=-5.6; p = .000). In the control group the discrimination for weak happiness was faster for ‘familiar’ than for unfamiliar faces (959 vs. 1013; t(11)= -2.8; p = 0.018).

Figure 6. Reaction time and error rates for the experimental and the control group of Experiment 1.

Error rates. Figure 6 displays the error rates of the experimental and the control group separated for familiarity and facial expression. According to RT results, both groups made less errors for strong when compared to weak expressions (F(1,22) = 139.0; p
< .01). In both groups the discrimination of facial expressions was more accurate for familiar when compared to unfamiliar faces (F(1,22) = 24.3; p < .01). It showed a trend towards this effect being dependent upon the between-subject factor group (F(1,22) = 3.9, p < .06). A small but significant effect of ‘familiarity’ was also found for the controls (14.9% vs. 17.0%; F(1,11) = 5.6; p = .04). In both groups the expression influenced the effect of familiarity (Fs (1,11) > 21.1, ps < .01). Less errors were made expecially for happy familiar faces (ts(11) = -4.5, p = .001). In addition, the control group showed more accurate decisions for disgust when displayed on unfamiliar faces (t(11)2.8; p = .016).

Discussion

As expected the discrimination of facial expressions was faster and more accurate for personally familiar faces in the present stimulus set. This main effect of familiarity is evident in the experimental group but not in the control group. Contrary to the expectation, this effect is most pronounced for personally familiar faces conveying strong happiness. Originally, an effect of familiarity was only expected for portraits with weak expressive intensity. If the processing of expression is faster than the processing of identity, an interaction between facial expressions and facial familiarity should only emerge when the expression discrimination is slow. Indeed, mean RT in the experimental group was rather slow (847 ms) and RTs for portraits with strong expressive intensity were not much faster (800 ms). Generally the task was a difficult one most probably due to the small size of the pictures and the closed mouth in all portraits. The relatively slow RT in the condition with strong expressive intensity might be the reason why the expected interaction of familiarity and expression emerged for portraits within this easier condition. On the other hand, increased RT variability due to slower response times in the condition with weak expressive intensity may have blurred the significance of a familiarity effect.

Unfortunately, the control group also showed a three-way interaction of familiarity, facial expression and expressive intensity for RTs. However, it only emerges for portraits with weak happiness, whereas, in the experimental group, an effect of familiarity was found for portraits with strong happiness. For strong happiness this effect is even absent in the control group when compared to the experimental group (t(22) = -4.8; p < .001). One might argue that the depiction of strong or weak happiness leaves the possibility of an incresed variety in expressive intensity, and therefore personally familiar people might have smiled stronger. Accordingly, the same familiarity effect which is present in the experimental group should also be found in the control group. This can safely be excluded. No main effect of ‘familiarity’ on RT is present in the control group. The three way interaction of facial familiarity, expression and expressive intensity shows a different pattern in both groups, respectively. In the experimental group there is a sizable familiarity effect for happy faces with strong expressive intensity, whereas, in the control group a smaller effect of ‘familiarity’ is found for happy faces with weak expressive intensity. Therefore, the facilitative effect of familiarity on expression discrimination cannot be attributed to differences in expressive intensity between familiar and unfamiliar faces per se but on the personal familiarity of a face.

In line with RT results the decision about facial expressions is more accurate for personally familiar faces in the experimental group. Regrettably, this was also the case for the control group. However, the effect was much smaller. Alongside this problematic ‘familiarity’ effect in the control group there is also an inverse pattern for faces displaying disgust. The accuracy for ‘familiar’ faces is decreased in the control group but not in the experimental group. If the effect of familiarity is based on differences in expressive intensity between familiar and unfamiliar faces, the same effect should be present in the experimental group as well. In contrast, in this group no difference in error percentage within faces displaying disgust is found.

By showing a main RT-effect of personal familiarity in the experimental group, which is absent in the control group, it can be concluded that personal familiarity facilitates the discrimination of facial expressions. This counts especially for personally familiar faces displaying happiness. Although familiarity also affected RT and error percentage in the control group, the effect of familiarity is much more pronounced in the experimental group, and thus, not explainable by the stimulus set per se. In addition, the small effects in the control group emerged on different factor levels when compared to the experimental group. For this and the above outlined reasoning the stimulus set is proposed for use in the following experiments of the present part. As is always the case, precaution is needed for the interpretation of further results. However, it is challenging to use personally familiar faces as a stimulus set for the sake of high ecological validity but at the price of experimental control.

Experiment 2
Rationale

In Experiment 2 the same expression discrimination task was used as in Experiment 1. Participants had to discriminate between the two facial expressions happiness and disgust. The same stimulus set as approved in Experiment 1 was used. Again, the presented portraits displayed either a personally familiar or unfamiliar face with a weak or strong facial expression.

The main purpose of the present experiment was to replicate the facilitating effect of personal familiarity on the discrimination of facial expressions. This effect should be manifested in RT and error rates. In addition, the functional architecture of the underlying processes is assessed by means of ERPs. In order to pinpoint the functional locus of an interaction between facial familiarity and facial expressions different ERP components will be used as time and latency markers for the processes that are involved. Amplitude and topographical distributions of the potentials can give a hint to the availlability of extracted and processed stimulus information.

Concerning RT und error rates, the same facilitating effect of personal familiarity on the discrimination of facial expressions is expected as in Experiment 1. This may not necessarily be the case for personally familiar faces with weak expressive intensity. Based on the results of the previous experiment, a facilitative effect of personal familiarity regarding the discrimination of facial expressions is expected, especially for faces displaying happiness.

Different possibilities arise for the functional locus of the interaction between personal familiarity and the discrimination of facial expressions. If personal familiarity facilitates the structural encoding of a face, the temporal properties for early perceptual processes - as indexed by the N170 peak latency (Rossion et al., 1999a) - should be reduced for familiar faces. Although worth mentioning, this possibility is rather unlikely because of a strong body of results showing that the peak latency of the N170 is not influenced by facial familiarity nor facial expressions (Eimer, 2000; Rossion et al., 1999a). If late perceptual processes and accordingly the perception of facial expressions are influenced, an effect should be found on the peak latency of the P300 component peaking earlier for personally familiar faces. In addition, no effect of familiarity should be present for the peak latency of the N170 component. In the case of a later functional locus, namely the response selection stage before hand selection, the S-LRP interval is expected to be shorter for familiar faces when compared to unfamiliar ones. In this case, the N170 and P300 latency should not be affected by familiarity. Otherwise, a functional locus on late perceptual processing stages cannot be excluded because of propagation of the advantage for personally familiar faces to the following stages. Effects might also be found on the LRP-R interval. If familiarity shortens the time needs for motor preparation beyond hand selection, a reduced LRP-R interval for personally familiar faces should be present in the data.

Method

Participants. 16 participants (11 women and 5 men; mean age = 24,6 years; aged between 20 and 32) took part in the experiment. They were personally familiar with half of the portrayed persons presented in the experiment. The participants fullfilled either course requirement or received a payment of 15 €. The mean handedness score (Oldfiled, 1971) was 78 (ranging from –82 to +100).

Design and Procedure. As in Experiment 1, participants had to discriminate the two facial expressions happiness and disgust. The same stimulus set was used as in the previous experiment. It was repeated three times with the exception that portraits of only 28 people were presented, half of them being personally familiar to the participant. For each participant 14 most familiar persons out of 16 potentially familiar ones were selected, respectively. The trial sequence was the same as in Experiment 1. The response side-to-expression key assignment was changed three times. This was done to calculate an LRP for the whole experiment and for every repetition of the stimulus set, respectively. Participants performed 40 practice trials by pressing the correct button according to the term refering to the crucial expression (“Freude”, happiness or “Ekel”, disgust) presented on the screen at the beginning of the experiment and before changing the key assignment.

Electrophysiological recording. The EEG was recorded from 31 electrode sites (Figure 7) including IO1, IO2, LO1, LO2, Fp1, Fp2, Fz, F3, F4, F7, F8, FT9, FT10, Cz, C’3 and C’4 (4 cm to the left and right of Cz, respectively), T7, T8, Pz, P3, P4, P7, P8, P9, P10, PO9, PO10, O1, O2, Iz and the right mastoid (M2) according to the modified 10-20 International System (Pivik, Broughton, Coppola et al., 1993). Tin electodes were used with ECI Electro-GelTM electrolyt paste and placed with an electrode cap (Electro-Cap International Inc. ). Electrode impedance was kept below 5 kΩ. The electrodes above and below the left eye (Fp1 and IO1) and next to the outer canthus of each eye (LO1 and LO2) served for controling EOG artefacts. All electrodes were referenced to the left mastoid (M1). A low pass filter was set at 30 Hz. The electrophysiological signal was digitized with 250 Hz and recorded continously together with triggers that marked stimulus onset and reaction.

Figure 7. Recording positions of the scalp electrodes for recording the EEG.

Offline the recording was segmented into epochs of 2300 ms for response synchronized onsets starting 1800 ms before the response. The EEG-data were bandpass filtered with high and low cutoff frequencies set to 0.01 and 8 Hz, respectively. All trials with incorrect responses, with signal drifts of more than 120 µV within the recording epoch or with other EEG artefacts were discarded. Blink trials were corrected by the method described by Elbert, Lutzenberger, Rockstroh, and Birbaumer (1985); if this was not possible, the trial was discarded. Stimulus synchronized epochs of 1200 ms length were generated by averaging the response synchronized epochs around a variable point representing the stimulus onset. It was calculated by subtracting the single trial RT from the response trigger for each epoch, respectively. An average reference was calculated disregarding the electrodes IO1, IO2, LO1 and, LO2. In order to calculate the ERPs, epochs were averaged according to the experimental conditions.

The LRP was derived by calculating the difference between the potentials contra- and ipsilateral to the responding hand at electrode sites C’3 and C’4 and averaged across hands (Coles, 1989). To assess possible influences of horizontal eye movements on the LRP, the lateralized horizontal EOG (LhEOG) was calculated for the electrode sites LO1 and LO2 in the same way as the LRP. This calculation was applied on response synchronized epochs as well as on stimulus synchronized ones.

Data analysis. Statistical analysis of RT and error rates were performed by means of Huyhn-Feldt corrected repeated measures ANOVAs, including the within-subject variables familiarity (personally familiar vs. unfamiliar), expressive intensity (weak vs. strong), and facial expression (happiness vs. disgust). As in the previous experiment Bonferroni-corrected significance levels were applied in the case of post-hoc comparisons by means of multiple t-Tests. In Experiment 1 it was hypothesized that personal familiarity can only act facilitatively on the discrimination of facial expressions when this process is slowed down, e.g. by weak expressive intensity. Surprisingly, a facilitation for personally familiar faces had been found within the strong expressive intensity condition. Therefore, an additional analysis of the RT data was applied based upon a median split. All trials were divided into trials with fast and slow RT according to the median of each condition and participant. A Huyhn-Feldt corrected repeated measures ANOVA was calculated including the factors RT-bin (fast RT trials vs. slow RT trials), familiarity, intensity, and facial expression.

For the event-related potentials a jackknifing based method (Miller, Patterson, & Ulrich, 1996) was applied to measure the onset-difference for any two conditions of the LRP and of the P300 peak latency. For the latter component the condition values at the electrode site Pz were determined. Because of the low signal-to-noise ratio of the LRP and sometimes indistinguishable P300-peaks within a single subject, signal quality was improved by averaging n times n-1 participants. The values of each new jackknifing-participant served as an estimate of variance for the participant which was left out. The t-values of the subsequent one-tailed t-Tests were adjusted by the value described by Miller et al. (1996). In order to assess possible influences of lateralized eye movements on the LRP, mean amplitudes over an interval of 100 ms around the point where the S-LRP and LRP-R onset occurred were derived from the LhEOG. Mean amplitudes were analyzed by means of Huyhn-Feldt corrected repeated measures ANOVA including the experimental factors familiarity and expression. For analysing the peak amplitude of the N170 component, measures were derived from averaged event related potentials at the electrode site P10. An Huyhn-Feldt corrected repeated measures ANOVA was performed with the factors familiarity, expressive intensity
, and expression.

Analysis of mean amplitudes was conducted with average referenced data on a subset of 28 electrodes (omitting electrodes LO1, LO2, IO1 and, IO2). ANOVAs were performed within intervals of 50 ms starting from 200 ms until 600 ms after stimulus onset. The within-subject factors electrodesite, familiarity
, and facial expression were used. In addition, Huyhn-Feldt corrected repeated measures ANOVAs for all intervalls were performed on vector-scaled data (McCarthy & Wood, 1985) in order to assure that found differences are ascribable to differences in the topographical distribution.

Results

Reaction time and error rate. Mean RTs and error rates for the different conditions of the expression discrimination task are displayed in Figure 8. Like in the previous experiment there was a strong effect of expressive intensity (F(1,15) = 159.8, p < .01) yielding faster RTs for portraits with strong when compared to weak intensity. However, there was no interaction of intensity with other factors. Participants showed also faster RTs on faces displaying disgust when compared to happiness (F(1,15) = 4.7, p < .05). Most important, RTs to familiar faces were faster than to unfamiliar faces (687 ms vs. 698 ms; F(1,15) = 9.9, p < .01). Again, this effect of familiarity is modulated by facial expression (F(1,15) = 5.2, p < .05) and only present in portraits with a happy expression (694 ms vs. 713 ms; t(15) = - 3.77, p
< 0.01) but not for disgust.

An additional analysis was performed by splitting RTs according to the median per condition and participant. The 2-level factor RT-bin was included. The main effects of the factors familiarity, intensity, and expression as well as the interaction of familiarity and expression are not mentioned again, as they correspond to the afore presented analysis. In case of significant interactions with the factor RT-bin two post hoc ANOVAs were calculated for the separate bins, respectively. Most important, the speed of RT (factor RT-bin) significantly influenced the effect of familiarity in the expression discrimination task (F(1,15) = 12.5, p = .003). A facilitation for personally familiar over unfamiliar faces was only present for trials with slow RT (801 ms vs. 817 ms; F(1,15) = 11.8, p = .004), but not for trials with fast RT (773 ms vs. 777 ms; F(1,15) = 2.2, p = .16). The advantage of disgust over happy expressions was affected by response speed (F(1,15) = 5.2, p = .038) and is only observable for slow RTs (797 ms vs. 821 ms; F(1,15) = 5.8, p = .03), but not for fast ones (p = .16).

Figure 8. Reaction time and error rates of the expression discrimination task of Experiment 2 separated for familiarity and facial expressions.

Event related potentials. Figure 9 displays the N170 component at the electrode site P10 for the different conditions peaking exactly at 170 ms after stimulus onset. It is obvious that there are no peak differences and only minor amplitude differences between the different conditions. For this reason I abandoned any statistical calculation of the peak latency of the N170 component. Concering the peak amplitude an ANOVA confirmed the supposition of the lack of any differences between conditions (F < 1).

Figure 9. The N170 component for the expression discrimination task of Experiment 2 at the electrode site P10 separated for familiarity and disgust.

Figure 10. The P300 component for the expression discrimination task of Experiment 2 at the electrode site Pz separated for familiarity and expression.

The P300 is most pronounced at the electrode site Pz (Figure 10). Based on the significant interaction of the factors familiarity and expression in RTs, I expected an earlier peak of the P300 component for familiar portraits when compared to unfamiliar ones only for the happy expression. The jackknifing based comparison of the conditions familiar happiness versus unfamiliar happiness at the electrode site Pz revealed a trend (tJ(15) = 1.6, p < .10) with a slightly earlier peak for happy familiar faces.

The onset of the S-LRP (Figure 11) within the expression discrimination task started earlier for personally familiar portraits when compared to unfamiliar ones (tJ(15) = 1.98, p < .05). This difference was most prominent for portraits with a happy expression (tJ(15) = 1.78, p < .05) whereas it was absent for faces showing disgust (tJ< 1).

No difference between conditions was found for the response-locked LRP (Figure 12).

An influence of the LhEOG on the S-LRP, and the LRP-R could be denied (ps > .10).

Figure 11. The stimulus-locked LRP for the expression discrimination task of Experiment 2 separated for familiarity and expression.

Figure 12. The response-locked LRP for the expression discrimination task of Experiment 2 separated for familiarity and expression.

Analysis of the mean amplitude distribution revealed significant interactions of electrodeposition by expression in all time intervals starting from 200 ms until 600 ms after stimulus onset (Fs(27,405) > 4.0, ps < .005; for all results see Appendix 6.1.). As can be seen in Figure 13, faces displaying disgut yielded higher amplitudes at parietal sites as well as lower amplitudes at temporal and occipital sites. In addition, in the same time intervals (200 ms until 600 ms post-stimulus), the interactions of electrode position by familiarity was also significant (Fs(27,405) > 5.2, p < .001). Higher amplitudes were observed for personally familiar faces at centroparietal sites as well as lower amplitudes at parietal and occipital sites.

Vector scaled data revealed topographical differences between faces displaying happiness and disgust in the intervals from 200 ms to 350 ms after stimulus onset (Fs(27,405) > 3.7, ps < .008). In addition, topographical differences between personally familiar and unfamiliar faces were evident in the time intervals starting from 200 ms until 450 ms after stimulus onset and again in the time interval from 550 ms until 600 ms after stimulus onset (Fs(27,405) > 4.3, ps< .003).

Figure 13. Differences of the mean amplitude distribution between unfamiliar (UF) and familiar (F) faces (top row) and between happiness (H) and disgust (D; bottom row) for the expression discrimination task of Experiment 2 in all tested time intervalls; a grey shading equals a negative difference.

Discussion

Although numerically smaller than in Experiment 1, a facilitation was observed for familiar faces when participants discriminated facial expressions. It was especially pronounced when the face displayed a happy expression. A speed-accuracy tradeoff cannot explain the pattern of results because error rate was decreased for familiar faces and especially for happy familiar faces. As was already evident in Experiment 1 and replicated here, a facilitation of familiarity was only found for happy faces. Hence, happiness might have a special role when recognizing familiar facial expressions. Next to the neutral expression it is probably the most frequently encountered facial expression on familiar faces. This could be one reason why there is an advantage recognizing especially this expression on personally familiar faces. Results from Endo et al. (1992) point in a similar direction, although their results are not exactly compatible with these results. In an identity discrimination task participants were faster in classifying personally familiar faces, as familiar, with a neutral expression and famous faces, as familiar, with a happy expression. The expression with which a person encounters the most might influence the representation of this persons face. In return, the access to the stored representation might be faster, the more similar the currently encountered face is to this representation. Hence, if personal familiarity was accessed faster for happy faces it could act facilitatively in the expression discrimination task. This might not hold true for faces displaying disgust.

Compared to the previous experiment the faciliative effect of personal familiarity on the discrimination of facial expressions was diminished. Possibly, the whole context of an electrophysiological experiment increased the vigilance and attention of the participants. Results of Baudouin et al. (2002) suggest, that selective attention is important for the independence of both processes. Increased selective attention might have diminished an interaction between facial familiarity and the discrimination of facial expression. Admittedly, in this experiment mean RT decreased by over 150 ms when compared to Experiment 1. An important pre-requisite of the main hypothesis was defused because of overall faster RTs. Hence, the impact of familiarity on the discrimination of facial expressions decreased. The additional analysis of RT by splitting all trails according to the median shed some light on the diminished effect. It was shown that the facilitative effect of personal familiarity on the discrimination of facial expression was most pronounced for trials with slow response speed. This was predicted by the initial hypothesis. Within the slower RT bin the expression discrimination lasted long enough that the processed information about personally familiar faces could act act as facilitative on the expression discrimination task.

An unexpected result was the decreased error percentage for unfamiliar faces when displaying disgust. Possibly, positive expressions (like a happy face) and negative ones (with a disgusted expression) act differently on the neurocognitive system. The detection of negative expressions, in order to decide quickly upon the approach and avoidance of a given situation, might be important for the survival of an organism. Indeed, the perception of disgust in the present experiment was faster when compared to happiness. Perceiving disgust involves structures that are also implicated in the evaluation of offensive stimuli (Phillips, Young, Senior et al., 1997). In this sense, unfamiliar faces may be perceived as more offensive than familiar faces and hence the recognition of disgust is more accurate for unfamiliar face.

Most importantly, but only based on a small effect, the facilitated perception of expression for personally familiar happy faces is reflected in the electrophysiological data. There was no effect of familiarity, nor an interaction with expression, on the peak latency of the N170 component leaving an early perceptual locus of the found interaction apart from consideration. The same holds true for the LRP-R interval and concurrently for the preparation of the motor system beyond hand selection. In contrast, a shorter S-LRP interval was found for personally familiar faces when compared to unfamiliar ones. Corroborated by RT results, a significant difference within the S-LRP data was also found when comparing only personally familiar and unfamiliar faces displaying happiness. In addition, a trend was present for the P300 peak latency which was decreased for happy familiar faces when compared to happy unfamiliar faces. This observed trend for happy faces might have been propagated to the following response selection stage, because there a significant difference between happy familiar and unfamiliar faces, as reflected by the S-LRP, was found. Therefore, an effect of familiarity on the response selection stage is not safe to conclude anymore. Concerning the trend of the P300 peak latency, the relatively high RT variability may have broadened the peak of the P300 component making the peak detection less reliable. Thus, the effect of familiarity within happy faces was blurred to only a trend.

The analysis of the topographical distribution revealed both an early significant difference for facial familiarity and facial expression. There was no interaction found between those factors. The discriminative information on facial familiarity as well as on facial expressions seems to be available as early as 200 ms after stimulus onset. It may be possible that the information about facial familiarity can influence later processing stages that are relevant for the participant’s task, and this may cause the facilitative interaction between facial familiarity and the discrimination of happy facial expressions which was found for personally familiar faces in the present data. Had an interaction of facial familiarity and facial expression been present for the topographical distribution it could be expected that the facilitative effect for personally familiar happy faces might be subserved by slightly different brain regions as when faces display disgust. However, no such interaction was present. The validity of this supposition remains unclear because brain regions might be involved which do not affect the topographical distribution.

Overall, the results point towards an interaction between the perception of facial familiarity and the discrimination of facial expressions for personally familiar faces. As intended, the event related potentials elucidated the functional locus of interaction within the information processing stream more clearly. They point to late perceptual or response selection stages as the possible loci of interaction between personal familiarity and the discrimination of facial expressions. Based on the presented ERP data no safe decision can be made favouring one or the other functional locus of interaction. Thus, further data are necessary in order to clarify this unclear functional locus.

Experiment 3
Rationale

It was the intention of the present experiment to replicate the results of Experiment 2 with a slightly changed design in order to increase the rather small facilitative effect of familiarity on the discrimination of facial expressions. As discussed before, the numerically small effect of familiarity and the relatively high variability in RT in the previous experiment may be one reason for the somewhat unclear effect in the P300 peak latency. In addition, the present experiment intends to get better control of the degree of personal familiarity by recording the skin conductance response (SCR) and applying a questionaire about people being personally familiar.

Prior to the experimental blocks the SCR was recorded for all personally familiar and unfamiliar faces. Within face recognition the SCR is highly linked to emotional arousal processes, orientating response, and attention. Hence, it can give a clue to the individual significance and emotional value of a face (Tranel et al., 1985) or the familiarity (Breen et al., 2000). In healthy participants but also in a clinical context, it is often found that the SCR is more enhanced for famous faces when compared to unfamiliar ones. Accordingly, the expectation is to find this pattern for personally familiar and unfamiliar faces in the present experiment.

The previous experiments gave the impression that the personal contact to some members of the teaching staff who served as personally familiar faces was either too little or too long ago for some participants. Therefore, after recording the SCR but before the start of the experimental blocks, an additional block was added presenting all personally familiar people with a neutral facial expression and some personal background information about each (e.g. age, children, biographical data, chair, lectures or seminars). It was intended to refresh the memory for personally familiar people. Subsequently after this block the experimental blocks were started.

Again, it is expected that personal familiarity facilitates the discrimination of facial expressions expecially for happy portraits. Because no interaction had been found with the factor expressive intensity in the previous experiment this factor was omitted and only portraits with strong expressive intensity were used. Thus, the hypotheses do not consider expressive intensity anymore. For ERP data more clear cut results are expected than in the previous experiment because of the omission of portraits with weak expressiveness and thus decreased variability in RT. As already implied by the trend concerning the P300 peak latency in the previous experiment it is expected that this latency should be reduced for personally familiar happy faces when compared to unfamiliar happy faces. Due to propagation, this facilitation may also be present in the onset latency of the S-LRP. No effects should be present for the N170 component as well as for the LRP-R. Hence, the expectation for the present experiment is to pinpoint late task relevant perceptual processing stages as indexed by the P300 as the functional locus of interaction between personal familiarity and the discrimination of facial expressions.

Method

Participants. In the third experiment 12 participants (10 women and 2 men; mean age = 25,8 years; aged between 21 and 54) took part. As in the previous experiments, they were personally familiar with half of the portrayed persons presented in the experiment. For each participant 14 of the most familiar people out of 16 potentially familiar ones were selected. Participants fullfilled either course requirement or received a payment of 15 €. The mean handedness score (Oldfiled, 1971) was 74,8 (ranging from –40 to +100).

Design and Procedure. The dedign and procedure differs from the previous experiments in that two blocks were added prior to the experimental blocks. At the beginning of the session, the SCR was recorded within one block. In order to keep the participants attention as well as to enhance the SCR, a familiarity discrimination task was required from the participants. They were instructed to view all presented portraits and discriminate personally familiar from unfamiliar faces by pressing one of two response keys on a computer keyboard with their index or middle finger of the dominant hand. Participants were told not to move the non-dominant hand where the SCR electrodes were affixed and which rested comfortably on a soft pad. After 3 warm-up trials at the beginning of the block showing unfamiliar people all 14 personally familiar faces as well as all 14 unfamiliar matches were presented successively and in random order with a neutral expression. The trial started with a fixation cross in the middle of the screen presented for a random duration between 11 and 12 seconds. In the last 5 seconds of the interval the fixation cross turned bold in order to capture the attention of the participant. It was replaced by a portrait of a familiar or unfamiliar person. All portraits remained on the screen for 2000 ms. Participants were supposed to make their familiarity decision within this time window. No feedback was provided.

After the SCR recording, the electrodes affixed to the hand were removed. Subsequently, participants had to perform a block to refresh their memory on the personally familiar people. All 14 familiar portraits were presented with a neutral expression together with information concerning age, children, position at institute, given lectures and seminars, hobbies, and biographical information (e.g. ‘was a competitive ice skater in his youth’). Participants were told that after the experiment there would be a short questionaire in order to test the information. Participants could look at the information at leisure.

Following the block to refresh memory the experimental blocks for the EEG recording started. As in the previous experiments, participants had to discriminate the two facial expressions happiness and disgust. The same stimulus set was used as in Experiment 2 with the exception that all portraits with weak expressive intensity were omitted because of the previous lack of interaction of this factor with any other factor. The trial sequence was the same as in the previous experiments. The whole stimulus set was repeated twice. The response side-to-expression key assignment was changed twice in order to calculate an LRP. At the beginning of the experiment and before changing the key assignment, participants performed 40 practice trials by pressing the correct button according to the expression (“Freude”, happiness or “Ekel”, disgust) written on the screen.

Electrophysiological recording. The SCR was recorded with Ag/AgCl electrodes (1 cm in diameter) and isotonic electrolyte gel (K-Y Jelly, Johnson & JohnsonTM). They were affixed to the thenar and hypothenar of the non-dominant hand. The ground electrode was placed on the forearm of the respective hand. A CoulbournTM Isolated Skin Conductance Coupler (Model V71-23) was used. The sampling rate was 200 Hz. Offline the signal was cut in 8200 ms epoch starting from 200 ms before until 8000 ms after the stimulus and converted in micro-Siemens (µS).

The EEG was recorded from the same 31 electrode sites as in Experiment 2 (see section 2.2.2.). For recording, the same settings and materials were used as in the previous experiment. Offline the recording was segmented into epochs of 2300 ms for response synchronized onsets starting 1800 ms before the response. The EEG-data were bandpass filtered with high and low cutoff frequencies set to 0.01 and 8 Hz, respectively. The criteria and procedures for artefact removal and blink trial correction was kept the same as in the previous experiment. Stimulus synchronized epochs of 1200 ms length were generated by averaging the response synchronized epochs around a variable point as in Experiment 2. An average reference was calculated disregarding the electrodes IO1, IO2, LO1 and, LO2.

The LRP at electrode sites C’3 and C’4 and the LhEOG at electrode sites LO1 and LO2 were calculated with the same procedure as described above (cf. Coles, 1989).

Data analysis. For calculation of the SCR data only artefact free trials with correct familiarity decisions were included. The prestimulus baseline was set at 200 ms. A valid SCR trial was counted if the maximum peak 1 to 7 sec after stimulus onset exceeded the criterion of 0.01 µS. If this was not the case the trial was counted as invalid and the value of the peak amplitude was set to zero. Different measures were calculated for familiar and unfamiliar faces. The magnitude was defined as the mean peak amplitude of all valid and invalid trials. For the peak amplitude and the peak latency only valid trials were included in the condition means. The frequency defined the percentage of valid responses. For each measure an ANOVA was performed with the two-level factor familiarity.

Statistical analyses of RT and error rate were performed by means of Huyhn-Feldt corrected repeated measures ANOVAs including the within-subject variables familiarity (personally familiar vs. unfamiliar) and facial expression (happiness vs. disgust). If post hoc comparisons were necessary, t-Test were calculated with Bonferroni corrected significance levels. Like in Experiment 2 all trials were split in two bins acording to the median for each condition and participant. Again, an ANOVA was performed with the additional two-level factor RT-bin (fast RT bin vs. slow RT bin).

As in the previous experiment a jackknifing based method (Miller et al., 1996) was used to measure the onset-difference for any two conditions of the LRP and of the P300 peak latency measured at the Pz electrode. As before, mean amplitudes were derived from the LhEOG for a 100 ms intervall in order to assess a possible influence on the LRP. For analysising the peak amplitudes of the N170 component measures were derived from averaged event related potentials at the electrode site P10. An ANOVA was performed with the factors familiarity and facial expression. Analysis of the mean amplitude distribution and the topography was performed the same way as in Experiment 2.

Figure 14. Skin conductance response as the mean of all valid trials for personally familiar and unfamiliar faces during a familiarity discrimination task.

Figure 15. Reaction time and error rates for the expression discrimination task of Experiment 3 separated for familiarity and expression.

Analysing the data after the median split revealed a differential effect of the factor familiarity for the bin which includes trials with slow RT when compared to the bin with fast RTs (F(1,11) = 9.6, p = .01). However, post-hoc ANOVAs, carried out for the two partitions seperately, showed facilitations for the expression discrimination task due to personal familiarity not just for the slow trials (773 ms vs. 817 ms; F(1,11) = 13.6, p = .004), but also for the fast RT bin (529 ms vs. 538 ms; F(1,11) = 10.5, p = .008).

Figure 16. The N170 component for the expression discrimination task of Experiment 3 at the electrode site P10 separated for familiarity and expression.

Event related potentials. Figure 16 displays the ERP amplitudes of the N170 component. It is most pronounced at the electrode site P10 peaking exactly at 170 ms. Statistical testing of the N170 at this electrode site confirmed, to all appearances, that there is no difference in peak latency nor in peak amplitude between the experimental conditions.

The P300 component (Figure 17) shows its maximum amplitude at the Pz electrode. According to the interaction of familiarity and facial expressions found in the present experiment, the difference between personally familiar and unfamiliar faces within the P300 peak latency was only tested for faces displaying happiness. Although numerically present (527 ms for personally familiar and 571 ms for unfamiliar happy faces) a t-Test based on jackknifing averages showed no significant difference in the P300 peak latency between personally familiar and unfamiliar happy faces. The mean amplitude of the P300 was measured at the Pz electrode. Statistical testing revealed higher amplitudes for personally familiar when compared to unfamiliar faces (9.7 µV vs. 9.8 µV; F(1,11) = 14.0, p < .01). In addition, higher amplitudes are present for faces displaying disgust when compared to the happy facial expression (9.8 µV vs. 8.9 µV; F (1,11) = 11.2, p < .01). No interaction was found between the factors familiarity and facial expression.

Figure 17. The P300 component for the expression discrimination task of Experiment 3 at the electrode site Pz separated for familiarity and expression.

According to the RT results of Experiment 3, onset differences in the S-LRP (Figure 18) between personally familiar and unfamiliar faces were only tested for faces displaying happiness. In contrast to the previous experiment no difference was found between these two types of faces within the expression discrimination task. As is evident in Figure 19, the latency between LRP-onset and overt response was of equal length for all experimental conditions. Based on statistical analysis, an influence of the LhEOG on the S-LRP, and the LRP-R could be denied (ps > .10).

Figure 18. The stimulus-locked LRP and LhEOG for the expression discrimination task of Experiment 3 separated for familiarity and expression.

Figure 19. The response-locked LRP and LhEOG for the expression discrimination task of Experiment 3 separated for familiarity and expression.

Statistical analysis of the mean amplitudes for the 8 different time intervals (for all results see Appendix 6.2.) revealed a significant effect of facial expression starting at 250 ms until 600 ms (Fs(27,297) > 4.1, ps < .002). As can be seen in Figure 20 more positivity for disgust is present at fronto-central and pareto-occipital sites. In addition, potentials for disgust are more negative at fronto-temporal sites. Amplitudes differ also significantly between personally familiar and unfamiliar faces starting at 250 until 600 ms (Fs(27,297) > 5.0, ps < .002). Additional calculation of the vector scaled data revealed topographical differences between facial expressions from 250 and 350 ms after stimulus onset (Fs(27,297) > 3.9, p < .003). Different topographies are also evident between familiar and unfamiliar faces in all intervalls from 250 ms until 600 ms poststimulus (Fs(27,297) > 3.5, ps < .01).

Figure 20. Differences of the mean amplitude distribution between unfamiliar (UF) and familiar faces (F; top row) and between happiness (H) and disgust (D; bottom row) for the expression discrimination task of Experiment 2.

Discussion

Although only revealing a trend, SCR data showed an increased peak amplitude for personally familiar faces when compared to unfamiliar faces. The statistical power of the calculated SCR measures was very low because they relied only on a maximum of 14 trials per condition and on n=12 participants in this experiment. Therefore, the trend for the mean peak amplitude can be taken as highly suggestive. It indicates a larger autonomic response for personally familiar faces and hence increased affective information processing. A possible alternative explanation of an increased signal value only for personally familiar faces is hard to justify, because all faces in this block were defined as targets. The SCR results differentiated between personally familiar and unfamiliar faces. Therefore, the results served as a good control for the degree of familiarity. It can safely be concluded, that the degree was fairly high for the personally familiar stimulus set when compared to unfamiliar faces.

Replicating results of the RT and error rates from the previous two experiments, personal familiarity facilitated the discrimination of facial expressions in general and especially when the face displayed a happy expression. As was intended by the initial block for refreshing the memory, the facilitative effect of familiarity in RT increased when compared to the results of Experiment 2. This was confirmed by a repeated measures ANOVA including the two experiments as a between-subject factor (F(1,26) = 5.3, p < .05). In the present experiment the difference in RT between personally familiar and unfamiliar faces was 27 ms; in the previous experiment the facilitation for personally familiar faces was, although significant, only 11 ms. This enhancement of the familiarity effect between this and the previous experiment is even more pronounced for happy faces (42 ms difference in the present experiment and, 19 ms in Experiment 2). The increased effect in RT cannot be explained by the pre-exposure of the faces preceeding the experimental blocks. For recording the SCR both personally familiar and unfamiliar faces were presented with a neutral facial expression. Although personally familiar faces were presented again within the block to refresh the memory, only portraits with neutral facial expression were displayed which were not used in the expression discrimination task of the experimental blocks. In addition, it may be possible and can never be excluded for personally familiar faces that participants have met some of them in the last days or even hours.

As was already seen in the previous experiment, the facilitative effect of personal familiarity was expecially pronounced in trials with slow RT, but also present in fast trials. This is an almost expected outcome based on the initial hypothesis to find an interaction of facial familiarity on the discrimination of facial expressions especially when processing is slow (Baudouin et al., 2000). In addition, finding a significant effect after the median split is more difficult because of the reduced statistical power based an a lower trial number. The robust significance underlines even more the genuine effect within the RT data. The facilitative effect of familiarity on the expression discrimination task also for fast trials might come along with the overall increased effect of familiarity in RT when compared to Experiment 2.

Surprisingly, the RT results were not corroborated by the electrophysiological data. Concerning the onset and peak latencies of the different event related components no differences were found between personally familiar and unfamiliar faces when collapsed over the two expressions, nor when considering only trials with a happy expression. Although the effect of personal familiarity on the discrimination of facial expressions was evident in RT and error rates and even enhanced when compared to Experiment 2 it was completely absent within all tested ERP components of the present experiment. This holds true to all appearances for the N170 component, the S-LRP and, the LRP-R interval. The only numerical difference was present for the P300 peak latency. Here the difference for personally familiar and unfamiliar faces was 44 ms which corresponds to the size of effect in the RT data. It can only be speculated why there was a lack of significance nor a trend. One reason could be the lower statistical power based on less participants (12 in the present Experiment vs. 16 in Experiment 2) and a reduced number of trials, because only portraits with strong expressive intensity were included. Omitting the trials with weak expressive intensity was in fact done to decrease the variability of RT and increase the quality of determining the peak and onset latency measures. In fact the standard deviation of the mean RT decreased only little from Experiment 2 (SD = 197.9) to the present experiment (SD = 187.7), despite using only trials with strong expressive intensity. Increasing the number of repetitions of the stimulus set has not been a useful alternative because of the growing perceptual familiarity with every repetition for unfamiliar faces. This, in fact, could decrease the already weak facilitative effect of familiarity on the discrimination of facial expressions.

Another difference between the present and the previous experiment is the later onset of topographical differences for facial expressions as well as independently for facial familiarity. In Experiment 2 different topographies for both factors started as early as 200 ms after stimulus onset. In the present experiment differential topographies were only evident from 250 ms poststimulus. The 50 ms displacement of the availability of information about facial expressions and familiarity may have disturbed the interaction of both processes.

In summary, the facilitative effect of personal familiarity on the discrimination of facial expressions was replicated for RT and error rates. By omitting the portraits with weak expressive intensity and adding a block to refresh the memory for personally familiar faces the numerically small effect in RT of the previous Experiment 2 was increased for the present experiment. Again, the facilitative effect of personal familiarity was especially pronounced for faces displaying a happy expression. It underlines again the special role a happy expression might have when recognizing the facial expression of personally familiar faces. Unexpected was the lack of effects within the ERP data. The slightly changed design, in order to increase the facilitative effect of personal familiarity on the discrimination of facial expressions and to decrease the variability in RT, only significantly affected the overt performance data. However, it failed to improve the ERP results and to clarify the functional locus of interaction between facial familiarity and the discrimination of facial expressions. One expanation for the unclear data may be the used stimulus set. It is always challenging to use personally familiar faces as stimuli because of high ecological validity. However, it is more difficult to control the influence of the stimulus set. A fully balanced design is hard to accomplish. In order to exclude a possible influence of the stimulus set, the following experiments use unfamiliar faces whereas particpants were familiarized with half of the presented persons.

Experiment 4
Rationale

In the previous experiments it has been shown that personal familiarity facilitates the discrimination of facial expressions especially for faces displaying happiness. ERP results suggested late perceptual processing stages – as indexed by the P300 peak latency – as the functional locus of interaction between both processes. However, due to high variability in RT, results have been unclear. The used stimulus set consisting of personally familiar and unfamiliar faces may have been one source of variance. It is the purpose of the present experiment to improve experimental control over the stimulus set by using only unfamiliar faces, whereas half of the stimulus set is experimentally familiarized within a learning block.

Although the degree of familiarity of experimentally familiarized faces is low when compared to personally familiar faces, an interaction could still emerge. This is conceivable when we consider that ERP results from Paller and coworkers (Paller, Bozic, Ranganath et al., 1999; Paller, Gonsalves, Grabowecky et al., 2000) found parietal ERP differences for visually familiarized faces when compared to newly presented faces. The authors ascribe this difference to the retrieval of stored visual face information. Striking evidence comes from Rossion et al. (2003) who found lower PET activation for unfamiliar versus learned familiarized faces within the right lateral fusiform gyrus and the right inferior occipital gyrus. Interestingly, the differences are found in visual extrastriate brain regions that subserve both, the categorization of faces on an object level as well as the discrimination based on previous encounters. An overlapping set of brain regions seems to be involved in face detection, individual discrimination, and presemantic recognition (Rossion et al., 2003). The involvement may just occur over different time scales, which ERP data suggests (e.g. Bentin et al., 1996; Eimer, 2000; Schweinberger, Pickering, Burton, & Kaufmann 2002a). Considering the mentioned results, neuronal differences between familiarized and unfamiliar faces are evident and reflected in ERP data. In addition, it is possible that the access to familiar face representations is faster when triggered by typical familiar stimuli such as an often encountered facial expression on a familiar face. Evidence comes from findings of Jemel, Pisani, Calabria et al. (in press), and Schweinberger, Pickering, Jentzsch et al. (2002) who found increased priming effects for the same facial stimuli when compared to different primes of the same face. Therefore, the expression with which a person is familiarized within a learning block is varied in the present experiment.

Only behavioural measures are recorded in Experiment 4 since it is the first time that I use this stimulus material. The expectation of the present experiment is to find an interaction between facial familiarity and the discrimination of facial expressions for experimentally familiarized faces. Thus, one half of the unfamiliar people will be familiarized in an initial learning block. As mentioned before, the visual experience with different facial expressions of the persons to be familiarized is varied. Within the learning block one half of the persons will be presented with a neutral and a happy expression. The other half will be displayed with a neutral and an angry expression. For the expression discrimination task this means that each familiarized person was seen before with only one expression, but not the other. The condition with the expression that was not encountered for a familiarized person is of particular interest. Previous results of Baudouin et al. (2000; Experiment 3) suggest that the facial expression of a person can be discriminated more accurately when it was familiarized with a happy when compared to a neutral expression. In contrast, it might also be possible that the previous encounter of a specific emotional expression can facilitate the discrimination of facial expressions. In this case it is hypothesized that, depending on the expression that was presented during the learning block, the discrimination is facilitated for a familiarized person displaying this already encountered facial expression. It could also be, that general visual familiarity of a face may facilitate the discrimination of facial expressions. In this case a facilitation should be present for all familiarized persons independent of the learned emotional expression.

Rating

In advance of the experiment a rating was conducted in order to select a stimulus set of 40 male and 40 female people out of 104 different unfamiliar people. For each person 3 pictures were available with either a neutral facial expression, happiness, or anger. Five participants (all female, mean age = 27.2 years, aged between 24 and 33) rated every portrait according to the six categories strong or weak happiness (“starke Freude”; “leichte Freude”), neutral (“Neutral”), strong or weak anger (“starker Ärger”; “leichter Ärger”), or none at all (“Keine von den angegebenen Kategorien”). Participants were instructed to use the last option mentioned sparingly. It did not belong to the rating scale but conveyed important information about the stimuli. Participants had as much time as they needed for rating the faces. Afterwards, the ratings were recoded on a scale ranging from -3 to + 3 and the percentage of correctly classified ratings was analyzed. People, whose portraits were less often correctly categorized were excluded, and 80 people (40 male) remained. They were divided into 4 subsets with 10 men and 10 women, respectively.

Table 1: Mean expression ratings of 4 subsets of unfamiliar faces.

anger

happiness

neutral

Subset 1

- 2.34

2.71

1.05

Subset 2

- 2.35

2.73

1.11

Subset 3

- 2.35

2.69

1.11

Subset 4

- 2.34

2.75

1.13

The decisive value was the mean rating of anger, because the variation was larger when compared to happy and neutral expressions. The latter two expressions were correctly categorized most often. The mean ratings for the three facial expressions happiness, anger, and neutral were comparable in each group (Table 1). That is, each group contained about an equal amount of people with high and low mean ratings of happiness or anger.

Method

Participants. Twelve participants (7 women and 5 men, mean age = 25.5 years, aged between 19 and 36) took part in the present experiment. All participants had normal or corrected to normal vision. They received either course credit or 10,00 € for participation. Handedness was not determined because electrophysiological data were not recorded.

Stimuli and Apparatus. All portraits of the stimulus set consisting of 80 persons were edited in Adobe Photoshop to 256-colour pictures with a bluish grey background. For the initial learning phase, the size of the pictures was edited to 269 by 350 pixels. For the following experimental blocks they were resized to 190 by 247 pixels. This was done to present the pictures in about the same size as the portraits in Experiments 1 to 3. All stimuli were presented on a 17-inch screen with a size of 10.8 x 14.0 cm for the learning phase which equals a visual angle of 6.1o horizontal and 8.0o vertical at a viewing distance of 100 cm. For the experimental blocks the size of the stimuli (5.9 cm by 7.7 cm) yielded a visual angle of 3.4o by 4.4o at the same viewing distance. ERTS® served as experimental software for stimulus presentation and response recording.

Learning-procedure. All participants had to undergo a 1 hour training session in order to become familiar with 40 people (two subsets of 20 people). One half of the presented faces displayed male faces. The assignment for the participants, to respond to two out of four subsets of faces, was counterbalanced. That is, half of the participants were familiarized with set one and two, whereas the other half was familiarized with sets three and four. For each participant, and counterbalanced between the participants, one set of people were presented only with a neutral and happy expression, the other set with a neutral and angry expression. In an initial learning block the participants viewed 80 portraits of 40 people with a neutral as well as a happy expression for the first subset or an angry expression for the second subset. All portraits were presented for 5000 ms. After a blank screen of 500 ms, the next portrait appeared in randomized order. Participants were instructed to view all presented portraits thoroughly in order to recognize them in the following blocks. No response was required. After a short break, 3 testblocks with a matching-to-sample task were added. On each trial, participants viewed 2 faces presented next to each other for 2000 ms. Within this time they had to decide, which of the two faces belonged to the previously learned ones by pressing the corresponsing right or left button on a computer keyboard. Subsequently, only the correct face remained in its position for an additional 1500 ms either with a green frame, when the previous decision was correct, or with a red frame in case of an error or a missed response. Reaction times over 2000 ms were also counted as errors. Forty different people served as distractor faces and were used only in the learning phase. In each block every portrait was presented twice with a distractor face of the same and of different gender. The two faces always had the same facial expression. At the end of each block, participants received feedback about mean RT, error count, and error percentage. After three test blocks, an error criterion of 5 % or lower had to be reached. If error percentage was above this criterion participants had to go on practising one, but at most three additional test blocks in order to meet this criterion. Overall, each person was viewed at least 14 times (52 s), whereas each single picture was viewed 7 times (26 s).

Design and data analysis. Following the learning phase, and a short break of about 10 minutes, the participants continued with the experimental blocks, that is, an expression discrimination task. The presented portraits comprised the 40 previously learned people as well as 40 completely unfamiliar people. Each person was represented with two portraits displaying either a happy or an angry expression. A fixation cross appeared in the middle of the screen for 500 ms. It was displaced by a target face displaying either a happy or angry expression. Participants had to discriminate the two expressions as fast and correct as possible by pressing the right or left key according to the key assignment. After pressing the button the screen went blank. After 1000 ms the next trial continued, while the order was randomized. If participants failed to respond within the maximum RT of 2000 ms, visual feedback for late response was provided (“Zu langsam!”, too slow). No feedback for wrong responses was provided because the stimulus set was presented twice and stimulus-to-reaction learning should be avoided. After one presentation of the whole set of portraits (divided into two blocks of trials) the key assignment changed. In order to prevent mistakes, it was practiced in a separate block at the beginning and before changing the key assignment. Participants responded to the stimulus words “Freude” (happiness) or “Ärger” (anger) by pressing the corresponding key. After each block, mean RT, error count and error percentage were provided as feedback.

Statistical analyses of RT and error rate were performed by means of Huyhn-Feldt corrected repeated measures ANOVA. The within-factors familiarity-type (familiar person with learned expression; familiar person with unlearned expression; unfamiliar person) and facial expression (happiness vs. anger) were used. If necessary, t-tests were calculated for post-hoc comparisons of conditions. If more than one comparision was necessary Bonferroni-corrected significance levels were applied.

Results and Discussion

Familiarization. Error rates for the three consecutive test-blocks of the learning procedure decreased from 6.6% over 3.4% for the second block to 2.7% for the last block. The decrease from the first to the second block was significant (t(11) = 5.5, p < .001).

Reaction time and error rates. Figure 21 summarizes RT and error rates for the expression discrimination task of Experiment 4. Faster RTs were expected for learned faces when compared to unfamiliar ones, especially when they were familiarized with a happy expression (Baudouin et al., 2000). As is obvious, portraits displaying happiness (506 ms) yielded faster RT (F(1,11)=11.9; p < 0.01) when compared to angry expressions (552 ms). Although not important for the main question of this dissertation, it has been found in previous studies that participants show faster RT to positive expressions of happiness when compared to negative expressions (Leppänen, Tenhunen, & Hietanen, in press; Crew, & Harrison, 1994; Kirita, & Endo, 1995; Hugdahl, Iversen, & Johnsen, 1993). It might be easier to recognize happiness, because less visual features may be necessary. Indeed, in the present stimulus set many happy faces display a salient smiling mouth. In contrast, the discrimination of angry faces may be more difficult. Already the ratings of the stimulus material conveyed more incorrect classifications for angry faces when compared to happy faces (t(4) = -5.6, p < .05). The main effect of facial expression in Experiment 2 with faster RTs for disgust when compared to happy faces does not stand against this interpretation. The closed mouth in the former stimulus set may have diminished the advantage for happy faces.

Figure 21. Reaction time and error rates for the expression discrimination task of Experiment 4 separated for familiarity and expression.

Unfortunately, the hypothesized effect of familiarity-type was not evident in a main effect nor an interaction (F < 1) with facial expression. For error percentage the data revealed only a trend for facial expression (F(1,11)=3.4; p=0.09) with slightly less errors for the angry (4,6%) when compared to happy expressions (6,6%).

As can be summarized from the present results no interaction was found between familiarity and the discrimination of facial expressions when using learned familiarized and unfamiliar faces. This independency is even more underlined by the fact that participants were not faster in discriminating the expression when they had seen the very same expressive portrait of a familiarized person during the learning block. Perceptual familiarity per se does not affect the perception or discrimination of facial expressions at all. Hence, other processes might have been involved which subserved the interaction between facial familiarity and facial expression discrimination for personally familiar faces in the previous experiments. An obvious difference is the missing personal importance and semantic information of the present stimulus set when compared to personally familiar faces. The latter mentioned faces may be more implemented in the neural system when compared to experimentally familiarized faces. Thus, in the present experiment the chance of finding an interaction between facial familiarity and the discrimination of facial expressions may have been smaller.

In general the mean RT of the present experiment was fairly fast when compared to the experiments which used personally familiar faces (Experiments 1-3). This may be due to the stimulus material used where the happy expression is clearly shown with an open mouth on all happy portraits. Faster RT for the happy facial expression when compared to anger underlines this supposition. Hence, because of the easy and fast expression discrimination task a necessary prerequisite of a possible interaction between facial familiarity and the discrimination of facial expressions is missing (c.f. Baudouin et al., 2000). On the other hand, the salient feature of an open mouth for happy portraits may have led the participants to a strategy where very little information is extracted from the internal facial features that are necessary to perform the task. There is evidence that the discrimination of facial expressions and identity share little featural information (Calder et al., 2001). Still, there is also overlapping feature information which is relevant for the discrimination of both processes. When participants have to use all featural information which is available and in part also relevant for identity recognition during an expression discrimination task, it can be assumed that an interaction might emerge. Possibly, the already mentioned lower degree of familiarity, a less distributed neural representation of familiarized faces, and also the lack of semantic knowledge and emotional importance may be more relevant for the unconfirmed hypothesis. To test whether the fast RT and the relatively easy discrimination of this experiment may have crossed a possible emerging of the expected interaction, a further experiment is conducted with a harder task in order to increase the mean RT for the expression discrimination.

Experiment 5
Rationale

As mentioned before, the mean RT of the previous experiment was very fast, presumably due to the stimulus material, which showed an open mouth as the most salient feature on portraits displaying happiness. In return, this may have led the participants to a strategy where most of the featural information of the internal face is ignored since it was irrelevant for the easy expression discrimination. In order to apply a harder condition and to increase the overall RT level different facial expressions were used within the expression discrimination tast of Experiment 5. Participants were asked to discriminate between neutral and angry facial expressions. If the lack of an interaction between familiarity type and discrimination of facial expressions is ascribable to the easy condition of the previous experiment, a facilitative effect of familiarity type for the discrimination of angry and neutral faces is expected in the present experiment.

Method

Participants. Twelve participants (9 women and 3 men, mean age = 24,8 years, aged between 17 and 37) took part in this experiment. They all had normal or corrected to normal vision. Participants received either course credit or 10,00 € for participation. As in the previous experiment handedness was not assessed, because electrophysiological data were not recorded.

Stimulus and apparatus. For Experiment 5 the same stimulus set was used as in the previous experiment. The size of the stimuli was kept for the initial learning phase and for the following experimental blocks. Again, all stimuli were presented on a 17-inch screen with the size of stimuli corresponding to the same visual angles as in Experiment 4. The viewing distance was kept constant at 100 cm. ERTS® served as experimental software for stimulus presentation and response recording.

Learning-procedure. Like in Experiment 4, participants had to undergo a 1 hour training session in order to become familiar with two sets of 20 people (half of them being male). In an initial learning block the participants viewed 80 portraits of 40 people with a happy expression and with a neutral or angry expression, respectively. All portraits were presented for 5000 ms in randomized order. The learning procedure including the test blocks was the same as in the previous experiment with the exception that all people were familiarized with a happy expression. In addition, one set was familarized with a neutral facial expression. The other set was familiarized with anger. As before, the distractor faces in the matching-to-sample task displayed always the same facial expression.

Design and data analysis. Participants continued with the experimental blocks after a short break of 10 minutes. In the subsequent experimental blocks it was asked for an expression discrimination where neutral and angry faces were now displayed. Except for the expression of happiness which was replaced by a neutral expression, the experimental procedure and design were the same as in Experiment 4.

Statistical analyses of RT and error rate were calculated with the same constraints as in the previous experiment.

Reaction time and error rates. The present experiment was conducted in order to exclude the possibility that the fairly fast RT, due to the easy discrimination in the previous experiment, might have been the reason for the lack of an interaction between facial familiarity and the discrimination of facial expression. As expected, mean RT was considerably slower in the present experiment when compared to the previous one (692 ms vs. 531 ms; t(11)=-4.3; p < 0.01). In Figure 22 it is evident, that portraits displaying an angry expression (639 ms) yielded faster RT (F(1,11)=34.4; p < 0.01) when compared to the neutral expression (729 ms). This result is in line with previous results showing that RT to expressive faces is faster than to neutral faces (Eastwood et al., 2001, Phillips et al., 1998). From an evolutionary perspective it is conceivable that it is important to recognize and react especially upon expressive faces since they may contain relevant information about an upcomming threat or a dangerous situation. There is a lot of evidence showing a more distributed neural network which is involved in the processing of expressive faces (Kesler/West et al., 2001; Sato et al., 2001). In addition, ERP results revealed that expressive faces are processed more rapidly within the brain (Eimer & Holmes, 2002).

Figure 22. Reaction time and error rates for the expression discrimination task of Experiment 5 separated for familiarity and expression.

Unfortunately, the expected effect of familiarity-type was not evident as a main effect nor an interaction (F < 1). No effect at all was present within error rates. Although the task of the present experiment was a lot harder when compared to the previous one, and response speed was fairly slow, no interaction emerged between the discrimination of facial expressions and facial familiarity. It can be supposed, that during the learning block, an extensive perceptual familiaritzation took place, which in return could have subserved an interaction. In contrast, the result of independency between the two processes are more underlined, because it is likely that participants had to use more featural information derived from the internal features in the fairly hard expression discrimination task of the present experiment. This information might even overlap with featural information which is relevant for perceiving identity.

From the present experiment it can be concluded that facial familiarity and the discrimination of facial expressions are independent processes for experimentally familiarized and unfamiliar faces. This also holds true in case of a harder task, which caused increased RT and possibly the increased use of information derived from internal facial features when compared to the easier task in the previous experiment. If an interaction between facial familiarity and the discrimination of facial expressions is not affected through perceptual familiarity, other features might be relevant for this effect. Possibly, semantic knowledge about a person may be sufficint to enhance the speed of expression discrimination. This can not be decided from these and previous results. Although a facilitative effect of familiarity was evident in Experiments 1 to 3, personally familiar faces comprise both semantic knowledge and personal importance. Both stimulus properties may be sufficient to subserve an interaction. Hence, the following experiment tries to solve this question by using famous faces, since they convey a lot of semantic information but are not necessarily of personal importance.

Experiment 6
Rationale

It has been shown in experiments 1 to 3 that facial familiarity can interact with the discrimination of facial expressions for personally familiar and unfamiliar faces. Personal familiarity facilitated the discrimination of facial expressions, especially for happy faces. The previous Experiments 4 and 5 contrasted these reuslts by showing no facilitation of expression discrimination for familiarized faces. It has been argued that the degree of familiarity, including the lack of semantic knowledge and personal importance, may have been the most obvious reason for the hypothesis to fail. Therefore, the present experiment will use famous faces since they already convey a lot of semantic knowledge. If pure perceptual familiarization is not sufficient to evoke an interaction of facial familiarity, and the discrimination of facial expressions, the question is raised whether this also holds for famous faces. Because of their higher degree of familiarity, the representation within the brain might be more distributed. Hence, they are processed differently to some degree. The basis for this assumption come from results by Paller et al. (1999; 2000). Participants had to view 40 faces. Half of them were presented with a name, and semantic information was given by a spoken voice. Completely new faces were added in a subsequent recognition test. ERP data revealed posterior differences between unnamed and new faces, which was interpreted as the retrieval of visual information. Most important, named faces elicted an additional anterior positivity when compared to new faces, which could be linked to the retrieval of semantic information. Thus, a small amount of semantic knowledge can alter the processing of previously learned faces when compared to completely unfamiliar ones.

Although the general neural processing of personally familiar and famous faces might be the same, it is conceivable that there are differences between them, because of a lack of direct encounter as well as the associated emotional importance which is not given for famous faces. Intracranial neurophysiological recordings show differences between these two facial types in the mesial and lateral temporal lobe and in the right amygdala (Seek, Mainwaring, Ives et al., 1993). These areas are linked to memory function as well as to emotional labeling (LeDoux, 1992), and might account for the differential processing of personally familiar faces on the one hand and famous faces on the other hand. Thus, similarities as well as differences in the processing of these face types are found. Therefore, the question is justifiable whether the observed interaction between facial familiarity and facial expressions for personally familiar faces also holds true when unfamiliar faces are contrasted with famous faces.

In addition, there are also differences in the processing of famous and unfamiliar faces. Already visual extrastriate areas are differently involved in the processing of both types of faces. Higher PET activation was found for unfamiliar versus familiar faces in the right fusiform gyrus and the right occipital inferior gyrus (George, Dolan, Fink et al., 1999; Gobbini, Leibenluft, Santiago et al., 2000). Evidence about differential processing of famous and unfamiliar faces within the brain also comes from ERP studies (Bentin, & Deouell, 2000; Eimer, 2000). Priming studies with parallel recorded ERPs suggest that these differences are already evident between 180 to 290 ms poststimulus (Begleiter, Porjesz, & Wang, 1995; Schweinberger, Pfütze, & Sommer, 1995). Schweinberger et al. (1995) proposed a face specific event-related component around 250 ms which emerges for repeated faces within a priming paradigm. For immediate repetitions this potential is higher in amplitude for famous faces when compared to unfamiliar ones. For unfamiliar faces the N250r component even disappeared with two to four intermediate items (Pfütze, Sommer, & Schweinberger, 2002). Converging results suggest that this potential “reflects the stimulus triggered access to stores facial representations in inferior temporal cortex as influenced by very recent visual experience” (Schweinberger et al., 2002, p 407). There are manifest processing differences between famous and unfamiliar faces which presumably lie in the availlability of a stored visual representation and of semantic knowledge. Referring to the purpose of this experiment, it is expected that these differences might also evoke a facilitation of the discrimination of facial expressions for famous faces.

For the present experiment a special stimulus set was created with german, international, and british celebrities. This experiment was planned in cooperation with J.Kaufmann at the University of Glasgow since the same experiment was to be conducted in Germany and Great Britain. The group of british celebrities was completely unfamiliar to the german participants. In a future experiment, to be conducted in Glasgow the german celebrities are thought to be completely unfamiliar to british participants. Hence, if an interaction between facial familiarity and the discrimination of facial expressions is found, this effect is expected to be reversed for the german and british celebrities in their respective country’s. For the present dissertation only the german experiment, conducted by myself, is described here.

Because of practical constraints and the accessibility of pictures, neutral and happy expressions were used. In contrast to personally familiar faces famous faces are emotionally less important, but the degree of percepual familiarity is high and semantic knowledge is available. The results from Experiments 1 to 3 point towards late perceptual processing stages which might show a facilitation for personally familiar faces. Possibly, not emotional value or attachment, but semantic knowledge only and therewith a more widespread neural distribution within the brain is the necessary pre-requisit for a facilitative effect. If so, it seems reasonable to expect the same facilitative effect for famous faces. Like in Experiment 1 to 3 this facilitative interaction should be evident in behavioural as well as in ERP data. As suggested by Experiment 2, a possible facilitation of the expression discrimination for famous faces should be reflected in a shorter peak latency of the P300 for familiar german and international celebrities when compared to unfamiliar british ones. Neutral and happy portraits may have a different effect when compared to the observed interaction between facial familiarity and facial expressions for personally familiar faces displaying happiness and disgust. Because both expressions are probably the expressions most encountered, the facilitative effect might generalize to both facial expressions. Hence, a pure main effect of familiarity within performance and ERP data is expected.

Method

Participants. Sixteen participants (12 women and 4 men, mean age = 25,8 years, aged between 19 and 42 years) took part in the present experiment. They received either course credit or payment for participation. The mean handedness score (Oldfield, 1971) was +75,9 (ranging from +25 to +100). All participants had normal or corrected to normal vision.

Stimuli and Apparatus. Pictures of British, German and International celebrities with neutral and happy facial expressions were used. Since it was planned to conduct this study in Germany and Scotland as a cross national comparison of the familiarity effect, celebrities had to be famous either only in Germany, or Scottland, or in both countries. Frontal to three-quarter views were allowed. A set of 197 celebrities was collected, each of them was represented with two portraits showing a neutral and a happy expression. Pictures were edited in Adobe Photoshop to 8-bit black and white pictues with 170 by 216 pixels and a black backround. In the pre-rating, as well as in the experiment, they were presented on a 17 inch screen with a resolution of 800 x 600 pixels. At a viewing distance of 100 cm this corresponds to a size of 6,8 by 8,6 cm and a visual angle of 3,9o horizontal and 4,9 o vertical. ERTS® served as experimental software for stimulus presentation and recording of behavioral responses.

Design and Procedure. In a two-choice reaction time task participants had to discriminate between neutral and happy facial expressions. In twelve consecutive blocks the whole stimulus set of 112 celebrities with 224 portraits, selected after a rating, was repeated three times. The possibility of participant determined breaks was given between blocks. Within each repetition of the stimulus set, the hand-to-key assignment was alternated once. At the beginning of the experiment and before every hand-to-key alternation participants performed a block of 24 practice trials. In each practice and experimental trial a fixation cross appeared in the middle of the screen for 1000 ms. Fixation was replaced by the stimulus which was presented for a maximum of 2000 ms or until the subject pressed the response key. For both, correct and incorrect responses, the next trial started after a fixed period of 1500 ms. In case of responses too early (under 100 ms) or too slow (above 2000 ms), feedback appeared for 800 ms after a blank screen of 500 ms. After the experimental blocks a familiarity rating was conducted. All 112 celebrities were presented to the participant with one picture at a time. The participants task was to rate the familiarity of each celebrity. No time limit was applied.

Electrophysiological recordings. The same experimental setup was used as in Experiments 2 and 3. The EEG was recorded from the same 31 electrode sites (see 2.2.2).

Offline the recording was cut into stimulus synchronized epochs of 2500 ms length (-500 ms prestimulus to 2000 ms poststimulus). Artefact treatment and ERP calculation were carried out with the same constraints as in Experiments 2 and 3. Response locked epochs were cut out of the stimulus locked epochs and averaged according to the point of time where the response occurred. An average reference was applied based on the same subset of 28 electrodes as in Experiments 2 and 3.

Data analysis. Statistical analysis of the behavioural data was performed by means of Huyhn-Feldt corrected repeated measures ANOVA including the within-variables familiarity (British, German, International) and expression (neutral vs. happy).

For peak latency measures of the P300 as well as onset latencies of the LRP, the jackknifing based method was used as outlined earlier. The F- and t-values of subsequent ANOVAs or two-tailed t-Tests were adjusted according to the method described by Miller et al. (1996), and by Ulrich and Miller (2001). Peak amplitude measures were derived from averaged event related potentials and ANOVAs were performed with the factors stimulus familiarity and expression. As in Experiments 2, and 3, mean amplitudes were derived from the LhEOG for a 100 ms interval around the S-LRP, and LRP-R onsets. Huyhn-Feldt corrected repeated measures ANOVAs including the factors familiarity
, and expression were calculated in order to assess possible effects on LRPs.

Topographical analysis was performed the same way as in Experiments 2 and 3. ANOVAs were calculated for amplitude measures and vector scaled data (McCarthy & Wood, 1985) for every 50 ms segment starting from 200 ms until 600 ms after stimulus onset. The within factors electrode site, familiarity
, and expression were included. In case of significant interactions of electrode site by familiarity post hoc ANOVAs were calculated including any two levels of this factor.

Rating

Before the Experiment a rating was conducted in order to choose a subset of the stimuli with the celebrities best representing the neutral and happy expression as well as being most familiar in either one or in both countries. In Germany as well as in Glasgow ten participants (11 women; mean age = 24,7 years, ranging from 19 to 32 years) took part in the rating. Participants had to rate each picture first on familiarity and on facial expression immediately afterwards.

Familiarity was rated on a 4-point scale (ranging from 0 to 3) with the categories: unfamiliar (0; “unbekannt”), slightly familiar (1; “kommt mir bekannt vor”), familiar but don’t know the name (2; “bekannt, weiß was, aber nicht den Namen”) and, know the name (3; “namentlich bekannt”). For the latter category it was supposed that when the participant knows the celebrity by name also other semantic knowledge is available (Burton, & Bruce, 1992; Hay, Young, & Ellis, 1991).

Facial expression was rated on a 3-category scale (ranging from 1 to 3) with the categories: neutral (1), happy (2; “eher fröhlich”) and, very happy (3; “sehr fröhlich”). An additional category “0” or none at all (“weder noch”) was added for pictures that did not represent any of the two facial expressions, neutral or happy. All of the 394 pictures were presented in randomized order on a computer screen. No time limit was applied for any of the ratings. On each trial a picture was presented at the top of the screen together with the familiarity rating scale. After the participant pressed the corresponding key for the rating choice with a left hand finger, the scale disappeared and the face remained on the screen for 500 ms. Afterwards, the facial expression rating scale was added to the picture. Then, the participant had to rate the same picture on facial expression with a right hand finger on four different keys on the computer keyboard. Visual presentation was terminated after the second key press corresponding to the expression rating. The next trial started after a blank screen of 1000 ms duration. The disconnection of the response hands was done to prevent participants from mixing up the successions of ratings which were accomplished on one picture in a single trial.

Mean ratings for each picture were calculated for both countries together (except for familiarity ratings) as well as for each country separately. To select the pictures a sequential strategy was used. The familiarity ratings served as a first criterion. The rating scale ranging from 0 to 3 provided the range limit of the mean value. British and german celebrities were selected if the mean value of the familiarity ratings for each picture was at least 1.6 by his/her own fellow countrymen and at the most 0.9 in the other country. Secondly, the expression ratings were considered. Here, the ratings of all participants from both countries were combined. The possible range of the mean values varied between 1 and 3. The lowest category (0; none at all) did not belong to the scale and was omitted for the calculation of the mean value for the expression ratings. Neutral pictures were accepted if a mean value of 1.4 was not exceeded. Happy pictures ought to have a minimum mean value of 1.5. The category ‘none at all’ (0) served as a final criterion on the expression rating scale. A maximum of 12 entries of a single picture was allowed. As the selection procedure was applied to a single picture a celebrity was only selected if both pictures fell within these criterions.

According to the procedure, a total of 112 celebrities (34 british, 37 german and 41 international) were selected, each represented by a neutral and a happy picture, respectively. The mean values of the familiarity and expression ratings are summarized in Table 2, and 3.

Table 2. Mean familiarity ratings for the stimulus set consisting of famous faces of Experiment 6 separated for the ratings in Scotland and Germany.

Ratings in Scotland

Ratings in Germany

Type of familiarity

Brit.

Ger.

Int.

Brit.

Ger.

Int.

Neutral pictures

2.48

0.23

2.72

0.23

2.53

2.38

Happy pictures

2.46

0.21

2.68

0.22

2.51

2.31

Combined

2.47

0.22

2.70

0.23

2.52

2.35

As can be seen in Table 2 the mean values of the familiarity ratings for british celebrities in Scotland, for german celebrities in Germany, and for international celebrities in both countries did not differ much. In addition, the mean values of the expression ratings (Table 3) for the three different groups of celebrities (British, German, and International) have only minor deviations as well.

Table 3. Mean expression ratings for the stimulus set consisting of famous faces of Experiment 6 separated for the ratings in Scotland and Germany.

Ratings in Scotland

Ratings in Germany

Ratings combined

Type of familiarity

Brit.

Ger.

Int.

Brit.

Ger.

Int.

Brit.

Ger.

Int.

Neutral pictures

1.06

1.05

1.05

1.11

1.15

1.13

1.09

1.10

1.09

Happy pictures

2.31

2.20

2.27

2.26

2.38

2.35

2.28

2.29

2.31

Results

Reaction time and error rates. Figure 23 summarizes the RT and error rates for the expression discrimination task of Experiment 6. RT to happy faces was faster when compared to neutral faces (590 ms vs. 614 ms; F(1,15) = 4.5, p = .5). No other effect in RT was observed. Only the factor familiarity had an effect on error rates (F(2,30) = 4.7, p< .05). When compared to British and International celebrities, error rates in the expression discrimination task were lowest for German celebrities.

Event-related potentials. Figure 24 displays the N170 component at the electrode sites P10 for all conditions split up after familiarity and expression. As is evident from the figures the component peaks at exactly 170 ms after stimulus onset for all conditions. Statistical analysis of the peak amplitudes, and peak latencies revealed no difference between conditions.

Figure 23. Reaction time and error rate for the expression discrimination task of Experiment 6 separated for familiarity and expression.

Figure 24. The N170 component for the expression discrimination task of Experiment 6 at the electrode site P10 separated for familiarity and expression.

The P300 component averaged according to the familiarity by expression
conditions is depicted in Figure 25. Calculation of the peak amplitude of the P300 component at the electrode site Pz revealed higher peak amplitudes for faces displaying happiness (M = 12.2 µV) when compared to the neutral expression (M = 10.9 µV; F(1,15) = 30.1, p < .001). There was no effect of facial familiarity (Fs < 1) nor an interaction of both factors (Fs < 1.6) for peak amplitude and peak latency of the P300 component.

Figure 25. The P300 component for the expression discrimination task of Experiment 6 at the electrode site Pz separated for familiarity and expression.

Figure 26 and 27 display the S-LRP and the LRP-R for all conditions, respectively. Both measures were not affected by horizontal eye movements, because there was no influence of experimental conditions on LhEOG (ps > .10). Despite small visible differences, jackknife based averages revealed no effect of any of the two factors expression nor familiarity. The same holds true for the response locked LRP.

Figure 26. The stimulus-locked LRP for the expression discrimination task of Experiment 6 separated for familiarity and expression.

Figure 27. The response-locked LRP for the expression discrimination task of Experiment 6 separated for familiarity and expression.

Analysis of the mean amplitude distribution over the scalp revealed a significant interaction of expression by electrode site starting from 200 ms until 550 ms (Fs(27,405) > 5.5, ps < .001) after stimulus onset (for all results see Appendix 6.3.). However, vector scaled data revealed topographical differences in the time intervals from 200 until 350 ms post-stimulus for the happy versus neutral expressions but not for later time intervals. Concerning the different amplitude distribution between the two expressions, difference maps (Figure 28) point to parieto-occipital and frontal electrode sites as containing the highest amplitude differences. Starting later from 350 ms after stimulus onset, the amplitude distribution depends also on familiarity (Fs(54,810) > 2.8, ps < .006). Different amplitude distributions are evident between british and german celebrities in the timerange from 450 until 550 ms after stimulus onset (Fs(27,405)>5.0, ps < .001) as well as between british and international celebrities from 350 to 400 and from 500 until 600 ms post-stimulus (Fs(27,405) > 3.9, ps < .009). No topographical differences for familiarity was found when comparing the vector scaled data of the three groups of celebrities.

Figure 28. Differences of the mean amplitude distribution between pairs of the British (B), German (G), and International celebrities (I; top row), as well as between the neutral expression (N) and happiness (H; bottom row) for the expression discrimination task of Experiment 6 in all tested time intervalls; a grey shading equals a negative difference.

Discussion

No facilitative interaction was found for famous faces and the discrimination of facial expressions. This is in contrast to the RT results of Experiments 1 to 3, where a facilitation was found for personally familiar faces displaying happiness. It was an unexpected result, since a facilitative effect for famous faces during an expression discrimination task was suggested by the results of other studies (Boudouin et al., 2000). The already mentioned differences between personally familiar and famous faces might possibly account for the absence of an interaction between facial familiarity and the discrimination of facial expressions in the present experiment. The most obvious differences between these faces are the lack of direct encounter and of associated emotional importance. The latter difference becomes obvious in studies using the SCR. Here, lower arousal levels are evident for famous faces when compared to personally familiar ones (Herzmann, Schweinberger, Sommer et al., in press; Tranel, Damasio, & Damasio, 1995). In addition, differences in the activation of the right amygdala were found for personally familiar and unfamiliar faces but not for famous and unfamiliar faces in studies using intracranial recordings (Seeck et al., 2001). Sugiura, Kawashira, Nakamura et al. (2001) presented corroborating results by showing differential PET activation of the amygdala, the hypothalamus, and the medial frontal gyrus for personally familiar and unfamiliar faces. These regions are linked to the behavioural significance of a recognized stimuli. It stands to reason that the lower emotional importance (or lower behavioural significance) of famous faces might explain the lack of an interaction between familiarity and the discrimination of facial expressions.

Another clue concerning the still unconfirmed hypothesis might be given by the different results concerning the amplitude and topographical distribution of the present and the previous experiments. For personally familiar versus unfamiliar faces (Experiment 2), topographical differences derived from vector scaled data were evident as early as 200 ms poststimulus. In contrast, in the present experiment different amplitude distributions for famous and unfamiliar faces started as late as 350 ms poststimulus. Most importantly, topographies remained the same for famous and unfamiliar faces in all tested time intervalls, because no significant effect was found for vector scaled data. Possibly, the discriminative information about familiarity within the present stimulus set is available later as for personally familiar and unfamiliar faces.

The only evident effect within the RT results were faster RTs for happy facial expressions when compared to neutral ones. That expressive faces are recognized faster within several paradigms is a finding often seen (e.g. Holmes, Vuilleumier, & Eimer, 2003; Läppänen et al., 2003). Other brain regions might be involved when recognizing facial expressions with emotional valence when compared to neutral faces (Kessler/West et al., 2001; Dolan, Fletcher, Morris et al., 1996). The topographical differences between the neutral and happy portraits starting at 200 ms poststimulus in the present experiment corroborate these results. In addition, emotionally valenced faces may boost attention mostly through the involvement of e.g. the amygdala (Adolphs 2002; Allison, Puce, & McCarthy, 2000; Whalen, Rauch, Etcoff et al., 1998) and hence, faster RTs can be observed. Another hint of exceeded saliency or attention which is captured by the expressive happy faces when compared to the neutral expression, is the increased P300 peak amplitude for happy faces measured at the Pz electrode. This is in line with quite a few results showing increased P300 amplitudes for happy versus neutral faces (Carretie, & Iglesias, 1995; Marinkovic, & Halgren, 1998; Herrmann et al., 2002; Vanderplog, Brown, & March, 1987) or emotional arousing versus neutral scenes (Cuthbert, Schupp, Bradley et al., 2000). All these results suggest that happy faces are processed differently from neutral ones in the present expression discrimination task. As mentioned earlier, these differences might emerge because of higher saliency and increased attention to happy faces which might be based on differential involvement of brain areas that subserve these processes.

To summarize, the present experiment revealed no interaction between facial familiarity and the discrimination of facial expressions for famous and unfamiliar faces. This was an unexpected result and suggests that emotional importance may be a necessary prerequisite for an interaction of both processes. Since emotional importance cannot be premised for famous faces in this experiment, but surely for personally familiar faces in the previous section the just found independence of both processes might be explainable. In addition, differences in amplitude distribution between famous and unfamiliar faces started later than in Experiment 2 and 3. This might suggest that differential information about familiarity is available fairly late and therefore, familiarity cannot act as a facilitative for the discrimination of facial expressions as found for personally familiar faces.

Summary and Discussion of Part I

In Part I the question was raised, whether there is a facilitative interaction between facial familiarity and the discrimination of facial expressions. Recent data observed within different paradigms suggest this possibility (Schweinberger & Soukup, 1998; Baudouin et al., 2000, 2000a). They contrast the assumptions that can be drawn of the functional model of face recognition by Bruce and Young (1986). However, studies which found a facilitative interaction between facial familiarity and the discrimination of facial expressions only used performance data (Baudouin et al., 2000, 2000a; Schweinberger & Soukup, 1998). Thus, questions arise concerning the functional and neural processes that are the basis of such an interaction. The experiments in Part I were conducted to replicate previous results with a simple two-choice RT paradigm. Participants had to discriminate between two facial expressions on successively presented portraits which depicted either familiar or unfamiliar faces. In addition, event-related potentials recorded in parallel should shed more light on the functional architecture of the processes involved.

The general hypothesis of the expression discrimination task expected that facial familiarity facilitates the discrimination of facial expressions especially when the discrimination of facial expressions is slowed down by a hard condition (c.f. Baudouin et al, 2000). To slow down the processing of facial expression recognition, the expressive intensity of the displayed expressions within the initial stimulus set (that is personally familiar vs. unfamiliar faces) was varied. Although normally the recognition of facial expession is very fast, an influence of familiarity might be possible in the hard condition and could facilitate the discrimination of facial expressions. Because expressive intensity did not affect an interaction between personal familiarity and the discrimination of facial expressions in an expected systematic way, this factor was not considered in the two other stimulus sets (experimentally familiarized vs. unfamiliar faces, and famous vs. unfamiliar faces).

In Experiment 1 participants had to discriminate the two facial expressions happiness and disgust on personally familiar and unfamiliar faces. The purpose was firstly to show a facilitation for personal familiarity on the discrimination of facial expressions within the stimulus set especially created for this experiment. Secondly, the possible explanation should be ruled out that the familiar faces used in the stimulus set were more expressive than the unfamiliar faces. This was accomplished by using an experimental group where all participants were personally familiar with half of the presented portraits. Here the facilitative effect of familiarity, mentioned above, was expected. In addition, in a control group - whose participants were unfamiliar with all presented portraits - no such effect was expected. The data of Experiment 1 suggested that the familiarity of a face facilitated the discrimination of facial expressions in general and especially when it displayed a happy expression. This effect was manifest in decreased RT and error rates for personally familiar faces when compared to unfamiliar ones. The variation of expressive intensity in order to slow down the perception of facial expressions and engender the facilitative effect of familiarity on this task, did not interact exclusively with familiarity. Contrary to the hypothesis, in the experimental group of Experiment 1 a strong facilitative effect of familiarity was found for happy faces with strong expressive intensity. Probably due to the mouth being closed in all portraits the task yielded fairly long RT in general. Therefore, the primary hypothesis of a facilitative familiarity effect only when the discrimination of facial expressions takes long enough, may also account for the condition with strong expressive faces. On the other hand, higher error rates in the weak expressive intensity condition in concert with increased variability of RT may have blurred the facilitative effect of familiarity on the expression discrimination in the experimental group. Although there have been some problematic effects in the control group of Experiment 1, it was outlined that they do not stand against the hypotheses, because differential effects of familiarity are not present within specific conditions in both groups. For the experimental group the effect of personal familiarity for happy faces is most pronounced within the strong expressive intensity condition. In contrast, for the controls this effect is completely absent within the same condition. Hence, an explanation based on differences in expressive intensity between personally familiar and unfamiliar faces is excluded. Therefore, the conclusion is justifiable, that the facilitative effect is based on facial familiarity and not just on differences in expressive intensity between familiar and unfamiliar faces. Along these lines the stimulus set was appraised as a basically sound one to use in the following two experiments.

In Experiment 2 participants performed the same expression discrimination task as in Experiment 1. In addition, event-related potentials were recorded in order to assess the functional architecture of the underlying processes. Although smaller, the same facilitative effect of personal familiarity on the discrimination of facial expressions was found as in the preceeding experiment. Independent of expressive intensity, performance was facilitated for personally familiar faces. Again, this effect is almost completely ascribable to the facilitative effect of happy familiar faces.

In line with RT results, the facilitative effect of personal familiarity on the discrimination of a happy expression was also present in event-related potentials. For the interval between LRP-onset and response, which represents the duration of motoric processes, no effect of personal familiarity was found. In addition, the peak latency of the N170 component was the same for personally familiar and unfamiliar faces. Hence the confluence of both processes on early perceptual and on motoric processing stages beyond hand selection is unlikely, because familiarity did not influence these two measures mentioned last. In contrast, the interval between stimulus and LRP-onset – indicating pre-motor processes respective response selection – was shorter for personally familiar faces when compared to unfamiliar ones, when the task of the participants was to discriminate between facial expressions. Although the same facilitative effect is more visible and numerically even bigger in the P300 peak latency, which represents the time needs of late perceptual processing stages, ANOVA only revealed a trend. This may be due to the overall small effect in RT. In addition, the fairly late peak latency of the P300 due to relatively slow RTs with high variability may have led to an increased variability of the elicitation point of the component. Hence the peak detection becomes less reliable through a broader apex. Due to less clear cut results concerning the P300 component, the response selection stage as an alternative facilitated cognitive process as indexed by the S-LRP results, cannot be ruled out. On the other hand, only if there is clearly no effect on the P300 peak latency could the response selection stage be safely adopted as the facilitated locus. Contrary, the numerical difference between happy familiar and unfamiliar faces is more pronounced within the P300 component when compared to the S-LRP interval. Therefore, the possibility of facilitated stimulus categorization time gains plausibility. Despite the reasons concerning the trend in the P300 peak latency and the numerically small effect in RT outlined above, I still consider late perceptual processes as the most likely functional process which is facilitated for personally familiar happy faces during the expression discrimination task.

In Experiment 3 it was attempted to decrease the RT variability by omitting the portraits with weak expressive intensity. In addition, a slightly changed design was introduced in order to have better control over stimulus familiarity. The SCR was recorded to personally familiar and unfamiliar faces. Preceeding the experimental blocks an additional block to refresh the memory was introduced. As expected, the facilitative effect of familiarity on RT was more pronounced than in Experiment 2. Unfortunately, no significant effect was observed in the ERP data although a numerical difference between happy familiar and unfamiliar faces was present for the P300 peak latency. Possibly, the reduced statistical power due to the omitted portraits with weak expressive intensity may have caused the lack of statistical significance.

The following Experiments 4 and 5 used only unfamiliar faces whereas half of the stimulus set was familiarized in a preceeding learning block. The advantage of using experimentally familiarized faces is to have a better control over familiarity and emotional expressiveness. In addition, the expression information which was encountered in the learning block was varied. Participants were familiarized with only one of the facial expressions which were presented in the following experimental blocks. Contrary to expectation no facilitative effect of familiarity on the discrimination of facial expressions was observed. There was also no advantage for the previously seen expression for familiarized persons.

To assess whether the lack of semantic knowledge might have been the reason for the unconfirmed hypothesis, Experiment 6 used a set of famous and unfamiliar faces. Again, no facilitative interaction between familiarity and facial expression discrimination was present within behavioural data and ERPs.

Before trying to interpret the observed facilitative interaction between personal familiarity and the discrimination of facial expressions, I have to discuss the unexpected lack of effects in Experiment 3. With this experiment it was attempted to overcome the problem of high RT variability in Experiment 2 by omitting the portraits with weak expressive intensity and to get a better control over personal familiarity. Accordingly, an increased facilitative effect of personal familiarity on the discrimination of facial expressions was observed in Experiment 3. Unfortunately, this was only present in the performance data of the experiment. The effect was not reflected in ERPs at all. However, a numerical but not significant difference between happy familiar faces and unfamiliar ones was present for the P300 peak latency. As mentioned, the intention to lower the RT variability was not completely accomplished. In addition, by ommiting the portraits with weak expressive intensity the number of trials was diminished and hence statistical power also. Using more trials is not the best alternative, because unfamiliar faces will become more familiar with more representations. In order to increase the degrees of freedom, more participants would have been necessary. The disconfirmed hypothesis within ERP data of Experiment 3 suggests, that the facilitative interaction between familiarity and the discrimination of facial expressions is rather transient and weak.

In an attempt to explain the observed facilitative effect of familiarity in the ERP data of Experiment 2 (that are also present in the performance data of Experiments 1 and 3), I will return to the results of the second experiment. How could this facilitative effect of personal familiarity on the late perceptual processing stage within the expression discrimination task be explained? Baudouin et al. (2000a) claim that the familiarity of a face increases the «fluency of the processing» and therefore improves the recognition of its expression. This interpretation of their results is rather unclear as it does not refer to any functional processing stage which should be facilitated. Possibly, Baudouin et al. (2000) assume perceptual processing stages to be facilitated for familiar faces when expression processing is slowed down by a hard condition. However, a strong body of results shows that early percepual processing stages, as indexed by the N170 component, are unaffected by facial familiarity and facial expressions (Eimer & Holmes, 2002; Herrmann et al. 2002). The present results point to late perceptual processing stages as a possible locus of facilitation for personally familiar faces. On the other hand, motor processes beyond hand selection and also early perceptual processing stages as indexed by the N170 component can be excluded as being facilitated. It was assumed, that the peak-latency of the P300 component represents the time needed for the perception of facial expressions. Although some results suggest, that the P300 latency is also influenced by response selection processes, it is only true when the task includes a response conflict (Leuthold & Sommer, 1998). This is not the case in the present task, making the latency to a pure perceptual measure for the task-relevant processing of facial expressions. Hence, based on present results, late perceptual processing stages are considered to be facilitated for personally familiar faces when participants discriminated facial expressions. An exception has been made in this sense, that it only holds true for the often seen facial expression of happiness. An explanation might be that, contrasting Bruce & Young (1986), the learned representation of a familiar face may depend on facial expressions that are encountered more often. For the stimulus set of Experiments 1 to 3 it can be assumed that personally familiar faces (lecturers of the University) are mostly encountered with neutral and happy expressions (when compared to disgust). Even if it is assumed, that the neutral expression is seen more often on personally familiar faces (Endo et al., 1992), happy expressions might share more overlapping features with a neutral expression than disgust. Hence representations might be more similar. In contrast, if it is not the characteristic of the stored representation of a face that might induce an interaction between facial familiarity and facial expressions, faster RT should also be present for familiar faces displaying disgust. This was clearly not the case. In their article Baudouin et al. (2000a) argue against the existence of expressive representations of familiar faces in the cognitive system. This is based on results with experimentally familiarized faces (Baudouin et al., 2000; Experiment 3). For faces that were familiarized with a happy in contrast to a neutral expression, the recognition of facial expressions independent of the presented expression (happy vs. neutral) was improved. This was not the case for faces which were familiarized with a neutral expression. However, recent data suggest, that at least the access to stored representations of familiar faces has an image specific influence. Increased priming effects are observed for same primes when compared to different portraits of the same person (Schweinberger et al., 2002; Jemel. Calabria, Delevenne et al., 2003). Admittedly, it can not clearly be decided by the data of Experiment 1 to 3, whether stored representations could affect an interaction between facial familiarity and the discrimination of facial expressions. The lack of an advantage within personally familiar faces displaying disgust indicates that stored representations of familiar faces may depend on the visual experience and hence, on the expression when a face is encountered. In this case, a learned representation of a familiar face might resemble the happy expression more and faster access might be possible. In return, faster recognition of identity might help facilitating the discrimination of facial expressions through the vast cognitive system.

The results of Experiments 4, 5, and 6 using experimentally familiarized or famous and unfamiliar faces, stand against an explanation which considers a possible influence of representations of familiar faces that are dependent on often encountered expressions. For the expression discrimination task of Experiments 5, and 6 no facilitation was found for expressions that were already seen on experimentally familiarized faces. For famous faces (Experiment 6) it can be supposed, that they are often encountered with a neutral or happy expression. Hence, a facilitation for famous faces should have been observed. This was clearly not the case. Contrary to the experiments which used personally familiar faces the results of Experiment 4 to 6 do not suggest that possible expression dependent stored representations are relevant for an interaction between facial familiarity and the discrimination of facial expressions. For the experimentally familiarized faces which were used in Experiments 4, and 5 it is likely that FRU-like stored representations for experimentally familiarized faces were built, because recognition was very good as error rates in the test blocks of the learning session decreased to under 3%.

Another possible explanation might be, that personally familiar faces possess a greater emotional valence and personal importance than unfamiliar and famous faces. There have been many results showing that emotionally valenced pictures experience a faster processing by the brain than their neutral counterparts (Eger, Jedynak, Iwaki et al., 2003; Adolphs, 2002, Eimer & Holmes, 2002; Sato et al., 2001). During a passive watching task, Cuthbert et al. (2000) found an earlier positive slow wave starting around 200 ms after stimulus onset at midline electrodes for pleasant pictures when compared to unpleasant or neutral pictures. After 400 ms both types of emotionally arousing pictures evoked greater positivity than neutral pictures. Because pleasant pictures also yielded an increased skin conductance response it was suggested by Cuthbert et al. (2000) that they show earlier and increased affective arousal. Although the present experiments only used expressive faces it might be of additional importance if the expression is posed by an unfamiliar or familiar person. Hence personally familiar expressive faces may also evoke an earlier and increased affective arousal than unfamiliar faces. Indeed, in Experiment 2 and 3 they showed an enhanced positive slow wave starting at 200 ms after stimulus onset at centro-parietal and inferior parietal sites. This effect of personal familiarity is also evident in the amplitude and topographical distributions which diverge significantly beyond 200 ms after stimulus onset. In addition, a slightly enhanced SCR response was evident for personally familiar faces in Experiment 3. Through the increased affective arousal of these faces the discrimination of facial expressions for personally familiar faces might be facilitated. Arousal may show a rather unspecific effect concerning the locus of interaction. In case of arousal modulation the affected locus might depend more on the temporal properties of involved processes and the task. By using simple visual and auditory stimuli of increased stimulus intensity results of Miller, Ulrich, and Rinkenauer(1999) pointed to shortened perceptual and pre-motoric processing stages, depending on the modality. Interestingly, no effect of stimulus intensity was observed on motor processes as indexed by the LRP-R. Motor processes were also unaffected in the present experiments, although the comparison between intensity of simple stimuli and an influence of arousal through complex stimuli like faces is questionable.

An effect of increased arousal may be modulated through the amygdala which is, inter alia, important for connecting faces to an emotional response (Rolls, 1999), and for directing attention to emotional valenced pictures (Krolak-Salmon et al., 2001; Vuilleumier, Armony, Driver et al., 2001; Breiter, Etcoff, & Whalen, 1996). It is important for learning conditioned responses and its rich connections are linked to brain areas which subserve the representation of reinforcer value - like the orbitofrontal cortex – or the response to emotional significant stimuli – like the anterior cingulate cortex (Cardinal, Parkinon, Hall et al., 2003). The amygdala was shown to be activated through facial expressions not only of fear but also of happiness (Whalen et al., 1998, Breiter et al., 1996). This shows, that the amygdala is also important for positively arousing pictures in general and for faces (Rolls, 1999). Thus it is not only relevant for pictures and faces with negative emotional valence or fear, as was thought previously (Morris, Friston, Buchel et al., 2001; Morris, DeBolis, & Dolan, 2001; Morris et al., 1998). Results of Gur, Schroeder, Turner et al. (2002) suggest, that the amygdala is not activated automatically but is dependent on the task. When using expressive and neutral faces, the amygdala was only activated during expression discrimination, but not during an age discrimination task. In contrast, Sugiura et al. (2001; see also Seeck et al., 2001) found the amygdala to be activated independently of the task during a familiar face detection task and a facial direction discrimination task for personally familiar faces when compared to unfamiliar faces. An expression discrimination task, or the posing of a facial expression per se may not be a prerequisite for amygdala activation. However, highly salient facial stimuli (as personally familiar faces are) may activate limbic structures and the amygdala. On the other hand, no activation of the amygdala was found for faces displaying disgust (Sprengelmeyer et al., 2003; Krolak-Salmon et al., 2001; Calder et al., 2001; Phillips et al., 1997). In addition, many results suggest, that the amygdala can modulate visual perception via back projections to the visual cortices (Adolphs, 2002; Armaral et al., siehe Anderson ea; Vuilleumier et al., 2002). Also the superior temporal sulcus receives projections from the amygdala (Allison et al., 2000), an area which is also important for the perception of facial expressions (Haxby et al., 2000, Narumoto et al., 2001), and identity (Rolls, 1999). Referring to the present Experiments 1 to 3, an amygdala activation is very likely for personally familiar faces displaying happiness. This might not be true for disgust, because the amygdala is not involved in the perception of this expression (see above). Hence, if an increased saliency and arousal level may affect e.g. the superior temporal area via projections from the amygdala, the exclusively found facilitation for personally familiar happy faces within the expression discrimination task might be explainable.

Although the amygdala may also be activated by personal familiarity for faces displaying disgust this expression might act differently on the neurocognitive system. As mentioned earlier, the perception of disgust involves different brain areas excluding the amygdala when compared to other expressions. The basal ganglia and insula are highly linked to the perception of this expression (Calder et al., 2001; Sprengelmeyer et al., 2003). In the ERP data of Experiments 2 and 3 an increased positive slow wave was also found for faces displaying disgust, when compared to the happy expression. If it is assumed that for these faces the affective arousal is increased in general, an increased response speed should be evident. Indeed, independent of familiarity shorter RTs were found for faces displaying disgust in Experiment 2. As to the value of disgust in order to prevent physical harm it might be important to react to this expression on short notice independently of facial familiarity. Therefore, an exclusive facilitation for personally familiar faces displaying disgust may not be found.

In the experiments using famous or experimentally familiarized and unfamiliar faces, no facilitation was found for familiarity on the expression discrimination task. Certainly, the visual familiarity for famous or familiarized faces might be comparable to personally familiar faces. Thus, if visual familiarity with specific facial expressions would be the important factor for a facilitative interaction such an effect should have been observed in Experiments 4 to 6. This was clearly not the case. As already mentioned, the observed interaction between facial familiarity and the discrimination of facial expressions for personally familiar faces as well as its absence for famous and familiarized faces stand against the agument of expression dependent representations of familiar faces. The lack of personal encounter and most probably of personal importance of famous and experimentally familiarized faces might be more important reasons when trying to explain the present results. If, as it might be the case for personally familiar faces, personal importance and hence, emotional arousal subserves an interaction via projections from the amygdala, the lack of an interaction for famous or experimentally familiarized faces is not surprising. In addition, when comparing the amplitude and topographical distributions for personally familiar or famous and unfamiliar faces, differences are evident concerning the effect of familiarity. Experiment 2 revealed differencial effects of personally familiar and unfamiliar faces for amplitudes and topographies as early as 200 ms poststimulus. This suggests, that the neural processing of personally familiar faces differs from the processing of unfamiliar faces. In Experiment 4, only different amplitude distributions emerged between famous and unfamiliar faces. In addition, they started as late as 300 ms poststimulus. No topographical difference was evident between both face types. The neural processing of famous and unfamiliar faces is probably more comparable. The observed late amplitude differences beyond 300 ms are possibly due to the recall of semantic knowledge about famous faces (Paller et al., 1999; 2000).

Based on the absence of a facilitative effect for famous and experimentally familiarized faces in Part I the question arises why an interaction was found elsewere (Baudouin et al., 2000, 2000a; Schweinberger and Soukup, 1998). The studies of Baudouin et al. (2000) and Schweinberger and Soukoup (1998) suffer from methodological problems. Concerning the asymmetric interaction between facial identity and facial expressions observed by Schweinberger and Soukup (1998), it has to be critically annotated that a small stimulus set was used. Facial expression and identity were only varied for two persons. It is possible that decreased RTs for the correlated condition when compared to the orthogonal one (see above) emerged only because expression discrimination could have relied on a picture based strategy. Due to different background shading or external facial features participants discrimination might have relied on these features instead on facial expression information (Kaufmann, 2002). Hence an interaction only emerged because both dimensions (facial expression and facial identity) were not varied independently as would be necessary for the Garner paradigm. In the study of Baudouin et al. (2000), an effect on RT was only found in Experiment 2. In the first experiment an interaction between facial familiarity and the discrimination of facial expressions emerged only for error rates. This shows that the interaction is transient and weak. In fact, in the easy condition of the expression discrimination task of Baudouin et al. (2000) famous faces were presented unconcealed and with a normal presentation time. Thus, it is comparable to the expression discrimination task of Experiment 6 which used also famous faces. As for the easy condition of Baudouin’s experiments, no facilitation of familiarity on the discrimination of facial expressions was found. This suggests, that an interaction between facial familiarity and the discrimination of facial expressions emerges under special circumstances only, like slowed down expression processing, or the presentation of personally familiar faces.

In summary, evidence has been given suggesting a facilitative interaction between facial familiarity and the discrimination of facial expressions. This contrasts the functional model of face recognition by Bruce and Young (1986), as well as many results favouring the independence of both processes. Importantly, the facilitative effect was only observed for personally familiar faces displaying happiness, but not for famous or experimentally familiarized faces or disgust. Although other studies found an interaction for unfamiliar or famous faces, it has been argued above, that these studies suffer from methodological problems. The facilitation for personally familiar faces was consistently found in behavioural data. However, event related potentials gave a more inconsistent picture. The ERP results of Experiment 2 suggested late perceptual processing stages – as indexed by the P300 peak latency - as the functional process which is facilitated for personally familiar faces displaying happiness. Unfortunately, this result was not confirmed by Experiment 3, although a nummerical difference was present within the P300 peak latency. Different explanations of the results have been discussed. It was ruled out by the results of Experiments 4 to 6 using familiarized and famous faces, that a facilitative effect could be caused by expression dependent neural representations of familiar faces. More likely an increased arousal due to personal importance may explain the facilitation of expression discrimination for personally familiar faces. The amygdala is possibly involved in subserving this effect since it is relevant for gating autonomic and conditioned responses to arousing and important stimuli. It also possesses rich connections to brain areas that are important for representing reinforcer value, and it plays a role in responding to emotional significant stimuli.

Since an interaction between facial familiarity and the discrimination of facial expressions could be shown within the constraints above mentioned the question arises, as to whether this interaction does also hold for the opposite direction – an interaction between facial expressions and the discrimination of familiar faces. In the following Part II this question is addressed by reversing the task. Participants have to discriminate the presented faces according to facial familiarity, whereas facial expression is varied independently.

Part II: Does facial expression affect the discrimination of facial familiarity?

The data presented in the previous chapter suggested a facilitative interaction for personally familiar faces and the discrimination of facial expressions. This does not contradict the assumption of different functional processing stages or neural brain regions which subserve both processes. However, it does not necessarily imply, that the opposite holds true – the interaction between facial expressions and the discrimination of facial familiarity. From everyday life it is conceivable, that facial expressions are very important cues for social interactions between familiar people. Expecially the happy expression is a special cue for familiar people when meeting with each other. In addition, emotions are more often expressed to familiar instead of unfamiliar people. Therefore, facial expressions might be a cue for familiarity.

As outlined already, the functional model of face recognition by Bruce and Young (1986) predicts an independency of the two processes in the direction just described. By using the Garner paradigm (see above), Schweinberger and Soukup (1998) only found evidence for an asymmetric interaction between facial familiarity and the discrimination of facial expressions but not in the opposite direction. On the other hand, the proposed model by Haxby et al. (2000), which is more related to the human neural system, allows for the possibility of an interaction between circumscribed brain structures and their cognitive functions. There is also data suggesting an opposite interaction between facial expressions and the discrimination of facial familiarity. People personally familiar were recognized faster as familiar displaying a neutral expression when compared to a happy one (Endo et al., 1992). On the other hand, the recognition of famous faces was also faster (Endo et al., 1992) and more accurate (Baudouin et al., 2000a) when showing a happy expression. In addition, a happy expression increases the rating of familiarity when compared to a condition in which the same face is presented with a neutral expression (Baudouin et al., 2000). There is also evidence, that changing of the expressive context of a face can modulate face specific processes in the medio-temporal cortex during short-term memory retrieval. Results of Guillaume and Tiberghien (2001) show an increased positive parietal slow wave during short-term recognition of faces, when the expression is changed from study to test. This indicates a more difficult classification of the face as ‘old’, even if the facial expression concerning the task is irrelevant.

The present chapter attempts to elucidate upon the question of whether an interaction is found between facial expressions and facial familiarity within a familiarity discrimination task. Considering the results from the previous chapter, the opposite interaction was only evident for personally familiar faces but not for famous or experimentally familiarized, and unfamiliar ones. If, as argued, increased affective arousal evoked by personally familiar faces may influence an interaction between facial familiarity and facial expressions, an interaction should be present in both directions. In addition it should only emerge for personally familiar faces. The temporal properties of both tasks should decide upon the functional locus of interaction. Thus the functional locus for both processes might be different from the proposed locus in the previous Part. In the following Experiment 7 the stimulus set with personally familiar and unfamiliar faces is used. In addition, Experiment 8 will use the set with experimentally familiarized and unfamiliar faces. Again, event-related components as temporal markers of the functional processes in question can help to pinpoint the process which is facilitated. A modified working model compared to the previous part is used. Facial expression processing is assumed to influence the discrimination of facial familiarity (Figure 29).

Within a familiarity discrimination task stored representations of faces certainly have to be accessed. If it is assumed that the stored representations of familiar faces are dependent on often encountered facial expressions, then there might be an advantage for such expressions (e.g. neutral faces or happy facial expressions). Thus it might be possible that an interdependency will also emerge for learned familiarized faces as well as for personally familiar ones. One correlat of familiar face perception within a priming paradigm is the N250r component mentioned earlier. No priming is conducted in the present paradigm, but the averaged waveforms include first and repeated presentations of the same person and also the same picture. Although not exactly comparable to the N250r (Schweinberger et al., 2002) the amplitude distribution for personally familiar and unfamiliar faces in Experiments 2, and 3 of the previous chapter clearly show effects in the time intervals that are typical for the N250r component. It was observed (e.g. in Experiment 2) that independently of each other, parieto-occipital negativity is enhanced for personally familiar and unfamiliar faces and for displayed disgust and happiness in the time intervals from 200 to 300 ms poststimulus. This amplitude effect in the present data is possibly a correlate of the same process that mediates the N250r component in a priming paradigm. In this case similar characteristics can be expected. This enhanced parieto-occipital negativity for personally familiar faces in Experiment 2 may reflect the accessability of the stored representation. The N250r component also emerges, although with reduced amplitude, for unfamiliar faces within a priming paradigm. Reduced parieto-occipital negativity was observed for unfamiliar faces in the same time intervals in Experiment 2. Recent results concerning the N250r suggest that perceptual factors may also play a role in the accessing of stored representations of familiar faces (Schweinberger et al., 2002). When different pictures of the same famous person were repeated a reduction was found. Hence it may not be surprising that facial expressions also affect the process that mediates the N250r.

Happiness may have a special role for the discrimination of familiarity, since it is the most universal social cue when aquainted people meet each other. According to this logic it can be predicted that a happy expression facilitates the decision for a face to be familiar. For the same reason it might be harder and more error prone to decide for unfamiliarity when an unfamiliar happy face is presented.

According to the findings mentioned above for the following experiments it is hypothesized, that facial expression affects the discrimination of facial familiarity. The highly familiar happy and neutral expressions might facilitate the decision for a face as being familiar. In addition, the decision for unfamiliarity might be harder when unfamiliar faces display these two expressions. Again, error rates should behave in the same way, being lower in case of a facilitated response.

Experiment 7
Rationale

In Experiment 7 the question is raised whether the observed interaction of personal familiarity onto the discrimination of facial expressions also holds true for an interaction in the opposite direction, that is, an effect of facial expression on the discrimination of facial familiarity for personally familiar and unfamiliar faces. Because the perception of facial expressions is a fast process and does not need to be slowed down by a hard condition in order to interfere in the functional processing streem, portraits with weak expressive intensity were omitted. Instead portraits with a neutral expression were added. Participants had to perform a speeded familiarity discrimination task in which all presented people displayed happiness, disgust, or a neutral expression in randomized order. It is expected that the frequently seen happy expression will facilitate the decision for familiarity (also see Endo et al. 1992). The neutral expression may also shed some light on the special role of the happy expression that was evident in all previous experiments. A neutral expression is probably the most frequently seen expression on personally familiar faces, but also on unfamiliar faces in general. If the pronounced facilitative effect of happiness is based on the frequency with which this expression is encountered on the face of a personally familiar person, comparable effects are expected for personally familiar portraits with happy and neutral expressions when compared to disgust. If the expressiveness resp. emotional arousal of a face is the crucial factor, the facilitation should only be evident for personally familiar faces displaying happiness, but not for a neutral expression. However, an expression of disgust may disturb the perception of familiarity, either because it is rarely encountered on a familiar face or because it may disturb the configuration of the internal features as a happy and certainly as a neutral expression more. In this case a facilitative effect is only expected for familiar happy faces.

Method

Participants. Twenty participants (all women; mean age = 25,0 years; aged between 20 and 34) took part in Experiment 7. They were personally familiar with half of the presented persons displayed in the experiment. Participants received either course credit or a payment of 12 €. The mean handedness score was 75 (ranging from –83 to +100; Oldfiled, 1971).

Design and Procedure. In the present experiment the same stimulus set was used as in Experiments 1 to 3 with the exception that portraits with weak expressive intensity were omitted and portraits with a neutral expression were added. Hence 2 x 3 different conditions arose: familiar and unfamiliar happy faces displaying a neutral expression, happiness, and disgust. In a two-choice reaction time task participants had to discriminate between whether the presented face was personally familiar or not. All 28 people of the stimulus set were presented with happiness, disgust, and a neutral expression in three different perspectives. Trials were presented in randomized order with the same trial scheme as in the previous experiments. The stimulus set was repeated twice, whereas, after a single repetition, the hand-to-key assignment was changed in order to calculate an LRP for all conditions. The order of the key assignment was counterbalanced across participants. Before the experimental blocks participants viewed all 28 persons, who were included in the stimulus set, with a neutral expression and made a verbal response about personal familiarity. This was done to avoid errors, because people in the stimulus set were displayed repeatedly. In a pilot study one participant happened to classify an unfamiliar person as familiar, and error rates increased extremely due to the repetitions of the people.

Electrophysiological recordings. In the present experiment the same electrode setup was used as in all previous experiments, which recorded event-related potentials. Furthermore, electrophysiological data were treated the same way as in the previous experiment and averaged to the above mentioned six conditions.

Data analysis. For statistical analysis the same tests and procedures were used as in the previous experiments, with the exception that the within subject factor expressive intensity was omitted. The factor expression contained three levels (happy, disgust, and neutral). Analysis of the S-LRP, and LRP-R onsets, the LhEOG as well as of the peak amplitudes for the N170, and P300 components were also the same as in the previous experiments. Peak latency measures of the N170 and P300 components were analyzed with a repeated measure ANOVA including both within subject factors mentioned above. The latency values of the P300 component were based on jackknifing averages. The F-values were corrected according to the equation Fc = F/(n-1)2 (Ulrich, & Miller, 2001).

Figure 30. Reaction time and error rates for the familiarity discrimination task of Experiment 7.

Results

Reaction time and error percentage. As can be seen in Figure 30 the classification for a portrait as familiar was faster than for unfamiliar portraits (559 ms vs. 609 ms; F(1,19) = 72.2, p < .01). In addition, the expression of a portrait also affected RT (F(2,38) = 4.8, p <.05). Post hoc comparisons revealed a trend for RTs on portraits with neutral expressions being slightly faster when compared to the expression of disgust (580 ms vs. 590 ms; t(19) = 2.4, p < .07). Most importantly, the classification of familiarity was affected by facial expression (F(2,38) = 5.1, p = 0.011). The classification of a portrait as familiar was facilitated when the face displayed either a happy (t(19) = -4.5, p < .01) or a neutral expression (t(19) = -3.1, p < .05) when compared to disgust. No effect of expression was found for unfamiliar faces.

The mean error percentage of 3.9 was fairly low. There were no effects in error percentage concerning the factors familiarity nor expression.

Event related potentials. The N170 component is most pronounced at the electrode site P10 (Figure 31). As is obvious in the figure, there is no difference in peak latency between conditions. Although based on small differences, there was an effect of familiarity (F(1,19) = 9.2, p < .01), and of expression (F(1,19) = 12.3, p< .01) on the peak amplitude of the N170 component. The mean amplitude for familiar faces was 0.4 µV larger when compared to unfamiliar faces. Furthermore, a smaller amplitude arose from faces with a neutral expression (-5.8 µV) when compared to faces expressing happiness (-6.2 µV; t(19) = -3.5, p < .01) or disgust (-6.3 µV; t(19) = -4.3, p = .01).

Figure 31. The N170 component for the familiarity discrimination task of Experiment 7 at the electrode site P10 separated for expression and for familiarity.

The P300 component was most prominent at the electrode site Pz (Figure 32). To all appearances there is quite a difference in peak latency between familiar (519 ms) and unfamiliar (575 ms) faces with the first ones peaking earlier than the latter ones. This was confirmed by a repeated measures ANOVA based on jackknifing averages (Fc(1,19) = 35.9, p < .01). There was no effect of expression on peak latency (Fc = 1.4) nor an interaction (Fc < 1). Familiarity also had a strong effect on peak amplitude (F(1,19) = 50.6, p < .01) yielding higher amplitudes for familiar (M = 11.6 µV) when compared to unfamiliar faces (M = 8.7 µV).

Figure 32. The P300 component for the familiarity discrimination task of Experiment 7 at the electrode site Pz separated for expression and separate graphs for familiarity.

Figure 33 displays the S-LRP for personally familiar and unfamiliar faces averaged for the different expressions. The expected difference within personally familiar faces for an earlier onset of happiness (M = 329 ms) when compared to disgust (M = 362 ms) was confirmed by a one-tailed t-test with jackknifing averages (tJ(19) = 1.75, p < .05). No differences were found between expressions for unfamiliar faces (tJ < 1).

Figure 33. The stimulus-locked LRP for the familiarity discrimination task of Experiment 7 separated for expression and separate graphs for familiarity.

Figure 34. The response-locked LRP for the familiarity discrimination task of Experiment 7 separated for expression and for familiarity.

There was no effect of facial expression in the interval between LRP-onset and response for personally familiar nor for unfamiliar faces (Figure 34). In addition, experimental conditions did not affect the LhEOG, which was calculated for stimulus-, and for response-locked epochs (ps > .10). Hence, an influence of horizontal eye movements on the S-LRP, and the LRP-R could be denied.

The statistical analysis of the mean amplitude distribution (Figure 35, Appendix 6.4.) of consecutive time intervals starting from 200 ms until 600 ms after stimulus onset revealed differences within facial familiarity (Fs(27,513) > 8.5, ps < .001), and for facial expression(Fs(54,1026) > 5.5, ps < .001) for all intervals tested (for all results see Appendix 6.4.). In addition, over all intervals starting from 200 ms poststimulus the calculation of vector scaled data revealed topographical differences for facial familiarity (Fs(27,513) > 3.5, ps < .007), as well as for facial expression (Fs(27,513) > 5.9, ps < .001). Because the factor expression included 3 levels, additional post-hoc analysis were calculated for any two of the expressions. Topographical differences between happiness and disgust started at 300 ms poststimulus (Fs(27,513) > 3.7, ps < .006). When comparing happiness or disgust to the neutral expression, topographical differences started early from 200 ms post-stimulus and remained significant (for exceptions see Appendix 6.4.) to the end of the tested intervals.

Figure 35. Differences of the mean amplitude distribution between pairs of happiness (H), disgust (D), and the neutral expression (N) as well as between unfamiliar (UF) and familiar faces (F; bottom row) for the familiarity discrimination task of Experiment 7 in all tested time intervalls; a grey shading equals a negative difference.

Discussion

As expected, the categorization for a face as being personally familiar yielded faster RTs than the categorization as unfamiliar. This is a result often found for tasks involving a familiarity discrimination (Endo et al., 1992; Phillips et al., 1998). Most important, facial expression affected the recognition of personal familiarity in the respect that the frequently seen happy and neutral expressions facilitated the decision for a face as being personally familiar when compared to disgust. No such effect was found for unfamiliar faces. Complementing the results of Part I, which pointed to an interaction between personal familiarity and the discrimination of facial expressions, the present experiment revealed an interaction in the opposite direction or personally familiar faces. Happy or neutral facial expressions facilitated the discrimination of a face as being personally familiar. A note of caution should be added. As mentioned before, all persons were presented with a neutral expression in advance of the experimental blocks, in order to exclude systematic errors based on classifying a person as familiar or unfamiliar by mistake. Hence neutral pictures of all people might have been primed, and interpretations which rely on the results of these pictures have to be carried out with caution.

In line with the RT results an erlier S-LRP onset was found exclusively for happy familiar faces when compared to familiar faces displaying disgust. No differences were found between personally familiar faces displaying happines and disgust when considering the LRP-R onset as well as the N170 or P300 peak latency. Together, the results clearly point to the response selection stage as indexed by the S-LRP which is facilitated for happy personally familiar faces within a familiarity discrimination task.

The results are in line with other findings of e.g. Endo et al. (1992), who found faster RT in a familiarity discrimination task for familiar faces when they expressed happiness or a neutral expression. Admittedly, for personally familiar faces they found decreased RT only for the neutral expression. Due to the mouth being closed in all pictures, it is likely that the participants in the present experiment were also familiar with this rather weak happy expression of all personally familiar faces. Therefore, the findings of Endo et al. (1992) concerning personally familiar faces might not stand up against the present results.

The expressions might have disturbed the spatial arrangement of the internal features to a different degree. It is conceivable that this is not true for the neutral expression, and only partly true for happiness, but mainly true for disgust. Hence, familiar faces should be most easily recognized with a neutral or a happy expression, whereas disgust should disturb the recognition the most. The lack of a difference between the neutral and happy expression is also explainable within the present stimulus set since happiness was expressed with the mouth being closed. This makes it more comparable to a neutral expression.

The response selection stage as the locus of interaction stands in contrast to the late perceptual processing stage, which was considered as the possible functional locus of interaction for the expression discrimination task in Experiment 2. Other processes may subserve the interaction between facial expressions and facial familiarity within a familiarity discrimination task. Since a happy face is the most common gesture between aquainted people, it is probable that the facilitation is not due to personal familiarity. It may depend on a strong stimulus-response learning between a happy facial cue and the feeling of familiarity. Hence the response selection stage as the facilitated process is the most likely to be expected. However, the facilitation for personally familiar faces with a neutral expression did not fit into this reasoning. Possibly, the initial pre-exposure of the neutral faces had a much stronger effect than expected. Thus, to proof this reasoning, the same facilitative effect of happiness displayed on familiar faces should also emerge for experimentally familiarized faces. If, on the other hand, an unfamiliar face displays a happy expression, the decision for unfamiliarity should be more error prone.

An additional conclusion can be drawn from the data concerning the locus of facilitation for the familiarity decision. It is a result often found, that famous or experimentally familiarized faces are categorized faster in an identity discrimination task than unfamiliar ones (Phillips et al., 1998). To my knowledge it has been never addressed on which functional processing stage this advantage for familiar faces is located. As cited above, there is numerous evidence that the peak latency of the N170 component is not influenced by familiarity. Therefore, early visual perception may not be faster depending on facial familiarity. Studies which compared amplitudes and topographies from event-related potentials found differences between familiar and unfamiliar faces starting around 300 ms after stimulus onset (Paller et al., 2000; Endl, Walla, Lindinger et al., 1998). In the present experiment a clear effect in P300 peak latencies was found with an earlier peak for personally familiar faces. This difference is still evident in the later response selection stage (S-LRP), indicating a propagation of the effect which is already present in the P300 peak latency. No other processing stage was influenced by familiarity. Hence the results clearly point to late perceptual processing stages that are faster for familiar faces within a familiarity decision.

Taken together, the results of Experiment 7 point towards an interaction between facial expressions and the recognition of familiarity for personally familiar faces only. In addition, the effect of facial familiarity on S-LRP onset points to the response selection stage as the functional locus of the facilitation found for happy and neutral expressions. However, the results do not clearly elucidate on whether the found interaction between facial expressions and the discrimination of facial familiarity is due to personal importance of the familiar faces, or because of the mere perceptual familiarity which may be stronger for the often seen neutral and happy expression. A third explanation might be the special status a happy expression has as a social cue when aquainted people meet each other. It is necessary to have better control over the learning experience concerning facial expressions, when a face becomes familiarized. Therefore, a last experiment is carried out using unfamiliar faces, where half of the faces were familiarized in a learning block with different facial expressions.

Experiment 8
Rationale

In the present experiment the question was raised, whether the found interaction between facial expressions and the discrimination of personally familiar faces also holds true for experimentally familiarized faces. Assuming that the representations of familiar faces also relies on perceptual factors like facial expression when the face is familiarized, appearance of the same expression should facilitate the recognition of a familiar face. Whereas, if personal significance and arousal for familiar faces is crucial for an interaction of both processes, an interaction should not emerge between both processes.

The stimulus material in the present experiment consisted of initially unfamiliar faces. Half of the faces were familiarized in a learning block always with a neutral expression, and either a happy, or angry facial expression. For the following experimental block an equal amount of unfamiliar faces was added. Participants had to discriminate familiarized from unfamiliar faces. Faces were presented in randomized order either with a happy or angry facial expression. Importantly, for half of the familiarized faces the angry expression was never encountered, for the other half of faces the happy expression was not previously shown.

It is the main hypothesis that the facial expression which was encountered during familiarization in the learning block would facilitate the decision for a face as being familiar. This should be reflected in reduced RT and error rates for these conditions but not for the conditions with the facial expressions which were not encountered in the learning block.

Method

Participants. Twelve participants (9 women and 3 men, mean age = 26,6 years, aged between 20 and 39) took part in the present experiment. All participants had normal or corrected to normal vision. They received either course credits or 10,00 € for participation. Handedness was not determined, because electrophysiological data were not recorded.

Stimulus and apparatus. For the present experiment the same stimulus set was used as in Experiment 4 and 5. The same size of the stimuli was used for the initial learning phase and for the following experimental blocks respectively. Again, all stimuli were presented on a 17-inch screen. The viewing distance was kept constant at 1m. ERTS® served as experimental software for stimulus presentation and response recording.

Learning-procedure. Participants had to undergo a 1 hour training session in order to become familiar with two sets of 20 people (half of them being male). The learning procedure was exactly the same as in Experiments 4. The presented facial expressions were the same. That is, one set of 20 people was familiarized with a neutral and happy expression, whereas the other set of 20 people was presented with neutral and angry expressions. The familiarized sets were counterbalanced across participants as described above.

Design and data analysis. Participants continued with the experimental blocks after a short break of about 10 minutes. They were then required to discriminate between whether the displayed portrait belonged to one of the 40 familiarized people or not by pressing a corresponding key on the right or left. All persons presented were displayed with the two facial expressions happiness, and anger. The whole stimulus set was presented in randomized order and repeated twice. The trial schema was the same as in all previous experiments. Participants had to perform 6 blocks of trials. In case of too slow (> 2000 ms) or incorrect responses feedback was provided after the trial by showing the words “Zu langsam!” (Too slow!) or “Falsch!” (Wrong!) for 500 ms on the screen. A summarized feedback for the block was provided during the breaks. At the beginning and in the middle of the experimental blocks the hand to key assignment was changed. In order to learn the key assignment, participants performed 40 practice trials by pressing the corresponding key according to the words “gelernt” (learned) or “ungelernt” (unlearned) presented on the screen.

Statistical analyses of RT and error rates was the same as in Experiments 4 and 5 with the exception that now the factor expression contained 3 levels (neutral, happy, and angry). In addition, the within-factor familiarity type was used. It contained the levels ‘familiar person with learned expression’, ‘familiar person with unlearned expression’, and ‘unfamiliar person’.

Results and Discussion

Familiarization. Error rates for the test-blocks of the learning procedure decreased from 11.0% over 4.7% (t(11) = 5.1, p < .001) to 3.5% for the third block.

Reaction time and error percent. Figure 36 displays the RT and error rate of Experiment 8. No effect at all emerged for the error rates in the present experiment. The RT strongly depends on the familiarity type (F(2,22)=15.3, p < .01). Post-hoc tests revealed faster RTs for faces that were familiarized in the previous learning block with (M = 608 ms) or without the displayed expression (M = 639 ms) when compared to unfamiliarized faces (M = 669 ms; ts(11) > -5.2, ps < .01). Although visible in Figure 36 and numerically evident, no statistical difference was found between the two types of familiarized faces (t(11) = -2.2, p = .15). Hence the main hypothesis of the present experiment was not confirmed. The discrimination of a face as being familiar was independent of the expression that was encountered during familiarization. However, the numerical difference between familiarized faces with learned expressions and familiarized faces with unlearned expressions lead towards a view that the perceptual experience with which a person is familiarized may play at least a minor role concerning the representations that are built up.

Figure 36. Reaction time and error rates for the familiarity discrimination task of Experiment 8 separated for expression and familiarity.

Summary and Discussion of Part II

In Part II the question was raised, whether emotional facial expressions affect the recognition of facial familiarity. In Part I it has been shown that the opposite also holds true, but appears to depend strongly on the degree of familiarity. Familiarity facilitated the discrimination of facial expressions only for personally familiar faces but not for famous or experimentally familiarized faces. In addition, this effect was specific for facial expressions, since it was only evident for personally familiar faces displaying happiness. Accordingly, facial expression affected the discrimination of facial familiarity. It was supposed that the expression of happiness again has a special role for the interaction between the processes in question. Possible reasons might be the increased frequency of happiness encountered on familiar faces, the special role a happy expression has as a social cue for familiarity, or the emotional arousal a happy face might evoke.

Two experiments were conducted using different stimulus sets with personally familiar, or experimentally familiarized, and unfamiliar faces. It was the main hypothesis, that the frequently encountered happy expression facilitates the discrimination of a face as personally familiar. In addition, a similar effect was expected for the experimentally familiarized faces with the objection that this time the effect could be modulated by the perceptual experience during the learning block.

For personally familiar faces it was shown that facial expression influenced the discrimination of facial familiarity. Personally familiar faces were recognized faster as familiar with the often seen neutral and happy expressions than when displaying disgust. No such effect was evident for unfamiliar faces. Event-related potentials clearly point to the response selection stage, which is facilitated for personally familiar faces displaying happiness when compared to disgust. It again has to be noted that in Experiment 7 all familiar and unfamiliar persons were presented with a neutral expression once before the experimental blocks in order to prevent participants from making systematic errors (see 3.2.2.). Therefore, the neutral condition was omitted from statistical analysis of ERPs. In contrast to the results of Experiment 7, no interaction between facial expressions and the discrimination of experimentally familiarized faces was found in Experiment 8. The decision for a face as being familiar was independent of the facial expression which had been encountered during the learning block.

According to accepted common sense it is not surprising to find the so called ‘smiling bias’ (Endo et al., 1992). That is, shorter response latencies for happy familiar faces. In everyday life displaying happiness is the most common gesture when someone familiar approaches us. The present results are in agreement with previous findings showing that familiar face recognition is facilitated by a happy expression (Baudouin et al., 2000; Endo et al., 1992). Baudouin et al. (2000) only found an interaction between facial expressions (happy vs. neutral) and a familiarity decision for their accuracy data and familiarity ratings but not for RT. Compared to a neutral expression more errors were made for happy unfamiliar faces. Familiarity ratings were increased for happy faces in general. The results of the present Experiment 7 extend these findings to response latencies but they show no effect on error percentage and the discrimination of unfamiliar faces. This might be due to lower error rates within the present data (under 5% when compared to the over 20% that Baudouin et al., 2000 found). In their article Baudouin et al. (2000) argue, that the effects found for both, familiar and unfamiliar, faces imply the decision stage as a common level of interaction. Based on the model of Bruce and Young (1986), the cognitive system gets two concurrent inputs from the face recognition unit and the facial expression analysis. The “smiling bias” is observable “when a strong smiling input is concurrent with a weak familiar input” (Baudouin et al, 2000a, pp 291). On the one hand, the results of Experiment 7 support this interpretation with the locus of confluence found in the response selection stage as indexed by the S-LRP. On the other hand it can be supposed that the ‘familiar input’ for personally familiar faces was strong in the familiarity discrimination task. The interpretation cited above does not apply to the present data, since an effect was still found for personally familiar faces. In addition, results of Experiment 7 revealed a strong difference in RTs between happy and neutral familiar faces and faces displaying disgust. Hence, the alternative explanation is excluded, that an emotional expression induces a bias per se, because people may feel more inclined to express their emotional state to familiar people (Baudouin et al., 2000). In this case, a facilitation of familiar face discrimination should have emerged for faces displaying happiness and disgust, but not for the neutral expression. This was not the case. Unfortunately, the interpretation based on the trials with neutral expressions has to be taken with caution, since all persons were displayed with a neutral expression before the experimental blocks in Experiment 7. The advantage for personally familiar faces with a neutral expression might be explained by the pre-exposure which may be comparable to a priming effect (Begleiter et al., 1995). This argument is reduced slightly by the fact, that also unfamiliar faces were pre-exposed and hence, a facilitation for the neutral expression should have emerged for unfamiliar faces also. However, decreased priming effects for unfamiliar faces when compared to familiar ones are often found (Jemel et al., in press; Schweinberger et al., 1995). Therefore, the pre-exposure of neutral faces might have primed the discrimination of unfamiliar neutral faces to a lesser extend.

In the previous Part I it was argued that the interaction between facial familiarity and the discrimination of facial expressions could be subserved by increased arousal elicited by personally familiar faces. The same interpretation might also account for the familiarity discrimination task of Experiment 7. The observed facilitation for the familiarity discrimination of happy and neutral personally familiar faces might be due to increased arousal for these faces when compared to unfamiliar ones. No interaction between facial expressions and the discrimination of facial familiarity was present in Experiment 8 using experimentally familiarized faces. No increased arousal is expected here for the stimulus set of unfamiliar faces. This argument is not negated by ERP results which pointed to a different locus of interaction for the two tasks of Part I and II. Late perceptual processing stages were facilitated for the expression discrimination task in Part I. The present data of Part II clearly point to the response selection stage as being facilitated for the familiarity discrimination task. If arousal processes are the basis for an interaction between facial expressions and the discrimination of facial familiarity the locus may to a large extend depend on the temporal properties of the processes involved and the task in general. The familiarity discrimination task of Experiment 7 yielded faster RT when compared to the expression discrimination of Experiment 2 (583 ms vs. 693 ms; t(34) = 4.2, p < .01). For this particular stimulus set of personally familiar and unfamiliar faces it can be assumed, that the perception of facial expressions takes longer when compared to the discrimination of facial familiarity. Hence, the facilitative effect of facial expressions on the discrimination of personally familiar faces emerged in a later processing stage (the response selection stage) because expression information is available late. In contrast, the facilitative effect of facial familiarity on the discrimination of facial expressions was observed at an earlier stage (late perceptual processing) because the information about facial familiarity might be available earlier.

Another explanation suggests, that the frequency with which a facial expression is seen on familiar faces could explain the observed facilitative effect. The representations of familiar faces might not be image-independent as it is supposed by the functional model of face recognition by Bruce and Young (1986). The facilitative effect of happy (and neutral) faces in Experiment 7 might just emerge because familiar faces are most frequently seen with a happy (and also a neutral) expression. In return, stored representations might depend on these expressions. However, a proof of this interpretation would have been a comparable effect for the expression with which faces were familiarized in Experiment 8. This was not the case, although a insignificant numerical difference between familiarized faces with and without a learned expression pointed into this direction. It can, however, be assumed that the perceptual familiarity for the familiarized faces in Experiment 8 was high because error rates in the learning session were fairly low and decreased from 11 % to 3.5 % from the first to the third matching-to-sample block. For this reason, expression dependent representations of familiar faces can be ruled out as an explanation.

The facilitated familiarity discrimination for personally familiar faces displaying a happy or a neutral expression may be based on the increasing importance of internal facial features for familiar faces (Ellis et al., 1979; Young et al., 1985). Although it has been shown that perceptual information which is used for the recognition of identity and expression differs to a large degree, there is also an overlap (Calder et al., 2001) which could explain an interaction between both processes. Accordingly, only personal familiarity leads to increased reliance on internal facial information which is important for both the discrimination of familiarity and of expression. In addition, the discrimination of expression largely relies on internal facial features. Hence, an interaction between facial expressions and the discrimination of facial familiarity would only be expected for highly familiar faces but not for faces with a low degree of familiarity or unfamiliar faces. For the type of faces first mentioned it might be, that internal facial features attract more attention when compared to unfamiliar faces. In contrast, the discrimination of unfamiliar faces might have relied more on external features which makes an interaction between facial expressions and the discrimination of facial familiarity less possible. This may go some way to explaining why Schweinberger and Soukup (1998) failed to show an interaction between facial expressions and the discrimination of identity (see 1.2.3.), since they only used unfamiliar faces within the Garner paradigm. The increased variation in the irrelevant expression dimension in the orthogonal condition did not affect the discrimination of identity at all, because participants may have relied on external facial features. The strategy of the participants might have been to ignore the internal facial features including the facial expression. Thus the lack of effect is not surprising.

Another important point might be the distortion of internal facial features through particular facial expressions. Distortion might be higher for expressions of disgust than for happiness, and might be completely absent for the neutral expression. In fact, happiness is the most easily recognized expression. This is supported by lower error rates for its recognition (Calder et al., 2001; Ekman, Friesen, & Ellsworth, 1972) and often found decreased RT (Leppänen et al., 2003; Kiria & Endo, 1995; Hugdahl et al., 1993). If happiness or a neutral expression causes less distortion of internal features when compared to disgust faster RT should be expected for the discrimination of highly familiar faces with a happy or neutral expression. This is not expected for unfamiliar faces, because the decision for unfamiliarity may not rely on internal features, but more on external features like hairstyle or face shape. The same might hold for experimentally familiarized faces. The results of Experiments 7 and 8 are in line with this interpretation.

To summarize, an interaction between facial expressions and the discrimination of facial familiarity was observed in Part II. This complements the results of Part I, which found an interaction in the opposite direction. Again, this interaction only emerged for personally familiar faces, but not for experimentally familiarized faces. ERP results point strongly to the response selection stage which is facilitated for personally familiar faces displaying happiness when compared to disgust. It was argued that a possible influence of facial expression on the stored representations of a familiar face is unlikely to explain the facilitation of the familiarity discrimination for happy or neutral personally familiar faces. Otherwise, such an effect should also have emerged for experimentally familiarized faces. It has been discussed that increased arousal for personally familiar faces might subserve an interaction between facial expressions and the discrimination of facial familiarity. Finally, the exclusiveness of the interaction for personally familiar faces may also be explained by the higher importance of internal facial features for highly familiar faces.

General Discussion
A short review of the presented data

I will now present a discussion on the interactions which were found in the expression discrimination task of Part I, and the familiarity discrimination task of Part II. However, it is not proposed to repeat again the discussion of the two different discrimination tasks separately. The reader is refered to the summaries and discussions of the two parts. Thus for comprehensiveness, results will be repeated in short.

It was the main question of the present dissertation as to whether there is an interaction between the discrimination of facial familiarity and facial expressions. The general hypothesis was to find a facilitative interaction between facial familiarity and the discrimination of facial expressions. Additionally, the opposite was also expected, namely that facial expressions might influence the discrimination of facial familiarity. By means of performance data and electrophysiological recordings I attempted to find the functional locus of interaction. Event related potentials were used as markers for the temporal properties of the functional processes that are involved. Although the functional model of face recognition by Bruce and Young (1986) presumes these processes t be independent, recent results suggest an interaction in one or the other direction (Baudouin et al., 2000, 2000a; Schweinberger & Soukup, 1998; Endo et al., 1992). In addition, the distributed neural system for face perception proposed by Haxby et al. (2000) allows for the possibility for such an interaction. This is based on interlinked brain regions which subserve the perception of invariant aspects like identity, and changeable aspects like expression. In order to elucidate the issue of the main question, a chronometric paradigm was applied. Throughout all experiments a two-choice RT-task was used demanding either an expression discrimination task or a familiarity discrimination. The task-irrelevant stimulus dimension was always varied independently of the task-relevant dimension. For the expression discrimination task half of the presented faces were familiar, and the other half was unfamiliar. However, all faces displayed both expressions which had to be discriminated by the participants. Accordingly, facial expression was varied within the familiarity discrimination task. By using three stimulus sets throughout the experiments, different degrees of familiarity were included. They contained either personally familiar, famous, or experimentally familiarized and unfamiliar faces. RT and error rates were recorded for all experiments. Event-related potentials were recorded only in some experiments. They served as time markers between the stimulus-response interval to separate the duration of proposed functional processing stages.

Results revealed the expected facilitative interaction between facial familiarity and facial expressions for the expression discrimination task. This only holds true for personally familiar happy faces, but not for faces displaying disgust. Late perceptual processing stages might be the possible functional locus of interaction as was suggested by event-related components. A trend was found for the P300 peak latency peaking earlier for personally familiar faces displaying happiness when compared unfamiliar faces. This facilitation propagated to the following response selection stage and was indicated by a significant difference for the S-LRP onset. Because of the somewhat unclear results based on the stimulus set with personally familiar faces, two further experiments were conducted using experimentally familiarized and unfamiliar faces. No interaction emerged between facial familiarity and the discrimination of facial expressions in both experiments. Perceptual familiarity per se is not relevant in explaining the interaction which was observed for personally familiar faces. Semantic knowledge about a person is also not the basis of an interaction between both processes. This was suggested by Experiment 6 which used famous and unfamiliar faces. Analysis of amplitude and topographical distribution revealed early differences starting at 200 ms poststimulus between personally familiar and unfamiliar faces as well as between happiness and disgust. No such topographical differences based on vector scaled data emerged for famous and unfamiliar faces although amplitude differences were evident.

In Part II the question was raised whether there is an interaction in the opposite direction. It was hyopthesized, that facial expression affects the discrimination of facial familiarity. Again, different stimulus sets were used comprising personally familiar or experimentally familiarized and unfamiliar faces. For the familiarity discrimination task it was expected that a happy expression would facilitate the decision for familiarity, because it is seen on familiar faces more often. In everyday life a happy expression is probably the most common social gesture when someone familiar approaches us and hence the decision for familiarity may be facilitated. Results revealed that the decision for a face as being familiar is faster when it displays a happy or neutral expression. Again, this only holds true for personally familiar faces but not for experimentally familiarized or unfamiliar faces. Event related potentials clearly pointed to a facilitated response selection stage – as was indexed by an earlier onset of the S-LRP.

In search of interpretations for the symmetrical interaction

From the results of the expression and the familiarity discrimination tasks it is obvious, that an interaction between facial expressions and familiarity was present in both directions. This contradicts the functional model of face recognition by Bruce and Young (1986) and a number of other findings (Young et al., 1986; Schweinberger, Klos, & Sommer et al., 1995a; Bobes et al., 2000). In addition, it contradicts previous findings which only found an asymmetrical interaction between facial identity / familiarity and the discrimination of facial expressions (Schweinberger & Soukup, 1998; Baudouin et al., 2002). However, previous results have also suggested an interaction between facial expressions and the discrimination of facial familiarity (Endo et al., 1992). These findings were extend in two important ways by the results of the present dissertation. Firstly, they show that a symmetrical interaction is observable under some circumstances, and secondly, with event-related potentials it was possible to observe, in which processing stage, in the information processing chain, the locus of confluence for the interaction of both processes might be represented. All previous studies which have examined the question of an interaction between facial familiarity and facial expressions have only used performance data (Young et al., 1986; Endo et al., 1992; Schweinberger & Soukup, 1998; Baudouin et al., 2000, 2000a) and, were therefore not able to answer such questions.

The interaction between facial familiarity and facial expressions was exclusively observed for personally familiar faces. No interaction emerged for famous or experimentally familiarized faces. A special characteristic for familiar faces is that they are familiarized through personal encounter. They often bear high personal importance. Therefore, personally familiar faces might induce a higher arousal level which may in return mediate an interaction between facial familiarity and facial expressions. This is supported by a slightly increased SCR amplitude to personally familiar faces when compared to unfamiliar ones in Experiment 3 (see also Herzmann et al, in press.). In previous studies it has been shown, that brain regions that are involved in face recognition can be modulated by affective and socially relevant information (Pizzagalli et al., 2002; Hariri et al., 2000; Morris et al., 1998). One might argue that no difference in affective information was present in the presented experiments, since both familiar and unfamiliar faces were always displayed with the same facial expressions. However, it might make a difference by whom an emotion is expressed. The social relevance might be higher when personally familiar faces display happiness. This, in return, may increase the level of arousal. Hence it is not surprising that an interaction between facial familiarity and facial expressions was only observed for personally familiar faces displaying happiness.

Neural structures subserving arousal processes are widely distributed throughout the brain (Robbins, 2000). Therefore, the effects of arousal on cognitive processing may depend on the task and temporal properties of the processes involved. Thus it can be expected that the faster to be discriminated stimulus dimension shows an earlier functional locus of interaction when the slower dimension is called as task-relevant. In this case, the relevant information is available earlier. Accordingly, the slower to be discriminated stimulus dimension should show a later functional locus of interaction when the faster to be discriminated dimension is task-relevant. Experiments 2 and 7 can be helpful to explore this prediction. Because the stimulus material also has a strong influence on temporal properties, only tasks should be compared which used the same stimulus material. This was the case for Experiments 2 and 7. A comparison of both experiments revealed faster mean RTs for the familiarity discrimination task of Experiment 7 (M = 583) when compared to the expression discrimination task (M = 693; t(34) = 4.2, p < .01). According to the expectation, an interaction between facial familiarity and the discrimination of facial expressions should emerge for earlier processing stages because facial familiarity is processed faster when compared to facial expressions. Hence an effect on earlier processing stages might be possible. On the other hand, facial expression should affect a familiarity discrimination task on later processing stages because this information is processed slower. Indeed, the results of both experiments are in line with this prediction which is illustrated in Figure 37. For the expression discrimination task of Experiment 2 an interaction emerged between facial familiarity. ERP results suggested late perceptual processing stages as the possible functional locus of interaction. By using the same stimulus material Experiment 7 employed a familiarity discrimination task. Here, ERP results clearly pointed to the response selection stage as the functional locus. Although a different functional locus of interaction emerged between facial familiarity and facial expression in both experiments, it might be modulated by the same mechanism. Depending on the temporal properties of the available expression or familiarity information, the functional locus of interaction is found either for earlier or later processing stages.

An alternative explanation of why there was only an interaction between facial familiarity and facial expression for personally familiar faces, might concern the neural processing of faces with a different degree of familiarity. The neural processing might differ between personally familiar, famous, experimentally familiarized, and unfamiliar faces. Intracranial recordings (Seeck et al., 2001) revealed differences in neural processing between personally familiar and unfamiliar faces within the right amygdala and the mesial and lateral temporal lobe. No differences were found for famous and unfamiliar faces at the same brain regions. This might be a reason why the interaction between the recognition of facial expression and facial familiarity only emerged for personally familiar faces but not for famous or experimentally familiarized faces. However, areas which subserve the representation of perceptual, and semantic facial information were not examined by Seeck et al. (2001). Differences between famous and unfamiliar faces would certainly be expected in these brain regions (Pfütze, Sommer, & Schweinberger, 2002; Schweinberger, Pfütze, & Sommer, 1995).

It is worth noting, that selective attention processes play an important role in the emergence of an interaction between facial familiarity and facial expressions. The importance of selective attention is underlined by studies of Schweinberger and Soukup (1998) as well as Baudouin et al. (2002). Both studies used the Garner paradigm (see 1.2.3.), a paradigm to study selective attention for different stimulus dimensions. Results of both studies point to an asymmetric interaction between facial identity and the discrimination of facial expressions. Hence, the ability to selectively attend to one or the other stimulus dimension or not may determine the emergence of an interaction. Using only unfamiliar faces can explain the observed asymmetrical interaction in the studies of Schweinberger and Soukup (1998) and Baudouin et al. (2002). As mentioned, external features like hairline or facial shape are more important for the discrimination of unfamiliar faces. The more familiar a faces becomes the more importance is gained by internal facial features. In both studies, it may have been easier for participants to selectively attend to facial identity in the identity discrimination task because decisions could exclusively rely on external facial features. Internal features, that are relevant for facial expression recognition, could have been ignored. In this case, for the identity discrimination task an interaction between both dimensions in the orthogonal condition is not expected. On the other hand, the discrimination of facial expressions has to rely on internal features. In this task it may be a lot more difficult to selectively attend to facial expression since there is more overlap with internal features that are also relevant for identity. Accordingly, it is much more likely to find an interaction between facial identity and the discrimination of facial expressions in the critical orthogonal condition. Results of Goshen-Gottstein and Ganel (2000) show that the differentiation between internal and external facial features is also relevant for an interaction between facial identity and the discrimination of gender. Both processes are assumed to be independent according to Bruce and Young’s model (1986). Their results strongly suggest that the reliance on internal features is important for an interaction to emerge between both processes. The authors employed a priming procedure within a gender discrimination task. Priming was only observed for faces with removed external features but not for whole faces including hair and hairline. Hence, in the previously mentioned condition, the reliance on internal features was implied and an interaction between both processes emerged. In the latter mentioned condition, the discrimination could have been easily relied upon external features and an interaction was absent.

Thus, if selective attention processes could exclusively explain an interaction, the lack of effects for the expression discrimination task in Experiment 6 using famous faces remains unresolved. It can be assumed, that internal facial features are important for the recognition of identity of famous faces. Hence an interaction of familiarity should have emerged for famous faces within the expression discrimination task. This was not the case in Experiment 6. Therefore, additional processes like arousal, due to increased personal importance, might subserve the interaction. The observed symmetrical interaction for personally familiar faces might have emerged, because it might be more difficult especially for personally familiar faces to maintain selective attention to facial familiarity or facial expression. In the familiarity discrimination task of Experiment 7, internal facial features for personally familiar faces may have captured attention more efficiantly because of their importance. Selective attention might have been harder to keep, and an interaction between facial expression and the discrimination of familiar faces should emerge. This does not neccessarily hold true for unfamiliar faces, because discrimination of unfamiliar faces could rely on external features. In this case, an interaction is not expected. This notion is underlined by the data of Experiment 7.

To sum up, increased arousal due to personal familiarity could be the basis for an interaction between facial expression and facial familiarity. This is underlined by the interaction exclusively found for personally familiar faces, but not for famous or experimentally familiarized faces. Due to temporal properties of the processes involved, the locus of facilitation can differ between tasks. This was the case for the expression discrimination of Experiment 2 and the familiarity discrimination of Experiment 7. However, also selective attention processes and the gained importance of internal facial features for highly familiar faces can partly explain the observed interaction of both processes. Arousal due to personal importance in concert with increased reliance on internal facial features for highly familiar faces might explain the symmetrical interaction between facial familiarity and facial expression.

Relevance of the data

The present data helped to shed more light on the controversy of whether and under which circumstances an interaction between facial familiarity and facial expression can emerge. Previous studies (Schweinberger & Soukup, 1998; Baudouin et al., 2000, 2000a), which contradicted the functional model of Bruce and Young (1986), were extended by finding a sym-metrical facilitative interaction and by localizing this facilitation within the information processing system.

From the observed interaction, implications can be drawn for further research in this field. The observation, that an interaction only emerged for personally familiar faces but not for famous or experimentally familiarized ones underlines the importance for models of face recognition to account for the degree of familiarity. It is also evident from functional imaging studies, that familiar and unfamiliar faces act differently on the neuronal system. Differences have been found within fusiform and ventral occipital regions (Gobbini et al., 2000), within the right middle temporal gyrus and the prefrontal gyrus (Phillips et al., 1998), and the right amygdala (Sugiura et al., 2001; Seeck et al., 2001). The brain regions involved may be more or less distributed depending on the degree of familiarity. Accordingly, differential assumptions can be drawn for different degrees of familiarity of faces which are presented. Especially for faces with a high degree of familiarity, additional processes like arousal or increased reliance on internal facial features might be relevant for hypotheses and results. The functional model of face recognition by Bruce and Young (1986) allows differential assumptions for familiar and unfamiliar faces, because FRUs are postulated for familiar faces only. However, the degree of familiarity is not taken into account. In addition, it postulates independence for facial expression recognition and facial familiarity. It was suggested, that the affective response and personal importance for faces might be the basis for an interaction between facial familiarity and facial expression. Taking this into account, the emergence of an interaction between both processes could depend on the degree of familiarity and hence on the strength of the linked functional processes (Figure 38).

Figure 38. A proposed modification based on the functional model of face recognition by Bruce and Young (1986), including the modification that was suggested by Breen et al. (2000).

As discussed above, an important prerequisite of an interaction might also be the temporal properties of the processes involved. Initially, it was expected to find an interaction between facial familiarity and facial expressions only in a hard condition, because expression discrimination was thought to be fast, and familiarity might only have a chance to interact with facial expression, when the expression discrimination is slow (Baudouin et al., 2000). Results showed that this manipulation was not necessary, since RT in Experiments 1 to 3 were fairly slow in general. However, in section 4.2. it was argued, that the temporal properties of both processes may explain the pattern of results for both discrimination tasks of Experiments 2 and 7. Faster RTs were observed in the familiarity discrimination task. Accordingly, the effect of familiarity on the expression discrimination task emerged for an earlier functional processing stage when compared to the effect of facial expression for the discrimination of familiarity. Thus, temporal properties of the processes involved should be considered in experiments which try to elucidate an interaction between different processes.

In order to draw clear assumptions for experiments concerning the processing of facial familiarity and facial expression, the kind of emotional expression also has to be considered. It has been shown in many studies, that a unique system can not subserve facial expression recognition. Different brain regions are involved depending on the expression. The amygdala plays an important role for the recognition of fear (Adolphs, 2002), but is also activated for happy expressions (Whalen et al., 1998). Basal ganglia and the insula are important for the recogntion of disgust (Sprengelmeyer et al., 2003). Hence, an interaction between facial expression and facial familiarity might only emerge for some expressions but not for others. Finding only a facilitation for personally familiar faces displaying happiness within the expression discrimination task might be due to the different neurocognitive processing of the other expression of disgust.

Another important point which emerges from using different expressions might be the distortion of facial features. It has been argued, that happiness is the expression easiest to recognize (Leppänen et al., 2003; Ekman et al., 1972). Other expressions might bear more distortion for the configuration of facial features. Hence differential effects can be expected, depending on the kind of expression. This was the case for the familiarity discrimination task of Experiment 7 as was discussed in section 2.7.

Perspective

In addition to the suggestions for further research and face recognition models which were drawn from the present results, there are still many open questions and implications for further experiments. The observed interaction between facial familiarity and expression for personally familiar faces was not always clearly cut, therefore, further experiments should be conducted with a better control of the stimulus material. By using different stimulus material, reduced variability in RT for the expression discrimination task might also increase the reliability of the P300 peak picking. Hence a significant difference for this component might emerge instead of a trend as was the case in Experiment 2. This would more strongly underline the suggested interpretation of facilitated late perceptual processing stages within the expression discrimination task.

It would also be interesting to see whether an interaction between facial expression and familiarity might emerge for famous and unfamiliar faces within a familiarity discrimination task. Such an experiment was not conducted in the present dissertation, because of liminations in time, but would have been a useful completion to the experiments presented. It might be, that the interaction between facial expression and the discrimination of familiarity is not just due to arousal and personal importance. Possibly, the advantage of internal features for familiar faces, might acount for an interaction between facial expression and the discrimination of familiarity. Therefore an interaction might be possible for famous faces since the importance of internal facial features is high.

An interesting variation of the stimulus material might be the removal of external facial features. In this case, the task-relevant information has to be extracted from internal facial features and an interaction might emerge due to overlapping features that are also important for the irrelevant dimension (see Goshen-Gottstein & Ganel, 2000), thus, not allowing participants to rely on external feature information.

Another important next step might be to use different facial expressions e.g. for personally familiar faces. It might be, that a facilitation of the discrimination of facial expressions for personally familiar faces could also emerge for neutral, fearful, or angry faces. At least for fearful faces the amygdala is activated because it is important for affective face processing and arousal (Rolls, 1999). Hence an interaction of both processes could be observed for personally familiar faces displaying fear.

Contrary to the influencial functional model of face recognition by Bruce and Young (1986), results of the present dissertation showed that an interaction between facial familiarity and facial expression is observable under some circumstances. Most important for such an interaction is the degree of familiarity because it only emerged for personally familiar faces. Event-related potentials pointed to late perceptual processing stages which are facilitated by familiarity within an expression discrimination task. On the other hand, facial expression affected the response selection stage for a familiarity discrimination task. It was argued that the affective resonse to personally familiar faces could be the basis of an interaction between facial familiarity and facial expression. In addition, the kind of facial expression in concert with the increased importance of internal facial features for familiar faces might also be relevant for an interaction to emerge. Implications for models of face recognition and for further research have been discussed. In summary, the present dissertation has elucidated upon important points concerning the functional interaction between facial expressions and facial familiarity. However, it also raised new questions and can be used as a basis for further research in this field.

Goshen-Gottstein, Y. & Ganel, T. (2000). Repetition priming for familiar and unfamiliar faces in a sex-judgement task: Evidence for a common route for the processing of sex and identity.Journal of Experimental Psychology: Learning, Memory, and Cognition
,
26
,
1198-1214.

Appendix 6.1.1
Results (p-Values) revealed by ANOVAs of the mean amplitude distribution, and of vector scaled mean amplitudes of Experiment 2 for 8 time intervals starting from 200 ms until 600 ms after stimulus onset. The effects of the factors expression, and familiarity are always the interaction with the factor electrodesite, because average referenced data have been used.

intervals

200-250

250-300

300-350

350-400

400-450

450-500

500-550

550-600

amplitudes

expression

.000

.000

.005

.003

.000

.000

.000

.000

familiarity

.000

.000

.000

.000

.000

.000

.000

.000

exp x fam

-

-

-

-

-

-

-

-

vector scaled data

expression

.000

.000

.008

-

-

-

-

-

familiarity

.003

.000

.000

.000

.001

(.051)

-

.001

exp x fam

-

-

-

-

-

-

-

-

Amplitude distribution and vector scaled data of Experment 3

Appendix 6.2.1
Results (p-Values) revealed by ANOVAs of the mean amplitude distribution, and of vector scaled mean amplitudes of Experiment 3 for 8 time intervals starting from 200 ms until 600 ms after stimulus onset. The effects of the factors expression, and familiarity are always the interaction with the factor electrode site, because average referenced data have been used.

intervals

200-250

250-300

300-350

350-400

400-450

450-500

500-550

550-600

amplitudes

expression

(.054)

.000

.002

.000

.000

.000

.000

.001

familiarity

(.038)

.001

.000

.000

.000

.000

.000

.000

exp x fam

-

-

-

-

-

-

-

-

vector scaled data

expression

-

.000

.002

(.038)

(.035)

-

-

-

familiarity

-

.011

.000

.000

.000

.000

.000

.000

exp x fam

-

-

-

-

-

-

-

-

Amplitude distribution and vector scaled data of Experment 6

Appendix 6.3.1
Results (p-Values) revealed by ANOVAs of the mean amplitude distribution, and of vector scaled mean amplitudes of Experiment 6 for 8 time intervals starting from 200 ms until 600 ms after stimulus onset. The effects of the factors expression, and familiarity are always the interaction with the factor electrode site, because average referenced data have been used.

intervals

200-250

250-300

300-350

350-400

400-450

450-500

500-550

550-600

amplitudes

expression

.000

.000

.000

.000

.000

.001

.009

.010

familiarity

-

-

.010

.006

-

.004

.000

.005

fam x exp

-

-

-

-

-

-

.008

.007

vector scaled data

expression

.000

.000

.000

-

-

-

-

.028

familiarity

-

-

(.059)

(.052)

-

-

-

-

fam x exp

-

-

-

-

-

-

-

-

Appendix 6.3.2
Results (p-Values) revealed by post-hoc ANOVAs for the factor familiarity of the 8 time intervals of Experiment 6. The mean amplitude distribution, and vector scaled mean amplitudes were used. Only two levels were applied for the factor familiarity: B = britisch celebrities, G = german celebrities, and I = international celebrities. The effects of the factor familiarity are always the interaction with the factor electrode site, because average referenced data have been used.

intervals

200-250

250-300

300-350

350-400

400-450

450-500

500-550

550-600

amplitudes

familiarity: B vs. G

-

-

-

-

-

.000

.000

(.050)

familiarity: B vs. I

-

-

.009

.004

-

(.082)

(.008)

.003

familiarity: G vs. I

-

-

(.034)

(.039)

-

-

(.079)

-

vector scaled data

familiarity: B vs. G

-

-

-

-

-

-

-

-

familiarity: B vs. I

-

-

.015

.005

-

-

-

-

familiarity: G vs. I

-

-

-

-

-

-

-

-

Amplitude distribution and vector scaled data of Experment 7

Apendix 6.4.1
Results (p-Values) revealed by ANOVAs of the mean amplitude distribution, and of vector scaled mean amplitudes of Experiment 7 for 8 time intervals starting from 200 ms until 600 ms after stimulus onset. The effects of the factors familiarity, and expression are always the interaction with the factor electrode site, because average referenced data have been used.