Category Archives: Cognitive Film Theory

We haven’t had any visual illusions on this blog for a while, and since the poster recently released for Ram Gopal Varma’s Bhoot Returns depends on a visual illusion this seems as good as time as any.

It’s surprising that more films do not choose to use visual illusions in their marketing materials, but some nice examples based on Disney films by Rowan Stocks Moore can be found here. The Peter Pan and Snow White posters in particular stand out.

Archimedes Lab has many different illusions and oddities from Gianni Sarcone and Marie Waeber, which you can access here. There is also a great selection of vintage illusions dating back 2500 years.

The finalists for this year’s Illusion of the Year contest can be found here, with attractive celebrities that turn ugly and a great interactive demonstration of the wagon wheel illusion. There is also an illusion inspired by the infamous twisting neck scene from The Exorcist which you can see below if you’re brave enough. The effect is much more eerie than anything you could do with CGI.

io9 has a dedicated illusions channel, which has lots of different examples of visual illusions and articles covering a range of issues including the art of anamorphic illusions and why our pupils contract when looking at illusions that are not bright lights.

This last example comes from the pages of Akiyoshi Kitaoka, and you can find details of his latest work here.

Finally, a good collection of illusions, including some of those listed above and in my other posts on this topic (here and here), can be found in New Scientist’s‘Friday Illusion’ column.

Following on from my earlier posts on the editing structure of slasher films, this week I have a draft of a paper that combines my early observations (much re-written) along with an analysis of the relationship between editing and the narrative structure of Friday the Thirteenth (1980)

Exploratory data analysis and film form: The editing structure of slasher films

We analyse the dynamic editing structure of four slasher films released between 1978 and 1983 with simple ordinal time series methods. We show the order structure matrix is a useful exploratory data analytical method for revealing the editing structure of motion pictures without requiring a priori assumptions about the objectives of a film. Comparing the order structure matrices of the four films, we find slasher films share a common editing pattern closely comprising multiple editing regimes with change points between editing patterns occur with large changes in mood and localised clusters of shorter and longer takes are associated with specific narrative events. The multiple editing regimes create different types of frightening experiences for the viewer with slower edited passages creating a pervading sense of foreboding and rapid editing linked to the frenzied violence of body horror, while the interaction of these two modes of expression intensifies the emotional experience of watching a slasher film.

Analysing the editing structure of these slasher films is only part of this paper. Another goal was to outline exploratory data analysis as a data-driven approach to understanding film style that avoids a specific problem of existing ways of thinking about film style.

Existing methods of analysing film style make a priori assumptions about the functions of style and then provide examples to support this assertion. This runs the risk of begging the question and circulus in probando, in which the researcher’s original assumption is used as a basis for selecting the pertinent relations of film style which are then used to justify the basis for making assumptions about the functions of film style. We would like to avoid such logically flawed reasoning whilst also minimising the risk that we will miss pertinent relations that did not initially occur to us. By adopting a data-driven approach we can derive the functions of film style by studying the elements themselves without the need to make any such a priori assumptions. Exploratory data analysis (EDA) allows us to do this by forcing us to attend to the data on its own terms.

Although this is a method developed within statistics, EDA can be applied not just to numerical data but to any situation where we need to understand the phenomenon before us. For example, I had not noticed that the number of scenes between hallucinations in Videodrome reduces by constant factor until I sat down and wrote out the narrative structure of the film (see here).

In this paper I discuss some relations between editing and the emotional experience of watching slasher films, and below are listed some interesting references that follow on from last week’s collection of paper on neuroscience and the cinema:

Some authors have speculated that the cognitive component (P3) of the Event-Related Potential (ERP) can function as a psychophysiological measure of sexual interest. The aim of this study was to determine if the P3 ERP component in a workload task can be used as a specific and objective measure of sexual motivation by comparing the neurophysiologic response to stimuli of motivational relevance with different levels of valence and arousal. A total of 30 healthy volunteers watched different films clips with erotic, horror, social-positive and social-negative content, while answering an auditory oddball paradigm. Erotic film clips resulted in larger interference when compared to both the social-positive and auditory alone conditions. Horror film clips resulted in the highest levels of interference with smaller P3 amplitudes than erotic and also than social-positive, social-negative and auditory alone condition. No gender differences were found. Both horror and erotic film clips significantly decreased heart rate (HR) when compared to both social-positive and social-negative films. The erotic film clips significantly increased the skin conductance level (SCL) compared to the social-negative films. The horror film clips significantly increased the SCL compared to both social-positive and social-negative films. Both the highly arousing erotic and non-erotic (horror) movies produced the largest decrease in the P3 amplitude, a decrease in the HR and an increase in the SCL. These data support the notion that this workload task is very sensitive to the attentional resources allocated to the film clip, although they do not act as a specific index of sexual interest. Therefore, the use of this methodology seems to be of questionable utility as a specific measure of sexual interest or as an objective measure of the severity of Hypoactive Sexual Desire Disorder.

Neuroimaging studies which investigate brain activity underlying declarative memory processes typically use artificial, unimodal laboratory stimuli. In contrast, we developed a paradigm which much more closely approximates real-life situations of information encoding.

Methods

In this study, we tested whether ecologically valid stimuli – clips of a TV news show – are apt to assess memory-related fMRI activation in healthy participants across a wide age range (22-70 years). We contrasted brain responses during natural stimulation (TV news video clips) with a control condition (scrambled versions of the same clips with reversed audio tracks). After scanning, free recall performance was assessed.

Results

The memory task evoked robust activation of a left-lateralized network, including primarily lateral temporal cortex, frontal cortex, as well as the left hippocampus. Further analyses revealed that – when controlling for performance effects – older age was associated with greater activation of left temporal and right frontal cortex.

Conclusion

We demonstrate the feasibility of assessing brain activity underlying declarative memory using a natural stimulation paradigm with high ecological validity. The preliminary result of greater brain activation with increasing age might reflect an attempt to compensate for decreasing episodic memory capacity associated with aging.

We measured long-term memory for a narrative film. During the study session, participants watched a 27-min movie episode, without instructions to remember it. During the test session, administered at a delay ranging from 3 h to 9 mo after the study session, long-term memory for the movie was probed using a computerized questionnaire that assessed cued recall, recognition, and metamemory of movie events sampled ∼20 sec apart. The performance of each group of participants was measured at a single time point only. The participants remembered many events in the movie even months after watching it. Analysis of performance, using multiple measures, indicates differences between recent (weeks) and remote (months) memory. While high-confidence recognition performance was a reliable index of memory throughout the measured time span, cued recall accuracy was higher for relatively recent information. Analysis of different content elements in the movie revealed differential memory performance profiles according to time since encoding. We also used the data to propose lower limits on the capacity of long-term memory. This experimental paradigm is useful not only for the analysis of behavioral performance that results from encoding episodes in a continuous real-life-like situation, but is also suitable for studying brain substrates and processes of real-life memory using functional brain imaging.

While much has been learned regarding the neural substrates supporting episodic encoding using highly controlled experimental protocols, relatively little is known regarding the neural bases of episodic encoding of real-world events. In an effort to examine this issue, we measured fMRI activity while observers viewed a novel TV sitcom. Three weeks later, subsequent memory (SM) for the narrative content of movie events was assessed. We analyzed the encoding data for intersubject correlations (ISC) based on subjects’ subsequent memory (ISC-SM) performance to identify brain regions whose BOLD response is significantly more correlated across subjects during portions of the movie that are successfully as compared to unsuccessfully encoded. These regions include the parahippocampal gyrus, superior temporal gyrus, anterior temporal poles, and the temporal-parietal junction. Further analyses reveal (1) that these correlated regions can display distinct activation profiles and (2) that the results seen with the ISC-SM analysis are complementary to more traditional linear models and allow analysis of complex time course data. Thus, the ISC-SM analysis extends traditional subsequent memory findings to a rich, dynamic and more ecologically valid situation.

Response reliability is complementary to more conventional measurements of response amplitudes, and can reveal phenomena that response amplitudes do not. Here we review studies that measured reliability of cortical activity within or between human subjects in response to naturalistic stimulation (e.g., free viewing of movies). Despite the seemingly uncontrolled nature of the task, some of these complex stimuli evoke highly reliable, selective, and time-locked activity in many brain areas, including some brain regions that often do not show much response modulation with conventional experimental protocols. This activity provides an opportunity to address novel questions concerning natural vision, temporal scale of processing, memory, and the neural basis of inter-group differences.

Hemodynamic activity in occipital, temporal, and parietal cortical areas were recently shown to correlate across subjects during viewing of a 30-minute movie clip. However, most of the frontal cortex lacked between-subject correlations. Here we presented 12 healthy naïve volunteers with the first 72 minutes of a movie (“Crash”, 2005, Lions Gate Films) outside of the fMRI scanner to involve the subjects in the plot of the movie, followed by presentation of the last 36 minutes during fMRI scanning. We observed significant between-subjects correlation of fMRI activity in especially right hemisphere frontal cortical areas, in addition to the correlation of activity in temporal, occipital, and parietal areas. It is possible that this resulted from the subjects following the plot of the movie and being emotionally engaged in the movie during fMRI scanning. We further show that probabilistic independent component analysis (ICA) reveals meaningful activations in individual subjects during natural viewing.

Cinema is a promising naturalistic stimulus that enables, for instance, elicitation of robust emotions during functional magnetic resonance imaging (fMRI). Inter-subject correlation (ISC) has been used as a model-free analysis method to map the highly complex hemodynamic responses that are evoked during watching a movie. Here, we extended the ISC analysis to frequency domain using wavelet analysis combined with non-parametric permutation methods for making voxel-wise statistical inferences about frequency-band specific ISC. We applied these novel analysis methods to a dataset collected in our previous study where 12 subjects watched an emotionally engaging movie “Crash” during fMRI scanning. Our results suggest that several regions within the frontal and temporal lobes show ISC predominantly at low frequency bands, whereas visual cortical areas exhibit ISC also at higher frequencies. It is possible that these findings relate to recent observations of a cortical hierarchy of temporal receptive windows, or that the types of events processed in temporal and prefrontal cortical areas (e.g., social interactions) occur over longer time periods than the stimulus features processed in the visual areas. Software tools to perform frequency-specific ISC analysis, together with a visualization application, are available as open source Matlab code.

The deployment of eye movements to complex spatiotemporal stimuli likely involves a variety of cognitive factors. However, eye movements to movies are surprisingly reliable both within and across observers. We exploited and manipulated that reliability to characterize observers’ temporal viewing strategies while they viewed naturalistic movies. Introducing cuts and scrambling the temporal order of the resulting clips systematically changed eye movement reliability. We developed a computational model that exhibited this behavior and provided an excellent fit to the measured eye movement reliability. The model assumed that observers searched for, found, and tracked a point of interest and that this process reset when there was a cut. The model did not require that eye movements depend on temporal context in any other way, and it managed to describe eye movements consistently across different observers and two movie sequences. Thus, we found no evidence for the integration of information over long time scales (greater than a second). The results are consistent with the idea that observers employ a simple tracking strategy even while viewing complex, engaging naturalistic stimuli.

This week some very interesting papers on how movement and screen size impacts on our experience and understanding of motion pictures. Particularly interesting is the paper that indicates small screens can be more immersive than big screens

Increasingly, television content is available to viewers across 3 different screen types: TVs, personal computers (PCs), and portable devices such as mobile phones and iPods. The purpose of this study was to see what effect physical and apparent screen size has upon ad effectiveness. Using a sample of 320 members of the Australian public, we found that TV ads can be just as effective on PCs and iPods. However, controlling for screen type, ads viewed from a closer distance (i.e. with a wider viewing angle) were more likely to be recalled the next day, and were associated with more favorable brand attitudes. Shorter programs, product relevance, and use of close-ups and detailed images made no difference to this general viewing-angle effect.

This study moved Presence into the realm of the smaller video format—comparing Apple iPod with a standard television presentation. Ninety-six students were exposed to one of two presentations on either an iPod or on a 32-inch television. Students saw either a 10-minute fast-paced (multiple cut) action sequence or a 10-minute slow-paced (long cut) conversation sequence from a feature length motion picture. The 2 x 2 design looked at differences in immersion, spatial presence and social realism. While previous research suggests that larger format presentations should generally result in higher levels of presence, this study found that subjects viewing the iPod reported higher levels of immersion. Social realism had a significant interaction with content/pace, and there was no significant difference between iPod and the 32-inch television in spatial presence.

An experiment investigated the effects of picture motion on individuals emotional reactions to images. Subjective measures (self-reports) and physiological data (skin conductance and heart rate) were obtained to provide convergent data on affective responses. Results indicate that picture motion significantly increased arousal, particularly when the image was already arousing. This finding was supported by the both skin conductance and the self-report data. Picture motion also tended to prompt more heart-rate deceleration, most likely reflecting a greater allocation of attention to the more arousing images. In this study, the influence of picture motion on affective valence was evident only in the self-report measures – positive images were experienced as more positive and negative images as more negative when the image contained motion. Implications of the results and suggestions for future research are discussed.

We examined the modulating influence of a small moving vs. static facial image on emotion- and attention-related subjective and physiological responses to financial news read by a newscaster, and on memory performance among 36 young adults. A moving-face newscaster was associated with high self-reported pleasure and arousal, but not with physiological arousal (electrodermal activity). Facial electromyographic responses to facial image motion were at variance with pleasure ratings. Facial motion was associated with decreased respiratory sinus arrhythmia, an index of attention, and improved memory performance for positive messages. A talking facial image on a small screen increases attention and knowledge acquisition.

The number of different screens that people confront is increasing. One potentially important difference in the psychological impact of screen displays is their size; new screens are both larger and smaller than older ones. A between-subjects experiment (n = 38) assessed viewer’s attention and arousal in response to three different size screens (56-inch, 13-inch, and 2-inch picture heights). Viewers responded to video images from television and film that displayed different emotions (# video segments = 60). Attention was measured by heart rate deceleration in response to the onset of pictures, and arousal was measured by skin conductance aggregated during viewing. Results showed that the largest screen produced greater heart rate deceleration than the medium and small screens. The large screen also produced greater skin conductance than the medium and small screens. For skin conductance, screen size also interacted with the emotional content of the stimuli such that the most arousing pictures (e.g., pictures of violence and sex) showed the highest levels of arousal on the large screen compared to the medium and small screens.

In the context of picture viewing, consistent and specific relationships have been found between two emotion dimensions ~valence and arousal! and self-report, physiological and overt behavioral responses. Relationships between stimulus content and the emotion-response profile can also be modulated by the formal properties of stimulus presentation such as screen size. The present experiment explored the impact of another presentation attribute, stimulus motion, on the perceived quality of the induced emotion and on its associated physiological response pattern. Using a within-subject design, moving and still versions of emotion-eliciting stimuli were shown to 35 subjects while facial muscle, heart rate, skin conductance, and emotion self-reports were monitored. The impact of motion was dramatic. Self-report and physiological data suggested strongly that motion increased arousal, had little impact on valence, and captured and sustained the subject’s attention to the image.

Last January I post a list of papers from 2010 on the broadly defined topic of cognitive film theory, and this year I’m doing the same for the past twelve months. There is something for everyone here (check out Neal et al‘s paper on eating popcorn in the cinema), but I would particularly point you in the direction of a special issue of Science in Context on ‘Cinematography, Seriality, and the Sciences’ from last year, which you can find here.

As ever, this list is not exhaustive but it should be accurate. I’ve included some interesting sounding papers that can be accessed as e-publications ahead of print, and so the year of publication may change when they move from digital to analogue form.

Lavaur J-F and Bairstow D 2011 Languages on the screen: is film comprehension related to the viewers’ fluency level and to the language in the subtitles?, International Journal of Psychology, 46 (6): 455-462. DOI:10.1080/00207594.2011.565343.

The Artist has been wowing audiences across the world. The film has already won some awards, and is hotly tipped for many more. It has also been attracting much interest in the press, and film scholars have been roped into this.

In an interview with the BBC, silent film expert Bryony Dixon of the BFI made a series of statements that are worth reflecting upon:

I am aware of no research that compares the viewing pleasures derived from silent films to sound films, and I have not been able to find any such research. In fact, what viewers find rewarding about the film experience is an under-researched area of film studies. If anyone knows of any research in this area please feel free to add a comment to this post listing the appropriate references.

This is just Dixon’s opinion, and we should not be surprised that an expert on silent films should prefer silent films. Other people will have their own opinions, tastes, and preferences. The difference is that other people will not have the opportunity to express them in the BBC under the heading ‘Expert on the rewards of silent film.’ This is problematic because it presents Dixon’s opinion as fact (‘An expert says …’). This may be the fault of the BBC and the way it has presented the interview, but from watching the video I doubt it.

Of course, a factor here is that there has not been much in the way of silent film since 1930 and so research on what viewers think about silent films has inevitably been extremely limited. The Artist provides an excellent opportunity for researchers to engage with this topic.

2.Watching a silent film requires the viewer to work harder

There is no research that I can find looking at the cognitive load of silent cinema (probably for reasons noted above), and the literature on cognitive load in film viewing is somewhat limited in general. An interesting place to start is this paper from Nitzan Ben-Shaul:

It is also worth reading Julian Hochberg and Virginia Brooks’s work on film viewing and visual momentum as it gives a general description of how observers attend to images (both moving and still) and how we cognitively process this information:

Cognitive load theory (CLT) might support the opposite conclusion to Dixon’s assertion. According to CLT, we have only a limited amount of working memory and the cognitive load of a task is determined by the number and complexity of the steps involved that use up those resources. The following example is from Gutashaw WE and Brigham FJ 2005 Instructional support employing spatial abilities: using complimentary cognitive pathways to support learning in students with achievement deficits, in TE Scruggs, MA Mastropieri (eds.) Cognition and Learning in Diverse Settings: Amsterdam: Elsevier: 47-70.

Watching a film in a language one does not understand but with subtitles is an example of an increased cognitive load over watching the same film in one’s own language. Now image watching a subtitled film with poor reading skills. The cognitive load increases dramatically (66).

Thinking along similar lines, we might think that because we do not have to attend to dialogue as well as images that the cognitive load in watching a silent film is lower than that when watching a film with synchronised dialogue that requires attention to multiple sensory modalities.

There has been no direct research on cognitive load that could answer this question, and so I make this argument as a hypothesis only, but as we see in relation to the next point the evidence indicates it is faster editing that increases the cognitive load on the viewer.

Dixon’s statement sound plausible, but without supporting research it is nothing more than a hypothesis and there are other hypotheses to be made and tested on this point. Of course, it may be that I just haven’t been looking for research in the right places and so if anyone knows of research demonstrating if this statement is true or not then please let me know.

There are a couple of things to consider here. First, contemporary film audiences are less likely to be familiar with silent films than they are with modern action blockbusters. Therefore, they may concentrate more on something unfamiliar than something commonplace and this would account for a difference in viewers’ experience. We may find that with increasing experience viewing habits may change so that viewers familiar with both silent films and contemporary cinema watch them in the same way. Again, this relates to the cognitive load placed in the viewer. This sounds plausible, but as noted above I have been able to find no research in this topic. In fact I can find no research on viewers’ ‘concentration’ in the cinema, and this leads us to our second problem: what is meant by ‘concentration?’ Dixon never defines the terms she uses, and it may mean the number of times a viewer looks at the screen, the length of time the viewer looks at a screen, the focus of the viewer’s attention when looking at the screen, etc.

If we take concentration to mean something similar to attention, then there is some research on this topic and it contradicts Dixon’s assertion that slower films require more concentration than fast edited films. Research on the limited capacity model of viewership has shown that rapid pacing in motion pictures requires increased allocation of perceptual resources. The research can be read in this paper:

The limited capacity model defines the viewer as an information processor faced with a variably redundant ongoing stream of audio-visual information, in the message content is the topic, genre, and information contained in a message. Therefore, ‘viewing is the continuous allocation of a limited pool of processing resources to the cognitive process required for viewers to make sense of a message.’ (I don’t like this definition of message content – it seems somewhat circular to me).

This research looked at the effect of production pacing and content on attention in the cinema, testing the hypothesis that both pacing and arousing content should increase the level of resources automatically allocated to processing the message. The results showed this is indeed the case: arousing content and fast pace increased self-reported arousal in television viewers, and that both factors increase the allocation of resources to processing messages.

This is also discussed in a subsequent paper (below), which showed that faster pacing resulted in the allocation of greater resources by viewers in attending to a television message and that self-reported arousal also increased with editing pace.

In summary, there is no evidence that slower films require greater concentration by film viewers but there is evidence that faster paced films – such as (but obviously not limited to) action blockbusters – do elicit greater allocation of information processing resources (including attention).

A final point to make is that we do not yet know what the distribution of shot lengths in The Artist, and so comparing its pace to other films is not yet possible. It will be interesting when the film comes out on DVD and we can look at it frame-by-frame to see whether its editing style is compatible with contemporary cinema or with silent films of the 1920s. However, as yet we cannot make any empirical statement about the contribution of editing to the pace of this film.

Dixon’s comments raise some interesting questions about the nature of film viewing and silent cinema, but in the absence of supporting evidence they are opinions and not facts. The danger comes when we accept the former as the latter without asking questions or referring to the existing research in this area. Empirical research leads us to reject incorrect and empty opinions by establishing the nature of those facts. This is what film studies is supposed to be for.

Other posts on this blog have listed links to papers addressing the indexing of video by analysis of the stylistic components of the video itself (shot lengths, colour, sound energy, etc). An alternative approach is not to look at the video but to look at the viewer watching the video or to look at the viewer’s brain whilst they watch. The papers presented adopt a range of approaches to understanding films by understanding viewers and to understanding viewers by understanding films. They are an example of the very interesting empirical research taking place across diverse subject areas that has yet to make any impact on film studies.

This paper describes research aimed at creating intelligent video recommendation engines for broadband media services in digital homes. The aim of our research is to harness physiological signals to characterise people’s video selection preferences which we plan to integrate into new generations of video recommendation engines. We describe an initial experiment aimed at determining whether videos produce useable variations in physiology and linking these with emotional changes elicited by video material. We discuss our results and consider the possibility of utilising physiological sensing methods to build profiles that can be treated as signatures. Finally, we conclude by describing the future directions of our work.

In this paper, we take a human-centred view to the definition of the affective content of films. We investigate the relationship between users physiological response and multimedia features extracted from the movies, from the perspective of narrative evolution rather than by measuring average values. We found a certain dynamic correlation between arousal, derived from measures of Galvanic Skin Resistance during film viewing, and specific multimedia features in both sound and video domains. Dynamic physiological measurements were also consistent with post-experiment self-assessment by the subjects. These findings suggest that narrative aspects (including staging) are central to the understanding of video affective content, and that direct mapping of video features to emotional models taken from psychology may not capture these phenomena in a straightforward manner.

In this paper we present a user-centric summarisation system that combines automatic visual-content analysis with user-interface design features as a practical method for home movie summarisation. The proposed summarisation system is designed in such a manner that the video segmentation results generated by the automatic content analysis tools are further subject to refinement through the use of an intuitive user-interface so that the automatically created summaries can be effectively tailored to each individual’s personal need. To this end, we study a number of content analysis techniques to facilitate the efficient computation of video summaries, and more specifically emphasise the need for employing an efficient and robust optical flow field computation method for sub-shot segmentation in home movies. Due to the subjectivity of video summarisation and the inherent challenges associated with automatic content analysis, we propose novel user-interface design features as a means to enable the creation of meaningful home movie summaries in a simple manner. The main features of the proposed summarisation system include the ability to automatically create summaries of different visual comprehension, interactively defining the target length of the desired summary, easy and interactive viewing of the content in terms of a storyboard, and manual refinement of the boundaries of the automatically selected video segments in the summary.

This paper presents an approach to affective video summarisation based on the facial expressions (FX) of viewers. A facial expression recognition system was deployed to capture a viewer’s face and his/her expressions. The user’s facial expressions were analysed to infer personalised affective scenes from videos. We proposed two models, pronounced level and expression’s change rate, to generate affective summaries using the FX data. Our result suggested that FX can be a promising source to exploit for affective video summaries that can be tailored to individual preferences.

This paper presents an approach to detect personal highlights in videos based on the analysis of facial activities of the viewer. Our facial activity analysis was based on the motion vectors tracked on twelve key points in the human face. In our approach, the magnitude of the motion vectors represented a degree of a viewer’s affective reaction to video contents. We examined 80 facial activity videos recorded for ten participants, each watching eight video clips in various genres. The experimental results suggest that useful motion vectors to detect personal highlights varied significantly across viewers. However, it was suggested that the activity in the upper part of face tended to be more indicative of personal highlights than the activity in the lower part.

n this paper, we propose a novel system for automatically summarizing home videos based on a user experience model. The user experience model takes account of user’s spontaneous behaviors when viewing videos. Based on users’ reaction when viewing videos, we can construct a systematic framework to automate video summarization. In this work, we analyze the variations of viewer’s eye movement and facial expression when he or she watching the raw home video. We transform these behaviors into the clues of determining the important part of each video shot. With the aids of music analysis, the developed system automatically generates a music video (MV) style summarized home videos. Experiments show that this new type of editing mechanism can effectively generate home video summaries and can largely reduce the efforts of manual summarization.

This paper describes a research project conducted to study the relationship between videos and users’ induced physiological and psychological responses. Firstly, a set of 43 film clips are carefully chosen, and 20 subjects are invited to participate in our experiment. They watch several of chosen clips while their EEG signals are recorded synchronously. After each clip, the subject is required to report his real induced emotion using emotional valence, arousal, basic emotion category and intensity. Secondly, several classical movie features and EEG features are extracted, and feature selections are conducted by computing the correlation between each feature and the arousal or valence. Thirdly, selected movie features and EEG features are used to simulate the arousal and valence respectively by employing the linear relevance vector machine. Fourthly, selected movie features are used to simulate the EEG feature values, and vice verse. The results show that arousal/valence can be well estimated by either video features or EEG features. Apart from that, they also indicate that there exist certain relationship between the videos and induced EEG signals, and some relation models are acquired. Finally, clustering is conducted to map the emotion dimensions to emotion categories. Thus, the gap between videos and emotion categories, as well as the gap between the EEG and emotion categories, has been bridged to some extent. This result could provide a reference to applications in brain-computer interaction field.

A couple of weeks ago I posted about some empirical research on editing and the viewer’s experience of pace in the visual media from the 1970s.

A fascinating read on the same topic is the UNESCO report on mass communication research published in 1961, which presents a comprehensive and global bibliography of research on the influence of the cinema on children and adolescents. The report can be accessed here, and it is definitely worth taking an afternoon to read through it.

UNESCO 1961 The Influence of the Cinema on Children and Adolescents: An Annotated International Bibliography. Reports and Papers on Mass Communication 31. Paris: UNESCO.

The report provides details on 491 different pieces of research from around the world, and provides an insight into the type of research that was done before film studies came along. Areas covered include the social effects of cinema on young people, the use of film in education, film and juvenile delinquency, motives behind film choice, and there is good coverage of what we would now call cognitive film theory.

There are all sorts of interesting things to discover. For example, item 62 on the list provides a fascinating insight into the habits of younger viewers in Michigan in the 1940s.

Gibson, Harold J. (Mrs .) and Nahabedian, Vaskey (Mrs .) . A Survey of the Reading, Radio and Motion Picture Habits of Royal Oak Public School Students and their Parents. Royal Oak, Michigan, Royal Oak Public School, 1949, 21 p.

The average pupil in the school surveyed attends the cinema much more frequently than his parents. At the age of 8, he goes to the cinema once a week; until the age of 12 he attends the Saturday afternoon performance. When he reaches junior high school he goes to the cinema on Friday evening, generally with a friend. His parents help him in the selection of films, and he generally appreciates the films his parents consider suitable for him. Westerns, cartoons and animal films are his favourites; later his interest in westerns wanes and his interest in musicals grows. He now chooses films on the basis of cast and publicity. When he reaches high school, he will be more influenced in his choice by official film criticism, and he tends to have the same criteria as his parents.

Some of the research is a bit prosaic: item 61 is a study of the cinema-going habits of Italian young people and concludes that as they get older ‘boys go more frequently with girls.’ Isn’t that what the cinema is for?

However, I’m really not sure about the study from 1949 that showed Robert Flaherty’s Nanook of the North (1922) to some Italian 8-to-14 year olds (no. 130) that concluded that children had difficulty understanding the film. An 80 minute silent documentary about an Eskimo is hardly suitable viewing for children as young as eight. I know I’ve never been that enamoured of this film, but I’m certainly glad I didn’t have to watch it as a child. The comments about spatial awareness and recognition of regular geometric forms do sound more interesting though.

A study of the reactions of 576 children, aged 8 to 14, to Flaherty’s film “Nanook’. Four hundred and ninety children were questioned: 86 made unsolicited comments. Particularly apparent were the many errors in observation and the discrepancy between what actually occurred in the film and what the children thought they had seen. The rapid succession of images, the inability to understand clearly, to compare precisely and to interpret exactly when drawing up a report has the following results for children: real difficulties in making accurate comparisons as to sizes and likenesses, in recognizing regular geometric forms, in establishing the position of persons in relation to a known object, and in interpreting some of their movements and attitudes. Such difficulties as these do not seem to lessen proportionately as the child grows older. Further research is recommended to study the choice of motion-pictures for children of different age groups.

Perhaps the researchers might have asked the children if they wanted to watch Nanook?

There is an extensive series of entries describing quite detailed studies by the Japanese Ministry of Education on cinema attendance among young people from the 1930s that sound very interesting.

In his ‘Foreword’ to Moving Image Theory: Ecological Considerations, David Bordwell wrote that film studies ‘got off on the wrong foot methodologically. Instead of framing questions, to which competing theories might have responded in a common concern for enlightenment, film academics embraced a doctrine-driven conception of research’ (2005: xi, original emphasis). [Bordwell D 2005 Foreword, in JD Anderson and B Fisher Anderson (eds.) MovingImage Theory: Ecological Considerations. Carbondale: Southern Illinois University Press, 2005: ix-xii].

This is may be an accurate description of film studies, but it is not an accurate description of the study of film.

What stands out from reading much of the research in the UNESCO report is that pre-film studies research in the cinema is (1) primarily concerned with psychology of the cinema and (2) that it is empirical research and is NOT doctrine-driven. And yet this research has had relatively little impact on film studies as it is taught in universities today. The institutionalisation of film studies as an academic discipline does not appear to have drawn on any of this tradition going back well into the silent era. Why not? And what are the consequences of this? What did these earlier researchers understand about the cinema that we have forgotten?

For example, Tim Smith has written about viewer’s eye movements when watching Hollywood films. You can find his blog describing his research here and his guest piece about eye movements in watching There Will Be Blood on David Bordwell’s blog is here. But if we go back to 1964 we can find this paper, which was addressing the same questions some 47 years ago.

This paper, like those in the UNESCO report, does not feature in the film studies curriculum due to the collective amnesia of film scholars who, it would seem, simply ignored decades of prior research when creating university courses in film. Why this should be the case is one of the most important and most interesting questions in film studies.

In the comments on the last update to my bibliography on cognitive film theory, someone asked why I hadn’t included the French Filmology research of the 1940s and 1950s. You can find the bibliography and the comments here. Part of my response was that I simply did not come across this research that often and that translations of this work are relatively rare. It is much harder, for example, to find the works of Gilbert Cohen-Seat in English than it is to find those of Christian Metz. Why should this be so?

The study of film existed before film studies, and it existed as a body of empirical research that looked at how viewers experienced and comprehended the cinema, at the behaviour of audiences, and at the social impact of cinema. And it did so by asking questions years before Bordwell began writing about a mid-level research programme as a means of moving forward.

Three differently edited versions of the same TV presentation consisting of ‘rapid,’ ‘moderate,’ and ‘original’ pacing rates were shown to 120 grade 10 Ss to ascertain their differential effectiveness in changing viewer attitudes and maintainance of viewer interest. Ss were randomly divided into three TV viewing treatment groups and one non-viewing control group. Attitudes were measured by a 32 item post attitude questionnaire with reliability of m0.90. Interest was measured by a 20 item post interest questionnaire with reliability of 0.96. Single classification ANOVA and HSD Tests revealed significant differences (P < 0.05) on the attitude measure between the treatment groups and the control group and between the faster paced versions and the original version. the rapid paced version was found to be the most effective. While no significant difference were found in interest levels, the rapid pacing rate significantly increased viewer attention.

The thesis can be downloaded from the repository at Concordia University here.

It is worth reading this piece because it gives us an insight into early empirical research of film and film viewership at a time when the hegemony of contemporary film theory was being established. 1974 was the year in which Screen published Colin McCabe’s ‘Realism and the cinema: notes on some Brechtian theses’ and Raymond Bellour’s ‘The obvious and the code;’ Jump Cut was founded; Film Theory and Criticism first appeared; and Christian Metz’s Film Language: A Semiotics of the Cinema and Language and Cinema were unleashed upon the world.

The thesis begins with a review of early research on film in an education context, and provides a series of references of research on the cinema before film studies. There are several references to research conducted on behalf of the US Army on the impact of the Why We Fight series. There are references to research going back to the early 1920s that are concerned with the nature of the viewer’s experience.

A couple of things stand out regarding this early work:

The concern with practical consequences for this early research – the apparent purpose behind much of this early research on educational film and television is to inform how future programmes should be produced to achieve their pedagogical aims

The concern with how people understand media texts – we would now consider this research to be a part of cognitive film theory as it is looks at the relationship between style (editing pace) and experience (attention and interest). The section on ‘Effects of Pacing on Cognitive and Affective Learning’ (from page 20) provides a summary of work in this area from the 1950s, but which would not be taken up in film studies until long after.

On page 6, Marks bemoans the fact there has been little research on style and its relationship to the viewer:

Although a great deal of film theory is based on the editing process, and has been since the beginning of motion picture production, there has been little to no systematic testing of the effects that the editing of films or television programs have on viewers.

It was not until the mid-1990s that film scholars began to address this lack as cognitivism began to become recognised as a central part of film theory. Marks sets out to address precisely this issue and asks a very simple question:

What relationship exists between the rate of pacing a highly visual information TV presentation and the attitude scores and interest levels of viewers?

This is carried out as a psychological research project, and clearly sets out the hypotheses being tested, the methodology being used, and the statistical analysis conducted. It therefore adopts a very different approach to the mainstream of film theory that had emerged in the examples given above.

This thesis is an example of how film studies could have been done – but instead it chose a different path.

Back in January, when I posted an bibliographical update of works published on cognitive film theory, I was asked if I would do an update of the larger bibliography of research in this area that appeared the year before. Six months later, I have finally done this, and the link below will take you to the new version as a pdf file:

This bibliography now contains 425 different items, which means that 70 new references (highlighted in red) have been added since the old version was put on-line. By ‘new’ I mean only that items did not appear in the earlier version, and though many of these are from as recent as 2010 some are much older. Again, this bibliography makes no claim to being exhaustive, but what is there should be accurate. (I’ve also tidied up a few things since the last version but these were very minor).

I haven’t included any references from 2011 becasue the year isn’t over yet, and you’ll just have to wait until next January for that update.

One area that I have also left to one side, but which could make a very large contribution to the bibliography is research on cognition that uses film clips to explore cognition but which is not related to film studies (i.e. the research doesn’t necessarily use commerical films). One area of considerable research using film in this way are studies of ‘affective style,’ and a handful of references with links to papers by Richard J Davidson are listed below as examples. I may decide at a later date to add such research to the bibliography, but for the time being it remains in limbo.