Sunday, March 04, 2012

New Article: Comics and the brain

I have a new paper available online (pdf), and I'm proud to say that this is my first brainwave study on comics. In this paper, now published by Cognitive Psychology, we argue that sequential images use a narrative "grammar" to distinguish coherent narrative sequences from random strings of images. We conducted two experiments measuring reaction times and brainwaves to examine the contributions of narrative structure and meaning to processing sequential images. Our findings provide evidence that sequential image comprehension uses a narrative structure that goes beyond "transitions" between panels.

Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.

Thanks for the comment Seth. I actually have a copy of the Art of Comics and have only thumbed through it so far.

My most interest, of course, is on the "Language of comics" chapter, which seems to be completely unaware of my research, though it is the most prominent approach that compares comics and language.

It's also worth noting that the main trouble that article has with the idea that sequential images can be language-like is with regards to a "visual syntax", which my brain study posted above shows neurocognitive evidence for their being.