The Science Of Comic Strips

Our brains recognize them as a distinct, and complex, "visual language."

Language is more than just a series of words strung together. A sentence must have some essential structure, some system of rules governing words and clauses—a grammar. You don't have to be Strunk or White to recognize this system at work; it's automatic in the brain. In Noam Chomsky's famous example, people know that the meaningless sentence "colorless green ideas sleep furiously" is grammatically correct the very first time they see it.

Psychologist Neil Cohn of University of California at San Diego believes comic strips operate the same way. Far from just a series of panels strung together, comics represent to Cohn a coherent, complex "visual language." His research over the years—culminating in a new book called The Visual Language of Comics (Bloomsbury)—suggests that our brains register the presence of a grammatical system in the funny pages just as they would in a book's pages.

"We would expect that the brain would treat, say, grammars of different systems in similar ways," Cohn tells Co.Design. "That's kind of the motivating assumption."

Cohn says any language has a "holy triumvirate" of elements: expressive form, grammar, and meaning. Comics, he argues, meet all three requirements. Their expressive form is the visual strip. Their grammatical structure consists of a basic vocabulary (such as stink lines or speech bubbles) and a syntax (a hierarchical panel structure). And, when done right, the images have a semantic relationship—a clear message.At a strictly cognitive level, people treat comic strips as a normal sentence.

The panel structure that forms the basis of comic grammar deserves a bit more explanation. Take the following six-panel Peanuts strip from one of Cohn's research papers. At the top of the hierarchy is a narrative arc, more or less equivalent to a sentence. From there it's broken down into "initial" and "peak" sequences that behave a little like clauses. These then break down into various components somewhat analogous to nouns, verbs, and other parts of speech.

"Comics end up being visual languages because they obey these properties," says Cohn. "They have a sequence, and they also have a systematic ways of drawing things."

Some of the best evidence that comics function as a visual language was published in a 2012 issue of Cognitive Psychology. For the study, Cohn and his collaborators showed test participants four types of comics strips. One was a "normal" strip that represented a meaningful, grammatical sentence. A second strip had panels with a semantic relationship but no grammatical structure. A third strip was grammatical but meaningless, in the spirit of Chomsky's "colorless green ideas sleep furiously." The fourth was a scrambled mess.

Sitting at a computer screen, the study participants were shown a "target" panel to identify in a strip. They were then shown the whole strip and had to push a button when the target appeared. The study was designed to echo a famous sentence structure experiment conducted in the 1970s, which found that people process "normal" sentences more quickly than ones that violate linguistic rules. Cohn wanted to see whether his participants reacted to comics the same way.

Sure enough, they did. Cohn and his fellow researchers found that reaction times to the target panel were fastest in the "normal" sequence. Target reactions in the semantic- and syntax-only sequences, shown second and third above, were slower than normal (though similar to one another). Reactions in the scrambled strips were slowest of all. At a strictly cognitive level, people did seem to treat normal comic strips as a normal sentence.A lot of the complexity of comics is not recognized.

In other tests, Cohn dug deeper into the brain. Using electrode caps, he measured the neural responses of test participants as they read various types of comics. He found that participants reading a "normal" comic strip showed the same brain activity that people show when processing semantics in standard sentences. He also found that participants reading an abnormal comic—with a blank panel inserted mid-strip, to interrupt the underlying structure—showed the same brain activity that people show when registering grammatical violations.

"I'd say a lot of the complexity [of comics] is not recognized—certainly not the idea of grammar," says Cohn. "That is controversial, and people often are very surprised by the results that I show."

Cohn's research probably says more about the brain than it does about comics themselves, but it does have creative applications. Understanding the "visual language" of comics should help artists draw strips with more engaging narratives. The work also suggests that young artists who want to learn this language would be better off buying a book on how to draw comics (the equivalent of a Rosetta Stone program) than an actual comic book (the equivalent of a novel).

Understanding the visual language of comics could help artists draw strips with better narratives.

Before becoming a research scientist, Cohn was a professional comics artist. As a kid he spent lots of time at Comic-Cons, and he's collaborated on full-length graphic novels. This background as a native comics speaker, if you will, gives Cohn a leg up in the laboratory. He can create panel sequences that reflect proper semantics and syntax, and therefore create studies that measure true linguistic understanding, whereas other researchers might just be stringing panels together. "The fact that I was a comic creator, and have been doing this for so long, means that I'm fluent in the visual language," he says.

Add New Comment

20Comments

There is actually a debate amongst comic books scholars as to whether they can be compared to a linguistics. I am not an expert in the field yet, but I am aware of Groensteen's assertion that comics should not be analyzed in the literary tradition and that they cannot be broken down into morphemes like Cohn suggests.

I think there is a difference between a visual language and what Barthes calls langue. Langue, which Cohn points out is not always arbitrary, is in fact coded while art and images create meaning from their focus on depiction.

On McCloud. Even if McCloud had said everything that needed to be said (and McCloud would be the first to challenge that assumption), it remains that his work is simply that of a practitioner trying to explain how his craft works.

It isn't an academic work, and he doesn't pretend that it is. And we do need rigorously vetted academic studies of the language of comics.

And yes, those studies will eventually have to cover all the same ground as McCloud did in "Understanding Comics" and the companion books, the difference being that everything has to be theoretically grounded, empirically tested and thoroughly peer-reviewed.

In the course of that academic work, any weaknesses in McCloud's work is sure to be revealed, ideas challenged, some ideas confirmed and a deeper understanding of comics will be possible, from an academic perspective.

From an academic perspective McCloud's work is an extremely interesting road map, but it is inadequate as basis for further theoretical work. If you were to use McCloud as a basis for, let's say, an analytic study of Neil Gaiman's Sandman, you could refer to McCloud's taxonomy, but you would first have to show how that taxonomy relates to already vetted and respected academic theories on semiotics, composition or communication.

That is why academic studies and works are needed that appear to cover the same ground that has already been "covered" by McCloud.

Think of it as McCloud being the psychich who told you who the killer was, how he did it and why, and the academics as the cops and forensic scientists who have to find the proof that will stand up in a court of law.

i'm for correct attribution, but the photo belongs to the photo owner. if you look carefully as well, schultz's signature is on each one of those strips (it appears vertically). this indicates that the comics were authored by schultz. if someone took a picture of a picasso, i would cite who took the photo rather than picasso himself, who would have had nothing to do with its genesis. a photo here is a primary source, assuming charles schultz (or his estate) have consented to reproduction in photographs.

It gets even more confusing - the correct attribution in this case is to the book in which the photo of the cartoon was published. As this is research, the reproduction of what was drawn by Schultz is deemed to be 'fair use' undermost jurisdiction's copyright law.

I'm not sure this work has anything new or worthwhile to say that hasn't been said by Scott McCloud or Marshall McLuhan before him. That comics is a visual media with its own syntax is not crying out to be proven by anyone.

It's called "science", Josh. It's a process in which we study things to determine whether they're true or not, rather than accepting them because a smart cartoonist or philosopher said them and they seem right. In the process we learn a little more about how and why.

Didn't have to say anything new. I've never heard of those other works. now that I've seen this, and seen your reference to them, I think I'm interested enough to look them up. This is an article that piques an interest in finding out more. It did the job nicely.

I think we encounter a problem here, Eric. It's a great study, and it's so interesting to see what makes (especially Charles Schulz' genius work) comic strips such a fun and powerful medium. But once everyone "understands" what makes the comic strip work, there's a danger that we'll start building (or encouraging young artists to build) comics based on the deconstructed pieces of the research, rather than the more organic intuition of the artist. Sorry - the idea reminds me of the first baboon to go through Brundle's Telepod in "The Fly": we don't want to end up with synthetic monkey.Thoughts?

I don't think there's any danger in teaching cartoonists how and why the medium works, any more than there is in teaching grammar to essayists or composition to photographers. No one capable of intuiting how to tell a story thru sequential art is going to stop using that skill and instead start doing it by formula.But understanding how it works provides a tool for when that intuition fails (not everyone is a Schulz, Kirby, Eisner, or Watterson). If readers complain that my comics are confusing to read, I can evaluate the work in the context of these terms and figure out what I'm doing wrong.

Great point, Jason, and well put, for sure - thanks. I just worry (perhaps needlessly - most are probably smarter than I think) that budding artists will rely more on a formula than on their own courage and creative instinct. It is magnified as our "hyper-encouragement" culture keeps telling young people they can do anything, instead of helping them find their unique talent.But you're right - no matter how nice your car is, you have to 1) learn how to use it, and 2) tune it up and fix the broken bits every now and then...

I love the explanation of the various sequence structures. When I taught English to middle school students, I would use comic strips to explain what an inference was. I would eliminate the final panel and students would have to draw the last panel as they envisioned it, based on the other panels. It was a great lesson, and the sequence of the panels always influenced how close the students' inference was to matching the actual final panel. Great article.

Expecting a 1000-word article about a specific body of research to cover the entire field of study is a bit unrealistic. And I'll wager $100 that McCloud is referenced in the book that this article is about.