How to integrate text and images interactively

‘Show, don’t tell’ dictates the famous credo in creative writing. Maybe it should say: show and tell. Communication often benefits from the combination of text and images. Within one message, some meaning is best encoded using written language while other information is best expressed in a picture, a graph, a diagram or a map. Combinations of these representational modalities are part of all kinds of genres, like news reports, textbooks, scientific articles, advertisements, comics, and even novels.

Although text and images can complement each other and thus can benefit comprehension, their combination also forces the readers to split their attention between the two. To reduce this split attention effect it is mandatory that text and image are optimally integrated. This can be achieved using design (for example by placing related textual and graphical elements close together, or linking them with lines or arrows) or by creating linguistic relations, like a textual reference to an image (‘see figure’), or its content (‘as the figure shows, there was a peak in January’).

Digital media have added a powerful new way to integrate text and images: interactivity. Let me explain by example. The model on the right, based on the dissertation of Cengiz Acartürk (pdf), shows how a combination of correlated text and image (a so-called multimodal constellation) consists of a text layer and a image layer. The first contains the written text, the second is divided in two sublayers: one containing graphical elements the other textual elements (for example a title, axis labels, numbers and verbal annotations). All layers and sublayers can be linked together by referential relations. Examples are written words in the text layer that refer to the image as a whole (link 1) or to the content of the graphical sublayer (link 2). Within the image layer, textual elements can be linked to graphical elements (link 3), for example by pointers. As this paragraph illustrates, all these links also can be interactive.

Several researchers have experimented with such hypertext connections between text and images. In the late 1990s, human-computer interaction researchers Pete Faraday and Alistair Sutcliffe studied interactive co-references between text and animations and measured a positive effect on comprehension. More recently, educational psychologists Tina Seufert and Roland Brünken also concluded that hyperlinks can guide the attention of the reader to relevant connections between text and images.

Interestingly enough, interactive text-image references are rarely used on the web. Sometimes they appear in long forms, the storytelling format that has become quite popular the last couple of years. For example in Snowfall, the archetype of this new genre created by The New York Times, hyperlinks in the text can be used to start correlated video. More sophisticated are Bret Victor’s explorable explanations and reactive documents, that allow users to manipulate text and images thus enabling what Victor dubbed ‘active reading’. Be sure to try out his redesigned version of Watts and Strogatz’s seminal paper on ‘small world’ networks.

From there it’s a small step to interactive links between text and data visualizations. Especially when visualization is used to communicate rather than to analyze data, good annotation is essential. Accompanying text can tell the audience not only what the visualization is about but also how to read and use it, for example by highlighting interesting parts or applying filters.

Let me again illustrate with a example. This chart shows the increase and decrease of the population of Amsterdam during the last decades. The red line indicates the total growth or decline. Except for the years 1994, 1995 and 1996 the population has been growing. This is mainly due to natural increase: more births than deaths. The last couple of years the net domestic migration also has been positive: more people in The Netherlands move to the capital than vice versa. The same is true for the net international migration although the influx of citizens from abroad is considerably lower than in the early nineties.

This chart is part of a project I did for the Research and Statistics Bureau of the City of Amsterdam that contains several more examples. If you have different thoughts on the interactive integration of text and images, or other examples, feel free to share them in the comments below!

4 comments on “How to integrate text and images interactively”

This is an exciting field to explore and the prototype already shows some interesting patterns, like corresponding colors between highlighted text and chart elements.

This approach requires some new design and interaction patterns. For example, how do we show to the user that there is an interaction between text and visualization? Underlining text may suggest a link towards another page. Do we have to develop a new icon for this that may become part of our common icon knowledge? Or can we think of other ways to indicate interactivity?

If we have several charts, do we have to show how interactive elements in the text are mapped to what chart?

Perhaps we also have to structure the text to facilitate the understanding of the interaction. It should not become an easter egg kind of game where one is invited to find hidden treasures and thereby losing the focus on the message or textual content.

Thank you for your reply, Eugene. I think you are right that the design of the links is an important factor to make these interactive references work. As you noticed, I tried to match the colors of related elements when highlighted, as is common practice when applying brushing and linking techniques in data visualizations. To distinguish contextual links from regular hyperlinks I used dotted underlines, as is convention with contextual elements like <abbr> and <acronym>.

Off course there’s also the issue of cognitive load: each link (contextual or external) is a small distraction from the actual message, so the challenge is to find when the benefit of contextual links in lifting the split attention effect is optimal and the negative effect of distraction is minimal.

Bas, what an interesting and smart way to turn data into a story that readers can understand and remember. I recently retired from Statistics Canada, where we frequently used charts like the one you use in your example. This seems an excellent way to blend figures into text, as we recommended in our analytical and plain language writing courses.

I’m curious about accessibility. Do these adapt well to screen readers for the vision-impaired?