New evidence that bullet-points don’t work

At last, we have some scientifically rigorous evidence to show that slides full of bullet-points don’t work.

The research is the work of Chris Atherton, a cognitive psychologist. Chris recently delivered a presentation at the Technical Communication UK Conference and has put up her slides on slideshare. There’s been a tremendous amount of interest in them, but as they were designed to complement Chris’s talk – they only tell half the story.

In this post I’ll explain the findings of Chris’s research. I’ve written the post based on Chris’s slides and asked Chris to comment on various aspects. Chris has also reviewed this post to make sure I’ve got all the science right.

The research

Chris tested the effects of using two different types of PowerPoint slides in a presentation. Students were randomly assigned to two groups. One group attended a presentation with traditional bullet-point slides (with the occasional diagram) and the second group attended a presentation with what Chris calls “sparse slides”, which contained the same diagrams, but minimized the amount of text, and broke up the information over several different slides. Both presentations were accompanied by the same spoken narrative.

Here are samples of the slides used:

A. Traditional bullet point with the occasional diagram

B. Sparse slides

Chris tested the students’ learning in two ways – multiple choice questions and short essay answers. There was no significant difference between the groups on the multiple choice questions. Chris comments:

This is most likely because it’s not very hard to pick out the correct answer from among distractors when you have only recently been exposed to the material and your memory of it is quite fresh.

Before marking the short essay answers, Chris worked with two independent people to identify the themes of information in the presentation. They identified around 30 themes by consensus. The short essay answers were then marked by counting how many of those themes the students wrote about.

As you can see the students who were in the presentation with the sparse slides did much better than those who saw traditional slides.

Theory behind the research

There are a number of theories which can be used to explain these results (if you’re not interested in the theories, scroll down to the next section “What does this mean for your presentations?”):

1. The limitations of working memory

Even the students who did well in recalling themes, remembered only 6-7 themes out of a possible 30. Chris suggests this is due to the limitations of our working memory. Recent work (Cowan 2001) has estimated working memory capacity to be around 4 chunks of information:

2. Two processing pathways

The brain has two major pathways for processing information.

The auditory cortex and the areas around it are involved in processing language – both spoken and written.

When a presenter uses bullet-point slides, they’re not using both pathways as effectively as they could. The audience member has to read the words on the slide and listen to the presenter at the same time, leading to overloading of the language areas whilst leaving the visual cortex with very little to do:

Chris notes:

The visual cortex is involved in reading the words on the screen – it works on the lines and features to assemble the words that are being read, but it’s not really being used to the full, since there’s usually little color or texture information.

3. Cognitive load

The theory of cognitive load was developed by John Sweller. Cognitive load is the amount of work required to understand or learn something. There are two main types:

Chris suggests that the sparse slides may minimize extraneous cognitive load by creating fewer competing demands on attention — that is, because we don’t need to spend very long processing the visual elements, we have more attention for what the speaker is saying. She adds:

Having anything on a screen invites people to look at it, the same way their gaze would keep returning to a TV screen in a pub. Since you can’t control the audience’s visual attention, it’s all about controlling what visual information you make available at any given moment, and minimising what is there so it’s not distracting from the spoken narrative, while also ensuring that it is congruent with what you are actually saying.

4. Better encoding of information

Encoding is the process of putting something into your memory. McDaniel and colleagues have shown that a little more effort at the encoding stage can be beneficial to learning. Chris suggests:

Sparse visual cues could lead to better encoding of information — that is, having to work a little bit harder to integrate the speaker’s narrative with the pictures might actually improve our storage of the information (obviously this is only true up to a point; having to work too hard at integrating the two could actually be counterproductive, effectively producing a situation with high extraneous cognitive load).

What does this mean for your presentations?

1. Don’t say too much

Limit what you cover in a presentation. Your audience has limited capacity to take it in.

2. Split the load

Take advantage of the brain’s two pathways. Design your slides so that they can be processed quickly by the visual cortex, allowing the language areas to focus on what you’re saying. This means using more pictures and as few words as you think you can get away with.

3. Get rid of visual clutter on your slides

Do what you can to minimize the extraneous cognitive load on your audience. For example:

Only put on your slides things you want the audience to focus on.

Split information between slides rather than having it all on one slide, so that you can direct the audience’s attention where you want it.

4. Make your audience work

There’s some evidence that making your audience work a little to understand your point will make your point stick better. A big caveat to this is that obviously you mustn’t make it so hard that they don’t get your point at all. Some ways of doing this are to:

Show a picture that the audience has difficulty relating to what you’re saying. Either ask them to guess the relationship, or explain the relationship to them.

Show them the axes of a graph, and ask your audience to guess the way the data goes (give enough clues that they’re fairly likely to get it right – without making it too easy).

Thank you Chris

It’s great to have some solid experimental evidence on the use of slides in a live presentation, to back-up what so many presentation authors, trainers and coaches have been saying. I’m also deeply indebted to Chris for her help with this post. I’m looking forward to whatever research Chris does next.

56 Comments

Olivia – This is terrific. Thank you! I am a speech coach and have been waiting a long time for this kind of research. Thank you for posting. I am adding your blog to my reading list and invite you to add mine!

Thanks for sharing this research. So much of this applies to e-learning courses too. I think many of us applied some of these approaches to our online course design, but now we have the research to back it up.

In the online learning area there’s a lot of research especially by Richard Mayer. In the past, in the presenting world we’ve relied on this research. So yes there is definitely overlap between the two areas of research.
Olivia

I agree with the above comments, and love this research POV! Anyone who has been in one of my presentation or facilitation skills classes knows my perspective on this topic: Slides are not the main event. Slides are there to enhance and support the understanding of key messages, but we as presenters are the key visual aid to driving that understanding. If the slide upstages you, you’re doing something wrong.

If I’m reading the research correctly, it looks like the primary problem arises when asking the brain to do two things at once by talking as the presenter while showing a bulleted slide. One way I’ve found to help with this problem, in addition to slimming down slides to the bare minimum (They’re not speaker’s notes!) is to preview a slide I’m about to show, and give the audience a focused viewing task for any bulleted slide. For example, if I’m about to show a traditional ‘Learning Objectives’ slide, I’ll tell the group, “You’re about to see a slide of bullets showing the objectives of this session. Once I show it to you, I’d like you to skim through it and let me know what looks most interesting to you on that slide.” Click to reveal the slide. Then, I provide them with time to read the slide (while I read it silently to make sure I’m allowing enough time), process it, and share whatever they find most interesting. Amazingly, using this approach, you can have a great dialogue about something as mundane as learning objectives, providing the presenter with input as to how to adapt and focus the remainder of the session. It’s the same bulleted slide used differently to better results.

Again, I’m no fan of bullets. However, if you have to have them there for some reason, using this ‘previewing and focused viewing’ approach can help to increase engagement and retention.

I really like this suggestion — even in my own teaching, where I am striving to eradicate slide text and bullet lists wherever I can, sometimes I still need to show some text on the slide. I love your approach of reaching out to the audience and effectively inviting them to do some metacognition — breaking the fourth wall of presenting/teaching, if you like 🙂

Another excellent, informative post. Thanks for publicising this research – I’ll be following up on Chris’s work – interestigly she’s based 20 miles up the road from me.

I think there are some difficulties translating the “less is more” approach to slides into practice for highly technical presentations – more work is needed on this.

A bigger problem is convicing clients. They are so used to the old style slides crammed full of words that it can be difficult to convince the that the “less is more” approach is more effective. I’ve had to face this recently.

In a discussion about your post at work a colleague wrote this in response: “I want to make a point (that may seem like splitting hairs, but the cognitive psychologist in me insists) and that is this: I don’t think that Chris Atherton said anything about the efficacy of bullet points per se, rather that in his test condition, the words and pictures interfered with one another, that is where the pictures and words said the same thing, they caused cognitive competition that got in the way of encoding the information. Also, that a cluttered slide makes it hard for the brain to figure out where to look. Simplicity, but done in a way that people have to think about what’s being shown or said on the slide (increase extraneous cognitive load), as opposed to being mentally confused by the same info entering two channels, was the ultimate message, not that bullets or words don’t work. (I hate lists of bullet points myself, and don’t want us to throw the baby out with the bathwater here.) I love the article on presenting by Steve Jobs [http://bit.ly/1TVRbx], which adds something to our understanding of this whole issue, I think: “You see, neuroscientists are finding that information is more effectively recalled when the ideas are delivered as text and pictures instead of text by itself.””

Your colleague’s point about bulleted text is interesting: while we weren’t explicitly testing text vs. pictures, there was relatively little text on the non-traditional slides, and certainly less text present on any given slide than in the traditional condition. So I think one could infer from this that bulleted text is less useful than short, simple text, but it depends what you consider to be “bulleted text”, I guess.

Cognitive psychology theory suggests that information goes in more easily when we don’t have to process words with our ears and our eyes at the same time … there are times when I think it’s advantageous to show a bullet-list (Ken De Loreto gives a nice example in the comments above) but my problem with slideware is that it routinely suggests bullet lists, far often than they might actually be genuinely useful.

[effusive praise] Have I told you lately how much I appreciate your blog? This is fantastic information. I’ll have to come back to reread it because I think I overloaded my working memory with RSS feeds. [/effusive praise]

Thanks for the great post concerning Chris’ research. It of course confirms and gives a good solid grounding for what many of us have been arguing for years. To Mike’s point about “highly technical” slides, I would respond that I’ve worked with many highly technical people and their slides and always found ways to simplify without distorting or losing information.

Great post! The lack of a strong editing process is usually the culprit of overloaded presentations. We struggle to cut “good” content to focus on the “best” elements. Thanks for showing how keeping all those good points can dilute overall learning.

Ah, the often skipped task of editing! Agreed. I’ll add to that the often skipped task of rehearsing. When working with individuals or teams responsible for creating customer-facing presentations (for example for sales ‘pitches’) two of my favorite tips are:

1. Edit: Create what you consider to be the ‘perfect’ presentation, and then cut it in half.

2. Rehearse: If you’re working on the slides the day before or of the presentation, you’re focusing on the wrong things, and probably avoiding the larger, more daunting task of preparing to engage the audience.

Bottom line for me: Slides are not the presentation. We are. However, it seems to be easier, and almost seductive, for presenters to dump everything they need/want to share onto slides. Their thinking is, “If it’s ‘in the presentation’, then the presentation must be good.” Oh, the siren’s song of PowerPoint. A metaphorical crashing onto the rocks happens as a result of this thinking. The presenters (or sailors to stay with the metaphor) never even notice that they’re creating dense, murky, and lifeless slides that will prove cumbersome when live with an audience. By not editing down the slides, and not rehearsing how to engage the audience with the edited slides prior to going live, many presenters don’t discover the traps of certain slides until they’re live with an audience. Not the time to realize that, particularly in a selling (or sailing) situation.

Thankyou Olivia for adding the meat to the excellent bones of Chris’s slides! I’ve also just read through all the comments, so many good points and examples. I too have had (and continue to have) some difficulties with some clients. Specifically these are clients who have asked me to design learning resources for face-to-face workshops. I have no trouble introducing learning activities, preparing background resource materials BUT so many clients want EVERYTHING on Powerpoint – I mean everything!! I’ve been doing some work trying to explain the learning benefits of altering their use of Powerpoint (facilitators who read their text-dense slides to learners), so thanks to you and Chris for supplying me (us all) with some research that may help make the task more successful!

So when I’m working with clients, we look at finding alternative safety nets. On our courses, we cover preparing notes before we even get to the PowerPoint. Knowing that they’ve going to have notes – makes a huge difference to their receptivity to the PowerPoint message. (When we used to do notes afterwards – we had a lot more resistance – nowadays we hardly have any).

There are different challenges when you’re preparing Powerpoint slides which may be used by a number of different presenters – as in your case. But it’s still about making sure that you provide an alternative – it could be using the notes pane of Powerpoint or providing a separate facilitator guide.

I’m afraid I have to agree with the comments of Simon’s colleague. The point that bullet slides do not work as well as sparse slides may or may not be true but it is not proven by the experiment if the sample slides are indeed typical. The slide described as “bullets with occasional diagram” is not in fact what it says. The sample is bullets **interrupted** by a diagram and thus confusing – existing research already confirms that this will not work. The situation is made worse by the confusing sequence of the text (or content) itself.

Assuming that the “essay” method of listing themes is a valid way to check recall (I am not sure about the way it was done) we can agree that of “sparse” slide worked better. But it has nothing to do with proving or disproving the more general conclusion that slides with bullets do not work if they are created right (cognitive guidelines) and presented right (not all at once).

I agree with everything that Olivia suggests in 4 points to make presentations work. The experiment stands on its own to prove that the sample “traditional” slide doesn’t work. But wider conclusions about general efficacy of bullets from this limited and “straw man” experiment are not required.

If we were to take a really strict interpretation all we could say was that on that day, using those slides, one set worked better than the other.

The issue is how far we generalise from that conclusion. Chris Atherton commented on her own blog:

“I can actually live with overgeneralisation in all but its most egregious forms, because to some extent all research has this. A little generalisation is okay; else all research would be hopelessly, pointlessly niche.”

I feel comfortable saying “there’s evidence that slides full of bullet-points are not as effective as visual slides”. Of course there’s a whole lot of qualifications that can go into that eg: the visual slides have to be relevant, well-designed etc.

Olivia

Deano647
on May 10, 2010 at 4:02 pm

Chris wrote:
“Having anything on a screen invites people to look at it, the same way their gaze would keep returning to a TV screen in a pub. Since you can’t control the audience’s visual attention, it’s all about controlling what visual information you make available at any given moment ..”

I very much disagree with the above and suggest that as presentation designers, we need to be aware that how we design visuals, directly affects WHERE the audience looks, WHEN they look and for how LONG … and as we all know, with their gaze goes their attention.

When you take into account cognitive load, this means part of the design process is literally deciding where you want the audience to look at a given time in the speech. Some clients find it hard to believe that we can tell 6 weeks before a presentation where the audience will be looking right at an exact second in a presentation, but when they are in the audience watching the presentation being delivered (not the actual presenter) They finally get it. And it’s very consistent across the audience.

If trying to communicate a complex point or concept, sometimes it’s very effective that the audience just LISTENS carefully. This can be aided by ensuring the accompany visual is actually quite simple, so audience attention (visual and aural) is focused solely on the speaker.

How often have you seen a very complex process or information flow represented in a slide ? Does it fit on screen ? … well yes. Is it nicely, designed, laid out and clearly labeled and readable on the day ? … well yes. But unless the speaker steps throughout hat chart, in detail and is a good enough presenter to know how to address it properly, that slide may hurt more than help … no matter how well designed or laid out.

The ability to put oneself in the seat of the audience is not just from a design/layout perspective (is this going to be readable from the back of the room?) … but also from the point of view of knowing what contrasting effect ‘busy’ and ‘simple’ slides will have on an audience and the subsequent success or failure of that information transfer … and concurrently, presentation effectiveness.

Ok… it could be true that bullet points are less effective than the presented alternative… at least for the audience’s benefit. But, when I started my speaking career, I had the bullet poiints for ME, not for the audience (while, of course, they were under the impression that the slides were for them).

Many speakers – especailly beginner speakers – use PowerPoint slides and bullet points as a crutch. In my first two years, or so, in my speaking career, I would have been “dead” without my PP slides and the bullet points. I was terrified that I’d forget what to say next, and the bullet points helped me stay on track.

Now, 7 years later, I only use PowerPoint in about 40% of my presentations and I use no more than 7 to 10 slides even in a full-day workshop… I use it mostly to show graphs or data.. or rarely some acronyms… well, and some games or humor here and there…

So, yes, I agree – bullet points migh not be very effective in getting the message across, but it is a much-needed tool for the beginner speaker… and I dont’ think that will change as a result of knowing about this research 🙂

With all that said, thanks for sharing. I always love to learn about new developments in Public Speaking.

Hi EG
I totally agree that bullet-points are immensely useful to the presenter. They are their notes. And most people need notes to keep on track.

I encourage presenters not to show their notes to the audience. In more recent versions of PowerPoint (2007 and 2010) you can have the notes on your laptop screen without showing them to the audience (you can also do this in PP2003 but it is very clunky). Or I recommend hard copy note cards – these have the advantage of being technology-crash proof!

I agree that PowerPoint is a crutch for people. Not only in the way that you’ve pointed out, but also because it gives another focal point for the audience, which many nervous presenters appreciate. Part of saving the world from boring bullet-point presentations is giving presenters the confidence to present without bullet-points.
Olivia

Chris Pantages
on June 11, 2010 at 6:23 am

I learned this years ago from one of my law professors. I forget what subject it was (complex litigation maybe), but his class was unique because it was all about presentation over substance. Always use the same font in all communications, take full advantage of multimedia elements whenever possible, and no power point slide should have more than 15 words. I got in the habit of preparing a paper script with all the content I wanted to use to describe each slide so I could create clean slide decks.

Unfortunately, speakers use bullet points not because there is any evidence to suggest they help the audience. Speakers use bullet points because they are lazy and it is the easiest thing to do. Thanks for the solid evidence on why this shouldn’t be done.

There is very definite scientific explanation for why you are more effective if you limit the use of literature in slides which are used for live presentations. This goes back to the work for which Roger Sperry won the Nobel Prize in 1952, on the way the left side of the brain analyzes grammar—what your researcher apparently refers to as the verbal cortex.

The functioning of the verbal cortex, the part of the brain which analyzes language, is SERIAL. That is, the brain can only process ONE source of grammar at a time. If you have literature on your slides, and you are talking at the same time, that is two sources which the verbal cortex must analyze, using the rules of grammar. The brain cannot do that.

Most people already know this instinctively. You can sit on a beach with your spouse or lover, holding hands, and have a conversation while you are both looking out at the sunset, and not feel that they are ignoring you. But, if you come home from work and try to talk to your spouse, and they are reading the newspaper, you KNOW they cannot READ the paper and pay attention to what you are saying. They have to use the verbal cortex to analyze both, and the verbal cortex can only analyze ONE source of grammar at a time.

(The analytical function can ALSO be overloaded by complex visual information. You can be driving along the freeway and hold a simple conversation with a passenger. But if you then exit the freeway and come to a junction with lights and signals and signs to analyze to figure out what to do—you have to stop the conversation.)

This is different from the way the brain manages chunks. Chunks are packets or items. You can IDENTIFY and manage in your short term attention a number of chunks, four plus or minus one according to Cowan, seven plus or minus one according to Miller. The difference really dependings on the complexity of the chunks. You can manage a greater number of simple chunks, say, marbles lying on the floor, and a lesser number of complicated chunks, say, a set of values.

You can IDENTIFY and manage more than one chunk at a time, you CANNOT analyze more than ONE source of grammar at a time.

So, you want to construct slides without literature as much as possible. Why? Well, if you put up a slide with literature in it (grammatical constructs) and you are talking (more grammatical constructs) the audience cannot process both sources of verbal input at the same time. Several disastrous effects result.

First, the audience has to choose. Which source do I analyze? So, now there are THREE things to do. Analyze the grammar in the slide, analyze the grammar in what you are saying, AND figure out which to analyze. Oh, this is in addition to processing the visual input through the visual cortex.

Two of these functions cannot be done at the same time, reading the slide and understanding what the speaker is saying. The third function, choosing which to analyze, cannot be done intelligently—because you can not really know what you are choosing until you actually do the analyzing. So you have to choose which to analyze without being sure what it is that you will be analyzing, as you don’t know what that is until it has been analyzed.

The imposition of these three functions simultaneously and continuously, slide after slide, leads to a fourth problem—the brain disconnecting. This kind of overload creates discomfort at the biomechanical level. The brain is not masochistic. It tends to go away from anything which overloads it. Any functioning system does that. So, people start to get distracted, look away, check their notes or their phone messages, even get drowsy and fall asleep. The problem is not content, it is delivery of the content in a way which causes neurological discomfort.

Does this mean you cannot have literature in a slide? No, not necessarily. It means, if you do have literature in a slide, then you have to stop talking and let the audience read it. As in, “Let’s take a look at our mission statement.”, click, silence while the audience reads it, and then you go on…. Bear in mind you do not want to do this too often, because to some extent, even if you give people time to read the slide first, some of them will be tempted to go back and read it again as you continue talking. So, unless you are going to highlight some particular part of that literature, you would do well to move on to another slide, perhaps even a buffer slide. A slide which has a simple thematic picture which reinforces your main message without giving the audience any literature or other complicated visuals to distract them.

So, in my workshops, I give people the slogan, where power point is concerned, “Death to literature.”, and we also give people a three step approach to using power point, called VIP, the Visual Image Protocol, which makes sure that you sequence oral and visual input, and make active use of the slide thereafter.

The virtue of having a set protocol is, you don’t have to think about it any more. You just make the protocol a good habit, and it takes care of the rest!

John Pavel said:
“You can be driving along the freeway and hold a simple conversation with a passenger. But if you then exit the freeway and come to a junction with lights and signals and signs to analyze to figure out what to do—you have to stop the conversation.”

Which is why the fines for using your mobile ph except for handsfree are such BS. It’s not the *holding* of the ph that causes the lack of awareness … it’s the conversation. It is just as distracting to talk on the hands-free as it is holding the phone.

Olivia Mitchell
on October 12, 2010 at 7:29 pm

Wow! John

You take the record for the longest comment on my blog!
Thank you for adding your expertise to the debate.

In reading this research again, I am struck by the importance, clarity and simplicity of the findings. What is mind-boggling to me (and so much a part of why I have a job) is how difficult these findings are to embrace for so many presenters:

– ‘Less is more’ is exposing, leaving an unconfident presenter feeling unsupported.
– ‘Talking over a slide’ happens because the presenter has no idea what is next, and can’t transition to or position the slide without seeing it himself/herself.
– ‘Cramming too much in’ happens because editing is a second part of the process all too often overlooked in presentation preparation. Often, this results because the presenter is unclear as to the real problem s/he is trying to address. So, the only way to deal with that is to put too much in. It is like having an important guest to your house for dinner and not knowing in advance what they like to eat, or if they have any food allergies. In your desire to please them and appear to be a good host, you make too much food, still never feeling ‘ready’ and confident that you’ve made the right things. Lesson: Ask what someone wants/needs to eat before planning your menu.

Ah, when will we learn? On the plus side, having used these techniques in my own work for some time, and having just reworked a subpar client course using these techniques, I know they work. How? People remember things. People learn. People walk away wanting to do things differently. Isn’t that the real purpose of any presentation: To drive better performance and to solve problems? If not, put it in an email and just press send.

My bottom line: Just because you said it, doesn’t mean they got it. If you want them to get it, present it differently.

What this demonstrates once again is that presentation and rhetorical skills are a way of raising the level of what you could call “social” understanding for both the speaker and the audience, as the speaker upgrades and refines his own understanding in order to make it understandable for the audience in an effective way.

Your email address will not be published. Required fields are marked *

Comment

Name *

Email *

Website

Notify me of follow-up comments by email.

Got an important presentation coming up?

Got an important presentation coming up and:
You have so much content that you can't figure out what to leave out?
Don't know where to begin your design process?
Worried that your material won't be of value?
Feeling overwhelmed and can't get started?
Can't figure out your theme?
Concerned you won't be engaging?Time is running out?