Rethinking a Dangerous Art Form with Dr. Will Thalheimer

Smile sheets (i.e., learner response forms, student reaction forms) are perhaps the most widely used tool in the learning industry for gathering feedback and attempting to assess the effectiveness of a learning event. The problem is, there is almost no correlation between smile sheets and learning results. The good news is that if we are thoughtful in the design our smile sheets, there is the potential to get reasonably good data related to learning effectiveness.

In this episode of the Leading Learning Podcast, Celisa talks with Will about the dangers of smile sheets, ways to encourage learners to take them seriously, and how to make them a more effective measure of learning. To tune in, just click below. To make sure you catch all of the future episodes, be sure to subscribe by RSS or on iTunes. And, if you like the podcast, be sure to give it a tweet!

[02:56] – An introduction to Will and some background information about what he does.

[04:42] – Your recent book, Performance-Focused Smile Sheets, is subtitled A Radical Rethinking of a Dangerous Art Form. Can you give our listeners a sense of the dangers inherent in traditional smile sheets? Will defines smile sheets as the questions we give to learners or conference attendees at the end of a session. He explains that the danger is we are getting information that isn’t telling us anything of value. He shares the results of two meta-analyses, which showed there was almost no correlation between smile sheets and learning results.

[07:07] – You advocate doing both end-of-learning and after-a-delay measurements of learning. Can you talk about why there’s the need for both and do you have any tips for how to encourage learners to respond to our smile sheets or other measurement mechanisms? Will explains that the benefit to asking immediately is you get people’s view of how it went from a top-of-mind experience. However if you ask them later, you can focus more in on whether there’s value and utility of what they learned. He reveals that the number one thing we can do to encourage learners to take our evaluation forms seriously is to ask them good questions. The second thing he says you can do is to make a passionate plea for the value of the evaluation forms. Will adds that it helps if you make changes based on the evaluations and share this with learners so they take them seriously.

[10:30] – Many of our listeners are at organizations that hold an annual conference with educational sessions. How would you recommend those organizations handle measuring the learning at such an event? Will says the number one thing to think about when you’re measuring is what do you want to measure and why? He explains the importance of giving opportunities for both overall conference feedback and session level feedback.

[12:30] – At multiple points in your book, you point out that what you’re after is better measurement—not ideal measurement. Can you talk a little bit about that progress-over-perfection perspective and what that does for learning professionals? Will shares that we need to ask ourselves, “What are we aiming to measure?” and “Do we have a reasonably good measure of that metric?” He talks about his journey to try and make smile sheets better and how it’s an ongoing process (even asking learners how he can make his smile sheets better). Ultimately, it’s about having questions that give good feedback and support the learners in making decisions.

[16:26] – A discussion about the importance of the fourth pillar (after-learning follow-through) and how this can be difficult when the learning isn’t in an organizational training setting. Celisa and Will talk about the notion of subscription learning or having an accountability buddy to follow up with the learning.

[18:15] – Will explains how the four pillars relate to measurement and how they embody the science of learning. He explains that we need to design based on the science of learning but also ask questions based on these principles. He shares a few examples including a question from a specific smile sheet to illustrate this.

[20:27] – One of the points I loved in your book is the idea that the smile sheet questions can send stealth messages. Would you explain what stealth messages are and why they’re an important part of the value smile sheets can deliver? Will says that when it comes to human cognition, although we think we are proactive, we are actually reactive. He explains how there are things we can do on a routine basis that can send stealth messages. He shares an example of this to illustrate that the questions you ask can send a message about what you think is important.

[23:22] – A further discussion about stealth messages and how you can use them in a conference setting to communicate something about your brand. For example, if you send the smile sheet to speakers in advance, they can design their presentations based on what you are emphasizing in the questions being asked and everyone who fills it out sees what’s important to the organization. Will shares an example.

[25:14] – Another point you make in your book is that we need to have a pre-established idea of the acceptability of responses to our smile sheet questions. Can you talk a little more about setting standards for question responses? Will says with smile sheets, we typically use a Likert-like scale (Strongly Agree, Agree, Disagree, etc.) or a numeric scale and he explains how this often leads to bias and paralysis. He suggests not using either of these scales, but rather concrete answer choices. Will recommends getting with stakeholders in advance to establish what the choices are and what the acceptable answers are which will ultimately make it easier to determine your effectiveness/next steps.

[28:47] – When you think about the future of learning measurement, what do you think or hope might change in the next five years? Will reveals that he hopes we have better smiles sheets and better mechanisms for gathering additional information. He also talks about the value in asking scenario-based questions.

[31:12] – What’s your approach to your own lifelong learning? Will talks about the importance of keeping an attitude of openness to experiences and information.

A reminder to check out the Leading Learning Symposium, an event designed specifically for senior leaders at organizations in the business of lifelong learning, continuing education, and professional development. The event will help you maximize the reach, revenue, and impact of your education business and takes place this year on October 24-25 in Baltimore, Maryland.