Session Evaluations

I do a lot of public speaking. I write, teach, present and communicate at many levels. I love to do those things. And I love to get better at them. And one of the ways you get better at something is to get feedback on how you did.

That being said, I have to confess that I really despise the “evaluations” I get at most venues. From college to technical events to other locations, at Microsoft and points in between, I find these things to be just shy of damaging, and most certainly useless. And it’s not always your fault.

Ouch. That seems harsh.

But let me ask you one question – and be as honest as you can with the answer – think about it first:

“What is the point of a session evaluation?”

I’m not saying there isn’t one. In fact, I think there’s a really important reason for them. In my mind, it’s really this: To make the speaker / next session better.

Now, if you look at that, you can see right away that most session evals don’t accomplish this goal – not even a little. No, the way that they are worded and the way you (and I) fill them out, it’s more like the implied goal is this: Tell us how you liked this speaker / session. The current ones are for you, not for the speaker or the next person. It’s a popularity contest.

Don’t get me wrong. I want to you have a good time. I want you to learn. I want (desperately, oh, please oh please) for you to like me. But in fact, that’s probably not why you went to the session / took the class / read that post. No, you want to learn, and to learn for a particular reason. Remember, I’m talking about college classes, sessions and other class environments here, not a general public event.

Most – OK, all – session evaluations make you answer the second goal, not the first. Let’s see how:

First, they don’t ask you why you’re there. They don’t ask you if you’re even qualified to evaluate the session or speaker. They don’t ask you how to make it better or keep it great. They use odd numeric scales that are meaningless. For instance, can someone really tell me the difference between a 100-level session and a 200-level one? Between a 400-level and a 500? Is it “internals” (whatever that means) or detail, or length or code, or what? I once heard a great description: A 100-level session makes me say, “wow - I’m smart.” A 500-level session makes me say “wow – that presenter is smart.”

And just what is the difference between a 6 and a 7 answer on this question:

How well did the speaker know the material? 1 2 3 4 5 6 7 8 9 10

Oh. My. Gosh. How does that make the next session better, or the speaker? And what criteria did you use to answer? And is a “10” better than a “1” (not always clear, and various cultures answer this differently). When it’s all said and done, a speaker basically finds out one thing from the current session evals:

“They liked me. They really really liked me.”

Or, “Wow. I think I may need to schedule some counseling for the depression I’m about to go into.”

You may not think that’s what the speaker hears, but trust me, they do. Those are the only two reactions to the current feedback sheets they get. Either they keep doing what they are doing, or they get their feelings hurt. They just can’t use the information provided to do better. Sorry, but there it is.

Keep in mind I do want your feedback. I want to get better. I want you to get your money and time’s worth, probably as much as any speaker alive. But I want those evaluations to be accurate, specific and actionable. I want to know if you had a good time, sure, but I also want to know if I did the right things, and if not, if I can do something different or better.

And so, for your consideration, here is the evaluation form I would LOVE for you to use. Feel free to copy it and mail it to me any time. I’m going to put some questions here, and then I’ll even include why they are there. Notice that the form asks you a subjective question right away, and then makes you explain why. That’s work on your part.

Notice also that it separates the room and the coffee and the lights and the LiveMeeting from the presenter. So many presenters are faced with circumstances beyond their control, and yet are rated high or low personally on those things. This form helps tease those apart.

It’s not numeric. Numbers are easier for the scoring committees but are useless for you and me. So I don’t have any numbers. We’re actually going to have to read these things, not put them in a machine. Hey, if you put in the work to write stuff down, the least we could do is take the time to read it.

It’s not anonymous. If you’ve got something to say, say it, and own up to it. People are not “more honest” when they are anonymous, they are less honest. So put your name on it. In fact – this is radical – I posit that these evaluations should be publicly available. Forever. Just like replies to a blog post. Hey, if I’m an organizer, I would LOVE to be able to have access to specific, actionable information on the attendees and the speakers. So if you want mine to be public, go for it. I’ll take the good and the bad.

Thanks for giving us your time today. We know that’s valuable, and we hope you learned something you can use from the session. If you can answer these questions as completely as you can, it will help the next person who attends a session here.

Your Name:

What you do for a living: (We Need your background to evaluate your evaluation)

How long you have been doing that: (Again, we need your background to evaluate your evaluation)

Paste Session Description Here: (This is what I said I would talk about)

Did you like the session? NoMeh Yes(General subjective question – overall “feeling”. You’ll tell us why in a minute.)

Tell us about the venue. Temperature, lights, coffee, or the online sound, performance, anything other than the speaker and the material. (Helps the logistics to be better or as good for the next person)

1. What did you expect to learn in this session? (How did you interpret that extract – did you have expectations that I should work towards for the next person?)

2. Did you learn what you expected to learn? Why? Be very specific. (This is the most important question there is. It tells us how to make the session better for someone like you.)

3. If you were giving this presentation, would you have done anything differently? What? (Helps us to gauge you, the listener, and might give us a great idea on how to do something better. Thanks!)

4. What will you do with the information you got? (Every presenter wants you to learn, and learn something useful. This will help us do that as well or better)

Healthcare Data Analyst (aka Report Writer, small-fry SQL developer using SSRS/SSIS and ad hoc TSQL, with some trending and analysis of medical cost thrown in for good measure). I also used to be a jazz performer for a living, was in customer service and have been a supervisor for a couple years over a call center, so some experience with training, evaluations and their role in process improvement.

How long you have been doing that:

3 years for the analyst position, 10 for the performance arts, 5ish for the other

Paste Session Description Here:

What Session Evals should look like

Did you like the session? (No Meh Yes)

Yes

Tell us about the venue. Temperature, lights, coffee, or the online sound, performance, anything other than the speaker and the material.

n/a, the webpage looks fine. Buck looks...never mind. n/a

1. What did you expect to learn in this session?

What a polished, practiced professional professes to possess that the paeons of the public are not privy to.

2. Did you learn what you expected to learn? Why? Be very specific.

Yes, your opinions were very well stated and practical, giving excellent feedback on how to respond in a useful manner. Ergo, comments.

3. If you were giving this presentation, would you have done anything differently? What?

Perhaps put forth a direct call for major organizations (a la PASS, INETA, etc) to adopt a standard feedback mechanism such as this. Not that your opinion does not hold great sway in the community, but it needs to be implemented to make a difference. Are there feedback mechanisms like this built into LiveMeeting? If not, can you lean over and ask that team to add them?

4. What will you do with the information you got?

I will endeavor to answer these questions, or some form of them, for every eval that I fill out, regardless of the prompted questions. Having done training on a one-to-one and team level, I wholeheartedly agree that these are the important questions, and will benefit everyone in the future. I will also strive to present to these points, and to hold myself to a higher standard than 1-5.

**side note, I know SQL Saturdays and other events like using feedback surveys for raffles at the end, but I think this could seriously degrade the quality of the feedback, and necessitates something like the 1-5 scale. Also, the last one I attended the forms were smaller than an index card, so there was limited space to comment, even when using the back side for extra space. If you want serious feedback from your audience, allow them time (end your session a little early?) and space to supply it, or give them another avenue.

Buck, I think you are on to something here but there are some points that come to mind as I read your whole piece.

1. Your form will take longer to complete. This has a few disadvantages;

- people are in my experience looking for somewhere to complete the form - a hotel lobby, a tea/coffee area, in a hall between sessions. This means that there will be queueing for the space and I would suggest fewer forms completed as the people will want to get to the next session and so on. As you say, they have paid to be there.

- If it is one of 6 or 8 sheets to do in a day then I would expect to see declining interest in completing the forms as the day progresses. This may be true already no doubt but I think a longer form filling experience will exaggerate the drop-off.

- ticking/circling a scale is a lot quicker to manage when using your napkin as a desk than writing an appraisal

2. Having this sort of feedback is something for the individual speaker to analyse rather than the event organisers to aggregate and distribute. Not necessarily a bad thing but - will this mean that event organisers will want to know from attendees about speakers too, as well as the location, catering etc. thus giving attendees a extra set of forms to complete. I think any attempt at ranking speakers or sessions will become extremely difficult.

Mike Walsh

17 Jun 2011 7:11 PM

Great minds think alike. I started going down this road a month ago (www.straightpathsql.com/.../on-feedback-and-a-plea-to-event-organizers) after seeing a couple posts about session feedback. I really like your idea for the form and we even agree on some of the questions (especially the "what will you do" idea) VERY IMPORTANT to get that. It's why we do speak/present!