Changing Classroom Culture

This project was part of a d.school class taught at Stanford (d.science). Here is the class blog. The goal of this project was to provide ways for students to share their level of understanding or confusion during a class lecture. We developed several prototypes of a phone-cliker which is a kind of “awareness tool”; students can see in real time how confused the class is. This facilitates action taking (stopping the professor, asking for clarifications) and community building. The following video shows how one of our prototype would work in a real classroom:

USER GROUP

Freshmen taking science classes (CS, math, physics…) in college.

EXISTING WORK

There is a general trend of ACTIVE / PARTICIPATORY learning for cognitive engagement in the classroom:

Clicker / Personal Response systems widely used

Also some clickers that communicate student understanding to teacher, but not necessarily to other students

Nothing addressing the stigma of asking a strange question in class… nothing helping students to understand each other’s collective knowledge or to engage the teacher in cooperation

existing systems are generally expensive; no low-cost solution to this need

Current work aimed at increasing teacher feedback is primarily focused on increasing student cognitive engagement. The intent of these systems is to keep students engaged by answering questions, with a secondary benefit of giving the teacher feedback on in-the-moment student understanding.
However, these systems fail to address two needs: the needs of students to feel safe and not judged for actually asking a question by raising their hand during class, and the need of the teacher for post-lecture data about student understanding.

EMPATHY OBSERVATIONS

Most of our observations come from a computer science class (“probability for computer scientists”). We observed TA’s office hours / lectures / interviews with students. Large lecture hall with hundreds of students. In situ, totally appropriate location and user groups. We were not intrusive. Students in lecture were watching and taking notes or surfing the web; sometimes students ask overly-advanced questions that aren’t helpful for the class. Students in office hours were working, getting help from the TA, or waiting.

SYNTHESIS

STUDENT 1

“There is a crucial moment when you learn a new formula, and you have to stop listening to the professor in order to mentally compute each steps.”

“Lack of big picture; you’re just flooded with information and you need to recall as much stuff and procedures as possible.”

“Video tapping is helpful because you can stop and do some consult other resources when needed.”

“I love when the teacher says a joke after a complex equation, because it gives me time to conceptualize it.”

“A huge cognitive load comes from the fact that you have to mentally replace the indices and variables in addition to knowing what they represent; writing their name in terms of the problem (concretely) helps a lot.”

TA

“Students usually get it when I explain a concept to them; at least for a moment. The insight often doesn’t last.”

“I try to make problems more familiar to students by using concrete examples, or by breaking it down in smaller problems.”

“It’s difficult for students to generalize from one example; you change a small thing and they lose it.”

STUDENT 2

“The class is very useful but also painful. I feel like I’m really spending a lot of time on it.”

“I need to look at the slides 3-4 times before class and again 3-4 times after class to feel like I have understood the lesson.”

“Students have to learn to ask conceptual questions to the TAs; office hours have become sections because there is too much content in a lecture.”

“It’s easier for people who are always doing maths; for others, there is a huge learning curve.”

OTHER STUDENTS

“I never raise my hand.”

“I’m slow, so it’s difficult to quickly come up with a good question.”

“I’m okay asking questions 1 on 1, but I don’t do it even in small groups.”

“Often I have a basic conceptual question and I don’t want to ask it because I should know the answer.”

“It would be helpful to know what the rest of the class thinks.” Repeated by all students!

INSIGHTS

Students generally aren’t comfortable giving direct performance feedback to the teacher, especially during class; they want to do it privately and anonymously to save face for themselves and the teacher.

Students are more comfortable asking questions if they know that others have the same question.

Students are more comfortable asking questions if they know the other students personally.

It’s hard to speak up during a lecture for two reasons – classroom culture, and size / anonymity.

Points of View

POV 1

INSIGHT: People joke about not having time to write stuff down, but they don’t ask the teacher to slow down!

NEED: student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb.

POV: “I don’t want to be ‘that guy’ in class who is holding everyone back – I want to do something that helps me but is also good for everyone else.”

POV 2

INSIGHT: Teacher doesn’t realize that they’re going too fast.

NEED: Feedback from student. A way to create a classroom culture in which it’s okay to ask the teacher for things.

POV: “I’m doing the best I can for my students, and I don’t want any assumptions to hold us all back from creating a great learning experience.”

BRAINSTORMING SOLUTION

We want to come up with a way to have students loose their inhibition towards asking the professor to slow down, and this will probably happen if they realize they’re not alone in that desire. Let’s give students a way to discreetly let the rest of the class know that they feel rushed and want the lecture to slow down; clicker systems have been used for similar feedback before, so let’s make one with that goal.

PROTOTYPE 1

NEED: student needs a way to feel better about asking the teacher to slow down without pissing off other people or looking dumb.

IDEA: Give students clickers so they can show they want the professor to slow down; then show the dynamic results of this to all the students during the lecture, to show they’re not alone in that desire.

VARIABLE: Will students ask the professor to slow down if prompted by a passive indicator?

We asked students what they thought of this feedback mechanism and concept, but did not test it in a real lecture setting.

INSIGHTS

Students liked and wanted to use the feedback mechanism; it was like a game.

Students said they would be encouraged to give feedback, but if no one does, the professor still does not know about class confusion.

If students have their phones out, will be tempted to goof off? Prevent students who leave the app from giving feedback?

Is a generic indicator of “confusion” enough information to convey to students about their peers?

What about students who don’t have smart phones?

Students might down-vote difficult sections that are still necessary. They are not the best judges of learning content.

Feedback isn’t constructive. This puts a lot of demand on the teacher to be a good performer; will this frustrate teachers who get bad feedback?

PROTOTYPE 2

NEED: teachers need to refer to previous feedback to identify complex concepts or slides that are confusing, since students are not giving the lecturer the feedback in-class.

IDEA: Build an histogram of students’ confusion (updated over time as class goes on).

VARIABLE: Will the lecturer use the immediate feedback to help with explanation?

Based on insights from previous user testing, we choose to improve our previous prototype “clicker” system to include a way to explicitly give the professor immediate feedback. Students still are able to press a “help” button on their clicker or phone and a program records when during the class this button was pressed.

We show the professor, in lecture, a graphic that shows “student confusion” as a function of time. They can see how well understood their comments just were, and it allows them to go back after class and look at what parts of their presentation need to be clarified and improved, by looking at what timestamps had the most confusion.

We tested this system in a small lecture given by one of David’s labmates. The lecturer could see a plot of when during his talk students were having the most confusion; the audience had no feedback, other than knowing when they themselves pressed “help” on the clicker page.

Histogram of students’ confusion during lecture:

Line chart directly next to the slides:

INSIGHTS

Audience felt guilty about using clickers: “I should just raise my hand, not use this thing.” Even after one testing session, we might be changing classroom culture!

Attendees did not use clickers very much. A class-facing component seems to be necessary to keep faith and engagement in the system.

Presenter did see chart clearly, and tried to act upon the new information.

“I didn’t know what to re-explain.” Presenter though that this would be more useful after lecture, when he would have time to re-assess the specific trouble spots.

Prototypes need to be tested in environment that mimics the intimidating environment of a large intro class; we were in a smaller, optional lecture, that was very low stress.

Presenter might need training as to how to deal with immediate feedback usefully.

The system needs both prototypes simultaneously to be functional; lecture environment can be changed more if there is an audience-facing screen, lecture confusion can be reduced more if there’s a professor-facing screen.

PROTOTYPE 3

NEED: Teachers needs to be able to identify what was confusing during their lecture (a histogram may not be enough); name of confusing concepts.

IDEA: Allow students to directly type a keyword on their phone / laptop

VARIABLE: Will students be willing to document confusing concepts? Can the lecturer incorporate current confusing topics into their talk in real-time?

We had previously ignored how the lecturer would react to this feedback, but now that our last prototype gave us specific feedback on that, we decided to incorporate more informative feedback into the system. We still wanted to keep the mechanism by which the students asked for help very straightforward, so we show a list of most frequent topics on people’s phones and allow students to easily type in new topics at the bottom. The current most requested topics are shown on the presenter / class-facing screen along with the previous histogram data.

We were unable to find an appropriate lecture that was willing to let us test out our clicker system before the end of the quarter; based on our previous testing, we knew that we needed to find a class that was as close as possible to the original lecture situation we performed needfinding on. Unfortunately, the larger lecture classes were unwilling to introduce things that might delay class this late in the term. We instead asked people and presenters for feedback on this updated system.

INSIGHTS

Presenter still was able to interpret data, and said he could incorporate it into his responses easier.

Presenter said that this would be especially helpful in reviewing how a presentation went after the fact, and knowing what topics to focus on / review in later lectures.

Similar to how people use hashtags during conference talks on Twitter; this shows the principle could be well received.

Now that it’s possible to directly and anonymously communicate with the professor, are we preventing classroom culture from changing? Will people be less likely to ask questions now that students are aware the professor already knows their question?

FINAL THOUGHTS

Testing that any device truly changes classroom culture will be difficult and require a longer term study than we were able to complete. We are still hopeful that further testing and answering some of our outstanding questions on the interplay between social norms between student and teacher, and between students will let that goal be achieved. The fact that after one testing session one student already said, “I should just raise my hand, not use this thing,” leads us to think it’s possible. In a sense, we’d like our project to be self-destructing; we think that any system that encourages students to ask questions and succeeds will positively reinforce the act and, hopefully, will make itself unnecessary.
Team: Daniel Green, David Selassie, Bertrand Schneider