Crowdsourcing Sociology 101

Millions of students around the world are enrolled in hundreds of MOOCs (massive open online courses), reports the New York Times. To evaluate students’ progress, Princeton Sociology Professor Mitchell Duneier is crowdsourcing his Introduction to Sociology class, which enrolls 40,000 students.

“It was really intimidating at the beginning to do these lectures with no live audience, no sense of who was listening and how they were reacting,” Professor Duneier said. “I talk about things like racial differences in I.Q., Abu Ghraib and public bathrooms, and I worried that my lectures might come across as examples of American ethnocentrism.”

Feedback came quickly. When his first lecture went online, students wrote hundreds, then thousands, of comments and questions in online discussion forums — far too many for Professor Duneier to keep up with. But crowd-sourcing technology helped: every student reading the forum could vote questions and comments up or down, allowing him to spot important topics and tailor his lectures to respond.

Each student must score the work of five classmates to get their own score, which is an average of what classmates have given them. To see whether peer grading matches traditional grading, Professor Duneier and his assistants graded thousands of midterms and finals.

“I had to announce to the students that some had gotten scores that were higher than they should have been,” he said. “And as data, the midterm scores are useless. But it helped us learn more about writing rubrics.”

. . . So far, he has found an impressive correlation of 0.88. The average peer score was 16.94 of 24 possible points, compared with an average teaching-staff score of 15.64. Peer graders give more accurate scores on good exams than bad ones, they found, and the lower the score, the more variance among graders.

About 3 percent of students copied from Wikipedia.

Princeton doesn’t offer a certificate of completion for MOOCs and less than 5 percent of sociology enrollees took the exams. That added up to 2,200 midterm exams and 1,283 final exams, still a heavy scoring burden without crowdsourcing.