Giving As You’d Like To Receive – How to Benefit from MOOC Peer-Assessment

“If I be penalized because English is not my native language and some peers do not understand what I say, well, I may also penalize other peers when I will not understand…”

– Student 1

“Okay everyone, I’m going to be the jerk on this thread. I read your essay —, and it was terrible. There were no coherent ideas. It was, to use an Americanism, a ‘hot mess’ that is virtually indecipherable. Grammar and syntactical atrocities aside, your essay did not make sense. It was not so far above our simple-minded understanding that we just ‘couldn’t get it’–it doesn’t make sense.”– Student 2, in response to Student 1

The two excerpts above are from an ongoing exchange in a public forum in a MOOC I took this spring. The quoted thread continued in this heated way for weeks with the the first student’s posts became increasingly defensive in response to the increasingly hurtful comments typed by an anonymous reviewer. Even worse, the community used the vote mechanism in the forum to down-vote the first student’s legitimate (albeit vindictive) concern about the grading system and to up-vote the sarcastic harassment.

The point of this story is not whether the latter student’s claims are valid or even whether the former student’s complaints are warranted. Rather, the point is to recognize the destructive communication happening between peers. The obvious tension between the two sides resulted in two polarized positions that did neither of them nor other students in the class any good.

If a community continues to support such behavior, it is unlikely the space will sustain a healthy learning environment, and the cruel language suggests an outstanding need to outline some guiding principles to help learners get the most out of MOOC peer-assessment. Otherwise, learners such as the first student above will become discouraged, embarrassed or shamed away from the learning community. Learners such as the second student will continue to grow more frustrated and disheartened, angry that his or her language isn’t getting through to the student in need of critical peer-review. There is a break in communication, the feedback loop comes to an abrupt halt and learning stops.

Above all else, education is about mutual respect and trust. Participants in a learning environment must speak politely with due consideration, knowing that each word chosen has a specific impact on the individual reading it. And they must understand that the impact may vary from person to person.

A recent article by John Duhring shows just how much MOOCs are depending on peer assessment. As courses become more peer-oriented, this responsibility falls more directly on the learners themselves; there is no instructor to police language or to guide a problematic conversation back to a safe learning zone.

We, the learners, however, do not need to wait for systematic improvements to solve these issues. Collectively, the MOOC user population is the learning culture. And it is this culture — not the platform infrastructure — that will make or break the learning experience. Everybody should do his or her part in maintaining a positive learning culture.

Below are some ways participants can make peer-assessment a more successful learning experience, for both the author and the reviewer.

Guidelines for the author

Concentrate on the constructive – While explicitly useful feedback may be sparse, a handful of marginally useful comments may lead to its own revelation.

Ignore the destructive – Making mistakes is a powerful and authentic learning experience; do not let anyone shame you into thinking your mistakes mean you’re a failure. Mistakes mean you are learning.

Always find a takeaway – Spend time with the feedback you receive. There is always something valuable to learn.

Analyze the reviewers’ analyses –This one’s particularly important. The reviewer’s words are not outside the realm of interpretation. Study them. This meta-cognition will help you reflect on your own writing process. Cross-reference the various reviewers. Look for patterns. Look for outliers. What do these trends mean?

Extract the subtextual assumptions – Sometimes what the reviewer doesn’t say is the most important part of the review. Remember that your reviewers are representatives from a global population. Many will not share your native language. They will have come from various socioeconomic communities and different cultural backgrounds. All of these elements affect the individual’s understanding of writing and reviewing culture. Recognize the assumptions you bring to your writing, and acknowledge that reviewers bring different assumptions to the process. Failure to acknowledge the differences often leads to problems in communication.

Remember that your peers are your equals – Their words are not definitive or final. They are suggestions. You have the right to feel they do not apply to your work. At the same time, you have the responsibility to give them the benefit of the doubt.

Feedback is not a blame game – The more energy you pour into blaming this or that person for a negative mark is less energy you can afford trying to extract the positive. It is impossible to satisfy all writing cultures in a single essay; one reviewer might be looking for the traditional five-paragraph form, while another expects the organization of an online article.

Tough out the low marks – There is no single rubric, nor a single level of difficulty. Even seasoned and professional writers have received low marks through the peer-assessment system. Do not let this variance overshadow the utility of the assignment and of the recursive feedback process.

Reflect on the assignment – Do not fall into the trap of letting the grade mark the end of the assignment. After reading the comments, reread your own work and write out a reflection. Self-assess, analyze the comments, apply the feedback where you can and record your overall feeling about how the assignment went. Compare this reflection to other assignments in the series.

Actually rewrite the paper – Even if it is not assigned, the best writing practice is recursive. Writing an essay once focuses largely on comprehension and content. A rewrite, however, shifts the focus to more formal considerations: organization, language and rhetoric. While working with content issues will help you master the course subjects, an analysis of these latter concepts will hone your composition skills more directly.

Guidelines for the peer reviewer

Feedback is a crucial part of the learning process – Do not consider peer-assessment as a means to grading some 10,000+ papers. Peer-reviewing allows you to practice critical reading skills. It challenges you to analyze and interpret a wide variety of texts from a diverse and culturally rich population of writers.

Read the text closely, slowly – There is as much to learn about composition from peer papers as there is from reading essays of history’s greatest thinkers, albeit perhaps not as explicit. From peer papers, you are able to trace alternative reader perspectives, identify how different thinkers tackle the same question and analyze various organizational methods. Consider each sentence as a unit and a part of the larger construction so by the end you have a wealth of examples to use as feedback.

Be specific – Direct the author to exact lines and language in his or her writing when making a claim about the essay. Generalized comments such as “confusing” or “sentence structure needs work” will almost always be useless to the author without an example of where such issues are occurring. If the author made the mistake in the first place, how could a single-word claim such as “confusing” give enough information to help guide the author to the problematic areas?

Time is of the essence – The time you spend peer-reviewing not only benefits your own learning, as stated above. Additionally, every minute you spend being more specific and thorough, the original author of the essay benefits exponentially. Pay it forward and give the author a useful critique.

When you think you’re done, give it one more look – As some MOOCs assign up to five peer-assessments per week, the process is at risk of becoming mechanized. Do not let this happen. If you find yourself skimming rather than deeply considering an essay, take a break. Typos and other oversights will detract from your trustworthiness and your review will be taken less seriously. Read the essay, write the feedback, read the essay a second time and finally review your feedback before submitting your work.

Identity with the international, multi-language audience –While it may be hard to imagine yourself into writing cultures involving languages and environments you have never experienced, it is essential to remain cognizant of this multicultural convergence. I, studying from the United States, should not apply an essentially American critique to a Brazilian or European writing. In addition to language barriers, which are their own legitimate concern, there are cultural histories of writing which must be taken into account.

There is no global writing style – This makes assessment difficult. If a particular essay doesn’t match what you believe to be the proper style for the essay, or mirror what you’ve seen in your personal composition education, it doesn’t necessarily mean it’s incorrect.

Be open minded – MOOC composition is writing on a scale never even possible before now. There are bound to be countless difficulties in navigating this process. Expect the unexpected and be careful and considerate when you assess.

TL;DR? Simply follow the golden rule of peer-assessment: give as you’d like to receive.

I leave you with one last quotation from the forum debate we started with. It is three lines written by the first student later in the debacle. Take it as both an authentic embrace of peer-assessment as well as a demand for explicit commentary. Take it as an affirmation to never give up, despite the negativity and shaming that does occasionally sneak through the cracks in the system. Take it, as I have, as oddly reminiscent of mystic poetry:

Hello! I am interested in and dedicated to research, education, and production/publication within the field of Digital Humanities. This includes Education technologies, hybrid pedagogies, and postmodern theory and literature.
Furthermore, I am interested in these subjects historically, with a particular attention paid to the English Enlightenment and concurrent print culture established throughout Europe.
I have taken several MOOC courses and closely follow the development of this and myriad other educational technologies as they continue to emerge, advance, and transform.

5 Comments

Dr. Melissa Kaulbach / May 30, 2013

Adam,
Excellent post! Constructive communication among peers is essential in any healthy learning community. You have provided practical and solid tips for the author and peer reviewer. I love the golden rule of peer assessment!

A good post with useful tips. The issue that those of us who are teachers can forget is just how hard assessment is. Although even inexperienced peer assessors can usually be relied upon to produce a fairly accurate result (at least as a body), producing individual feedback is much more demanding.

xyz / September 3, 2013

First, I’m both a student and a teacher.

Rudeness:
If there are no moderators (err… Teaching Assistants) and no consequences for bad behavior then rudeness is inevitable.

Peer Grading:
I have taken on-line classes and I enjoy the student discussions but I don’t want to be graded by my fellow students. I want the insights of someone who has the experience and knowledge that will result in useful (both negative and positive) comments that help me learn.

I’m getting a second degree at a university that is in the top fifty. Too many of my fellow students are well off, glossy suburbanites who can’t write a coherent sentence because their teachers and parents were too busy telling them that they were wonderful. These students seriously believe that they should get an A for effort regardless of the product. I always write thoughtful responses but too often these entitled students become upset if they receive less than a standing ovation. At the same time their feedback on my work is often incoherent and irrelevant. How are these students my peers?

Sometimes teacher feedback needs to be stern. E.g. No grade until the student rewrites an obviously slapped together essay. A fellow student has no authority to insist on a rewrite that will help the other student learn. Also, there is privacy in the teacher/student relationship that allows a student to absorb useful critique without shame.

I have given grades to fellow group project members and often there is someone who does very little work. Shouldn’t they be given the wake up call of a failing grade? But I never do because I don’t want to be mean.

On the other hand it’s been found that students grade high performing students more harshly than the instructor would have.

This same study also found that peer grading did not result in student learning.

The bottom line is that these MOOCs are brought to us by corporations that need to make money. Moderators, teaching assistants and instructors cost money. Without the crucial elements of interactive instruction and feedback then this is not education but edutainment.

‘XYZ’, while I might agreee with some of your comments, overall I think that you have unrealistic expectations of MOOCs. Is peer-assessment ideal? No. Is it necessary? Yes. If a course requires written work to be assessed then the choice is to use peer assessment or simply ask the students to self-assess; the preferred option of professional assessment is simply not available to a free course with tens of thousands of students.

Your reference is slightly misleading. While it is true that the study found no learning benefit to assessors in peer-grading, the results were closely correlated to the teacher’s grading. It would also be worthwhile highlighting that this study looked at middle-school students, quite a different group from those typically enrolled in MOOCs. The study did find a tendency to mark the highest achievers more conservatively than did the teachers. This is almost certainly due to a lack of confidence. I see the same response among my less experienced teaching colleagues; an unwillingness to commit to the highest grades even where clearly earned.

As I noted in my earlier comment, writing useful feedback is a very difficult task and I have few expectations in that regard. Although I have rarely received any useful feedback from peers either in MOOCs or in formal education, the grading has always seemed reasonable. If you really can’t live with peer-assessment then the only solution I can see is to pay the appropriate course fees to have professional assessment. Maybe this is a possible monetisation route for MOOCs to explore; free entry for peer-assessed students, a $50 cover charge for assessments from an expert (most likely a ‘TA’ or contract assessor).

“MOOCs are brought to us by corporations that need to make money”. The point really is that no course offered free at the point of delivery could possibly provide assessment by teachers or even TAs. My estimate is that a small course of 10,000 students with five short text assignments would cost around $200,000 in assessment fees alone if professional teachers were employed to complete the grading and feedback.

xyz / September 3, 2013

I’ve found that even at a college level students grade high performers more harshly. As a high performer this is significant to me! Why would I choose a learning format that penalizes me?