Monday, August 27, 2012

New Forms of Assessment: measuring what you contribute rather than what you collect

Once again, as we do at the start of every school year, we are hearing about the rampant cheating that goes on, especially online, but in fact, everywhere, and without remorse or regret.

As Nikhil Hoyal writes, "Cheating is an epidemic in schools across the nation. A 2010 survey of
2,000 Stuyvesant students revealed that more than 72 percent of
students copied their homework from others and about 90 percent of
seniors cheated on tests."

In the past I've commented on the likelihood that students will emulate their role models, and so if there is a cheating epidemic in schools, it is likely the result of a cheating epidemic in society in general.

The cheating hasn't stopped with this 2005 article, and it's not limited to the U.S. Indeed, in the wake of the 2008 crash and findings of widespread manipulation of things like the stock market and key interest rates, it appears that cheating has become mainstream. No wonder our students find cheating to be the most reasonable response to assessment and evaluation.

I have often wondered what society would look like if we took wealth out of the equation. What would it look like, say, if we limited salaries to a million dollars a year, corporate incomes to $100 million, and set corresponding limits on the accumulation of wealth (to reduce the hoarding that is wrecking our economy)?

Perhaps we can get some sense of this by taking the traditional incentives out of education. Typically, students perform tests, write essays, or complete projects for grades from instructors. Cheating occurs when they perform something other than the intended task in order to produce the result - copying answers, for example, in order to achieve higher grades than they would have earned on their own.

Even when the incentive is minimal, when we set up learning based on grades accumulated in the traditional way, it leads to cheating. Thus we see recent lamentations in Forbes, the Chronicle and others "concerningrecentarticles on cheating and skepticism of online learning," according to Dick Lipton.The response from companies like Coursera and Udacity was predictable: an honour code prohibiting multiple identities (and authentication if that doesn't work), the long-standing idea of mastery learning, orcreating in-person testing centres. And if none of that works, "the online exams are regarded as “practice” insofar as not counting toward a certificate of value."

To me, this is a failure of imagination. Surely, both in schools and society, we can do better than the dog-eat-dog accumulation of numbers representing real or fictional value for power and profit. Surely we can reflect the achievement of individuals in some way that does not resemble a big stack of stuff.The first thing that both the financial system and the grading system devalue is the worth of assistance and generosity to others. Oh, sure, there is a token 'charitable donations' check-box in your income tax form. But imagine your income went up if you gave time, money or resources to charity, even if you were living on social assistance!In the schools, too, there is no reward for helping others (indeed, it is heavily penalized). Suppose educational achievement was measured at least partially according to how much (and how well) you helped others. The value of the achievement would increase if the person is a stranger (and conversely, decrease to zero if it's just a small clique helping each other) and would be in proportion to the timeliness and utility of the assistance (both of which can be measured).The financial system also values mass. That is to say, it favours the creation of consolidated institutions that act as a single entity. There is limited incentive to work with others - indeed, it is often more worth your while to go to court against long-standing business partners. A limit on corporate size, by contrast, would create an incentive to cooperate with others. Changes in patent and trade law would make costly lawsuits counterproductive. But all this is for naught if people have been educated since birth to engage in cutthroat competition with each other. Sure, we hear about fair play, but it is so rare that organizations like the Olympics issue special awards when it happens. Far more often students see cheating, doping, and plain bad manners. Indeed, they learn it doesn't even matter.Suppose instead students were rewarded for cooperation. Not collaboration; this is just the school-level emulation of the creation of cliques and corporations. Cooperation, which is a common and ad hoc creation of interactions and exchanges for mutual value. Cooperative behaviours include exchanges of goods and services, agreement on open standards and protocols, sharing of resources in common (and open) pools, and similar behaviours.Imagine receiving academic credit for contributing well-received resources into open source repositories, whether as software, art, photography, or educational resources. Imagine receiving credit for long-lasting additions to Wikipedia or similar online resources (we would have to fix Wikipedia, as it is now run by a gang of thugs known as 'Wikipedia editors'). We can have wide-ranging and nuanced evaluations of such contributions, not simple grades, but something based on how the content contributed is used and reused across the net (this would have the interesting result that your assessment could continue to go up over time).

Society does not in general reward contributions to the public good. Indeed, quite the opposite - in order to earn profit, corporations and individuals bribe governments to act against the public interest. Companies are more interested in seeing services privatized, instituting user fees, or other measures designed to wring wealth out of what might otherwise be a universal program. As for long range public good, such as environmental protection, or society-wide public good, such as energy and information access, more money is to be made ignoring the public good than supporting it.

Imagine it was the opposite. Imagine private enterprise and individuals were rewarded for supporting the public interest - suppose, for example, they were rewarded according to the actual good they produced, after they made the investment, rather than through some contract or billing system. Imagine a phone company made money, not from privatizing the telephone system, but by adding value to the existing public system. Imagine rewarding energy companies for producing the more environmentally sound energy, not the cheapest.

There is, again, no reason why public service cannot be incorporated into individual assessment. Adding value to fire and police services by means of monitoring and reporting (not the piece-work model of something like CrimeStoppers, but actual prevention), supporting environment by counting birds, sampling water, servicing sports events by acting as a timer or umpire - all these can add to a person's assessment.

I'm not thinking of the simple sort of tasks grade school students can perform. Indeed, a person hoping to attain a higher level qualification would need to contribute to the public good in a substantial and tangible way. Offering open online courses (that are well-subscribed and positively reviewed by the community) should be a requirement for any graduate-level recognition. The PhD used to be about offering a unique research contribution to the field; now it's about paying tuition and being exploited as a TA.

These three things - helping others, being cooperative, contributing to the public good - are obviously not easy to assess. To be sure, it's far easier to ask students simple questions and grade the number of correct responses. But asking students simple questions, far from measuring putative 'content knowledge', is really an exercise in counting without any real interest in what is being counted. It acts as an invitation to cheat, as it places self-interest ahead of the values it is actually trying to measure.

This list of three types of assessment is intended only to stimulate thought. No doubt there are many other forms of assessment along similar lines, all based on measuring what you contribute rather than what you collect. And until we begin measuring achievement - and wealth - in this way, we cannot expect better than dysfunctional students and a dysfunctional society.

Fascinating article. My sceptical side wondered if the societal changes you describe are a little utopian but then I considered that many aspects of the open source, open educational, open badges movements (at their most pure) already align to a certain extent with some of what you propose. Thanks for the fresh and mind-lifting angle :)

Fascinating! I would like to incorporate more of the concept of "credit for contributions" into the online course I teach. The course is about incorporating the Internet into the teaching of second/foreign languages. Perhaps, I could give credit to students for their contributions to the Peer Help forum or for providing meaningful feedback to each other on products they create. Hmmmm.

Say that seems like a good idea...going beyond mere participation marks..I wonder if it's cheat proof?How would you determine meaningful feedback? Would that be self-reported? would students then give each other credit for meaningful feedback? would you have to teach how to give meaningful feedback? would you ask students to tell you if your own feedback was meaningful?At least this kinda feedback broadens intellectual competency beyond merely feeding one's self to sustaining a community.. acquiring intellectual

Hi Stephen - I agree with the comments of others who applaud your suggestions. I too believe this is the 'game changer' we need in how we can better harness ICT in assessment. It is also an extension of the emerging focus on curation in learning. Also sounds complimentary to the Flipped Classroom approach except it would be Flipped Assessment :-) Are there any marking schema or rubrics that you know of for assessment using the approach you have suggested?

We should record and document the things we value, rather than credits and grades on transcripts. The old accounting systems put in place for measuring faculty productivity have long become the defacto measures of success. What would your ideal transcript look like? Who will create it? Would you...What badges would you give? I like this thinking!

This article is really on to something. The social norm today seems to be that as long as something is strictly legal and has the ability to make money it is accepted. How much better the world might be if we changed that norm to be so long as something measurably benefits the public good. We need to develop a better "accounting system" to handle this type of incentive.

Stephen as usual your article is great.For many years i taught the Educational measurements to teacher trainees. My students were mostly paramedics ie nurses, physiotherapist occasionally some medical doctors. Assessment in the medical world in a matter of life and date.Surprisingly even medical students cheat and in a big way too. Why? the society. Nobody wants to be labelled a failure. Pressure from peer and family make students to succumb to this pressure. Secondly, and this ridiculous for a competency based profession.A lot of weight is disproportionately given to theoretical examination.The world needs to move to more cooperative learning. Portfolios including e and peer assessment should be employed. More practical examination ie one Objective Structured Clinical Examination- where students are assessed on selected practical skills- would go a long way in producing more responsive and competent health providers

Hi Stephen Yes, utopian and ideal but substantively naive and impractical. As with any measure it is the measure that has to be met and which sets the criteria as well as the limit. Not everyone can meet or exceed the measure yet that is how the system will judge for advancement and to whichpersons will strive to acquire, meet, or demonstrate. As with all the chatter doan twitter and the persiflage propagated on the net, folks will figure the game and the rules to meet the measure and we have game on whether sports or school

In an in-between moment at a conference on e-learning in Singapore some years back, Dale Spender and I sat dreaming along similar lines.

Surely it should be possible, we thought, that a student could post one contribution to an online course/event, and if that post completely changed the course of the discussions that followed, the student should be awarded a 'masters' or 'mastery' award for that one posting.

The question is how to set up the tracking of both the network of interactions, and the network of thought.

I love this idea - and I feel that I have seen it take place in environments like openStudy, P2PU and various Wiki-projects. It is valued what you give and how you interact with others - not necessarily how much you produce or accumulate.

Thanks a lot for this article – it is truly inspiring and thought-provoking.Let me share one experience with you: At the beginning of this year I finished a mediation training that had taken 22 months. Part of this training was to work in a peer group. When the training started I knew nobody (well, same for nearly everybody there) and the actual choice of the group was more or less based on sympathy and geographical vicinity. We were seven in our group – coming from totally different backgrounds: somebody working for a bank, somebody working for a public institution, somebody working for a church, somebody working in her husband’s company, somebody working in a big chemical plant and me, a freelance lawyer. We all had the focus to learn mediation and conflict management and to fulfill all requirements to receive the certificate at the end. We knew from the beginning that there would not be any grading. We only had the following requirements: being present on the training days (10 weekends over 22 months), taking part in peer group activities (no required time minimum or proof), writing a paper (about a concrete project) of 20 pages and to present the project at the last training weekend. We all had our various experiences with groups, meetings etc. – me too, so I felt I wanted to try something out. I wanted to see if group work can actually work out if there is anything like a “foundation” or a “set of rules”. Based on a book about proper quarreling (“the Dispute School”) which contained numerous exercises for a group learning how to deal with conflict, I chose some of these exercises as a proposal for our first meeting. So actually within our first meeting we negotiated our own set of rules – talking about topics such as punctuality, breaks, food and drink, personal responsibility of each of us, tolerance, fairness, respectful treatment, what will I do if I feel bad about something, personal needs and structure of the meetings. Moreover we discussed what would make the peer group worthwhile for us and what would actually “disturb” us. We wrote a “protocol” about all these points and worked on this basis. Sure enough there were meetings when we kind of deviated from our own rules (beginning late, talking too much about other topics, not following our prepared agenda etc.), but in the whole we had a frame for our meetings. When the time to start writing the paper drew nearer our work meetings intensified – at least for five of us. Though we do not live in the same cities we met 5 times during the last two months and spent considerable time reading emails, draft papers, commenting on progress, talking on the phone – just helping each other when we were stuck and finally rehearsing the presentation. It was hard work, it was very time consuming – but it was really worthwhile! Though the actual extent of our peer group work is not part of our certificate we have all profited very much from this situation and – more important – we continue to meet and to work/share on the topic of mediation. Even without any proper assessment or “incentive” our cooperative work turned out to be source of important learning and experience for all of us. It is certainly difficult to translate this experience in any way to a more formal educational approach – however if we could include courses that promote cooperative learning experiences we might all “profit” from this experience.My reward from this experience actually is: a better understanding of myself, a good basis with regard to mediation and new friends whom I can trust!

Very timely, we just added a learning module that does exactly this, students are given credit for their contributions and discussion around those contributions. The whole course consists of contribution.

Thanks again for an inspiring article Stephen! It may be a utopian ideal, but game-breaker ideas always start out that way. Now to explore ways to incorporate this thinking into the work I do :-)

PHP training in jaipur :-Website Designing Institute JaipurThe sort of course we have developed start with an introduction to the net and the web. We are much eager to tell you the exact norms of web designing technology. We train, we place and we support because over the years we have acquired that power to suffice you with complete regards. We have in possession the perfect designing software and the workable Firework Logos to help you out with perfection. Adobe Photoshop and Adobe Flash are several effective tools that we have in hand and we make use of perfect images and design banners for initiating a pleasing website layout.

in my technology course (University of Padua, Italy) I assess the articles editing process in Wikipedia of my students. Articles are related to the course content.Students are satisfied and happy to perform a task useful to the entire community.

Hi Stephen. What an interesting article. The idea reminds me of the “bucket-filling” notions that have been circulating in my son’s elementary school for the past couple of years. I often wonder what learning/school would look like if we took numerical-based grades out of the equation. My question is how do we implement the notion of contributing-based assessment in a classroom that prepares students for university/careers, when this is not the current practice of most universities or places of work? It seems a shift in our society would be needed for this to be appreciated. Maybe the bucket-filling notion in elementary schools is the start of a new way of thinking about assessment!

Thank you for that article, that makes a lot of sense. I work in an alternative education setting and have found that when traditional assessment, and "the agenda", are removed, there is a shift. I have had the opportunity to assess students based mainly on Social-Emotional competencies, along with some Numeracy and and Literacy competencies, as well as conferencing. This model takes away the "any means to the end" attitude so often found in school (such as cheating for the good mark). It seems when we remove the assessment of "what you know" and replace it with the assessment of "what kind of a person are you and what can you contribute", the desire of the "any means to the end" attitude can be extinguished. I was wondering Stephen what do you think about using these kinds of Competency Assessments and do you think they will ever go beyond Alt Ed or Elementary School? (the District I worked in for 5 years and just moved from is now doing a conferencing model without letter grades for student assessment). Thank you. Darci, OLTD 501 (VIU)(posted by Mary O on behalf of Darci)

I think teachers are slowly becoming more aware of this issue as an an online teacher myself we are trying to find ways to avoid cheating and create innovative ways of assessing students.

My question to you Stephen, is that are our old cultural teaching methods holding us back, are teachers today teaching in only the same way they were taught? Perhaps it is only a matter of time and a shift in this way of thinking that will push teachers in becoming facilitators rather than deliverers of content.

Hi Stephan,I fully believe that a shift needs to happen along the way. Being an online teacher we do struggle with these same issues of cheating and creating new and creative forms of assessment.

My question to you is do you believe that teachers today only know how to teach in a method in which they were taught?

Perhaps, as we continue to move forward in education NEW teacher programs will begin to incorporate more innovative assessments or cooperations as you suggest. I think as the shift happens it will work from the ground up. Most teachers that I know that have been in the education system for years are resistant to change and feel that they don't even know where to begin the change process. But if new teachers begin to see the importance of being facilitators rather than a method of delivering content then perhaps we can see a break through!