Tuesday, March 29, 2011

Outcomes and Assessments: Point of Order

Some of you may remember that I am not antipathetic towards the outcomes and assessment movement. In fact, I'm rather a fan of the idea of faculty sitting down together and seeing how the students are doing and coming up with ways to make sure students graduate with a degree that means something. I like the idea of giving a common exam to all students on a course and having double-blind readings of the scripts. I like the idea of outside examiners. To me, those things are signals that there is some sort of conversation across a field about what is important, and it would certainly encourage me to work harder on content and on making sure I was better organized. I don't find such things threatening to my autonomy, as do many of my colleagues at SLAC and in other places. They are just reminders that I have to cover a certain body of knowledge, and not all students will answer all questions anyway.

I feel similarly about the sort of outcomes my department has, which are largely skills-based. Here I'm on very shaky ground and likely to really piss people off. Different historians have different sorts of skills expertise. We can't all be experts in all of them. I have a colleague who uses local archives all the time - I've never done local history, nor used an archive; at the same time, some of my colleagues have very different ideas of what it means to write a primary source analysis than I do. I think a student can write a perfectly good, and more importantly, rigorous, 5000-7000 word research essay that asks a question of a single source, perhaps a saga, or Orderic Vitalis' Ecclesiastical History; or perhaps of a few dispute charters or traditiones. Close reading is integral to what I do (so much so that sometimes I skimp on the literature more than I should). But if there is a single course that all students have to take to teach the skills we say we assess, it seems to me that faculty need to work together to define the skills and they need to give up some autonomy to make sure that, whoever teaches the course, the students can go on to all other classes in the department and be fairly successful without any one faculty member having to teach from scratch the stuff that students in their classes need.

In a way, outcomes and assessments done well and meaningfully are sort of a reflection of Rousseau's Social Contract. We all give up being able to teach only what we want,* or just our way of doing things, in order to make sure our students will be successful in other departmental courses, or if they transfer elsewhere. They have to have faculty buy-in and contribution at all levels, and faculty, even those who are willing, need to see the bigger picture.**

Lots of things can gum up the works. Lack of faculty buy-in, administrators who take far too dictatorial an approach (other than saying, "I don't care how you do it, but it needs to be done by X or we lose funding and accreditation."), Accreditation agencies and governments who are far too hung up on "accountability" when they have no bloody idea of what is important in a field (and everybody think they know what is important in history -- dog forbid that it requires anything like training to be an expert!).

But this week, I realized what the number one problem with having valid outcomes and assessments is. You could have the most cooperative faculty in the world and the most supportive administrators. It doesn't mean a damned thing if the students aren't prepared for university level work. If they can't manage, or won't do, the readings; if they don't come to class sessions, especially seminars and discussions, prepared; if they are ignorant of geography, how can we teach towards the outcomes we've set? How can our assessments and measurements of those outcomes be valid?

They can't.

If we teach at the university level, we have to have outcomes that reflect that level. But such outcomes are based on the idea that our students can work up to that level over the period they are at the university. How do we measure when we have to (and yes, this is true not just at my SLAC, but at colleges and universities all over the US) spend time on how to study, how to write an essay, how to read effectively,*** take notes, write an essay exam, become familiar with the most basic world maps of the present (let alone the past)...? Those things are just plain inappropriate as university level outcomes. Some of them belong in elementary school, for goodness' sake!

And how do we measure that the reasons students cannot achieve the outcomes we set, outcomes that should match up to those at other institutions like ours, are not necessarily because of our teaching, or even of student learning, are because the underlying assumptions of what it means to be ready to study for a university degree have not been met? Technically, it's not all that difficult to document, I suppose. We can ask students to self-report average time spent studying on a course (the are surprisingly honest!). We can scan and save student writing samples to demonstrate that students are not prepared, or that they have learnt something, but it may not be what they were supposed to learn for the course (mine usually write better at the end -- at least "write a well-argued analytical essay that answers a historical question, supported with specific detail," is one of our outcomes!

But I ask you -- how can assessment be meaningful if we spend as much time teaching students to be students as we do our subject? and how can the teaching in our subject not suffer if we are taking so much time away from it to give students the skills they need to succeed (to a point -- if students really are clueless and hopeless, I will ask them to drop). I have colleagues who simply fail such students, but there has to be a better resolution.

What do you all do, if you have to deal with such things? How will you, when you do?(which you might not if you are in the UK, since the Big Society mavens seem to be very happy with the idea of limiting uni education to the elite -- although they seem to have bugger all in the way of ideas as to what's going to happen to everybody else).

*Yeah, I know: most of us don't teach just what we want, but you'd be surprised at how many people think that academic freedom means complete control over the curriculum. There are things I don't teach, but it's not because I don't want to -- it's generally because I can't get to them because the most specialized class I teach is Ren-Ref!

**I will smack the first one of you that says this is a dean-ish comment. Just because I can think like an administrator doesn't mean I want to be one!

***this week I have had the fun of discovering that a student thought an incredibly common expression meaning "people were so focused on this thing that it became the driving force behind government and social policies," instead meant, "people discarded this thing and got as far away from it as possible." There was also the joy of having students tell me the only way they could learn was to bring their books to class and read along while we were discussing the information. No joke.

8 comments:

What I'd written before was along the lines of agreeing that it's difficult when we have students admitted who're unprepared. The problem is in determining exactly where they're lacking in skills, from basic issues in map-reading to study skills and beyond. Then, once we have that information, how do we address those deficiencies?

It shouldn't be individual prof by individual prof. Our U has mandatory and optional library workshops. Something along those lines, for incoming students whose pretests or questionnaires show weakness in some areas might help. But some students, whatever is offered, will resist having to spend "extra time" to build capabilities and workshops that we might want to rely upon won't be as effective as hoped, etc., etc.

So true. And our English department flat out refuses to add remedial courses. They think we should only admit people who can do the work. Sadly, this is entirely unrealistic for a tuition-driven SLAC --especially ours. We don't have the sorts of conditions that allow that. Faculty are not as productive as they need to be, facilities are not good enough, faculty loads are too high to be the kind of faculty that you get at more selective institutions...

This is not a criticism of SLAC -- it's just not our niche, nor is it our mission, which includes being here to give a second chance to people who might not be able to get into more selective places. But that means we need to make sure we get them up to speed...

We're finally facing these realities, I think, at Field College. Next year we're adding Developmental Writing (so necessary!) and a three-credit first-year seminar that will aim to teach students how to succeed in college--through study skills, but also through practice engaging in critical reading, writing, and discussion. I hope that it helps.

The only solution I see is trying to devise ways to combine disciplinary work with skill-building exercises in activities like writing, reading, and research. And no, I don't expect this to be popular, but otherwise you have to grade the papers that result from neglecting such things.

Well, one idea that immediately springs to mind is College Entrance Exams, the old-fashioned kind. Can students read a map, do a close reading, etc.; then at least you know what you're starting with and why your outcomes wind up being what they are. But the last thing we really need is yet another test for high school teachers to teach to. More reading (more vocabulary building) without any more writing or anything else would fix a lot of the problems we're both seeing. Not all of them, but a lot.

Slightly off topic, I wish historians would include more maps in their books and papers. I find anything that turns on communications or climate or relative position virtually incomprehensible without a map and while I think it is perfectly reasonable to expect people to know where, say, Egypt is it's probably not reasonable to expect people to know where Ragusa stands in relation to the major Alpine passes.