Archives for December 19, 2018

My adventures in assessment began in 2004 when I was assigned to the General Education Assessment Committee. I have served on that committee since then and have become the co-chair. I have also always done the assessment for the Philosophy & Religion unit (it is part of a larger department formed during the consolidations of the 1970s). As would be expected, assessment for the unit is just an extra smooshed into the infinitely expandable 20% of my Assignment of Responsibility (which also includes advising, research, web mastering, facilitating, publishing, and 4-9 committees).

When I first started in the assessment
business, most of it seemed quite arbitrary. For example, we were told by the
assessment guru that we needed five outcomes to assess. I still assess five
outcomes. As another example, we were told that our goal should be that 80% of
Xs do Y; for example, that 80% of students achieve a rating of competent or
better in an outcome. When one asked why 80% was picked, the answer was simply
that this was what was to be done. To be
fair, almost everyone uses 70% as the cutoff for a C grade because that has
what has always been done and what everyone does. Which is to say that we use
an appeal to tradition and an appeal to common practice.

As is always the case, the years have
seen more and more added into the assessment process—I now attend multiple
committee meetings, collect data from all the faculty in the unit, and create
stacks of spreadsheets with sweet pie charts. I then do an analysis of each outcome,
craft improvement narratives and write an extensive reflection. All of this
gets loaded into a LiveText system for review by multiple levels of administrators.
Its final fate is to be part of various collective reports. While much could be
written about any part of this system, the focus of this essay is on the improvement
narrative.

As noted above, the performance of the
unit is assessed in terms of five outcomes. For example, the written
communication skills of students are assessed as part of the written
communication outcome. Once the data for the academic year is gathered and
properly spreadsheeted, I do an analysis and then an improvement narrative for
each assessment method within the outcome. For example, I do an analysis of
draft performance relative to final paper performance as part of the written
communication outcome and then write an improvement narrative that maps out a
timetable and budget for the improvement. The budget part is easy: we do not
get a budget for any improvement, so that is a simple copy-paste operation.

The idea of writing an improvement narrative
can be reasonable. After all, if the unit is falling short on an outcome or
otherwise encountering problems, then it makes sense to develop a plan for addressing
these shortcomings or problems. For example, the students tend to do poorly in
the area of written communication. One obvious reason for this is that students
are generally less prepared in K-12 in writing argumentative papers than they were
in the past. As such, the faculty must address this shortcoming in student preparation
in order to bring our results up to the 80% target. That said improvement narratives
can also be problematic.

One practical concern is that there is a
practical limit to improvement. To use an obvious analogy to sports, an athlete
can only improve so far before the cost of improvement becomes prohibitive and there
is a point at which further improvement is no longer possible. The same is also
true in education: even if improvement is possible, the cost of doing so will
be prohibitive and hence there will be a point at which improvement is not
rational. One example of this is when trying to improve in one area must come
at the expense at making another area worse. To illustrate, the more a philosophy
professor tries to make up for the K-12 educational shortfall in writing skills
the less time they will have to address everything else—such as philosophy. Continuing
the athlete analogy, it makes sense that professors also have skill caps—they will
reach a point at which they cannot get any better and improvement will no
longer be possible. It might be objected
that a professor can always improve something and hence there must always be
improvement narratives. This leads to the more abstract concern of endless improvement.

As might be imagined, one cannot simply
write “we are meeting our goals and need no improvement” as part of the improvement
narrative. There must always be an improvement narrative that includes improvement
narratives. This assumes the possibility of endless improvement. On the one
hand, it could be claimed that there is always room for improvement—unless the
professor is a perfect being teaching a perfect class to perfect students. This
is obviously true. On the other hand, endless improvement is a practical
impossibility. One could even say that there is a seeming paradox here:
imperfect beings can always improve, but imperfect beings will always hit their
limits. As such, imperfect beings cannot always improve. So, what should be
done about improvement narratives?

One approach is to be practical and
realistic: focus on actual problems and areas that can be improved while having
the freedom to claim that areas where the acceptable levels are being
maintained are not going to be improved. That is, to say that the performance
is good enough given the available resources and pay. This is what I strive to
do and is what good assessment should be like. I take the same approach to my
running: I know where I can improve and, more importantly, that time and injuries
have put a hard cap on my performance. That is to say that without magic or
technology, I’ll never run another 2:45 marathon or a 33 minute 10K.

Another approach is to engage in the narrative
for the sake of the narrative: if people must write an improvement narrative
for everything, even when they know damn well that there is no way that
something is going to be improved, people will craft suitable fictions to
appease the administration. On the plus side, this will check all the requisite
boxes. On the minus side, it is demoralizing to do this and, to be honest,
dishonest. I try to avoid doing this.

One especially interesting thing about
improvement narratives is that there is the expectation, as noted above, of
endless improvement. However, for most faculty their salary is largely detached
from their performance and improvement: they receive no improved compensation
or benefits for improving the results of outcomes. The only practical incentive
is to stay above the level at which one can be fired for cause and to perform
well enough for tenure and promotion; other than that, there is no practical
incentive to improve. Most faculty, of course, are motivated by pride and professionalism
to do a good job and improve—so it is good that people are not solely motivated
by gain. But higher education is being pushed ever deeper into the business
model and a key concept in business is that you get what you pay for. So, if
constant improvement is expected, then there should also be a corresponding
increase in compensation based on this. If such improvement is valued and
required, then there should also be adequate support provided to achieve that
improvement. Expecting someone to always improve without any improvement in compensation
and without any increase in resources is to expect far too much. Now, back to my
data analysis and narrative crafting.