Impact Measures

From Institutional Effectiveness and Accreditation

Impact Measures using LONs and PIPs

A long standing problem in assessment involves the reporting of course impact information. The problem occurs whenever a course is taught by a single instructor. If the impact information is made public, it could affect their employment evaluations. That, as the traditional strongly holds, is tantamount to killing assessment. As written in earlier postings, Learning Outcomes Networks (LONs) involve every faculty member assessing every student once per course using multidimensional developmental rubrics across an entire program. Such networks provide rich data for determining course impacts (see the attachment on LON Impact Analysis originally posted in “Analyzing Learning Outcomes Networks: Basic Steps, July 21, 2011). The solution then was to calculate the impact scores only if assured by senior administration that the data would not be requested and would not be used for evaluative purposes. Then, I offered to share the results with the individual faculty who taught the course. Such was the best I could think of at the time.

Recently, work with a faculty committee at Virginia Wesleyan College (VWC), who are charged with evaluating and enhancing a five year assessment plan that I created, a solution became apparent. In 2010-11, VWC underwent a major curricular re-structuring based on moving to a 4x4 model (students take four course that are four credit hours each). As part of justifying the additional credit hour, each faculty member who taught a previously 3-credit course had to specify enhancements. This led to a database of enhancements that was one of the key factors that attracted me to relocate to the college. And I was not disappointed. The list of enhancements looks like a list of Potentially Innovative Practices (PIPs). When analyzed, the number of such PIPs taken by students was even found to be related to retention. But that is only the starting point.

The proposal generated by the 5-year assessment plan committee involved three steps: (1) further develop the checklist created in 2010 into a more comprehensive list of PIPs, (2) add it to each of the courses listed on each faculty member’s annual Professional Activities Form, and (3) analyze the LON data by aggregating across all courses that use the same enhancement.
One of the big plusses of LONs is that they provide a rich data set even for small programs. By aggregating across a variety of courses taught in a variety of disciplines we will have the opportunity to learn what practices have what kind of impact, at what steps in programs during a student’s undergraduate experience. That, to me, provides an unprecedented and broad assessment opportunity that deepens the promise of the LON approach. Learning outcomes at an undergraduate college, even a small one with 1,400 students, is an astonishing complex, interactive system only rivaled by the rainforest. Linking LONs with PIPs provides a data map into the learning outcomes system that can help to determine points of health and points needing nourishment. This can be accomplished with an hour or so of work per faculty member per year. And it can be accomplished with no threat to anyone.