University-wide Assessment

Comprehensive Education Program 1998 Assessment Report

(Condensed version) University of Nebraska–Lincoln

Hard copies of this report (both condensed and full versions) have been sent to the deans of all UNL colleges and to the Teaching and Learning Center. Questions concerning this report or university assessment activities should be directed to the Director of Institutional Assessment.

Introduction

The 1997-98 academic year was a time of transition for assessment, both of the Comprehensive Education Program (CEP) and of programs across the UNL campus. The intense concentration on assessment prior to the 1997 North Central Accreditation review was followed by the slow, difficult work of building or reinforcing an infrastructure to ensure ongoing and useful assessment at the college and department levels. Those responsible for CEP assessment wrestled with the growing realization that the Office of the Vice Chancellor for Academic Affairs was not the platform from which student learning outcomes attributable to UNL's new general education program could be measured. Such assessment must occur at the site of instruction and learning, and it must be integrated with other assessment efforts in departments and colleges. Throughout the year, the CEP Assessment Team1 addressed three issues:

How could those responsible for the maintenance of the Comprehensive Education Program help departments and colleges merge CEP assessment with their other assessment efforts, avoiding unnecessary duplication and resulting in educational benefits to students and faculty in the units, as well as pertinent information about a university-wide program?

What assessment standards pertinent to CEP were available or could be developed to be employed by the wide variety of undergraduate programs at UNL?

What assessment activities are appropriate at the Academic Affairs level?

To reformulate a large-scale assessment effort begun only a year earlier is slow, complex, and sometimes frustrating work. Some avenues explored by the Assessment Team proved to be dead-ends; others yielded only minimal results. Nonetheless, the year's efforts have resulted in the gathering of very useful information and a level of institutionalization for CEP assessment that one would have hardly thought possible in November of 1997.

The report that follows will concentrate on the response to the third issue above and comment on the second. The report was prepared by Robert Bergstrom, under the direction of Associate Vice Chancellor Stara. The individual survey reports are the result of a collaborative effort by Professor Bergstrom and Rachel Ante, Brian Cannon, Ann Reither, and Cheryl Wiese, from the Bureau of Sociological Research, all working under the direction of Professor David Johnson, Director of the Bureau.

SURVEY RESULTS

In the fall of 1997, the Bureau of Sociological Research mailed surveys to 800 UNL freshmen, selected randomly based on college or General Studies affiliation, and in numbers proportional to the population of each college. In the spring of 1998, the Bureau mailed similar surveys to 800 seniors, observing the same college proportion, and to 1,600 alumni. Each set of surveys included: (1) A section asking about high school preparation, difficulty of classes at UNL (these two data sets only in the first-year and senior surveys) and demographic data; (2) a section asking the respondents to rate their degree of skill or knowledge development in areas pertinent to the Comprehensive Education Program; and (3) a section asking respondents to rate a number of educational and life goals. The senior and alumni surveys added a section on career support at UNL, and the alumni survey, yet another section on the respondents' postgraduate situation. One half of the senior surveys and all of the alumni surveys asked the respondents to rate the university's contribution to their skill and learning development. All the surveys provided a space for open-ended comments.

The return on the first-year survey was 56.9% (n = 455); on the senior survey, the return was 51.8% (n = 414); and on the alumni survey, the return was 46.8% (n = 748). These numbers are large enough to yield significant results, and the instrument seems generally reliable. Moreover, the returns were roughly congruent with the proportion of UNL students enrolled in the undergraduate colleges (and, for first-year students, in General Studies). Thus, baseline data now exists to which to compare future first-year students, current freshmen when they are seniors, and current students governed by the Comprehensive Education Program when they are alumni.

In Section I of the first-year and senior surveys, respondents were asked to rate how well their high school prepared them for the university and how difficult they found academic work at UNL. In both surveys, fewer than 14% of respondents rated their high school preparation inadequate. Twenty-nine percent (29%) of the seniors and 31% of the first-year students judged that they were "very well" prepared. Of the first-year students, 19.9% judged university work "much harder" than expected, while 8.2% thought it was "much easier." Only 9.9% of the seniors thought the work was much harder than they had expected, and another 9.9% judged it much easier. All three surveys asked respondents about their participation in or attendance at a variety of university or community activities at UNL. Table 1 compares the results.

Table 1: Participation in College Activities

First-Year (%)

Seniors (%)

Alumni (%)

Athletic Events

65.4

59.3

64.0

Student Organizations

48.4

55.5

54.0

Community Service

28.1

34.0

34.8

Cultural Events

25.3

33.5

36.5

Student Performances

29.7

31.8

39.6

Intramural Activities

50.2

31.3

47.6

Internships

3.7

28.3

30.9

Fraternity or Sorority

30.0

19.8

32.1

Campus Government

15.7

8.3

12.4

Learning Community

3.5

5.5

3.9

Study Abroad

0.7

4.8

7.9

Note: The cumulative total is greater than 100% because alumni were asked to respond to "all that apply."

In Section II, respondents were asked to rate their level of knowledge or skill in over 40 specific areas corresponding to the Essential Studies (ES) domains and the aims of the Integrative Studies (IS) requirement. [The results for each of these items can be found in the full-length report.] Then the results were clustered in seven general areas. The tabulations for these scaled areas are reported in Table 2. [The response categories were: 1 (None), 2 (Little), 3 (Some), 4 (Much), and 9 (Don't Know - not included in the calculation of the mean).]

Table 2

First-Year Student Mean

Senior Student Mean

Alumni Mean

Critical Thinking

3.16 (.44)

3.35 (.42)

3.35 (.41)

Writing

3.39 (.48)

3.44 (.47)

3.31 (.61)

Oral Communication

3.26 (.44)

3.42 (.42)

3.48 (.43)

Arts and Humanities

2.92 (.85)

2.99 (.91)

2.75 (.89)

Quantitative Reasoning

3.13 (.57)

3.27 (.55)

3.22 (.57)

Human Behavior

3.11 (.48)

3.26 (.46)

3.22 (.44)

Personal Development

3.46 (.39)

3.56 (.39)

3.62 (.33)

The first-year respondents rate themselves just over the 3.0 mark in all categories except for arts/humanities. They rate themselves highest in personal development and writing. The senior respondents rate themselves higher than the freshmen in all categories, the largest advance being in critical thinking (.21 added to the mean), and the smallest in writing (.05) and arts/humanities (.07). Only personal development is rated over 3.5 on a scale of 4.0. Thus, we see a slow but perceptible advance in self-reported knowledge or skill in five areas between freshmen and seniors, but the alumni rate themselves higher only in oral communication and personal development. They rate themselves lower in writing and arts/humanities, and the rest of the ratings are nearly unchanged. Even allowing for increasing rigor of judgment as people become more mature, these results do not support the image of a rapid increase of skill and knowledge from the first to the final year of a UNL education beyond what might be expected by natural intellectual growth as age advances. (Some seniors and alumni say precisely that in their open-ended comments.) The alumni responses indicate an actual degradation of skills (specifically writing) and knowledge since graduation (though, of course, the seniors and the alumni are a totally different set of people). It would seem that many students do not fully recognize the value of general education until they graduate, but it is also true that the comments of seniors and alumni indicate dissatisfaction with career preparation and real-world application in their undergraduate education.

The seniors who filled out the long form rated the university's contribution to their learning, on average, about .5 lower than the rating of that learning, ranging from 1.0 or more in developing ethical standards and using the computer to .2 in using mathematical concepts for analysis. In answers to individual questions, the seniors rated the university's contribution over 3.0 in only five: Defining and solving problems; writing a paper that includes your own opinions, responses, or ideas; incorporating data or research into a paper; analyzing and drawing conclusions from data; and accessing and using a variety of information sources. Results in the alumni survey (all of the respondents using the long form) were similar. To some extent, the gaps between the achievement rating and the university's contribution may be the result of a natural desire to take credit for one's own accomplishments, but they may also reveal an undercurrent of dissatisfaction with the respondents' university experience. In any event, all of these comparative data will provide a useful baseline for future freshman and for seniors who are completing their work under the Comprehensive Education Program.

Section III asked all the respondents to rate the importance of a variety of educational goals provided in the survey. The response categories were: 1 (Not At All Important), 2 (Not Very Important), 3 (Important), 4 (Very Important), and 9 (Don't Know). Table 3 compares the results from each survey.

Here are the comparative rankings of educational goals by the three survey groups:

Though there is some shifting of place among the three survey groups, the general ordering remains unchanged, an ordering that places the knowledge categories central to the academic definition of the liberal arts lower than the skill categories. If these three groups are at all comparable, it would appear that the undergraduate experience of the older respondents has not expanded or reprioritized the view of education with which they entered the university. It will be interesting to see whether this fact and the pattern of self-reported learning change as more students move through the Comprehensive Education Program.

Section IV asked students to make a similar ranking of their life goals. The results are reported in Table 4.

Table 4

First-year Students Succeeding in work Strong friendships Health and wellness Committed partner Interesting things to do Continuing to learn Raising a family Contributing to society Being a leader Religious belief Making much money Geographic mobility

Seniors Succeeding in work Strong friendships Continuing to learn Health and wellness Committed partner Interesting things to do Contributing to society Raising a family Being a leader Religious belief Geographic mobility Making much money

Alumni Succeeding in work Health and wellness Continuing to learn Committed partner Strong friendships Interesting things to do Contributing to society Raising a family Being a leader Religious belief Making much money Geographic mobility

Section V was included on the senior and alumni surveys but not on the first-year survey. The senior students and alumni were asked to rate how well the university prepared them in a number of career areas. The response categories were: 1 (Not at All), 2 (A Little), 3 (Some), 4 (A Great Deal), and 9 (Don't Know). Again, the "don't know" responses were not included in determining the mean. Tables 5 and 6, respectively, report the results for seniors and alumni.

Table 5

N

Mean

Standard Deviation

Developing skills employers need

398

3.20

.71

Learning about career options

410

2.87

.81

Selecting courses to prepare you for employment

404

3.03

.87

Discovering career interests

410

2.87

.91

Formulating career goals

411

2.71

.92

Improving chances for successful job seeking

401

3.04

.87

Finding a job

354

2.63

1.03

Formulating a resume

379

2.51

1.05

[The high standard deviation scores indicate that many students answered these responses to one extreme or the other.]

Table 6

N

Mean

Standard Deviation

Developing skills employers need

739

3.16

.73

Learning about career options

739

2.58

.89

Selecting courses to prepare you for employment

735

2.87

.90

Discovering career interests

738

2.61

.93

Formulating career goals

739

2.41

.96

Improving chances for successful job seeking

732

2.82

.97

Finding a job

729

2.36

1.03

Formulating a resume

737

2.45

1.04

The lower ratings by the alumni in Table 6 than by the seniors in Table 5 might best be seen in the context of three other sets of results from the alumni, only, reported in the following self-explanatory tables.

Table 7: What Are You Doing Now?

Frequency

Percent

Cumulative Percent

Employed full-time

587

80.7

80.7

Employed part-time

40

5.5

86.2

Not employed/seeking employment

6

0.8

87.1

Not employed /not seeking employment

17

2.3

89.4

Attending graduate/professional school

77

10.6

100.0

Total

727

100.0

Table 8: Difficulty of Seeking Employment

Frequency

Percent

Cumulative Percent

Much easier than expected

103

14.6

14.6

About on target with what I expected

425

60.1

74.7

Much more difficult than expected

179

25.3

100.0

Total

707

100.0

Table 9: Relationship of Job to College Major

Frequency

Percent

Cumulative Percent

Specifically related

384

52.3

52.3

Somewhat related

154

21.0

73.3

Not related

158

21.5

94.8

Not employed

38

5.2

100.0

Total

734

100.0

Finally, each survey offered respondents a chance to write comments on their educational experience at UNL. The first-year students wrote very little. The seniors, however, wrote 126 comments, all of which are included in the full report. These pages are well worth reading, both because they add a context for the data reported above and because they support some implications of those data. For example, seven students say explicitly that most of their learning and growth came not from classes, but from peers and learning activities outside of the classroom. The only response more numerous than that were the eight who complained that UNL's career counseling was inadequate. In general, complaints outweighed praise, though five students said, unequivocally, that their university experience was very positive. Perhaps revealing were the five students who said that, in their experience, the university did not push students toward excellence, and the three who said that many UNL students have no discernible thirst for growth and knowledge. Four students said that the Comprehensive Education Program was too rigid or confusing, especially for transfer students, and five students expressed disagreement with the whole concept of liberal education (as distinguished from career training). Three students expressed support for liberal education and one for the goals of Integrative Studies. Three students asked for more opportunities to apply knowledge. Four complained in general terms about the teaching at UNL and five about advising. One student wondered how a small sample of seniors could represent the whole, revealing a lack of understanding of statistics.

The alumni wrote 234 comments, also available verbatim in the full report. To begin with encouraging news, fully 60 of the respondents expressed general or complete satisfaction with their undergraduate experience at UNL. Fifteen acknowledged the value of a liberal education and the general education (not CEP) they received at the university. Two actually wished for more, and three (all identifying themselves as engineering graduates) would have liked more communication classes. Four asked specifically for more and better instruction in the use of computers. Three respondents decried the amount of attention given to gender and multicultural issues, one arguing that learning more history of various cultures would have been beneficial, and another complaining of "brainwashing" in English classes. Two other graduates, however, bemoaned that lack of diversity at UNL. Two major complaints were that the institution provided insufficient career counseling and support (25) and that the curriculum needed more attention to practical, "real world," hands-on application of skills and knowledge (22). Eight respondents stated that the university provided little or no incentive to attain excellence (some included the students, themselves, in that criticism). Four said that many classes and teachers do not encourage critical or creative thinking. A few respondents found the institution less that user-friendly. Others said that had wished for more and closer contact with faculty, two of them saying that they felt "alone" even while getting a good education. Seven "nontraditional" graduates, some of whom were otherwise positive in their responses, said that UNL was not a friendly place for older, working students and those who are parents. Some also commented that our survey assumed that all students moved directly from high school to the university.

CONCLUSION

The three surveys, the results of which are summarized in this report, have served two purposes. First, they have provided a self-portrait of first-year students, seniors, and alumni with regard to the aims of general education at UNL and the effect of a university education on students' and graduates' lives. Second, they have established baseline opinions which can be compared to future members of each group. This is of particular importance to the assessment of the Comprehensive Education Program, since its various components are intended to produce a measurable improvement of educational outcomes across the undergraduate colleges. It is well to remember that these surveys do not, and could not, measure actual learning. That kind of assessment will arise slowly from the efforts of faculty and units to look closely at the actual outcomes in the classroom in order to both clarify learning objectives and improve instructional methods. Nonetheless, the 1997-98 surveys do more than measure student satisfaction with the educational process. They uncover attitudes, priorities, and opinions of current and past students, some of which change as the respondents mature, and some of which do not. Both patterns are significant, and it will be interesting to see whether or not they repeat themselves in a few years' time.

RECOMMENDATIONS

In order to take advantage of the results of the surveys in this report, they ought to be repeated in due course with different sets of students. The instruments might be fine-tuned, but it is important that they match closely the ones used in 1997-98 so that the results can be compared easily and accurately. Academic Affairs should repeat the senior survey in 1999-2000, as the first students governed by the Comprehensive Education Program move into their senior year. Then, two years later, survey the 1997-98 freshman class which was studied this year (not necessarily the same students, of course) and the first-year students of 2001-2002. In addition, that same year would be a useful time to survey UNL alumni again, focusing on those who had entered the university in 1995-96 and 1996-97. The goal of the senior and alumni surveys would be to analyze what effect the Comprehensive Education Program has had on students' learning and attitudes as they perceive them.

Relevant portions of this report will be distributed to the members of the Freshman-Year Task Force. Of particular interest are the findings that indicate that students' understanding of and attitudes toward a university education seem to change very little from the first to the last year of the course of studies. That implies that habits of mind developed in the first two years (when students take many of their general education courses) may determine students' approach to their educational experience throughout their time at UNL. For example, consider the different outcomes if students come (or continue) to believe that their academic courses are obstacles to be overcome on the way to a career, or if they see their studies as an opportunity to grow in intellectual power and knowledge on the way to becoming thoughtful, productive, and responsible citizens.

The goals of a general and liberal education are reflected in the Essential Studies and Integrative Studies requirements, but they are hardly new to UNL. Even so, the surveys reveal both that many elements of general education are not very important to our students and that they do not rate their gain in knowledge very highly, particularly in some traditional academic areas. These results may be disturbing, but there is evidence that they are not, by any means, limited to UNL2. They do, however, highlight the need (a) to measure through outcomes assessment what our undergraduates actually learn from their general education, and (b) to clarify our learning goals, particularly in reference to Essential Studies. For this reason, it is vital that the UNL academic community support the proposal to connect the Peer Review of Teaching Project with formative assessment of general education (for which funding is being sought from the William and Flora Hewlett Foundation). This initiative will engage faculty teams from departments that are major suppliers of general education courses in formulating learning goals, constructing assessment instruments, assessing student learning, and feeding the results back to the faculty for the purpose of instructional improvement. As it develops and expands, this effort will not only aid in the evolution of the Comprehensive Education Program by involving faculty and departments in the clarification of goals and methods, it will also elicit outcomes assessment results to guide the faculty toward making CEP the powerful agent of general education envisioned by its creators.

Assessment of the Integrative Studies component of CEP has been, to some degree, infused into outcomes assessment efforts established in colleges and departments across the university because (a) a useful set of IS learning objectives has been developed and distributed to assessment groups, and (b) at least three of the four condensed IS goals (critical thinking, writing, and oral expression) are standard expectations for graduates in nearly all fields. The three surveys examined in this report, however, may suggest the need for immediate attention to one IS area and concern about another.

Writing: A comparison of self-perceived skill in writing among the three sets of survey responses reveals little advance from the first-year students to the seniors and a lower rating by alumni; whereas, ideally, one would expect a higher level of confidence and skill as students mature intellectually in an academic setting. It is not unreasonable to speculate that first-year students overestimate their writing ability, but such an assumption would seem to cast doubt on all of their self-evaluative survey responses. Ultimately, only true outcomes assessment of student writing can determine the relative quality of writing by our freshmen and seniors3, but one variable in the current survey may be important. Most of the first-year students who took the survey were likely to have taken one composition course already, and many may have been currently enrolled in a second. Their concentration on writing improvement might well contribute to over-confidence in their skills. Seniors, however, whether they wrote little or much in their last year, were far less likely to be receiving direct or substantial writing instruction. We may discover in future assessments that the emphasis on writing in Integrative Studies has changed this pattern, but that outcome is far from certain. A study by Professors Ritchie, Minter, and Goodburn (English), sponsored by the Teaching Council, revealed that teachers of IS classes across the university are, indeed, assigning and commenting on student writing; but it also concludes that many such teachers are unsure of how to make good writing assignments or how to respond to them. This is hardly surprising, but it is worrisome. One or two classes of first-year composition are insufficient to fully prepare students (especially younger students) for the complex writing tasks they will confront in higher-division classes, their majors, or their careers. Merely writing papers and reports in students' last three or four years at UNL does not, in itself, result in increased sophistication and power. Such enhancement of their ability depends on carefully constructed assignments and writing instruction by faculty members who are not experts in such instruction but do understand how writing is done in their disciplines. Thus, the Integrative Studies writing component must be buttressed in colleges and departments by providing help for faculty who are striving to meet the spirit of the IS requirements and for students who are engaged in a continuing effort to improve their writing ability. The College of Business Administration has instituted such a program, and other models are available.

Human Diversity: Respondents to the three surveys consistently rank knowledge of human diversity low on their lists of educational goals. It is not surprising that they rank their knowledge of this aspect of their education quite low, as well. The Comprehensive Education Program is intended to address the fact that our students and graduates must live and work in a diverse society, nationally and globally, by making human diversity a focus of both Essential Studies and Integrative Studies. Once again, we may hope for better results from future studies of UNL seniors and alumni; however, the three CEP surveys from 1997-98 indicate that our efforts are working against serious obstacles. First, students and alumni understandably wonder about the emphasis on diversity when the university and the state are so homogeneous in terms of race and culture. Second, the opinion persists that the aim of diversity education is essentially "political," not fundamental to academic knowledge or even personal development. A change in these perceptions will depend not only on requirements and classroom instruction but also on the university's efforts to diversify the faculty, staff, and student body and to make cultural understanding a natural part of learning, development, and career preparation.

Despite the fact that the three surveys analyzed in this report focus heavily on the academic aspect of general education, it would be unwise to ignore either the respondents' very high rating of career among their educational goals or the relatively high frequency of criticism of career preparation in the senior and alumni surveys. It might seem as if this is a "Student Affairs problem," but that is likely to be too narrow a conclusion. The surveys were not intended to assess the work of Career Planning and Placement; nor do they uncover substantial evidence that that service is, in itself, lacking. Two factors may be at work here. First, while some colleges and departments stress career preparation from the time students declare a major, students in other units seem to wait until they are near or at the point of graduation before they become actively concerned about careers. Then they become frustrated with the process of self-examination and career exploration that is the first step of the Career Office's help. Second, as students move from course to course, instructor to instructor, they are not conscious that they are building skills and knowledge that will help determine career choices and success in finding a job. What seems to be lacking here is an integration of academic learning, personal development, and career preparation throughout students' course of studies, an integration that students may not accomplish for themselves. While Career Planning and Placement needs to intensify their outreach effort to students, their success is unlikely without encouragement from the faculty and academic advisors before the final semester of students' lives at UNL, including efforts to increase students' involvement in internships and other applied experiences and in-service learning.

See George D. Kuh. "How Are We Doing? Tracking the Quality of Undergraduate Experience, 1960s to the Present." The Review of Higher Education 22 (Winter, 1999), 99-119.Back to text

The portfolio study carried out in the 1996-97 CEP assessment, flawed by inconsistent sampling results, indicated that the writing of upperclass students had not kept pace with the increasing complexity of writing tasks, relative to the work of first-year students.Back to text