An Assessment Primer – the Black Box Metaphor

In science and math, a "black box" problem is one in which you can see the inputs and the outputs from a closed system, but you do not know what is going on inside the system. You can infer what is happening inside the "black box" based on your knowledge of the "inputs" and "outputs." For example, if the inputs to the system consist of oranges and the output is orange juice, you can reasonably infer that the "black box" is a juicer that somehow also manages to get rid of the orange peels. Within an academic context, surprisingly enough, the metaphor of the "black box" can help us understand some of the key terms and motivations related to classroom and program assessment.

Within this assessment context, the "inputs" to the system are the students, before they undergo a particular educational experience. The "black box" is the educational experience you will be providing. It can be a course, a program/major, or the students’ full undergraduate experience. The "outputs" are the students as they emerge from that educational experience:

When you are trying to set learning goals (or "student learning outcomes") for your course, major, or program, the first question you need to ask is, "What will our students be able to do when they have finished this particular experience?" In other words, if the "black box" is a single course, the question can be framed as, "What will my students be able to do as a result of taking this course?" Another way to generate learning goals is to turn that question into a statement: "As a result of taking this course, students will be able to . . . ." You would then complete that sentence (multiple times) to generate your learning goals.

I focus on what students will be able to do because those goals tend to be more specifically defined and more directly measurable than goals articulating what students will "know" or "understand." For example, in a basic microbiology course, a learning goal for a particular unit might be articulated as follows: "As a result of completing this unit, students will understand the difference between a prokaryote and a eukaryote." The typical follow-up question with a learning goal is, "How will you know that you have successfully achieved that goal?" In other words, how will you know that students understand the difference between prokaryotes and eukaryotes? An answer to that question might be, "Students will explain on a midterm exam the properties of eukaryotes and prokaryotes and will differentiate between the two types of organisms." The "explain" and "differentiate" parts of that sentence are the action words that describe something students will do, so it makes more sense to articulate the learning goal based on those actions. The learning goal can thus be rephrased as, "As a result of completing this unit, students will be able to articulate the basic properties of prokaryotes and eukaryotes and to explain clearly the differences between the two types of organisms." If you do indeed ask a question on the first midterm that requires students to describe the basic properties of a prokaryote and eukaryote and to distinguish between those two basic types of organisms, the scores on that particular midterm question will help you determine what percent of your students have successfully achieved that goal. Many writers have linked these "action verbs" to levels of Bloom’s Taxonomy. For a sample list of such verbs, see (for example): http://www.fresnostate.edu/academics/oie/documents/assesments/Blooms%20Level.pdf.

This basic example also provides a sense of how learning goals are linked to "summative assessment practices" – or, in less jargoned language, how learning goals help you determine what forms of assessment you will use to determine if students are achieving the goals you have established. If your learning goals express the need for students to "articulate" or to "explain" something, your assessment of that goal will need to give students the opportunity to explain that concept. If (for a composition course, for example), your learning goal is that "As a result of taking this class, students will be able to present a sustained argument with a clear thesis, well-defined paragraphs, clear and compelling evidence, and an effective and uplifting conclusion," then you will need to assign one or more essays of that type to allow students to demonstrate their mastery of those skills.

To return to our "black box" metaphor, if the "outputs" are the students as they emerge from the particular experience, the learning goals express what those students can now do as a result of having undergone that experience (course, major, program, degree, etc.). To help you determine what has occurred in the "black box" and how successful those activities have been in promoting student learning, it helps to have some understanding of the "inputs" as well. For the example of a course as our "black box" experience, what were the students able to do at the beginning of the course (and how well), and what can they do as they emerge from the course? Pre- and post-tests (or early and final writing samples) are very often used in assessment because they provide a relatively clear sense of the "inputs" and "outputs" of the course, in terms of students’ particular knowledge and skills and how that knowledge and skills have changed over time.