OCRI
Symposium
“Investigating the efficacy of teacher
work sample methodology in teacher
preparation”
AACTE
February 25th, 2007
OCRI is a joint effort between the Oregon Association of Colleges of Teacher
Education (OACTE) and the Oregon Association of Teacher Educators (ORATE).
This Symposium is Dedicated to the
Memory and Legacy of Dr. Del Schalock
Dr. Del Schalock, Professor, The Teaching
Research Institute, Western Oregon University
Lead Developer and Advocate of the Teacher
Work Sample Methodology (TWS)
Symposium Overview

Introduction

Paper 1: Collaborative research design in teacher
preparation: Blueprints and processes

Paper 2: Evolution of policy implementation of teacher
work samples in Oregon

Paper 3: Perceptions of teacher educators and candidates
on the use of teacher work samples

Paper 4: Teacher work sample effects on the learning of
K-12 students

Discussion
Introduction
Paper 1
Collaborative research design in
teacher preparation: Blueprints and
processes
Ronald A. Beghetto (University of Oregon)
Linda Samek (Corban College)
Imagine…

A focused, coherent multi-institutional collaborative
research initiative
 Nearly a dozen institutions (public and private)
 Engaged in meaningful and sustainable teacher
education research.

This vision is a reality
 With several partnerships across the nation
 And, most recently, a new network in the state of Oregon
 The
Oregon Collaborative Research Initiative (OCRI).
Blue Prints and Processes




Under what conditions do such partnerships
develop?
How do collaborative groups develop and
carry-out their research agendas?
What seems to support such efforts?
What challenges do such efforts face?
Historical Context
The Diversity of OACTE Institutions

19 institutions
 6 public universities


3 research and/or urban
3 regional
 12 private non-profit colleges and universities


6 religious
6 secular
 1 for profit university

Small to large
 From newly approved with no completers to
 Hundreds each year

New to old
 Multnomah Bible College program to begin fall 2007
 Western Oregon University, preparing teachers since 1882
Precursors to OCRI




All subject to rules governing programs from the Oregon
Teacher Standards and Practices Commission
Mid 1990s – Oregon Collaborative for Excellence in
Preparation of Teachers, a 5-year, $5 million NSF
collaborative that included public, private, and community
colleges
1998-2001 – Oregon Quality Assurance in Teaching, a 4year, multi-million dollar grant that involved the 16 four-year
institutions with teacher preparation programs
Common interests in the education of children through
OACTE
Perfect Storm that Spawned OCRI






Leaders from all institutions have common goals
Oregon TSPC requires evidence of competent
performance and student learning
All student teachers are required to produce
Teacher Work Samples
More Oregon institutions interested in NCATE
accreditation
Decentralized education enterprise leaves
everyone under-funded
OACTE had treasury funds that needed to be
used
Sowing the Seeds of
Collaboration
Building a Flexible Blueprint

Approach
 Listen, synthesize, and verify

Workflow
 An iterative, multistage process of planning and activity
Research Designer
CORT
Project Strand Leaders
Local Participants & Partner Sites
Primary Goals of OCRI

The Oregon Collaborative Research
Initiative (OCRI) was established with
the purpose of:
 Engaging Oregon's teacher preparation
institutions in the collaborative study and
dissemination of a meaningful and
sustainable set of research projects.
Meaningful

Meaningful in that the projects address
questions of common interest, are
aimed at improving teacher
preparation, and have real-world
implications for teacher development
and PK-12 student learning.
Sustainable

Sustainable in that the research
projects do not overburden
participants but rather offer feasible
and rewarding opportunities to engage
in and disseminate the findings of
collaborative inquiry.
Collaborative Focus

Our initial collaborative endeavor has
focused on examining the efficacy of
teacher work sample methodology in
preparing teachers in the state of
Oregon.
TWSM: An Initial Project

Three interrelated questions:
 How has the TWSM evolved?
 What is the experience of teacher
educators and candidates with the TWSM?
 What effect has the TWSM had on PK-12
students?
Example Planning Tools
Evolution Study


Goal: Examine how TWSM first became a
mandate and how implementation has
evolved over time.
Brief overview: Using a “Theory of Action”
framework, juxtapose the enactment of
TWSM with its intended and espoused
theory.
Experience Study


Goal: Document the perceptions of teacher
educators and candidates regarding TWSM.
Brief overview: Participants’ perceptions
(representing public and private institutions)
regarding the meaningfulness, usefulness,
and applicability will be examined and
reported on.
Effect Study


Goal: Examine how TWSM has impacted PK12 learners.
Brief overview: Analysis of completed work
samples to examine how work samples are
impacting K-12 learning. This effort also
focuses on the development and refinement
of research/measurement tools most
appropriate for analyzing the impact of
TWSM.
Supports and
Challenges
Key Metaphors

“Cyclical scaffolding” and “loose coupling” of
parallel efforts as metaphors for supporting
collaborative research design
 Key challenges:
Anticipating and providing “just in time” assistance
 Helping to navigate research strand “autonomy” and
“relatedness”
 Issues and challenges with respect to intellectual
property (e.g., datasets), secondary analyses, and
dissemination of findings.

Concluding Thoughts

Even though the Oregon group is still in the early
stages of this process,
 the project has already attained no small measure of
success.

Only time will tell what the Oregon group is able to
accomplish in this ambitious endeavor.
 W are highly optimistic and, indeed, energized by
the idea that
 multiple institutions can come together
 To
forge a strong link between the research and practice of
teacher education, and
 Work towards enhancing teacher preparation and, in turn, PK12
student learning.
Evolution study
Paper 2 – Evolution Study
Evolution of Policy and Implementation of
Teacher Work Sample in Oregon
Presenter:
Hilda Rosselli - Western Oregon University
Research Team:
Dew Anna Brumley - Warner Pacific University
Maria Cirielllo - University of Portland
Hilda Rosselli - Western Oregon University
Research Questions
1.
2.
3.
What led to teacher work samples
becoming mandated?
How have teacher work samples evolved
over time within the state of Oregon?
To what extent do dominant patterns of
implementation run counter to the major
intended premises of the state’s policy?
(espoused theory and theory in use)
Research Design

Phase 1
 Historical--4 qualitative interviews
 Key players involved with the early stages of
teacher work samples policy development

Phase 2
 Survey data collected from 8 institutions
 Early implementation, evolution of practice, challenges that have
evolved

Phase 3
 Document analysis
Methodology

Four individuals selected for interviewing
representing varying perspectives




Del Schalock—Teacher researcher
Dave Myton— Teacher education policymaker
Kathy Wiper—Teacher on state commission
Holly Zanville—Higher education official
Methodology






Conducted 4 semi-structured interviews
Transcribed interview notes
Conducted membership checks
Coded transcripts individually
Compared coding schemes
Reviewed and refined themes
Theme 1: Original Intent

What were teacher work samples designed
originally to accomplish? What were they a
reaction to or remedy for?
 An evidence-based process designed to demonstrate
that student teachers can teach and that students learn
under the guidance of a student teacher.
 Legislators were looking beyond teacher mandated
standardized tests for more accountability from teacher
education programs about their graduates.
Theme 1: Original Intent

What were teacher work samples designed
originally to accomplish? What were TWSs a
reaction to or remedy for?
 Oregon Teacher Standards and Practices Commission
were looking to output standards for program review
purposes
 Going beyond what do teachers know and do to
producing evidence that indicated student learning.
Theme 1: Original Intent
“Document teacher performance beyond knowledge
and connected to the standards”
Del Schalock
“…analogy of the driver’s test. Not enough to just take
a written driver’s test. Need to demonstrate the skill to
drive.”
Holly Zanville
Theme 1: Original Intent
[It also] “ arose out of a dissatisfaction with the
practice of using tests to demonstrate teacher
competence”
Holly Zanville
“At Western it was clearly to go beyond the wall we had
hit with competency based teacher ed where the issue is
not what teachers know and can do. By 1974 we knew
we had to go somewhere else because that mode didn’t
help us so we realized we needed to connect teaching
and learning.”
Del Schalock
Theme 2: Roots and Context

What else was happening at the same time
that influenced the development and
adoption of TWSM? National, regional, state
contexts?
 Intersection of key events
Development
of competency based programs
Creation of state curriculum standards in K-12
Increased accountability for teacher prep
Theme 2: Roots and Context

Other factors
 Elementary Secondary Act and engagement with
schools (late 60’s and 70’s)
 Com-Field- a study funded through NWREL offered a
model for competency based, field-centered, and
personalized teacher preparation (65 - 74)
 Formation of TSPC, the nation’s first independent
commission for teacher education (1971-72)
 Commitment to context for research: Teaching
Research Institute and College of Education at WOU
 TSPC revision of programs to standards-based (1973)
Theme 2: Roots and Context

Other factors
 Mid-Valley Consortium to assist districts with goal-based
education and integration of Curriculum, Instruction, and
Assessment (1975 - 1984)
 Publication of Nation at Risk
 Report to Joint Boards referencing work samples (1986)
 21st Century Schools Act in Oregon (1991)
 Analysis of this act and its implications for teachers.
 Recasting of teacher education standards to outcome
based (1997)
Theme 2: Roots and Context
“ The school context in which teacher ed was
being fashioned became much more outcome
oriented, e.g. what we want our kids to know and
be able to do and hold ourselves to responsibility
for assessing that. Wasn’t that big a jump to
apply this to teacher ed which led to TWSM.”
Del Schalock
Theme 2: Roots and Context
“ComField was field centered—a graduated set
of field experiences—moving from observations,
to small group teaching, to full responsibility for
teaching. That was a frame of reference that
was firmly in the faculty minds so we married
that with the goal based curriculum instruction
demands for Oregon’s standards based
movement.”
Del Schalock
Theme 2: Roots and Context
“Behind closed doors [they] were able to
pinpoint right to the classroom door where
students were meeting the expectations….what
they were getting at with the student test data
that they were beginning to be able to look at
effective schools.”
Dave Myton
Theme 2: Roots and Context
Differing opinion
“I don’t think TWSMs were intended to address
the credibility of the institutions but it did allow
conversation with legislators in which Colleges
could say we are using multiple measures to
assess quality.”
Holly Zanville
Theme 3: Instrumental Players

What individuals or agencies or political issues
were instrumental in the start up and then
evolution of TWSM?
 Del Schalock
 Theory and practice context at WOU
 TSPC
Theme 3: Instrumental Players

Who sat down and crafted the first draft?
“I think I did.”
Del Schalock
“…75% probability that the language of TWSM came
out of the research team in the mid 70s. The last half of
the 70s was where it was being formulated. Tested at
WOU in the late 70s or early 80s. Had sufficient testing
within WOU before it went to TSPC.”
Del Schalock
Theme 3: Instrumental Players
“They [TRI/WOU] built and still maintain a huge
database of student test results…got the superintendents
at the table looking at the data and it was all over the
place. So their concern was how to ensure that kids are
getting not only a good education.
Gerald????so he created a formula. It filled up the
whole front of a classroom…but what it was to see if
there were correlations with any of the inputs.”
Dave Myton
Theme 3: Instrumental Players
“The 21st Century Schools Act didn’t mention
teacher licensing but the Board of Ed and the
Board of Higher Ed both indicated that they
were going to do something, they were going to
take Teacher Ed back, if the Commission didn’t
do something pretty drastic.”
Dave Myton
Theme 3: Instrumental Players
“So in the end, it kind of turned out to be the
commission taking hold of it and saying we want
them [student teachers] to have a work sample,
you can waive anything else.”
Dave Myton
Theme 4: Evolution of Intent

What other purposes did teacher work
samples attempt to serve over time?




Stronger clinical supervision model
Follow-up on teacher candidates performance
Quartile learning gains for groups of students
Shift of emphasis from learning gains to also
more focus on student teacher performance
Theme 4: Evolution of Intent
“Maybe unintended but one of the purposes was
to create a closer link between student teachers
and mentor teachers in the classrooms. Became
a tool for tightening the expectations for student
teaching and an opportunity to work together
with the school personnel. Mentoring a student
teacher couldn’t be a time to go sit in the
lounge.”
Holly Zanville
Theme 4: Evolution of Intent
“The responsibility for the evaluating of work
samples never became the CTs [cooperating
teachers’] responsibility because of the OEA
opposition.”
Kathy Wiper
Theme 4: Evolution of Intent
“And people were thinking, ‘now we know we have
good people coming in, and we can see them in
practice, but what we want to know though, is how are
they when they leave? Not only the door of the
university, but how are they 1, 2, or 5 years out? And
frankly, we never get it, partly because the turnover rate
in teaching is so high. And second because nobody had
resources to go out and observe them in the field.”
Dave Myton
Theme 4: Evolution of Intent
“ In the early iterations of the work sample, we
were talking about quartiles of students. If we
were to go back and collect the old graphs of the
standards…The goal of the work sample was to
show that students had learned by quartiles, as
opposed to each student.”
Dave Myton
Theme 4: Evolution of Intent
“Work sample purposes have evolved from
teaching. Looking for student gains first…the
meaningful gains that were focused on the CCGs
(Common Curriculum Goals), then shifted to
looking at the student teaching…demonstrating
knowledge and skills.”
Kathy Wiper
Theme 4: Evolution of Intent
“The language that went into the TSPC standard was
faithful to the idea of TWS but what didn’t go into the
implementation were performance demands around
learning gains. Became an exercise. Can you do it?
TSPC never really put any quality control mechanisms
in place. How those standards were interpreted and
used? TSPC believes that this is partly handled through
the program review.”
Del Schalock
Insights

1. Del Schlock is the “father” of work sample
methodology
 - he had access to the research based on his position at
Teaching and research
 - he had funds through the Early Models program
 - he worked with the mid valley consortium

2. The state and national context in which TWSs
first emerged was essential to understanding the
full intent of the espoused theory as was the state
of affairs in BOTH K-12 schooling and teacher
preparation.
Insights

3. Although TWSM is required in order to complete
a teacher education program and receive a
license, the specific content and implementation
process has been left to the discretion of a
particular teacher education program. It has never
been exactly prescribed by TSPC. Therefore
different kinds of work samples and evolutions
have flowed naturally from each institution which
tailored the requirements to fit the philosophy and
needs of the specific program.
Maria Ciriello
Insights
4. “In 2002-03, the detractors started coming out
and telling the legislators that they were being
burdened with this extensive TWS project
requirement. And it’s more than any teacher
should ever need. They had the opportunity to
make teacher education a real profession and a
self-governing profession. And by running back to
the legislature saying TSPC is going to make it too
hard to be a teacher, it undercut a lot of work.”
Dave Myton
Insights
“I thought one of the changes that went into
place that was helpful was when it was mandated
that all campuses were to do TWSMs. Then a
collaborative approach emerged. There was
sharing and that was an important tool around
which campuses, both private and public, could
collaborate.”
Holly Zanville
5.
Insights

6. “In a perfect world, we would have spent
the energy required to get solid observations
in place and quality measures of student
learning and its interpretation. But that not
the case. So, in some respects, it’s all been
a little vacuous but faculties in Oregon have
continued to play with that concept over
time.”
Del Schalock
Experience study
Paper 3 – Experience Study
Perceptions of teacher educators and
candidates on the use of teacher work
samples
Presenters:
Blaine C. Ackley – University of Portland
Melina Dyer – Lewis & Clark College
Research Team:
Blaine C. Ackley – University of Portland
Roy Bunch – Corban College
Emily de la Cruz – Portland State University
Melina Dyer – Lewis & Clark College
Mollie Galloway – Lewis & Clark College
Experience Team Members











Roy Bunch, Corban College, Chair
Blaine Ackley, University of Portland
Janine Allen, Northwest Christian College
Emily de la Cruz, Portland State University
Melina Dyer, Lewis & Clark College
Mollie Galloway, Lewis & Clark College
Mary Johnson, George Fox University
Ann Matschiner, Pacific University
Tisha Morrell, University of Portland
Paula Politte, Pacific University
Amanda Smith, Western Oregon University
Survey Instruments

Twelve quantitative response questions using a Likert
scale from 1 (strongly disagree to) to 5 (strongly agree)
 Three questions about the TWS methodology
 Nine questions about elements of TWS

Two qualitative questions:
 In what ways is the work sample process and product beneficial in
the development of an effective teacher?
 What, if anything, would you change about the work sample
requirements to make the process and product more beneficial in
the development of effective teachers?
Survey Data Responses



8 Teacher Education Institutions
33 Faculty Respondents
242 Candidates who had completed at least
one Teacher Work Sample (TWS)
Quantitative Survey Results

No statistically significant
differences in candidates
perceptions of the work
sample by:
 Institution type (public or
private)

For candidates there was a
statistically significant moderate
correlation between elements of
TWS and the overall work sample
(.627, p<.01)

Faculty rated elements of the TWS
more highly than candidates:
 Graduate and
undergraduate teacher
education programs
 Gender
 Number of work samples
completed (1 vs. more
than 1)
 t (316)=5.38, p<.001
 Faculty M=4.20, SD=.65
 Candidate M=3.66, SD=.65

Faculty and candidate felt the TWS
best aided them in lesson planning
and assessment:
 M=4.07, SD=.93 for planning item
 M=3.88, SD=.77 for assessment items
Qualitative Analysis of TWS Benefits
T W S B e n e fits
C a n d id a te s
F a cu lty
80
%
60
40
20
0
P la n n in g
A sse ssin g R e fle ctin g
O th e r
T e a ch in g
“ B ig
P ictu re ”

Faculty find the TWS more beneficial overall than
candidates.
 Candidates focus on the specific elements of the TWS
while Faculty focus on the entire process.
Respondents’ Exemplars:
Benefits of TWS

Teaching
 Candidate: “… Writing and putting the work sample
together was a necessary evil, but the experience
teaching is what made the work sample important to
me.”
 Faculty: “It helps them understand the intricacies of
teaching…

“Big Picture”
 Candidate: “It helped me to think about the large picture
as well as the small details in my teaching.”
 Faculty: “The entire process forces the student teacher
to plan, teach, and assess a unit of instruction. “
Qualitative Analysis of TWS Improvements
T W S Im p r o v e m e n ts
C a n d id a te
F a cu lty
140
120
%
100
80
60
40
20
0
P la nning
As s e s s ing Re fle c ting
O the r
Te a c hing
“ Big
P ic tur e ”

Logis tic s
M odify the
Ne w
P r oc e s s
M e thod
Faculty and candidates agree improvements
should be made to the logistical implementation of
the TWS.
 Faculty to a greater degree than Candidates
indicated that TWS modifications are needed.
Respondents’ Exemplars: Improvements to TWS

Logistics
 Candidate: “The handbook that was supposed to guide us
and provide requirements was scarce, poorly organized,
with bad examples.”
 Faculty: “We seem to be forcing them to concentrate
more on the wording than the teaching process, and they
put a lot of time into duplicating pages of information from
the lesson plans…”

Modify the Process
 Candidate: “The work sample does not allow for
flexibility. There were some days I had to completely
change my lesson, and this has no place in the work
sample other than reflections
 Faculty: “The work sample process often becomes one of
jumping through hoops for students. Is it pretty enough?
Is their grammar correct? “
Respondents’ Exemplars: Improvements to
TWS (2)

New Method
 Candidate: “Work sample requirements could
be more flexible, perhaps shorter.”
 Faculty: “The work sample both as a process
and product is valuable, if and only if, the
students get a chance to discuss it with friendly
cooperating teachers and peers. Otherwise, it
tends to remain superficial.”
Next Steps

Spring Survey
 Most candidates will have completed a second
work sample.
 We will add a code to distinguish between
elementary and secondary candidates.
 We will administer the qualitative questions
before the quantitative questions.
Effect study
Paper 5 – Effect Study
Teacher work sample effects on the learning of
K-12 students
Presenters:
Tom Greene, Co-Chair, University of Portland
Mark Girod, Co-Chair, Western Oregon
Jan Albrecht, Concordia University
Dana Barbarick, Cascade College
Karen Buchanan, George Fox University
Mickey Caskey, Portland State University
Jane Kovisto, Concordia University
Linda Samek, Corban College
Jacqueline Waggoner, University of Portland
The Effect Group

Consists of self-selected volunteers from participating
public and private institutions
Co-Chairs, one public, other private

Guiding Principle:
What kinds of information do teachers need to support and
measure student progress toward learning in standardsbased systems of schooling most effectively?

Judging Instruction and
Assessment…

Candidate level variables - gender, type of program,
endorsements, authorization level
 School and classroom level variables - SES, mobility,
race/ethnicity, program eligibility….
 Instructional methods by lesson
 Type of assessment strategies by outcome
Judging Alignment…
Judgments about alignment
between:






State Standards
District Standards
Goals
Objectives
Instruction
Pre/Formative/Post
Assessments
Judging Learning Gains…





By individual student
By categorization of
student services - ELL,
IEP, TAG, None…
By Race/ethnicity
By type of learning
outcome
Met, Progress, Not met
Pilot Studies

How will TWS data gathering process
function at different institutions?

Are there instrumentation and procedural
issues across institutions that need to be
modified prior to full implementation?
Two Pilot Studies
Pilot Study
Time
I
II
Spring, 2006 Fall, 2006
Number of
TWS
88
9
Number of
P-12
Students
1907
214
Results: Pilot Studies

TWSs addressed the following subjects: math, language arts, social
studies, science, world languages, health and PE

Teaching methods - Most frequent: Student- centered (62%);
Teacher-centered (27%)

Assessment Methods - Most Frequent: Selected Fixed Response at
the knowledge level; Student-constructed response at the
knowledge level
Results: Pilot Studies

92% of the P-12 students had learning gains

There were no significant difference in the learning gains or
no learning gains based on race, TAG, SPED, ESL, or in
schools with poverty designation

Greater frequency of learning gains in high
school…Function of instrumentation?
What is Next?






Use revised measures
Gathering faculty support
Consider consequences for
pedagogy
Local point people will clean and
organize data and submit to a
central database
Full implementation across
seven institutions
Continue cross validation
Challenges








Ownership of data - Institutions, Authors, Associations?
IRB process across all institutions
Communication - frequent management meetings
Institutional integrity - compliance with protocols
Congruity of definitions - developing statewide common
terminology
Support from stakeholders
Time, Money, Resources
Theory in action—differences in perspectives
General Discussion