Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

6.
Validity: what are we assessing?
• Curricula have changed from an input
orientation to an output orientation
• We went from haphazard learning to
integrated learning objectives, to end
objectives, and now to (generic) competencies
• We went from teacher oriented programs to
learning oriented, self-directed programs

8.
Knows
Shows how
Knows how
Does
Knows
Knows how
Shows how
Does
Validity: what are we assessing?
Standardized
assessment (fairly
established)
Unstandardized
assessment (emerging)

9.
Messages from validity research
• There is no magic bullet; we need a
mixture of methods to cover the
competency pyramid
• We need BOTH standardized and non-
standardized assessment methods
• For standardized assessment quality
control around test development and
administration is vital
• For unstandardized assessment the users
(the people) are vital.

15.
Messages from reliability research
• Acceptable reliability is only achieved
with large samples of test elements
(contexts, cases) and assessors
• No method is inherently better than
any other (that includes the new
ones!)
• Objectivity is NOT equal to reliability
• Many subjective judgments are pretty
reproducible/reliable.

17.
• All learners construct knowledge from an inner
scaffolding of their individual and social
experiences, emotions, will, aptitudes, beliefs,
values, self-awareness, purpose, and more . . .
if you are learning ….., what you understand is
determined by how you understand things,
who you are, and what you already know.
Peter Senge, Director of the Center for
Organizational Learning at MIT (as cited in van
Ryn et al., 2014)

18.
Messages learning impact research
• No assessment without (meaningful)
feedback
• Narrative feedback has a lot more
impact on complex skills than scores
• Provision of feedback is not enough
(feedback is a dialogue)
• Longitudinal assessment is needed.

19.
Overview
• From practice to research
• From research to theory
• From theory to practice
• Conclusions

20.
Limitations of the single-method
approach
• No single method can do it all
• Each individual method has
(significant) limitations
• Each single method is a
considerable compromise on
reliability, validity, educational
impact

22.
Programmatic assessment
• A curriculum is a good metaphor;
in a program of assessment:
– Elements are planned, arranged, coordinated
– Is systematically evaluated and reformed
• But how? (the literature provides extremely
little support!)

24.
Building blocks for programmatic
assessment 1
• Every assessment is but one data point (Δ)
• Every data point is optimized for learning
– Information rich (quantitative, qualitative)
– Meaningful
– Variation in format
• Summative versus formative is replaced by a
continuum of stakes (stakes)
• N data points are proportionally related to
the stakes of the decision to be taken.

25.
Continuum of stakes,
number of data point and their function
No
stake
Very high
stake
One
Data point:
• Focused on
information
• Feedback
oriented
• Not decision
oriented
Intermediate
progress decisions:
• More data points
needed
• Focus on diagnosis,
remediation,
prediction
Final decisions on
promotion or selection:
• Many data points needed
• Focused on a (non-
surprising) heavy decision

32.
Physician-clinical investigator program
• 4 year graduate entry program
• Competency-based (Canmeds) with emphasis on
research
• PBL program
– Year 1: classic PBL
– Year 2: real patient PBL
– Year 3: clerkship rotations
– Year 4: participation in research and health care
• High expectations of students: in terms of
motivation, promotion of excellence, self-
directedness

35.
Maastricht Electronic portfolio
(ePass)
Comparison
between the score
of the student and
the average score
of his/her peers.

36.
Every blue dot
corresponds to
an assessment
form included in
the portfolio.
Maastricht Electronic portfolio
(ePass)

37.
Coaching by counselors
• Coaching is essential for successful use of reflective
learning skills
• Counselor gives advice/comments (whether asked or not)
• He/she counsels if choices have to be made
• He/she guards and discusses study progress and
development of competencies

38.
Decision-making by committee
• Committee of counselors and externals
• Decision is based on portfolio information & counselor
recommendation, competency standards
• Deliberation is proportional to clarity of information
• Decisions are justified when needed; remediation
recommendation may be provided

41.
Overview
• From practice to research
• From research to theory
• From theory to practice
• Conclusions

42.
Conclusions 1: The way forward
• We have to stop thinking in terms of individual
assessment methods
• A systematic and programmatic approach is
needed, longitudinally oriented
• Every method of assessment may be functional
(old and new; standardized and unstandardized)
• Professional judgment is imperative (similar to
clinical practice)
• Subjectivity is dealt with through sampling and
procedural bias reduction methods (not with
standardization or objectification).