10
10 Defining Expectations: Benchmarks/levels Year 1 Year 2 Year 3 Year 4

11
11 Measuring Expectations Appropriate assessments – provide evidence that students have achieved what we want them to know and be able to do Appropriate rating criteria –fair, students know what is expected, provide feedback to students and stakeholders

12
12 Rating methods Objective and subjective evaluation If we view objectivity and subjectivity of evaluation along a continuum, we can represent various assessment and scoring methods along its length.

14
14 Rater reliability In assessment, we strive to ensure two types of reliability: inter-rater (raters agree with each other) and intra-rater (a rater gives the same score to a performance rated on separate occasions)

15
15 Rubrics & Scales Scales and rubrics differ in their specificity: scales generally can be applied to more than one task; rubrics are more closely aligned with particular tasks. Holistic rubrics and scales are used to assign one integrated evaluation to a performance, e.g., ACTFL Proficiency Guidelines. Analytic rubrics and scales are used to evaluate the various dimensions of a performance.

20
20 Disadvantages Provide limited information about how to improve performance. Do not indicate relative quality of performance.

21
21 Rubrics rating scales used with performance assessments increasingly used to evaluate performance assessments. provide an indication of quality of performance/student work Provide detailed feedback to learners

22
22 Why use rubrics? Set anchor points along a quality continuum rather than right or wrong Increase construct validity Align assessment to curriculum and instruction Focus on the most salient goals Expectations are clearer to students (and to yourself)

23
23 Provide specific feedback to students Well-designed rubrics increase assessment reliability by setting criteria that can be consistently applied by raters The Virtual Assessment Center at CARLA provides an extensive tutorial on rubrics http://www.carla.umn.edu/assessment/vac/Evaluation/p_4.html

24
24 Rubrics should be: Clear - define excellent, good or work needing improvement Democratic – consistent, consistently reinforced, understood by students that grading is not subjective Involving – students should be involved in creating rubrics Engaging – allow students to evaluate own work From Blaz, C. (2001). A collection of performance assessment tasks and rubrics. Eye on Education: Larchmont, NY.

28
28 Multitrait Rubric for Recipe Presentation Organization and Clarity Fluency and Pronunciation Use of the Imperative (or other form)

29
29 Primary trait rubrics Performance is scored on the main criterion for success on the task. Example: Task*: Write a persuasive letter to the editor of the school newspaper.

30
30 Primary Trait: Persuading an audience* 0Fails to persuade the audience 1Attempts to persuade but does not provide sufficient support. 2Presents a somewhat persuasive argument but without consistent development and support 3Develops a persuasive argument that is well developed and supported. * Tasks/rubrics can be found in Minnesota Articulation Project. (2002). Proficiency-oriented language instruction and assessment: A curriculum handbook for teachers (Rev Ed.). CARLA Working Paper Series. D. J. Tedick (Ed.). Minneapolis, MN: University of Minnesota, The Center for Advanced Research on Language Acquisition.

31
31 Creating a Rubric Step 1: Generate potential dimensions Rank order the potential dimensions from most to least important; eliminate the "non-negotiables".

32
32 Step 2: Select a reasonable number of dimensions 3-7?, 4-8? "How many is enough? … There's no one correct answer, but it might help if you consider your purpose for this measurement. If it's diagnostic and formative, err on the side of more dimensions rather than fewer. If you just want to be able to give a summative evaluation of your students' performance for this particular lesson, fewer dimensions are OK." (Triton/Patterns Summer Symposium. (1999). Creating A Rubric for a Given Task. San Diego City Schools. Available online at http://projects.edtech.sandi.net/staffdev/tpss99/rubrics/rubrics.html

39
39 Constructing a rubric to evaluate an ESL speaking assessment According to the number on your sheet, in groups of three or four: please read the purpose for which you will use this speaking assessment (see section 4, binder) Read the ESL speaking assessment on the handout make a rubric with criteria that meet the assigned purpose (to consider level, consult ACTFL guidelines in your binder, if needed) Listen to the student, and rate the sample using the rubric your group produced.

40
40 English CoSA/SOPI – Speaking Meeting your new tandem partner It's the second week of school, and the ESL program has arranged a reception for international students to meet their tandem partners. You are paired up with Paula, who is from southern Iowa, and studies linguistics. Task 1 Paula wonders about you: where you're from, how old you are, what you're studying, and so on. Wait until you hear Paula's question, then tell her about yourself.

41
41 Task 2 Paula asks you about your classes: she's curious about the ESL program. You tell her which classes you're taking and your opinion of each class. You also tell her a few things that you do in your classes. Wait until you hear Paula's question, then describe your classes to her, and include the activities students do in each class. Task 3 You and Paula begin talking about dating in the United States. She wants to know how you meet people in your home country, and what dating customs are like there: are parents involved, do people go out in groups, and so on. Wait until you hear Paula's question, then tell her about dating in your country.

42
42 Task 4 Later that day, you run into Paula in the library—you're both tired of studying, and you start to talk about families. She tells you all about her family in Iowa, and then wants to know about yours: your sisters, brothers, parents, etc.: their ages, where they work, what they do in their spare time, and so on. Wait until you hear Paula's question, then describe your family to her. Task 5 You would like to learn more about Paula and her life in Iowa: her friends, her interests, the town she lives in, etc. Wait until you hear Paula speak, then ask her at least five questions to find out more about her life in Iowa.

43
43 Task 6 You start thinking ahead to the weekend, and Paula wonders what there is to do here at the University of Minnesota and in the Twin Cities. She wants to get to know the area and make some new friends. Wait until you hear Paula's question, then make at least 4 suggestions as to what she might do this weekend. Task 7 You and Paula decide that you will meet later for lunch to make plans. Wait until you hear Paula's question, then tell her where you go to eat lunch, what you like to eat, and that you will see her later.

44
44 Appropriateness of the instrument In your group, answer the following: How appropriate is this speaking assessment for your group’s defined goal? How appropriate are the tasks? What about face and construct validity? Discuss in groups