6. Evaluating Student Projects

The most common form of evaluation is summative, used to judge performance and provide grades. In addition, formative evaluation can be used to support students’ learning. Most evaluation instruments such as quizzes and examinations are summative whereas active learning techniques such as experiments and cooperative learning tend to be formative. The remainder of this section will highlight some key points to consider in developing criteria and processes for three dimensions of evaluation. It is critical that instructors develop formative and summative evaluation practices at three levels:

research skills,

economic analysis skills, and

communication skills.

6.1 Research skills

The advantage of grounding the research project in a learning taxonomy (such as Hansen’s proficiencies) is that it explicitly identifies the research skills we want our students to develop. Consider, for example, a research project that is designed so that students create new knowledge. This requires that students use their research skills to understand a body of literature, choose a topic and then develop an effective economic research question. The summative part of the evaluation process would include a measure of the students’ success in demonstrating that they have a command of the foundational knowledge that supports such a question and the extent to which they could justify it in terms of the components – problem-oriented, analytical, interesting and significant, amenable to economic analysis and feasible (Greenlaw, 2006: 14–18 ) – required for the question to be effective. But it is also critical at this juncture to provide constructive feedback (formative evaluation) to the student as they simply cannot continue until their research question is viable.

Similar arguments can be made for other research skills that would be expected of a student for a project that is intended to have them create new knowledge including: locating key literature relating to this question, properly identifying an area of contribution, developing economic evidence in support of their hypothesis, and contextualising the results of their work.

6.2 Economic analysis

Developing structured arguments using economic analysis is one of the hardest tasks that students face in the research process. Detailed descriptions of project components can communicate the form and quality of economic analysis that is expected in a research project However, even when students are generally competent in summarising and synthesising literature, striking out on their own to conduct original analysis is completely foreign to them. This is one instance in which the process by which professionals conduct their own research is a natural model for students. Rarely does a professional paper become published that is not subject to review and revision. Often the revision entails strengthening the economic analysis and arguments: exactly what we want for our students. Challenging student arguments, offering alternative perspectives and correcting inaccuracies are critical inputs into the research process. It is important, however, to make sure that a student learns from this process and does not simply wait for feedback and revise the work accordingly. Feedback on multiple stages of a project can begin by providing nurturing comments in early assignments that signal expectations and provide editorial comments designed to show students how to improve their analysis. As the project progresses, however, these comments should simply point out problems and allow the student to develop the solution.

Special consideration needs to be given to the extent to which students are expected to create new knowledge. Undergraduate research projects have great potential to push students beyond the boundaries of their current knowledge in a variety of ways. The use of undergraduate research as a pedagogical tool is all about stretching students’ minds and providing them with skills to act like economists. A word of caution is appropriate, however, since the structure of undergraduate research projects closely mimic professionally developed publications; it is tempting to develop too high expectations for students. To avoid unnecessary frustrations it is important that new knowledge creation be judged in the context of the students’ knowledge base and demonstrate the achievement of the proficiencies rather than using our professional knowledge base as the benchmark.

6.3 Communication skills

As suggested above, presentation matters in the case of undergraduate research more than in most other learning exercises. Student learning as a result of the research process is more sophisticated and it requires a complementary level of communication, either oral or written.

In the case of oral skills one can model expectations and provide formative feedback through class discussions regarding specific components of the research process. For example, students can be asked to provide an oral justification of their research question to a small group of classmates. This provides a good opportunity for students to practise formulating their thoughts in a public forum and to receive constructive feedback that will improve their question (and subsequent research project). Oral communication skills can also be evaluated during one-on-one meetings. Since students must explain the status of their project and their future plans, the tutor can observe communication skills in an informal setting and provide guidance on how such skills may be improved. Summative evaluation can be achieved through presentations to peers and academics.

Research (Lehr, 1995) suggests that an understanding of the revision process is a key component for developing and improving writing. Writing skills can be enhanced when students are required to develop sections of their projects through revision and resubmission. This is likely to help students to improve their arguments and it is also likely to help with two key components with which students often struggle: organisation and grammar. As with the process described for economic analysis, feedback on students’ written communication should progress from nurturing and providing examples of solutions to the point at which they simply point out errors and require the student to develop the necessary revisions. Once again, the one-on-one meetings can help this process because they provides a more collaborative forum for discussing recurrent problems and encourages students to see the value in this formative evaluation of their work and the opportunities that the revision process provides.

Other helpful processes that encourage good writing include the use of grading rubrics and model writing. Signaling expectations through the use of grading rubrics that include sections for both content and communication skill evaluation encourage students to take the communication of content more seriously, especially if they are also required to evaluate their own work using the rubric. While presenting models of good student writing from previous semesters might also help students understand expectation levels, it can provide some students with a crutch leading to papers that are mirror images of the models. A more effective way to use sample writing as a model is to provide a single first draft paragraph along with the multiple revisions that demonstrate the development into a much-improved final version.

In summary, it is important to recognise and provide opportunities for summative and formative methods of evaluation that can occur formally via written evaluations of work and informally during class exercises and one-on-one meetings.

6.4 Getting Students to Evaluate the Quality of their Own Work

How do we get our students to recognise the quality of their own work? My experience suggests that we need to provide guides and models that students can use as indicators of quality. Such guides should run the gamut from general to specific. For example, three handouts I have used to help my students develop a better understanding of what it means to do quality research include:

grading scale models

a guide of writing tips based on common errors

peer/self review guides.

Students are less likely to have accurate understanding of tutors’ definitions of top-quality work if these definitions are not disclosed to them. The course guide I provide all my research students with is over 100 pages and contains descriptions of expectations, research and writing tips, course assignments and evaluations (to be completed by the student, peers and the instructor). The guide aims to:

signal that this is a serious endeavour;

provide a clear structure for students’ work; and

highlight the multiple components of the research process.

One of the very first documents in this guide is a grading scale that specifically identifies what it takes to earn an A, B, C, D or F. Each grade is described in terms of what the written report includes, what it does not include and the way that information should be communicated. There are many such grading scale models available from which one might develop a specific guide for students. See for example:

Since mistakes made by novice researchers tend to be consistent over time, I provide my students with a series of research tips before they even begin their research projects (Figure 7). These tips focus on the research process as well as the expected output. More importantly, the class discussion that accompanies this handout is focused on why these items are important.

Figure 7: Research Tips for Students

While many of these statements are specific to the final research product, now is a good time to get an idea of my expectations and problems to avoid.

Keep track of your research. The semester is short and we have a lot to do. It is in your best interest to keep detailed notes as to the research you have completed, both the process you have used and the results you have found. Don't make the mistake of having to look for a source or a page of a citation at the last minute. If you look ahead in the course schedule you will notice that this will be facilitated through the creation of a research binder that will be turned in at various points during the semester.

Thesis statements. Written assignments must have a thesis, a single major point. A thesis is a guide for the reader, one that tells your audience the purpose of your essay. It is a statement that encourages the reader to continue. Thus, it should be explicitly stated at the onset of your work. The reader should not have to search for your thesis statement. Good thesis statements often answer the questions 'How?' and/or 'Why?'

Your title. This is one place to show your creativity. A title grabs the attention of the reader, makes them want to read your work, and provides information as to the content of the paper.

Organisation. Written work should be organised around a purpose. The beginning should tell the reader what it is you are trying to accomplish in your work. The middle should begin with the background (often in the form of a review of literature) needed to set up the question you are addressing followed by the evidence or arguments used to support your thesis. This should then be followed by a conclusion (more below).

Know your audience. Recognise that the readers for most of your written work are your peers in class as well as Dr. M. You should assume a general familiarity with economic terms, but be sure to include explanations of less commonly known concepts. Do not use jargon or terms not generally familiar to the reader.

Introduction. Don't start with an introduction that is too far reaching or is narrative in format. You are trying to motivate the reader to continue. Introduce the specific issues you will be addressing in your paper in a way that is interesting.

Answering your economic question. It is critical that in the body of the paper you actually address the thesis you put forth in your introduction. This requires detailed evidence in support of your ultimate answer. Do not simply link a bunch of quotes together and assume the reader understands the link to your thesis. Be specific and explicit in your linking of evidence to the economic question. In addition, avoid unsupported statements or ones that are clearly supported only by opinion. Hint: a paper with a lot of quotes implies that you did not understand the evidence enough to put it into your own words. This also suggests that you are unlikely to convince the reader. In her book, Economical Writing, McCloskey notes that: 'Writing is Thinking' (p. 6). Bad writing implies poorly thought-out ideas. If you do not understand the content of your arguments, it will surely be reflected in your writing.

Tangents. Don't go off on tangents. Be sure that the evidence you include in your paper directly addresses your economic question. Keep asking yourself what evidence this contributes to support the ultimate conclusion.

Take us along for the ride. Don't assume the reader knows where you are going with your arguments. Make the path easy for them to follow. The key to this is to include well-formulated paragraphs in your essay. Each paragraph should develop a specific idea, and the first sentence or two of a paragraph should provide a transition from the previous paragraph and announce what is coming in the current paragraph. The content of the paragraph should then be directly related to what is announced in the first one or two sentences. Hint: read each paragraph and come up with a couple of words to describe their content. Place these words alongside your paragraphs in the margins. Then review the work by reading these summaries from start to finish. Do they flow? (They should if your outline was well formed.)

The conclusion. Don't just rehash the introduction. The conclusion is where you need to wrap it all up. You want to make that final push to convince the reader that you have answered the economic question you proposed in your thesis. One way to accomplish this is to put the work you have completed in a broader context that shows its importance. In other words, make sure the conclusion follows from the evidence. Greenlaw Ch 4, pp. 3-5

First person. Although writing in the fist person is not incorrect, it is best avoided. Using the first person often leads to subjective and imprecise statements. Writing in economics is a formal process and this project requires evidence in answering an economic question. Thus, your project should not be written using the first person.

Format. Your final paper must be typed, double-spaced and include a list of references (at the end of the paper, not as footnotes). The paper should be standard 12 pitch font, New Times Roman with one-inch margins on all sides. Do not justify the right margin. Draft papers and sections must be stapled in the upper-left corner. Final papers must be spiral bound. A title page must include the title of your paper, your name, the date, the course title and the complete pledge must be written in full and signed. To avoid interruption of the flow of your written work, all tables, graphs and charts should appear after the text but prior to the references page. Be sure that each table has the appropriate degree of self-explanation (can it stand alone, independent of the text?) and is appropriately labelled for text reference.

References. Remember to include only those sources actually cited in the paper. These are to be listed in alphabetical order at the end of the paper. Make sure to document all sources, direct quotations, paraphrases and any information that is not your own. Be especially careful about plagiarism, see the course syllabus for more information. When in doubt, cite it!

Computer problems. This will not be an excuse for late work. Do not wait until the last moment to complete or print out your work. Back up your research often, including sending an e-mail copy to yourself!

Writing Centre. Good writing is a form of communication. Your ideas will get lost if the reader keeps getting lost because of a lack of organisation or poor grammar. The writing centre staff are there to help you improve your written communication skills. Even good writers can benefit from this service. For more information visit http://writing.richmond.edu/.

The Writer's Web is an especially useful source of helpful hints and guidelines for writing at various stages of the process. I encourage you to review this resource before you begin your work!

References:

Finally, because students (not unlike ourselves) find it easier to be critical of another person’s work rather than see similar flaws in their own work, I implement a peer review process of all final drafts of research projects. The peer review document in Figure 8 has a combination of general and specific questions relating to both the content and presentation of the economic analysis. Not only does this review provide students with a new set of comments on their papers, but often reading another student’s paper provides another model for organising arguments, shows examples of common mistakes that are made, and stresses the importance of clear and concise communication. After students have reviewed a peer’s work, I ask them to re-read their own work (prior to receiving the peer comments). They are asked to identify (in much the same manner as they did in reviewing their peers’ work) the strengths and weaknesses of their own research. Because the process of reviewing is fresh in their minds, they are more likely to be critical of their own research.

Figure 8: Peer Review of Paper Draft

Title of Paper: _________________________________________________________

Directions:

Answer the following questions thoughtfully, clearly and concisely. Use complete sentences and specific examples (or page references) to make your advice as understandable as possible. You will be evaluated on the thoroughness and helpfulness of your responses to your peer partner. To make your review as beneficial as possible, it is important that you follow the review steps in the order given below:

First, read the paper carefully, without pausing to comment, to get the overall effect of the paper.

In the space below, write down your general feelings about the paper. Is it interesting? Did it generally maintain your interest? Is it easy to follow (organisation is clear)? Did you get lost along the way?

Next, look over all the questions below and then reread each section of the paper and answer the related questions. As you reread the paper make margin notes to the author identifying which paragraphs/sections were clear and which ones were not.

Title and Introduction:

The introduction should present the topic and its importance. It should clearly define relevant concepts, provide an overview of relationships, present the thesis and indicate how it will be developed, and generally prepare the reader for the paper to follow.

Does the title reflect the content of the paper? Is it interesting?

Are the relevant concepts defined? Make a list of the concepts introduced and note which ones are not clearly defined.

Are the relevant relationships described? Make a list of the relationships that were introduced and note which ones were not clearly described.

Is the economic question clearly presented? What is the paper's thesis or main argument? Describe this in your own words in the space below and identify it in the paper using a margin note. (Provide the page number on which this appears.)

Did the introduction adequately convey the importance of this topic? Why or why not?

Rating: On a scale of 1 to 5 (5 being excellent) what rating would you give this introduction? Explain.

Review of Literature:

The review is designed to inform the reader of what other work has been completed on this topic and to motivate the current work. The review presentation should be organised (such as chronologically or by theme) and be directly related to the research project.

Does the depth of review provide enough background to understand the nature of previous studies? Explain.

Is the review sufficiently linked to the current project? Identify links in the paper using margin notes and provide the page numbers where these appear.

Rating: On a scale of 1 to 5 (5 being excellent) what rating would you give this review section? Explain.

The Contribution Section:

In reviewing this section you should mark each sub-section as excellent if it can be clearly identified in the paper, if it accomplishes what is expected, if you are not left with any content questions, and if it is well written (good flow between and within paragraphs, good sentence structure, etc.). Poor sub-sections are those that are not clear because of incoherent content or flow issues, are incomplete leaving you with unanswered questions or issues not sufficiently addressed, or missing altogether.

Please be sure to provide comments for each subsection!

Contribution Section

Excellent

Good

Poor

Statement of what will be demonstrated in the analysis
Comments:

Set up of relationships
Comments:

Hypothesis development
Comments:

Description of method
Comments:

Evidence
Comments:

Relation of evidence to general hypothesis
Comments:

Comparison of findings to previous research
Comments:

Effective use of tables, etc. (where applicable)
Comments:

Rating: On a scale of 1 to 5 (5 being excellent) what rating would you give this contribution section? Explain.

Conclusion:

A conclusion must go beyond a simple restatement of the overall argument of the paper. It should provide a final push to convince the reader that the economic question proposed in the thesis has been answered.

Is a short reiteration of the topic and hypothesis presented? Identify this in the paper by using a margin note and note on which page this appears.

Is the evidence synthesised sufficiently that you are convinced of the answer to the economic question? Explain.

Is the research contextualised in a broader economic issue that indicates its importance? State this broader issue.

Rating: On a scale of 1 to 5 (5 being excellent) what rating would you give this conclusion section? Explain.

Readability:

A research paper can only effectively convey its message if it is well written.

Was the paper clearly written? Highlight or make margin notes for any paragraph or section that you had to reread to understand or still were not sure what the contribution of the section was to the overall paper. (Note these section pages below.) Explain to the author what was not clear and any questions that you still have.

Is the language use appropriate? And the audience accurately gauged?

Does the paper suffer from grammar or organisational problems that inhibit the message of the author?