6. Assessment

The work on threshold concepts has begun to consider the implications for assessment. This section contains the insights that can be drawn from the approach and some initial discussion of question design.

The different types of conceptual change and their interrelationships carry some implications for assessment. First, the acquisition of a basic concept in terms of ‘understanding or not understanding’ misrepresents the situation. Students may have an understanding of a basic concept, without having the kind of understanding associated with expert knowledge in a discipline. Therefore, it is desirable that assessment should distinguish between limited and more complex understanding of a basic concept. The more complex understanding will be the way of understanding the concept that is aligned with a threshold concept. Thus second, we might therefore interpret a more complex understanding of a basic concept as an indication that a learner is, at least, becoming aware of a threshold concept.

Third, a learner’s journey towards incorporating a discipline threshold concept in their thinking requires that they learn to use modelling concepts to generate structures of thought in analysing applied questions. For example, an important stage in introductory macroeconomics is learning to combine a number of basic concepts (such as the distinction between injections and withdrawals, savings and investment, stocks and flows, real and nominal values) in a model of expenditure flows. Without appreciating the importance of the modelling concept of equilibrium the set of basic concepts that are pertinent to understanding the model are not made to act in concert to produce a coherently structured understanding of an economy as a system and this leads to a limited understanding of concepts such as the multiplier. This is not to say that most students, when asked, will not be able to recall the equilibrium condition in such a model. The concern is over their being able to integrate this in their analysis. For instance, a study of the examination answers to applied questions on income/expenditure flows in the Embedding Threshold Concepts project indicated that many students describe the multiplier process as a continuing sequence (a decrease in government expenditure leads to less employment and lowers income which leads to less consumption which leads to less employment and income, etc.) rather than one with a limit with the endogenous variables changing as the system returns to equilibrium.

These implications can be related to assessment criteria. At level 1, for instance, the approach suggests we want to examine:

the development of understanding of a range of basic concepts

the ability to use the modelling procedures of economics (particularly comparative statics at level 1)

the emergence of understanding of threshold concepts.

However, the expected understanding of threshold concepts may be limited and patchy at this level and, as we have argued in section 2, this may affect the understanding of even basic concepts. Students need some understanding of basic concepts to progress to threshold concepts, but also understanding the threshold concepts may deepen the understanding of basic concepts (which does create a problem for the learner in that they need to progress without full understanding and continually reassess their comprehension). This complicates judgements on the progression of students – understanding is not a nice linear sequence and we need to take this into account in our judgements.

Question design and the assessment package

The assessment package, and within that the design of questions, needs to create opportunities for students to demonstrate a range of levels of understanding through their answers. However, given the discussion above on the interrelationships between understandings of concepts, this is not a simple matter. We want the design to uncover students’ understanding rather than their rote knowledge. The understanding we want to test is ‘how to think like an economist’, and not simply an ability to manipulate a given model (even if in a complicated manner). Students need to show the ‘framing’ skills necessary to analyse the problem and we need to develop these ‘framing’ skills in the teaching and learning activities we undertake. In order to uncover students’ understanding the question should present something unfamiliar to the students which they then have to use their understanding to unravel, rather than be able to reproduce more or less verbatim a rehearsed answer.

However, the ‘patchy’ understanding we may expect because of the complex nature of concept acquisition means we have to be careful to give

borderline students a good chance of demonstrating their ‘just passing’ level of understanding. Such students may well be rote learning in some parts of the curriculum because they have not gained the integration that goes with the acquisition of the relevant threshold concepts. There is a problem in assessing such learning as there is no clear continuum from this level to understanding. Such learning may be deemed irrelevant if it never progresses, but if the student revisits the concept it could provide the first stage in the later development of understanding. These problems may encourage the design of questions that give students considerable support (such as through directing them towards a concept/model they should use in their answer). However, the threshold concepts approach argues against this and that it is important that students show development in understanding (although not necessarily showing a deep understanding across the whole of the curriculum).

These arguments suggest that the ‘assessment package’ needs careful design which allows students to reveal the depth of their understanding. This requires the setting of questions that allows for this and we have proposed certain criteria for questions:

a) Problem–focused rather than straight theory

The ability to answer problem focused exercises is a good indication of student progress with regards to ‘thinking like an economist’ – it is a good way of finding out the level of understanding of theory and concepts. The focus of the problem needs to be an aspect that has not directly been covered in the module or we may just be picking up rote learning (this does not preclude a question on a general applied area that has been covered, but it must be sufficiently far from replication). The problem needs to be a question that students can relate to and they will have sufficient information on the background.

b) Does not explicitly direct the use of particular theory/concepts

We need to see that students can start to recognise the importance of certain concepts, rather than being directed all the time. Recognising the web of concepts and being able to pick out the important aspects for a particular question is an indication of the depth of integration achieved.

c) The complexities of the question

There is a need to design questions that allow students to show their understanding of the web of concepts that relate to a particular question and within that show an emerging understanding of threshold concepts. However, this is tricky, in that given the web of concepts some applied questions need a large number of concepts in their analysis. How many can we expect students to integrate? We may want to accept ‘good’ answers that are limited in scope and we need to be careful that the complexity of the question does not stop students attempting the question.

These criteria suggest that we should reduce the ‘scaffolding’ (the direction to using certain concepts and models) that is currently present in many economics examination questions. However, basing questions on the criteria above needs to be part of an integrated approach to teaching, learning and assessment. The approach developed in the Embedding Threshold Concepts project suggests that we need to use applied questions in helping students learning during our courses, whilst providing appropriate support, but in testing we can only examine depth of understanding by asking students questions that allow them to ‘frame’ their answer.

As examples, Table 2 gives some past examination questions on two different areas and a discussion of their merits, given these criteria. The first set of three questions is in microeconomics and relate to the threshold concept of opportunity cost. The second set of three questions is in macroeconomics and are concerned with the effect of a fall in aggregate demand on the economy, related to the threshold concepts of interactions between markets and cumulative causation (in this case the multiplier). Given the criteria set out above, question (a) would not be seen as appropriate in the microeconomics set and neither would question (c) in the macroeconomics set.

Top Tip

Test depth of understanding in assessment by asking applied questions without providing clues to the answers expected (by for instance telling the student what model or concepts to use). But only do this having tackled this type of question, with feedback, during the course in seminars/web-based exercises, etc.).

Microeconomics questions

a

What is meant by 'opportunity cost'? Illustrate this concept by reference to a production possibility diagram. Why do economists use the concept of opportunity cost when referring to the marginal cost of production?

This question directs the student to the concepts of interest and is not problem focused. It can be answered by rote learning.

b

Why might an economist expect the price of Christmas trees to fall substantially in the last few days before Christmas? How do sellers try to prevent this?

This is problem focused and does not direct the student to the concepts/theories. It requires students to consider the role of price as an incentive and opportunity cost. They need to recognise that the economist would use the comparative static framework of demand and supply in answering this question.

c

‘Economics is about scarcity, choice, and opportunity cost.’ Explain what this means and give examples to illustrate your answer. How can principles of choice based on opportunity cost be related to the decisions of parents with small children about whether or not to seek paid employment.

This question directs the student to the concepts of interest. However, the second part is problem focused and does test the depth of understanding of these concepts if the particular example has not been pursued in class. In the example the students need to consider the modelling framework of supply (and demand) for labour.

Macroeconomics questions

a

A recent fall in the stock market has hit consumer confidence and reduced consumption. Explain how the economy is likely to react, both in the short and long run, to this event.

This is problem focused and does not direct the student to the concepts/theories. Students need to decide which model they have covered in their module is appropriate to answer the question. It is testing their ability to recognise the need for and use appropriate economic modelling in answering an applied question. A good answer will consider the threshold concepts of cumulative causation and interaction between markets.

b

A recent fall in the stock market has hit consumer confidence and reduced consumption. Explain how the economy is likely to react, both in the short and long run to this. Use aggregate demand and supply curves to show the likely effects on prices, output and income in the short and long run.

This question directs the student to the model of interest. However, the setting is problem focused and does test the depth of understanding of these concepts if the particular example has not been pursued in class. A good answer will apply the model to this case and will consider the threshold concepts of cumulative causation and interaction between markets.

c

Use the aggregate demand and aggregate supply model of the economy to explain how the economy is likely to react, both in the long and short run, to a fall in aggregate demand.

This question directs the student to the concepts of interest and is not problem focused. It can be answered by rote learning. It tells students what model they should use, in a non-applied situation covered in texts, which makes it difficult to distinquish rote learning from deeper understanding .