J.

CURRICULUM STUDIES,

2003, VOL. 35,

NO.

4, 459–478

On the evaluation of curriculum reforms
STEFAN THOMAS HOPMANN
The paper considers the current international trend towards standards-based evaluation in a historical and comparative perspective. Based on a systematization of evaluation perspectives and tools, two basic patterns of curriculum control are discussed: process evaluation, and product evaluation. Whereas the first type has dominated the Continental European context for two centuries or more, and the latter type the English-speaking world for most of the 20th century, the last two decades have brought about a situation in which both types are more and more intertwined. The paper discusses the historical preconditions, empirical findings, and possible prospects around this development.

a rather simplified systematization of the tools of evaluation available within these systems, of the two dominant Western traditions of controlling, changing, and evaluating the curriculum of public school systems.2 I will start out by giving rough sketches of (1) the basics of evaluation, and (2) institutionalization; I well know that much could be added and that reality is much more blurred than any systematization of main characteristics can reflect. However, drawing on my systematization, I will outline briefly how the two main patterns for evaluating the curriculum came into being. Finally, I will discuss how current developments may lead to an alignment of the two traditions identified. In all of this, I focus on the historically prevalent forms of evaluation as a social practice, leaving aside the issue of what else evaluation might be.3 Evaluation Logically, there are three basic levels in any social practice which can be the focus of some kind of evaluation:
᭹ ᭹ ᭹

the people involved, the processes engaged in, or the products emerging from these processes.

We look primarily at the personal level, the people involved, if we believe that personal characteristics, i.e. abilities, attitudes, etc., are decisive for the quality of what they do. For instance, we might look if the people involved get along with each other, or if there is a mismatch of personalities or skills of some kind — which might have negative consequences. This approach works, to some degree, when we evaluate a few people in a narrow setting largely shaped by those involved rather than by the environments in which they act. However, in larger social systems this approach doesn’t lead far enough. If organizations depend overwhelmingly on the personal characteristics of their members (as in charismatic organizations, to use Weber’s terms), they tend not to be stable outside the direct reach of these individuals. Any significant change in leadership or staff puts the organization at risk. Larger social systems, like institutions or organizations, are supposed to function irrespective of the idiosyncrasies of their members. In a factory or an administration, we want to rely on things working out, even if the people involved are replaced by others. If we want to change or improve such an organization or institution, we will look at the processes which shape the interactions within that system. In a totally routinized environment, which deals almost exclusively with well-defined problems, i.e. if a, then b, as in a traditional factory, that might be sufficient. All we have to know is if the relations between the a’s and the b’s regulating the interaction are working properly. In modern organizations, however, the routines, and virtually all other processes, depend on how the people fit into their places within the organization. For instance, something might happen which requires more than a mechanical response of the type ‘if a, then b’. The modern answer to this problem is to professionalize the relation between people and processes,

they run the risk of going bankrupt. institution) offering with it. processes or products involved in a social action is difficult in cases where what is going on is doubtful. we have a kind of continuous scale of evaluation levels: we might start at one end or the other. However. This move is especially the case where the process being taken care of is comparatively ill-defined. there are three possibilities. to look only at the products of a social practice. evaluating) a product without knowing its origins cannot be sure that the product as they see it is indeed first and foremost a result of the production line (i.e. to develop professions in which a specific education and a specific set of standards ensure that their members fit into the processes the profession takes care of. For well-defined products (like groceries or machines).e. those buying (i. Thus. personnel.e. processes production. the market and its forces are the most powerful tools of such product evaluation and — because of the success of this approach in creating the Western economy — it is often believed that this is the most important way of evaluating all kinds of problem-solving. i. if the product itself is ill-defined.
Looking at the people. requiring some kind of situational adjustment and assessment. education). Thus. Otherwise.. it is easy to draw this line from the people and processes towards the qualities of the product. even though the product as such might be what they wanted. and pay-offs. or in-between. For instance. organizational. a second necessary differentiation looks at who is doing the evaluation.e. In the modern era. etc. However. The decision to ask a professional to do something is based on the trust that a professional background enables him or her to provide such situational judgement and to render appropriate services based on that judgement. financial. a lover may think differently about the outcomes of his love affair than the husband of his beloved. but without the slightest idea of how the product was manufactured and who was involved in its making. In principle. and products.EVALUATION OF CURRICULUM REFORMS
461
i. All of us undertake this kind of evaluation each and every day as we buy things based (at best) on a comparison of different products. A completely different approach would be to ignore the people and processes and only to focus on end-products. those providing a product also have to look at the process of its production. i. it may be that
.e. Looking at production may force them to also look at the processes and people involved. or destroying other pre-requisites of their problemsolving. as is the case with many social services (like health care. and choose to focus more on one factor or another depending on what we believe about the social action which is to be evaluated:
᭹ ᭹ ᭹
people professionals. the efficiency of the police or of divorce courts might be evaluated rather differently. and depends on who is looking: a thief has a different view on the products of a theft than those who owned the product before. at least in terms of trading. Accordingly. looking at people. costs.

T. with peers or external observers undertaking the evaluation. either by being a part of the same process or by having comparable personal qualifications or duties. The most basic form of external evaluation is the one done by someone close to us. These different perspectives can also be seen in a continuum where the different ways of evaluating can be completely detached from each other or intertwined:
᭹ ᭹ ᭹
self-evaluation peers from within. the more likely it is not to restrict evaluation to self-evaluation by the persons involved but to move towards more complex evaluation approaches. and who does it is different in different types of organization. depending on the social action in question. Institutionalization In the tradition of modern sociology. from Max Weber to Talcott Parsons and Niklas Luhmann. one is evaluated by an acquaintance or by someone who knows one’s work.462
᭹ ᭹ ᭹
S. If there is absolutely nothing which connects us in this sense to the evaluating person.e. We have different kind of peers: those close to us. HOPMANN
one is evaluating oneself. self-evaluation. To simplify somewhat. who knows who we are and looks at what we are doing. there will be some kind of external evaluation. Otherwise. including evaluation of the processes and products. or one is evaluated by someone who is not familiar with what is going on. like colleagues who share in our doing. and external evaluation. They are peers as long as they share something with us. we talk about peer evaluation. However. in social organizations such self-evaluation might be something which we are obliged to do. there is good reason to talk about a movement from the personal level of selfevaluation towards external product-control as the predominant type of evaluation (see Figure 1). e. peer evaluation peers from outside. is something we do every day — and is not noticed unless we communicate the results of our self-evaluation. and we have those who come from the outside and take a fresh look at what we do. ‘peers from within’. either informal or formal. the process of institutionalization is one main force
. we can say that the more complex a social action becomes. we may talk about a truly external evaluation. and there may be rules for how do it and report the results.
The first case. Selfevaluation is only adequate if what we do doesn’t affect others — if it is only our own business. i. those who buy the products we made or who evaluate us from a kind of ‘alien’ stance of no familiarity with the persons and processes involved.g. For modern public institutions. and has changed throughout history. In this case.
Which level of evaluation is used.

but they agree on the fact that we organize most of our social life with the help of institutionalizations which give our activities more or less stable frames of reference. What is striking in the history of modern societies are the differences in how attention to large-scale social problems has been institutionalized. or is.
shaping the structures of modern Western societies. In this respect. etc. and life-span (from short-lived to lives of many centuries). the production of well-defined goods is left to private institutions (from farms to companies) if they are available. As a rule — with exceptions — we can say that ill-defined social problems tend to end up as public matters. for the purposes of this paper it is enough to suggest that institutionalization is the modern way of dealing with social problems which do not solve themselves. we can draw a continuum indicating the degree to which social problems are defined. while well-defined social problems are mainly left to the people and their private institutionalizations. Thus. to market forces. then b ). In such
.4 The theories may differ somewhat in what they see as the driving forces behind this process (functional or other types of differentiation. i. from clear-cut well-defined problems where we know precisely everything that there is to know about this problem (if a. with public institutions moving in to ensure either the good’s availability or distribution in times of shortages. depending on how well-defined the problem was. institutionalizations differ widely in their size (from local to global). Of course. scope (from single purposes to all matters encompassing multiple societies).EVALUATION OF CURRICULUM REFORMS
463
Figure 1.). to ill-defined problems where we do not know what the problem is and what a possible solution might be.
Types and levels of evaluation. However. belief systems.e.

In this sense. HOPMANN
cases. Thus. e. etc. in one society the distribution of certain goods may be seen as highly ill-defined. solved. military. Politics. it is a question of social power. if the professionals concerned could be considered the best people available to deal with it. of course. What a problem is is not a feature embedded in some kind of ‘nature’ of the problem itself but in the definitions of the problem and its solutions which prevail. on the viewpoint of those governing a problem or its environment (e. Professions stabilize the interaction of people and processes.g. ch. the solution depends on whether or not the problem is seen as well-defined.e. to set up the necessary frame for solving the problem of education — either within the family (or whatever the primary group for child-rearing might be) or within institutions. the invention of the teacher as a professional is like the similar inventions of the medical. but those rearing them are seen as knowing what to do. is seen as ill-defined or well-defined depends. Thus. not adequate. T.g. or the availability of a problem-solution. or at least at present. as an early example). Thus. Aristotle. Institutionalization of education only occurs if there are doubts about the adequacy of ‘natural growth’ into adult life as a preparation for the future. And. the availability or distribution becomes ill-defined. rather. and public institutions move in to ‘relieve’ private providers of the risks which occur if the availability or distribution of the problem-solution is unclear. i. The modern answer to this problem was the invention of the professions: by leaving the problem solving and the running of institutions to professionals. from the origins of public schooling there have been huge differences in the views on the form of solution of the educational problem by this or that institution (see. where the education of the young is not seen as being problematic. It is left to the devices of families. based on the agreement of those involved. it is not by chance that almost all educational writers. there is no need for institutionalization. base the need of public education on the argument that privately-provided child-rearing is never. if education is seen as problematic. These rather abstract definitions come to life if we think about specific social problems. taking care of an ill-defined problem by institutionalization does not necessarily make the problem well-defined. that most institutionalizations of ill-defined social problems are accompanied by discussions on how the institution should treat the problem. because the situation is evaluated as being well-defined. there is no need for public involvement. It is no surprise. In short. whereas the same distribution in another society may not create a need for public institutions. it could be assumed that the problem would find the best possible solution. Whether a social problem. therefore. society). highly controversial. In a society. and legal professions. a way of limiting the risk of ill-defined problems by placing them within betterdefined structures. The process leading to other types of problem-institutionalization emerges only if there is doubt about the adequacy of privately-provided procedures for child-rearing. within the predominant frame of problemunderstanding. If young people ‘need’ education. from Plato to Luther to those writing school laws and regulations today.464
S. It is.g. However. 8. They do so by laying out standards for professional education (and the subsequent service-
. e.

i. i. etc. for example. exams. the schools run by professionals with the aim of fostering new professionals like. evaluations. the need of the emerging states to secure the loyalty of their inhabitants or as churches sought to secure the commitment of their constituencies by other means than pure force. monastery and cathedral schools. etc. public. it is difficult to specify means for
. These standards and the measures to enforce them can also be divided along the line of well-defined to ill-defined: well-defined.5 Thus. instead they looked for another professional if they judged the quality of the results to be poor.EVALUATION OF CURRICULUM REFORMS
465
rendering) which culminate in rites of passage (i. one can define the production of problem-solutions within these institutions — which processes were expected to deliver which products. etc. armies. of course.e. And. hospitals.e. respected professions are often allowed to define and defend the standards of professional education and service themselves. whereas less well-defined professions run the risk of their standards and membership rules becoming a non-professional. The second most successful type of school institutions were. what the product of the problem-solving process should be. should be defined externally. with. writing. schools. and such emerging institutions of higher education as the mediaeval universities. Whereas the distribution of the parts and pieces of knowing can be easily defined. matter.) either to enter or to leave (i. be thrown out of) the profession. arithmetic. i.e. Parents did not interfere in the ways these outcomes were achieved. Such locally-controlled contract schools were the prevalent type of schooling in many parts of western and northern Europe until the late-18th century. When. such schools function well if the problematic part of education is seen in terms of learning contents and skills which are difficult to teach in the home. it was (and is) natural for them to assert a need for the profession itself to undertake the definition of the processes — as the body arguably knowing most about the range of possible problem-solutions.e. This claim was often extended: only the product of their involvement. professionals were (and are) seen as having competence for their tasks. the ‘aims of education’. i. internships. the predominant form of pre-modern schooling in Europe was a kind of contract school where parents made contracts with the schoolmaster for what their children (of course typically only boys) were supposed to learn in school: how much reading. for example. The converse is also true: it is difficult to come up with processdefinitions if the product to be achieved is difficult to define.e.). but leaves the definition and control of the day-to-day rendering of services to the professionals involved. by measures of personal inclusion. It is easy to demonstrate historically how the level of problem-definition and the degree to which the professionals were trusted have shaped the process of institutionalization.e. The most common solution to such problems has been a kind of power-sharing: the public (or whoever represents it) defines the structures of the institution and its professions (for courts. for instance. In addition. The picture changed slowly and gradually as less well-defined problems moved into the field. Of course this is not easy to do: an ill-defined problem is an ill-defined problem because we don’t know exactly how to solve it.

e. producing specific types of professions and forms of institutionalization as well as different approaches to evaluation: Well-defined problems dealt with by well-defined
. Thus. i. depending on the economic and political sustainability of private solutions over churchrun or state-controlled schools. T. even when institutionalized. along with process regulations (i. social problem-solving has to be somewhat independent of personal idiosyncrasies in order to function on a larger scale and over the long run. An ill-defined problem remains ill-defined. The argument (as stated. and different kinds of product control (i. contract schools and the like. better-defined than another. the main focus for the evaluation of institutions becomes either the processes or the products involved.466
S. Therefore. every other combination is possible. the emerging public school system did not replace the existing types of privately-held or -controlled instruction. with the possible mix stretching from strong professions dealing with weakly-defined problems (like handicraft) to weak professions dealing with better-defined problems (like manufacturing) (see table 1).e. The development of public institutions of mass schooling followed. The expansion of non-public school systems continued in many areas in Europe until the mid-18th century. the definition of what is needed could no longer be left to the primary environments or the emerging professions alone but became a public matter.e. or was never finished — depending on where in Europe and under what circumstances the developments took place. for example. the evaluation of the involved people (insofar as they are accepted as professionals) tends to become subordinated and restricted to the question of whether the processes or the products provided by individual professionals are in accordance with professional and institutional standards. It is how the problem gets dealt with that receives placement within society — and the acceptance of the new placement as a better definition of the problem is not guaranteed. the new task of the schools goes beyond the deployment of knowledge and skills to the enforcement of habits. However. However. institutions. were invented. The evaluation of institutions and the institutionalizing of evaluation As I have suggested. once again depending on how welldefined the problem in view is considered to be by whom. in most cases the family) to provide an appropriate education.). etc.e. i. school laws. by Luther) continues to be the inability of the primary childrearing environments (i. Faced with such new problems.e. takes place by way of forms of evaluation. i. the process of change took up to 300 years after the Reformation. Whether the process or the product perspective dominates depends on how well defined the profession and/or the problem-solving (production) is. the professions and specified production lines connected with them. at least not immediately. HOPMANN
fostering and measuring attitudes or ways of being. Establishing that one solution is more powerful. from evaluating the people and processes involved to product evaluation by supply and demand.e. Thus. by means of confirmation and other examinations).

However. i.
. for instance. Traditions of process control This interaction between professions and their institutions proved to be decisive for the struggle between the private and the public school sector and initiated cycles of forms of evaluation which mixed different approaches to evaluation or switched from one type to another depending on:
᭹ ᭹
the interaction of school evaluation with other frames of public problem-solving. therefore. Thus.
Professions and problems. a form of inspection. a new dimension of external control. and by the first school laws of the Reformation outlining how schools should proceed. the strategies of both process evaluation and control only worked within the still-weak public sector.EVALUATION OF CURRICULUM REFORMS
467
Table 1. the more likely evaluation will look to find at least some welldefined features of the product to measure — while strong professions enjoy substantial freedom to define both the processes and products themselves. a typical visitation report would. a first step in the process towards a unified profession.e. conclude with the statement that the students showed considerable abilities in the three Rs — which was what contract schools already provided — but that the instruction or the students were obviously lacking the religious spirit — which was what the church and the state were looking for. Thus. initiated by Phillip Melanchthon’s detailed instructions on visitation. and could not. Also connected to the visitation approach was the requirement that teachers in public schools needed the approval of the respective authorities.
The first of these issues is well illustrated by the introduction of the visitation within Protestant (Lutheran) schooling. The visitation added to the then-prevalent forms of entrance and exit evaluation (mostly done by internal examinations). The weaker a profession is considered to be. Problems Ill-defined ←→ Well-defined
Well-defined Professions Ill-defined
professions will less likely create a demand for external evaluation than illdefined problems treated by ill-defined professions.. the evaluation of institutions spans a whole range of different approaches. and the power of the professions working within the institutions. focusing on processes of inclusion instead of products. or the more difficult the assessment of process qualities seems to be. etc.

a new move in the evaluation cycle followed. Thus. For such instruction. performed by assessing the students’ knowledge in externally-defined exams and confirmations. in those areas where the confirmation requirement was first introduced (as early as 17th century in some parts of Denmark) it only took about two or three generations before the school system included almost all available children. The pace of this professionalization process was remarkably different regionally. This intertwining proved to be so successful that. In such a school or seminary. maintaining his teaching of a division between the two worlds). as a living Christianity. as was the pace of school development as a whole. It is an historical irony that one of the most powerful forces in shaping the teachers’ move towards a unified profession was a counter-movement to the Lutheran view of schooling and confirmation.e. Pietism. which had become a formal pre-requisite for becoming a teacher. connecting confirmation to participation in at least some years of schooling. this was still not enough to break the prevalence of non-public schooling. As a result of a common education. In the early-19th century in parts of Denmark and Prussia. However. and then. the how’s and why’s of the staging could be made visible. parents were forced to send their children to school. i. HOPMANN
contribute very much to winning the competition with the private sector. where the public sector still struggled with positioning itself (as in England). The concept of the normal school as an exemplary stage is closely connected to the insight that successful instruction depends on a specific staging. it became possible for teachers to develop the self-understanding of a new specialized profession. The break only came about with help from the professionals running the schools.e. pedagogical supplementary courses were also initiated at universities for those who were to become teachers. for instance. almost all teachers had completed some kind of seminary education. and as a result knowing others working within teaching. In other school systems. The consequence of this idea was the invention of seminaries for teachers. it was not enough that the teachers knew what they were supposed to teach. i.468
S. this professionalizing transformation was not completed until well into the 20th century (or is still in the making). From the mid-18th century onwards. and to act accordingly. even if they did not see an educational rationale in this step (at least when compared to what could be learned in the privately-controlled areas of education). By means of a first step connecting certain civil rights (such as marriage) to confirmation. that is schools where future teachers could learn both the contents and the ways and means of staging a living faith. the introduction of confirmation in Protestant states (a step which Luther had refused to take. Thus. and from this the idea of transferring the script of the staged lessons by means of detailed curriculum guidelines added a very
. One of Pietism’s basic criticisms of the thenmodel of schooling was that teaching only the catechism and other more or less religious content was not enough to turn children into truly Christian adults. the next move in the evaluation cycle was added: external product control in addition to external process evaluation. in a second step. T. in addition they should be able to ‘stage’ instruction in a faith-filled way as praxis pietatis. Only instruction which brought the faith to life could turn knowing into belief. the teachers.

it seems obvious that the definition of the processes and products of their work should be left to themselves: they know most about their problems and the possible solutions to those problems. it had its greatest success in areas where there were comparatively well-established school systems with well-educated teachers (as in. licensing. an Anglican. the English Quaker. It left the teachers as the notvery-independent working tools of a teaching system. thus permitting students to teach each other and thereby allowing the teacher to take care of large groups of students (up to thousands. The state provided general curriculum guidelines (syllabi or Lehrplan ) outlining what to teach combined with prescriptions for who could teach. and dissenters like Joseph Lancaster. The so-called monitorial instruction which they developed and which. had enormous success throughout Europe and US in the late-18th and early-19th century. the most successful version of this approach did not come from the Pietist movement. This new structure was copied from the most effective model of organizing work in post-Napoleonic Prussia. both the state and the emerging teaching profession had an interest in diminishing the influence of local authorities over the schools. Denmark or Sweden).g. In the late-18th and early-19th century. a French Catholic. this unsatisfactory situation led to the introduction of a new type of control over and evaluation of schools: the freedom of teachers as to teach according to their own standards. because the system provided the emerging teaching profession with its own language and set of tools (whereas the Pietist concept copied the education and language of the clergy). but within centralized guidelines which provided frames of reference for their institutions and for the processes within them. the monitorial approach was a useful tool in unifying the teaching profession and in making its specific competencies visible. it was said). However. either to secure the dominance of the larger public school system or their independence from local expectations — or both. those who had passed the required teacher examinations. from Joseph Bell. The monitorial movement started out as a surrogate for more traditional schooling in places where the public school sector was not yet developed in order to provide for mass schooling.6 With this system in place. or from other seminaries. working in far-less-developed school environments. Thus. This interest coincided with the need of the state to gain greater control over the school sector as a whole. Monitorial instruction could help to achieving this. e.g. but from Jean-Baptiste de la Salle. it severely limited the teacher’s leeway to shape his own instruction. however. in a number of different versions. For professionals. the state could now rely on the support of the teachers when enforcing its school policies: the teaching profession received a stable frame of reference — the school as a public institution — which left its members considerable freedom and professional
. However. For an emerging profession.EVALUATION OF CURRICULUM REFORMS
469
strong tool of external process control to the evaluation tool-kit. The main idea of monitorialism was a very detailed prescription of the progression of learning and of the teaching environment. but left the how-to-do-it to what was called the pedagogical or methodological ‘freedom’ of the local teaching staff or school. e. provided the model for this development.

i. e. And. From around the 1820s on. Thus. these external controls either lost their connection to the school sector (in the way that confirmation did in the 19th century) or they were emptied of content (as the regulations for final examinations were in the 20th century). the knowledge of how to create a bridge. All these elements together created a situation in which any external evaluation of the outcomes of one or another curriculum guideline in terms of changing teaching procedures or learning products did not make much sense.e. such assessment had been subject of external outcome and/or entrance examinations. What happened within the frameworks could be different from place to place. between institutional frameworks and locally chosen activities became the centrepiece of the professional education of teachers in most European countries. a new curriculum rarely required substantial changes in the traditional forms of teaching. and it still constitutes the framework for steering the curriculum. T. The teaching profession gained still more influence over the definition of the institutional framework by means of its involvement in the production of these frameworks by.7 As one US observer reported with astonishment at the end of the 19th century: ‘Ask these teachers about whatever they did. up to the schools and teachers. (To that point. which then turned against all other forms of external school evaluation and control. This open system of process control enhanced the independence of the teaching profession.g. by means of argument. irrespective of what students actually knew. this system of curriculum control diffused through most of continental Europe. the transition from secondary schools to universities). HOPMANN
independence in their day-to-day activities. but today in Continental Europe there are virtually no parliamentary committees or branches of administration dealing with schooling in which the majority of those involved are not active or former teachers.
. on the one hand on the administrative and public discourse on schooling and.g. and they will come up with a comprehensive didactical explanation’ (Prince 1892: 7). depending on the local school cultures and the capacity of local teacher groups.) Another very visible side-effect of this was the general introduction of marks or grades instead of content-related characterizations as a main way of evaluating students (this shift — at first developed within the Catholic school system — occurred in most parts of the European world in the course of the 19th century). over time. i. on the other hand. Passing the final internal exams of one type of school became enough to gain access to the following stages (e. However. The definition of the specifics of the products (what was considered as a adequate student achievement) was. on teachers’ argumentation about why this or that could be considered as being within the framework. Didaktik. It took about a century to achieve this position. and by taking over more and more positions — both in school administration as well as in the political systems dealing with schooling. it was on the semantics of schooling. the commissions developing the curriculum guidelines. denouncing them as not being professionally grounded.e. If a new curriculum framework had any direct impact.470
S. thus. This gives the teaching profession a unique intertwined fabric of professional self-control and self-evaluation on all levels of the public school sector.

No unified concept of teacher education and of teaching standards was achieved. etc. Decisions about who was/might be appropriate as a teacher were left to the same local and regional authorities.g. However. following different sets of regulation. home schooling. either as a completely external control (as in externally-designed testing at the end of school stages) or as transition controls. Process controls (of the kind used in the larger East Coast cities’ hugely successful 19th-century monitorial instruction) never gained national status. The necessary pre-requisites for introducing the licensing principle were simply lacking.
The history of schooling in the former British empire provides many examples of this story. Those who can afford it can opt out of the public system whenever they like (using private schools. the most prominent example of this is seen in the US. with the states moving in slowly as regional supervisors. thereby leaving considerable influence with external product controls. and No unified institutional frame for schooling based on a common curriculum emerged. without the driving force of the combination of state or nation and church striving for inclusion by means of schooling.). was lacking. although most shared common features:
᭹ ᭹ ᭹ ᭹ ᭹
᭹
The school sector remained divided among many private and public providers. All this came rather naturally in that the process of nation-building followed a decentralized (federalist) and non-denominational pattern. not least in England and Wales itself (before the change in the control structures in the late 1980s). US schooling was organized in a way that is similar to the contract school tradition. schools are primarily seen as an instrument of the local community’s provision of methods for the distribution of knowledge and ability which go beyond the capabilities of the average families. where neither a unified national school system nor national system of teacher education existed. e. Nor could process controls contribute to the construction of an independent teaching profession in that the necessary pre-requisite.EVALUATION OF CURRICULUM REFORMS
471
Traditions of product control The development of assessment and teaching took a completely different direction in countries where either the public school sector or the teaching profession failed to secure the same prominence as it did in much of continental Europe.
. throughout the 19th century. Different paths of development occured within such systems. with school districts later moving into the position of collective contractors. a unified teacher education at least on the seminary level. and made it impossible to move from external control and evaluation towards internal control and evaluation. And. entrance examinations for subsequent school stages. Even today. The administration of schools was seen as a local affair. which made it impossible for the teaching profession to gain the same status and influence as it had within the ‘Didaktik system’.

472
S. seems to be well-defined. Tracking the impact of changes in the evaluation system makes perfect sense in that the how’s and why’s of local teaching do not count and the quality of the products of schooling. or in a few cases (like the federal government’s National Assessment of Educational Progress) on the national level. these frameworks are realized in a variety of local activities and outcomes. two different approaches to curriculum evaluation and control in the Western world:
᭹
the continental licensing or Didaktik system. Didaktik. inevitably encounter the principal difficulty around this approach when it is used to deal with ill-defined problems. As a result. schools are made accountable for the outcomes. rather. and
. given the professional autonomy of teachers. the ability to connect the institutional frame with the not-yetdefined variety of local day-to-day activities and outcomes by means of pedagogical arguments. at the end of the 20th century. the common core of professionalism is. efficiency becomes the core of the profession: the best teacher is the one who gets his students to ‘stand up and deliver’ at the right time — whatever the cause for their ability to do just that might be. It is different in product evaluation-based systems. i. one can observe. Not surprisingly (if we take the contract roots of the school system as a whole into account). In the product-centred approach. However.e. i. a national curriculum. external product controls. the one tool available as a result of this history was shared standards of product control — as with the Committee of Ten which defined college and university entrance requirements in the 1890s. Changing patterns of evaluation Thus.e. to what degree student achievements are indeed an indicator of the performance of the schools or. reflect factors beyond their reach (like socio-economic circumstances). In the licensing system. e. Tracking the impact of new curriculum frameworks does not make much sense in the licensing system in that. when disconnected from process controls. at the end of the 19th century. the need to enhance the standards of US public education was finally accepted as a national issue. as I have noted. the 20th century was characterized by an explosive multiplication of external product controls in the form of all kinds of testing and assessment.8 In line with the contract tradition. All subsequent initiatives to follow up the product approach by process controls. have failed. T. HOPMANN
When.g. whatever their origin might be. student achievement. developed and administered by local or state agencies or private providers. these product controls focused on the distribution of knowledge and abilities. It does not acknowledge the degree to which what is measured in fact represents a value that is added by the evaluated institution. It also follows that the teaching profession has different possibilities for showing its professionalism in the different systems (see table 2). exercising a weak control and evaluation of the processes and almost no external control of the outcomes of schooling.

Differences between process and product control
Process Input-based plans and frameworks as main instrument Different results allowed if the procedures are in accord with the plans/framework The teacher as main addressee pedagogical (didactical) legitimacy
Product Output-based output results/standards as main instrument Different procedures allowed if the results meet expectations The student as main addressee efficiency of learning
The basic claim of professional expertise DIDAKTIK EFFECTIVE TEACHING
᭹
the product-centred system of external controls as seen. teaching. etc. and the main effect of the external product-evaluation tools seemed to be standardization and stress. etc. or only little. And. level of detail. the results of evolving assessment research supported this public view:
᭹
᭹ ᭹ ᭹
most teachers reported that new assessment strategies had no.EVALUATION OF CURRICULUM REFORMS
473
Table 2. Alternative approaches — introducing. impact on their lesson-planning. in the 1980s and 1990s. in the US. in short. for example. assessment tools in licensing systems or curriculum packages or guidelines within assessment systems — have been almost always time-limited local events. In the US. either by issuing new curriculum guidelines in the case of the continental European systems or issuing new approaches to assessment in the case of the product-centred systems. higher stakes.9
. of assessment tools had no or very little impact on how students and teachers cope. for instance.. student involvement. However.
In the past. but neither innovation nor the enhancement of quality. size. etc. the public came to see the whole ‘nation at risk’ because of the shortcomings of the public school system. both traditions stumbled into a substantial crisis which might well change the traditional patterns. for example. lead to almost nothing or rather the opposite. student achievement. doubts about the performance of the school as public institution have been normally dealt with within the trajectories provided by these traditions. the format. added content.
Varieties of professionalism within the teaching profession. decades of ‘raising the stakes’ and calling for better performance had obviously not produced adequate outcomes.

and is. student achievement. levelling gender differences. level of detail.. broadening the access to higher education. Decades of reforming the curriculum again and again had obviously not brought about the changes which the curriculum authorities had hoped for. impact on how students and teachers cope. the most reasonable reaction was. size. etc. If a new curriculum presents a completely new challenge. and is. especially when they are low-achievers facing accountability expectations. to wait until it is replaced by the next generation of guidelines (which happens about every 5 –10 years. etc. higher stakes. new contents. irrespective of the content area and the school level) and to adapt their didactical argumentation rather then their teaching until the change comes — as they did. And. teachers do have the guidelines in mind when planning their teaching. with the ‘new math’ of the 1960s and 1970s and do today with the ‘project method’ and other ‘new’ instructional requirements. the research on the impact of new curricula could be read as neatly supporting this impression:
᭹
᭹ ᭹ ᭹
most teachers reported that new curriculum guidelines had no or only little impact on their lesson-planning. the emerging international comparisons of student achievement (as undertaken by the IEA and others)
. added content. the format.e. etc. led to almost nothing. T. the public and their politicians evaluated the situation in a similar way. the most natural reaction in such situations was. in neither system could these changes be reasonably attributed to the tools of control and evaluation in use — nor were the outcomes in terms of achievement or public impressions good enough for those who had to judge the change or be judged by them. even while constantly failing. Not least. or very little. and the main effect of the external process evaluation tools seemed to be legitimation and the distribution of new arguments around the curriculum.g.474
S. However. in short. students and the public! Of course. On the other side of the Atlantic. e. of the guidelines had no. They know them and relate to them if they have to defend why what they are doing fits within the guidelines.. However. HOPMANN
Not a good bottom line for an evaluation system which had eaten up billions of dollars and caused quite a lot of frustration among teachers. In both the US and Europe. or rather the opposite. schooling underwent tremendous change in the post-war period: i. but neither innovation nor the enhancement of quality. students. to narrow the scope of the curriculum to what the product control asks for as stand-up performance — which does not yield a long-term strategy for the improvement of the quality of outcomes as a whole and in the long-run.10
Not a good bottom line for an evaluation system which had produced literally thousands of guidelines and caused quite a lot of frustration among teachers. their students’ involvement. etc. teaching. and parents! Teachers and students do have these product controls in mind. new methods.

‘standards-based reforms’ and ‘systemic restructuring’ are terms found everywhere. more rigorous licensing of teachers and accreditation of teacher education according to national standards has come onto the agenda. etc. The best-known of these approaches is found. And the move towards evaluating schools by knowledge and ability distribution contributes to the re-surfacing of the contract school. Comparative product assessment has come to these countries in a significant way. the concept of opening the school sector up to market solutions by.
. The ‘stand-up-and-deliver’ pedagogy of assessment is too strongly ingrained in the day-to-day routines to be thrown out for good by far-away state guidelines. In the US. which fuelled a tremendous strive to climb the ladder on the comparative evaluation tables.). It remains to be seen what the outcome of this realignment or rather — taking the internationalization of educational politics into account — the merger of these two traditions will be like.e. a sentiment which was supported by international organizations like OECD and UNESCO. it seems that these new tools first and foremost serve to co-ordinate the assessment activity. all states but one have introduced state curriculum guidelines of some kind (i. most western European states have turned to the tool-kit of product control. Teacher unions throughout Europe have complained that these tools undermine the professionally-adjusted service-rendering which had been the characteristic of the European system. in England and Wales where a formerly locally-controlled school system is now subjected to both an impressive array of national assessment tools and a national curriculum which aims at high standards in every subject for every student. On the other hand.11 In short. other traditionally high-performing countries like Switzerland and Norway have undertaken large-scale evaluations of their school systems. the tool-kit of process evaluation has been introduced as never before. and has come to dictate the public discourse on schooling to a hitherto-unknown degree. combining this approach with a reduction of the traditional centralized process controls. as we look at the introduction of process tools in the assessment tradition.e.EVALUATION OF CURRICULUM REFORMS
475
seemed to support the diagnosis that schooling in Western societies was substandard. There are some indications that the inclusion of product controls may have a comparatively larger effect on the licensing system. in the years since the publication of A Nation at Risk (National Commission on Excellence in Education 1983). Much will depend on the ability of the teaching profession to use the new tools to re-define its professionality and to improve its public standing. curriculum standards. Countries like Sweden and Finland have broadened the number of test and assessment tools they use. To this point. led those responsible on both sides to look across the Atlantic for possible solutions. calling for more local school ‘autonomy’ or simply by private competition. Thus. This common feeling of crisis. introducing all kinds of assessment and testing to their systems in the hope that this will reassure the public that everything possible to improve the outcomes of schooling is being undertaken. but have no direct impact on local schooling. However. of course. all we have are guesses. for example. i.

(see Hopmann and Riquarts 1995. for instance. the US. Goodson et al.g.g. (2000). If that was the case. Traditional accounts of the history of evaluation tend to reduce its scope and content to the Anglo-Saxon component. 3. A useful summary of the pros and cons of assessment is given in Mehrens (1998). educational psychology. they miss the fundamental structures and processes which have shaped the differences between. one William Farish invented quantitative marks to assess student performance in 1792. and the Nordic countries (see. and many European countries. e. Sometimes. e. Westbury et al. 2000). (1986) and Chelimsky and Shadish (1997). Models of curriculum evaluation are discussed in.e. 5. Haft and Hopmann 1990). leaving out the much older roots stemming from. (1999). but still useful accounts may be found in Kaufmann et al. and — not least — all the children above average. The paper is based on research undertaken within the evaluation of the Norwegian curriculum reform of 1997 (EVA 97 supported by a grant of the Norwegian Research Council) as well as my contributions to a number of other national and international evaluation projects. A more detailed account will be published in Hopmann (in press). in the Anglo-Saxon tradition these subjects are assigned to different disciplines such as curriculum theory. 9. as in so many times in the past. Schulein ¨ (1987). all the men are goodlooking. In a very real sense. By proceeding in this way. as a typical example. Older. Schmalz-Bruhns (1989). There is a never-ending debate both within and outside the evaluation community on what these assessments do indeed assess and to what degree this is an effect of the institutions involved (see. the US. e. McLaughlin and Phillips (1991).12
Notes
1. It may be that the one best solution would be to send all the evaluators and their tool-kits off to the shores of Wobegon to evaluate what may be one of the most successful mergers of cultural traditions. if the merger of the two traditions does achieve all of its intended goals. and Nesje and Hopmann (2002). covering all subjects relevant for teaching. Teddlie and Reynolds 2000).476
S. 8.g. this leads to real errors of fact: that. The following historical account is based on historical studies which I have conducted in Germany. although there are restrictions intended to regulate the overall traffic system in order to sustain its overall coherence and order. and Leschinsky (1996). e. e.
. Didaktik means far more then the English term ‘didactics’. teaching methods. 2. and Jackson (1996). any comparative evaluation will show inevitably some systems and countries above and some below the international mean. Maddaus and Stufflebeam (2000: 4)). Hopmann 1988. see Kunzli ¨ and Hopmann (1998). this system is analogous to the privileges given to holders of drivers’ licences: with a licence a driver is free to drive anywhere. e. the UK.g. For examples and further references. it was a classical re-invention of a wheel which was widely known in Continental Europe by this time (see. T.g. 7. i. Didaktik is the centre-piece of professional teacher education in Continental Europe. See. 6. Luhmann (1964. 4. However.g. Europe’s 16th-century religious Reformation and Counter-reformation. Hopmann (2001). the traditions prove to be strong enough to cool the impact of whatever is added to the evaluation toolkit. HOPMANN
It may be that. find out how the Norwegian bachelor-farmers of Minnesota manage their educational processes so that all the woman are strong. 2000). etc. and each system will continue to follow the main lines of their institutional history. A good account of the various shapes and sizes of policy evaluation is given by Vedung (1997) and Stufflebeam et al.