An experiment by the Christian Commission for
Development in Bangladesh.

This paper has been published, with some variations, in Mosse, D..
Farrington, J., and Rew, A. (1998) Development as Process: Concepts and
Methods for Working with Complexity. London. Routledge/ODI, pages 68-83;
and in Impact Assessment and Project Appraisal, Vol. 16. No. 3,
September 1998, pages 243-250.

INTRODUCTION

The Christian Commission for Development in Bangladesh (CCDB) is a
medium sized Bangladeshi non-government organisation (NGO) with almost 550
staff. It's annual budget of approximately US$4 million is funded by a
consortium of Protestant donor agencies, in addition to its own internally
generated income. Its main programme is the Peoples Participatory Development
Programme (PPRDP), which involves more than 46,000 people in 785 villages in 10
districts. Approximately 80% of the direct beneficiaries are women. Development
assistance is made available to participants in three forms: group based
savings and credit facilities used to meet the needs of individual households,
grant assistance given to the same groups on a pro-rata basis and intended for
community level developments, and skills training, mainly for livelihood
purposes. The large scale and open ended nature of these activities poses a
major problem for the design of any system intended to monitor process and
outcome.

In 1994 an experiment in participatory monitoring was conducted with
CCDB's PPRDP programme that involved the deliberate abandonment of the use of
"indicators", a central concept in orthodox approaches to monitoring. The key
features of the new approach to project monitoring have been outlined in very
brief form below. Implementation took place in four PPRDP project areas in
Rajshahi zone of western Bangladesh, where 140 CCDB staff are working with
approximately 16,500 people grouped into 503 shomiti (associations).

The experiment was the outcome of a voluntary collaboration between the
author as an independent researcher and CCDB. CCDB adopted the approach
because, according to the Director, it appeared to fit their needs. The
experiment was not funded by or encouraged by CCDB's donors, nor was the author
in any way answerable to those donors. The author worked with a male staff
member from the Training Unit, located in the Dhaka office of CCDB, and saw
himself as answerable to the CCDB Director. This immediate allegiance was
moderated by a longer term need to generate information of value to a PhD
thesis. .

The first section of this paper outlines the methodology by detailing
the steps involved in it's operation. This is followed by a summary of the
state of the monitoring system as of March 1995, a year after the first steps
were taken to establish it. A series of contrasts are then made between this
participatory monitoring system (PMS) and what are described as orthodox
approaches to project monitoring. Questions are then raised about the
evaluation of monitoring systems. Finally, an interim conclusion about the
value of the experiment is stated, and two issues for further exploration are
identified.

AN OUTLINE OF THE PROCESS, AS IMPLEMENTED

Nine steps were involved. These are outlined in detail below.

Step 1 The selection of domains of change to be monitored.

Through the Director of CCDB the author facilitated a brief process
whereby Dhaka based senior staff identified three broad areas or types of
changes they thought CCDB needed to monitor at the project level. The number
was limited to three in order to keep procedures simple at the earliest stages
of what was initially an experiment.

The three types of change selected were phrased as follows:

"Changes in people's lives"

"Changes in people's participation"

"Changes in the sustainability of people's institutions and their
activities."

None of these types of change were precisely defined. Their boundaries
were deliberately left "fuzzy". Initially it would be up to the field level
staff to interpret what they felt was a change belonging to any one of these
categories. One additional type of change was included. This was "any other
type of change" as judged important by the project level staff. The
intention was to leave one completely open window through which field level
staff could define what was important and report accordingly. In the case of
the first three domains of change it was the Dhaka head office staff who had
proposed the boundaries or window within which events would be reported.

Step 2 The reporting period

Since the first trial of the method in March 1994 changes have been
reported for each of the months of April onwards in each of the four Rajshahi
project areas. An experiment was made with fortnightly reporting in April but
this was found to be too demanding of staff time at both the project office and
head office level.

Step 3 The participants

There were four groups of participants in the monitoring system: (a) the
shomiti members in the Rajshahi area, (b) the project staff in the Rajshahi
area, (c) the senior staff in the head office of Dhaka, and (d) CCDB's donors,
particularly those participating in the annual Round Table Meeting (RTM). The
structure of their participation determined how the information from the
monitoring system was analysed. This is discussed in detail under Step 5
below.

Step 4 Phrasing the question

The basis to the monitoring system was a simple question in the
following form:

"During the last month, in your opinion, what do you think was the most
significant change that took place in the lives of people participating in
the PPRDP project ?

The respondent was then asked to give an answer (written in Bangla) in
two parts. The first part was descriptive: what happened, who was
involved, where did it happen, when did it hap pen. The intention was that
there should be enough information written down so that an inde pendent person
could visit the area, find the people involved and verify that the event took
place as described.

The second part of the answer was expected to be explanatory. The
respondent must explain why they thought the change was the most significant
out of all the changes that took place in that month. In particular, what
difference did it make already, or will it make in the future ?

Significance was not expected in any absolute sense, but rather in a
relative sense, evident when the various changes that were observed to have
taken place in the same reporting period were compared to each other.

It was not expected that the explanation of significance will be
objective. On the contrary, it would be a subjective expression of the
respondents values and concerns. The purpose of the explanation was to help
bring these values into the public realm where they could be examined, compared
and selected.

The process of sampling changes that was involved was purposive
rather than random. The monitoring system was not reporting the average
state of the PPRDP, but rather what was tak ing place on the outer perimeter of
the programme's experience, the most significant events. If the reported
change was a negative one, then it was a type of change the PPRDP would want to
move away from, to avoid in the future. If it was a positive one, then it was a
type of change that the PPRDP would want to see become more central to their
programme, more typical of their activities as a whole, in the future.

Step 5 The structure of participation

In March 1994 a workshop was held with the senior staff of the four
Rajshahi project offices to plan the implementation of the monitoring system.
Each Project Office (PO) was told that at the end of each month thereafter they
would be expected to report to the head office one sig nificant change in each
of the four domains of change. Each Project Office was then asked to draw up a
plan for how their field staff would, each month, identify a range of
potentially im portant changes and how these would then be analysed in order to
identify the most important. This change would then be sent by the Project
Office to the head office in Dhaka. For research reasons no requirements or
constraints were imposed on who could or could not be involved in the
identification of significant changes within each of the Project Offices.
However, it would not have undermined the basic methodology if the head office
had imposed a specified com mon process. For example, all Project Offices must
involve shomiti representatives in this pro cess. In this experiment Project
Offices were not told that they had to include beneficiaries, or that they
could not include beneficiaries in this process. They were also told they were
free to copy from each others plans if they wished.

Some options concerning methods of selecting from an array of
significant changes were out lined, specifically the possibility of using
hierarchy (immediate bosses) or teams (of peers) to make the selection of the
most significant change out of all they had identified. There was no
requirement that the same approach be used in each project area. Nor was there
any require ment that the plan individual Project Offices made would have to be
rigidly adhered to thereaf ter. However, it was insisted that if the plan was
changed then the new plan should be made clear to the head office. The central
requirement was that however the changes were identified and then selected to
be sent to Dhaka, it should be transparent and accountable to those reading the
selected accounts. In practice, an average of 15 changes were documented by the
field staff at each Project Office, each month, out of which four were then
selected by more senior staff in the same Project Office as the most
significant and then sent on to Dhaka.

The same process was repeated at the Dhaka head office. The four sets of
four changes (one set of 4 from each Project Office) were brought to the head
office each month. The task of the head office staff was to select the 4
changes from the 16 which they thought were the most significant of all. In
other words, the single most significant change in peoples lives, in peoples
participation, in sustainability, and change of any other type. The choice of
participants was left to the Director. In practice between four and eight
senior staff attended each of the monthly meetings. The process whereby the
choice was made by the Dhaka participants was left up to that group. In
practice they decided that each participant would rate each story out of 10,
and the ratings would then be aggregated to produce the group response. The
rating process was preceded by an active group discussion of each account of
change. The single requirement was that they must document and explain their
choice, including who was involved in that process, and communicate it back to
the staff in the four Project Offices. In practice, the Dhaka office meeting
and discussion took about 3 hours of staff time per month.

The third level in this process of analysis involved the donors who
attended the 1994 Round Table Meeting (RTM) in Dhaka in November 1994. By the
end of September the CCDB head office had selected 24 accounts of significant
changes (4 domains x 6 months). Those changes were collated in the form of four
chapters in a report. The introduction outlined the methodology behind their
collection (as here), and each chapter thereafter focused on one do main of
change (with accounts of change ordered chronologically within the chapters).
The appendices detailed an analysis of shomiti and staff participation in the
monitoring system. It was proposed that donors should read each chapter and
select the one change in each chapter which they think was the most significant
according to their own values and concerns. As with other participants, they
should document the reasons for their choices. In practice, the presence of
wide range of people at the RTM enabled the 6 months' changes in the first
domain, that of the lives of the people, to be analysed in this way by 5
sub-groups (2 donors, 1 senior staff, 1 junior staff and 1 shomiti
representatives group).

The structure of participation described above meant that a very wide
range of people's life experiences at the shomiti level were subject to an
iterated process of analysis (choice- explanation-choice...) that eventually
selected a small number of stories of high value. At each level of CCDB a range
of stories of change were available and subject to a range of interpreta tions.
From amongst these some were selected, retained and forwarded on to the next
level in the organisational hierarchy. This process was re-iterated from the
level of field workers, senior Project Office staff and senior Dhaka office
staff. The process of iterated variation - selection - retention, taking place
here within the environment of a single organisation, is the essence of what
has been described by Campbell (1969) and others as the evolutionary
algorithm.

Step 6 Feedback

After each month's changes were evaluated by CCDB head office staff
their judgement of the most significant changes, and the reasons behind those
judgements, have been fed back to the Project Offices concerned. Similarly, the
results of the sub-group discussions at the RTM were also fed back via a
plenary session.

The purpose of regular feedback was so that those identifying the
changes in the first instance can take into account the views of CCDB senior
staff when in the process of evaluating changes. They could either passively
adapt their search for significant change according to the perceived concerns
of the head office, or more actively seek better examples and provide better
explanations for the significance of the types of changes that they
think are most significant. It was intended that if feedback was provided as
planned the monitoring system should take the form of a slow but extensive
dialogue up and down the CCDB hierarchy each month. In more evolutionary terms
it can be seen as a process of co-evolution of interpretative frameworks within
an organisational ecology.

Step 7 Verification

Those changes that were identified as the most significant of all were
precisely those stories where the most effort needed to be invested in
verifying the factual details of the event. Verification visits to the sites of
the described events can perform a policing function, ensuring that field staff
are kept honest in their report writing. They also provide an opportunity to
gather more detailed information about the event which was seen as specially
significant, and if some time after the event, a chance to see what has
happened since the event was first documented (another aspect of impact).
Initially follow up visits were made by the author, with his Training Unit
counterpart. Later in his absence CCDB sent head office staff from the CCDB
Information Unit.

The next two steps are optional extras, not ones central to the
process.

Step 8 Quantification

This can take place at two stages. Firstly when an account of change was
being described it was quite possible to included quantitative information as
well as qualitative information. Secondly, it was possible to quantify the
extent to which changes identified as the most significant in one location or
zone have taken place in other locations, within a specific period. In the case
of one significant change identified in March 1994 (concerning a women shomiti
member's purchase of land in her own name) a letter was sent by the Programme
Coordinator to all 10 PPRDP Project Offices seeking information on the numbers
of identical incidents that they were aware of having taken place in their
project area in the past year. However, no need was seen to repeat this
particular question every month thereafter, as in traditional monitoring
systems.

Step 9Monitoring the monitoring system

This step was not essential, but is described here nevertheless. Using
records generated by the above process it was possible to monitor changes over
time in the proportion of shomities and households that the monitoring system
is effectively sampling. An analysis in November 1994 showed that after 6
months of operation accounts of change had been written concerning 43% of the
shomities. The proportion has continued to grow since then. Similarly it is
possible to monitor the degree to which different types of staff (gender, age,
position, education) are ac tively involved in the process of monitoring
change, and of those actively involved, with what degree of success. Success in
this case being defined as having an account of change being se lected as most
significant at the Project Office and Dhaka level. CCDB has yet to do this type
of analysis formally, but informally there is awareness of differences between
staff in their par ticipation and success. Finally, with an existing data base
detailing features of the shomities that exist it is possible for specialist
staff, such as the CCDB Research Unit, to identify the correla tion between
changes reported as taking place and objective features of the shomities, such
as gender, size, and savings levels.

OUTCOMES SO FAR

CCDB has experimented on two previous occasions with monitoring methods.
One system was developed in 1989 by an external consultant and was wholly
quantitative in its focus, poten tially generating large volumes of tabulated
data. It was never implemented. The second was jointly developed in 1993 by a
new expatriate staff member working with the staff of the CCDB Information
Unit. It focused on the use of rating scales by field staff which were intended
to identify the variations in five predetermined types of participation taking
place in the peoples organisations. It was not implemented.

Although initially planned to operate for the six months until the RTM
meeting the monitoring system described above was continued on afterwards, on
the instructions of the Director. De lays in the reporting of changes from the
project level have not increased, but Dhaka staff were one month behind in
their analysis of changes, at the time of the last visit to CCDB in March 1995.
In January 1995 CCDB decided on its own initiative to extend the system to
include 3 more of the 10 PPRDP areas. A training workshop was organised by the
writers CCDB Train ing Unit counterpart in mid-January, making use of Rajshahi
zone staff who have experience with the method. Mention has been made of
extending it to other CCDB projects and to spe cialist support units with the
CCDB Dhaka office.

Rather than the contents becoming increasingly identical over time, and
the system reaching a form of steady state, new changes have continued to be
reported. The most notable of which concerned shomiti involvement in a
intra-family conflict over contraception use, reported in December 1995. Rather
than the Project Office with the most sycophantic Project Officer be ing the
most successful (as defined above), the reverse has been the case, success
seems to be more correlated with independence of opinion. While the Director
had previously identified one Project Office as the most successful, on the
basis of its good credit repayment record, the same Project Office was the
least successful in terms of it's ability to generate, through the PMS, a large
number of highly rated accounts of significant change.

In contrast to the very limited use made of the output of the CCDB
Research Unit over the same period, staff in the Information Unit and the
Materials Development Units of the Dhaka office have made extensive use of
stories of change in CCDB publications, videos and educa tional materials. In
addition, project office staff took visiting donor representatives to shomities
featuring the reported significant changes immediately prior to the RTM in
November.

The specific settings of the parameters of the monitoring system have
not remained static. In March 1995 CCDB staff were in the process of
considering a focus on "changes in the project management". In response to
demand from the field and head office the Dhaka staff who give the highest or
lowest ratings for a change which in aggregate has been selected as most
signifi cant had been asked to give their explanations. As a result of an
informal participatory evalua tion of the system carried out by the author in
February 1995 CCDB staff have become aware of the fact that collectively they
see a much wider range of objectives for the monitoring system than were
initially conceived by the author and the Director (who wanted evidence of
impact). In summary, the monitoring system has survived and is itself
undergoing evolution, both in the specifics of its procedures and in its
perceived purpose.

The fate of the system is by no means secure however. Differences of
opinion (some explicit) exist within senior CCDB staff, as to the value of the
process. In the absence of clear signs of external demand for the information
that it is producing, for example, from CCDB's donors, it's future is
uncertain.

The survival, use and extension of the system exceeded the researchers
initial expectations. Two of the researchers own expectations were not met. In
each account of change that went up the organizational hierarchy there were
associated arguments for that choice, added when ever that change was selected.
It was expected that middle and especially senior staff would focus on the
associated arguments and through their own choices, selectively reinforce
specific types of arguments. Decision premises would thus be subject to
evolution over time. In practice it seemed that all levels of staff focused
primarily on the descriptions, the empirical events, and gave only passing
attention to the criteria of choice used by those below them. Two explana tions
are possible. One is of "inappropriate" management style, that middle and
senior staff were guilty of micro-management. The other is that the behaviour
reflects the staff members understanding of how the system was supposed to
work, that they were in fact required to fo cus on the events themselves first
of all. Explanation of the option of focusing on the criteria of choice used by
more junior staff may have been needed.

The other expectation was that because of the diversity of possible
significant changes each set of staff faced they would be required to be
critical. They would have to make choices and this would involve comparisons,
and thus judgements of relative merit. Participation in meetings at both
Project office and Dhaka Office levels showed that staff could become very
animated and assertive in their opinions of relative merit. However,
documentation of the choices made, es pecially the reasons given, only rarely
reflected any sense of doubt or criticism, or made any explicit comparison with
other changes that had been examined.

A diversity of perceptions offers an organisation choice of direction.
One aspects of that diver sity is the extent to which the changes that are
reported are explicitly negative or positive. Per haps 90%-95% of the changes
that were reported were seen by participants as being about pos itive changes.
During the informal evaluation of the PMS it was clear that staff at all levels
were quite aware of this aspect of its functioning. Field level staff explained
the risks to their job security involved in more critical reporting of events.
Senior staff, less explicitly, indicated concern about donor and governing
board responses to negative changes. For CCDB staff the balance of reporting as
it existed was functional. Coming from an academic perspective it was somewhat
disappointing for the researcher, who initially saw the presence of critical
content as a potential indicator of the systems success. Two means of managing
balance in reporting were potentially available. One was where negative changes
were reported, senior staff could select them and quite consciously provide
feedback which lauded the act of reporting as a reason for its selection, along
with the specific contents of the report. One problem with this approach,
already encountered, is that subtlety of communication style (intended to lower
risk) will mean that what is intended as a report of a negative change is in
fact not recognised as such by the senior staff. The other means was for CCDB
staff to establish a fourth separate and specific domain where "negative
changes" had to be reported. Here the researchers interests as a re searcher
meant that this option was not spelled out as clearly, or encouraged, as much
as it might have been.

ORTHODOX VERSUS EVOLUTIONARY APPROACHES TO MONITORING

Current approaches to monitoring found in the larger NGOs, as well as
bilateral and multilat eral aid organisations are heavily influenced by a
planning ethos that places substantial empha sis on rationality, prediction and
control (Davies, J,. 1994). The approach documented in this paper is in many
respects the opposite. Seven differences are outlined below.

1. Perhaps the central feature of planning base methods of monitoring is
the use of "indicators", and the need for a common understanding about
them within the organisation if the monitoring system is to work as intended.
That understanding ideally includes: (a) the meaning of the "indicator", what
it is meant to represent, (b) the worth of monitoring the event represented by
the indicator rather than any other possibilities, and (c) the existence of the
event, did it take place ?. Confusion, especially over their existence or
meaning is seen as a threat to the systems functioning. Within the conventional
approach it is believed that differences in the subjective perspectives of
events, and the underlying value concerns of different observers need to be
controlled or ignored.

Under the evolutionary approach agreement on meaning of events is an
outcome at the end of a process (a months cycle or more), never final in its
form, and subject to revision in the light of new experience. The
identification of differences in interpretation is a central to the whole
process, they are to be brought to the surface and explored, not to be ruled
out. In some respects, specially the unfinished and tentative nature of
knowledge, the epistemology of the evolutionary approach is a more post modern
outlook, whereas the conventional approach is closer to positivism.

The PMS acknowledges the fact that different sets of values and
interpretative frameworks co- existence at different levels within an NGO, and
between the NGO and its donors. The NGO field level staff in effect are
creating menus of possible views of the world within broad categories defined
by their bosses (Step 1). Their immediate superiors choose from this menu of
views and in the process create a smaller menu based on their view of the
world. This menu in turn is chosen from by those above them, according to their
views. Although clearly located in a hierarchical structure of power it gives a
significant power to those at the base, more so than under conventional
systems. It also still enables the donors to address their concerns without
counter-productively riding on the backs of the NGO. The senior staff of CCDB
did not address gender issues in their original specification of the three
domains of change. Nevertheless, a large proportion of the accounts of change
that were processed up the hierarchy had very clear issues of gender equity
embedded in them. In the Round Table Meeting between CCDB and its donors the
most important differences between them in their interpretation and valuation
of the significant changes focused on the gender issues within the descriptions
of those changes. These were far more subtle and complex than an indicator ap
proach to gender issues would have revealed.

2. Planning based systems are almost universally very quantitative in
their content. Quantitative analysis is based on the ability to enumerate
things or events. Enumeration requires a basic as sumption about the identify
of events being enumerated i.e each of six apples is in fact an ap ple. At the
very basic level of counting quantification is also about the homogenisation of
expe rience. Differences between apples are extraneous and irrelevant. One
apple and one orange can only be added by regarding them both simply as pieces
of fruit. Within the daily experience of organisation those events which are
countable are summarised by a process of inclusion. Broad swathes of experience
are summarised by totals and averages.

The need to summarise is understandable given that most formal
organisations have a hierar chical shape. The daily experience of clients and
staff at an organisation's base has to be con verted into a form manageable by
the very small number of staff at the apex. In CCDB the Dhaka office staff had
to make sense of 16,500 person years of beneficiary experience of CCDB each
year, in the Rajshahi area alone. The question is whether there are not better
means of doing this than quantification via indicators.

Within the evolutionary approach experience is summarised by
selection rather than by inclu sion, it focuses on the exceptional rather
than the commonality, and it seeks to differentiate rather than homogenise. It
is about defining the meaningful edges of experience rather than identifying a
central tendency. Within this perspective an observer of a basket of five or
six dif ferent types of fruit might selectively summarise that experience by
reporting that the orange is best, because it is the sweetest. Other observers
might selectively summarise the same experi ence from a different perspective,
reporting that the apple should be removed because it has begun to rot and will
spoil the other fruit if left there. Philosophising together they might agree
that the rot is the more important feature because unattended to it will
preclude a range of later experiences, including those of sweetness.

3. Under the planning based approach to monitoring events of concern are
identified before their occurrence, rather than afterwards. In conventional
systems "indicators" are established at the beginning of a project, and data in
the form of statistics is gathered repeatedly throughout the life of the
project, concerning the frequency of those events. In more recent revisions of
this approach, described as "process" approaches, by the Overseas Development
Administra tion (ODA) and others, the identification of indicators may be
delayed until the project is estab lished, and may in the course of the life of
the project, be redefined more than once. The pro cess is strongly deductive in
orientation: start with a conception of the desired state and work down from
there what might be the empirical indicators of its occurrence.

The opposite is an inductive approach, where indicative events are
abstracted out of recent ex perience, and this process is renewed with each new
reporting period of the monitoring system. Instead of being predictable it is
open ended.

4. In most monitoring systems events of concern are defined by people
distant from those events which are to be monitored. Typically the
identification of indicators is carried out by senior staff in organisations,
either in specialist research units or senior executive staff. In some cases it
is made by executives in head offices in other countries, such as was the case
with Action Aid's attempt to identify a common set of indicators for its global
programme in the early 1990's. Distance can also be in the form of differences
in tribe, caste, class, language and education as well as in geography, between
those identifying the indicators and those whose lives they are expected to
relate to. Reformist approaches have consisted of taking the indicator
identification process down the hierarchy, in some cases, to the beneficiaries
themselves whose views are sought, through the use of PRA methods. Action Aid,
ACCORD and other northern NGOs have taken this route in more recent times. The
problem with such an approach is the difficulty the NGO then finds, while
operating within the conventional framework, in summa rising the information
produced by a diversity of locally identified indicators.

The alternative approach is to give those closest to the experience
being monitored (e.g the field staff) the right to pose to those above them a
range of competing interpretations of those events. The role of those in power
over them then becomes to respond, on a selective rather than inclusive basis,
to the menu of options provided to them anew each month. Diversity be comes a
source of opportunity rather than a conundrum.

5. Normally the analysis of events documented by an organisations
monitoring system is carried out on a centralised basis, at senior levels of
the organisation. Typically, field level workers do not analyses the data they
collect, rather they simply forward information up their hierarchies for others
to analyse. In the language of computer design this is a serial and central
processor based approach. From a Marxist point of view it might be seen as a
form of alienation.

The alternative, which can be described as parallel and distributed
processing, is seen both in recent approaches to computer design, and
theorising about information processing in biologi cal systems, and markets.
Information is not stored or processed on a centralised basis, but is
distributed throughout the organisation, and processed locally. Staff not only
collect informa tion about events but they make their own evaluation of that
information, according to their own local perspective.

6. Normally when conventional monitoring data is analysed it is in a
form and location that strips it of context. Centrally located analysts of
tables of statistics sent from field offices are usually well removed from the
site of field experience and have a very limited and static predesigned context
in which to interpret the meaning of the events that are summarised. Typically
few text comments accompany statistics sent up from field workers.

The alternative makes use of what Geertz (1973) has called "thick,
description", closely textured accounts of events, placed in their local
context, and where the role of the observer and their subjectivity, is visible.
In the world of ordinary people these often take the form of stories or
anecdotes. Within the evolutionary approach to monitoring outlined here these
"stories" are accompanied by their readers interpretations.

7. Most monitoring systems are largely static structures. Indicators
remain essentially the same each reporting period, and the same questions are
asked again and again. The focus remains the same. Those involved in the
monitoring system are also seen as unchanging, they simply do their duty. The
possibility of independent (constructive or subversive) staff adaptations of
and to the monitoring system is denied.

With the evolutionary approach the contents of the monitoring system are
potentially far more dynamic and adaptive (The reality of practice will of
course vary from organisation to organisation). Events reported reflect both a
changing world and a changing sets of perceptions within the organisation about
what is important within their world. Where quantitative data is sought on the
incidence of an event found to be significant by a number of levels of the
organi sation it can be done so on a once off basis, there is no intrinsic need
to repeat the same inquiry each reporting period thereafter. Though there are
relatively static domains of change being monitored the fuzzy nature of their
definition, and the process of definition by current best ex ample, means the
overall focus of the monitoring can move over time. Finally, the monitoring
system is simply another arena of organisational life where staff are expected
to adapt, but one where their adaptations should be more visible, and therefore
open to more deliberate and se lective reinforcement.

EVALUATING MONITORING SYSTEMS

The evaluation of a monitoring system is as problematic as the task of
evaluating the impact of CCDB's activities on the lives of it's beneficiaries.
The month to month functioning of the monitoring system is an event subject to
observation and interpretation just as much as that of the women shomiti member
who purchased land in her own name. There are multiple observers with varying
criteria of concern, some within CCDB and some outside. All are making
judgements based influenced by their current context and past history. What
weight should each of those judgements be given and how can they be aggregated
or summated ?

An approach based on an evolutionary epistemology would start with the
fact that the system has survived and treat that as evidence of its value in an
aggregate sense. It has some degree of fitness with it's environment. But this
is a minimalist judgement. In the words of Belew (1991:9), "the dumbest smart
thing you can do is stay alive". Associated with this complex recurrent event
called a monitoring system are the many interpretations of its meaning,
including values and preferences which sustain it, some of which are more
prevalent than others, and whose prevalence changes over time. Success, i.e
some form aggregate value, might also be judged, though with much more
difficulty, by identifying changes in prevalence of these sustaining values. It
could be argued that sustainability on this level is more important than the
mere survival of a set of organisational routines known as a "participatory
monitoring system". But on the other hand it could be argued that those values
need some form of substantiation to be fully meaningful and thereby to survive
as a set.

While the CCDB monitoring system shows that the structured and public
exercise of choice is a means of surfacing and managing some values within an
organisation, it remains to be established how such a process should be
structured when there is a need to evaluate an intra- organisational event in a
wider social and organisational context. Donors, government, consultants, and
academics all have may have an interest in the CCDB experiment, yet they are
much more independent from each other than staff within a single hierarchically
structured organisation. One possibility, yet to be explored, is the use of a
market as a means of summarising judgements. As with the use of hierarchy, its
value could be improved by making information about events taking place in that
market as transparent and public as possible.

CONCLUSIONS AND ISSUES REMAINING

The monitoring system that has been developed not only addressed the
needs of CCDB as an organisation to monitor the impact of its activities. The
concepts underlying its functioning are also relevant to the wider issue of how
to best analyse qualitative (as distinct from quantitative) data, and how to do
this on a participatory rather than solitary basis. The approach proposed
suggest that a key issue is inter-subjectivity, the extent to which different
observers of events or phenomenon agree and disagree with each other. In this
paper the focus has been on the per ception of change and of differences
between those perceptions. The associated practical issues are how to process
such information and at the same time how to represent it.. It is suggested
that iterated processes of variation - selection - retention, i.e the use of an
evolutionary algo rithm, are of value in dealing with both tasks.

If this approach to organisational learning is to be developed two areas
need more in-depth ex ploration. One is the concept of public and private
domains of knowledge in organisation. Studies of simulated multi-agent systems
have suggested that the capacity for learning involves a balance between order
and disorder (Kauffmann, 1994). The public domain of knowledge in an
organisation could be seen as highly ordered, one containing "that which
everyone knows that everyone knows" and one where areas of agreement on meaning
provide a further level of order. Many more private domains also exist in
organisations, and in their plurality and differ ence can be seen to embody
more organisational disorder. Many of these private domains are likely to
remain so, because of the contradictions in interests which could not co-exist
in public. What is the role of monitoring systems in relationship to these
public and private domains ? Widened agreement on an issue, and awareness of
that agreement, will almost by definition assist successful joint action by
people holding those views. Increased awareness of differences is more
ambivalent. It may enable innovations in organisations' objectives and practice
or it may threaten the very cohesion, and survival, of the organisation. The
question is then "How can these risks be managed creatively ?"

The second issue is that of internal and external demand for
information. The CCDB experi ment took place because of internal perceptions of
need, but that awareness included knowl edge of donors views and concerns. The
impact of external demands for information is medi ated not only by internal
processes of interpretation within NGOs, but also processes of inter pretation
within their donor organisations. Variations clearly exist in the extent to
which donors see their demands as legitimate or questionable (e.g ODA versus
Christian Aid), and burdens to be minimised or lightened versus requests which
are potentially educative and enabling. Em phasis also varies in the extent to
which they see themselves as seeking confirmation of pre-es tablished
expectations versus identification and understanding of news about innovation.
The fate of developments such as those within CCDB will undoubtedly be
influenced by how these tensions are resolved. In order to understand and
possibly influence this process attention will also have to be directed by
researchers and consultants to the nature of information demands within donor
organisations, to the influences on these demands, and how they can be managed
creatively.

--o0o--

Subsequent evolution of the CCDB system

Since 1994 a number of people and organisations have experimented with
variations of the method implemented by CCDB. Their names and contact addresses
are given below, for those who might want to contact them.

Leslie Dove is helping
the Women's Health and Safe Motherhood Project implement the system in the
Philippines. Five domains of change are being used: (a) Changes in women's
health practices, (b) Changes in women's decision making, (c) Changes in
women's participation and involvement in the community, (d) Any other changes,
(d) Lessons learned.

Jessica Dart, a Ph.D
student at the University of Melbourne. Jessica has been adapting the method to
Target 10, a dairy extension project in Victoria, Australia. See "Target 10
Evaluation Stories May 1998 - May 1999" for detailed examples of the products
so far.

Terry Bergdall, consultant
based in Zambia. Applied a variation of the method during the design stage of a
rural development project in Ethiopia.

Helen Wedgwood, and others at
ITDG, applied the method as a way of integrating information coming in to the
head office in the UK, from the country and regional offices.

Michael Schroll, of
IIRR in Kenya, has used the method in M&E training workshops with
NGOs.

And CCDB continues to use the method, though the original system has
been expanded and altered more than once. For information contact
Kalipada Sarker, who initially helped set
up the system, and who has been managing it since then

Mailing List

Please Note: There is now a "MostSignificantChanges" egroup dedicated to
information sharing and discussion about the use of the method described in
this paper.