The following is the established format for referencing this article:
Zorrilla, P., G. Carmona, Á. De la Hera, C. Varela-Ortega, P. Martínez-Santos, J. Bromley and H. Jorgen Henriksen 2009. Evaluation of bayesian networks as a tool for participatory water resources management: application to the upper guadiana basin in spain. Ecology and Society15(3): 12. [online] URL: http://www.ecologyandsociety.org/vol15/iss3/art12/Research, part of Special Feature on Implementing Participatory Water Management: Recent Advances in Theory, Practice and Evaluation

Stakeholder participation is becoming increasingly important in water resources management. In participatory processes, stakeholders contribute by putting forward their own perspective, and they benefit by enhancing their understanding of the factors involved in decision making. A diversity of modeling tools can be used to facilitate participatory processes. Bayesian networks are well suited to this task for a variety of reasons, including their ability to structure discussions and visual appeal. This research focuses on developing and testing a set of evaluation criteria for public participation. The advantages and limitations of these criteria are discussed in the light of a specific participatory modeling initiative. Modeling work was conducted in the Upper Guadiana Basin in central Spain, where uncontrolled groundwater extraction is responsible for wetland degradation and conflicts between farmers, water authorities, and environmentalists. Finding adequate solutions to the problem is urgent because the implementation of the EU Water Framework Directive requires all aquatic ecosystems to be in a “good ecological state” within a relatively short time frame. Stakeholder evaluation highlights the potential of Bayesian networks to support public participation processes.

Stakeholder participation is becoming increasingly important in water resources management (CEC 2000). There are a number of methods to facilitate the involvement of interested parties in decision making, but in many cases it is difficult to evaluate the quality of these methods because of the lack of appropriate benchmarks. This paper describes the development and the evaluation of a participatory decision support system based on Bayesian networks (BNs).

Research was carried out under the NeWater project (New Approaches to Adaptive Water Management under Uncertainty, Mysiak et al. 2010). The project aims to develop tools and strategies to cope with uncertainty, while also promoting the concept of adaptive water management. In this context, BNs were proposed as a suitable approach to underpin water management in the Upper Guadiana Basin in central Spain.

This paper begins with a description of the study area and the research objectives. The focus then shifts to describing and discussing the methodology and the results of the evaluation of the participatory process. The final section is devoted to outlining the main conclusions.

The Upper Guadiana Basin, in central Spain, presents a set of water-related
conflicts. These mostly stem from uncontrolled groundwater extractions,
and are rooted in the distant past. Since the 1970s, groundwater abstraction
has brought prosperity to the region by enabling farmers to switch from
dryland agriculture to irrigation. However, uncontrolled extractions have
considerably depleted the Mancha Occidental aquifer, the area’s main
water source. This has caused a variety of environmental problems, including
the desiccation of Ramsar wetlands (Coleto et al. 2003, Fornés
et al. 2005, Llamas 2005), leading to conflicts between stakeholder groups; i.e.,
environmental conservation groups, farmers’ associations, and the Guadiana
River Basin Authority. To date, environmental restoration attempts have failed
up due to poorly integrated water, environmental, and agricultural
policies (Martínez-Santos et al. 2008a,b). One example is
the Common Agricultural Policy of the 1980s and 1990s, which favored
water-intensive crops by coupling subsidies to production (Varela-Ortega
et al. 1998, Varela-Ortega 2007). Besides, control over groundwater extractions
is traditionally limited. Only half of the existing wells are registered.
For practical purposes, this implies that water authorities are unable to
enforce sustainable extraction patterns.

Urgent action is required to comply with European legislation. Indeed, the EU Water Framework Directive (CEC 2000) urges all member states to attain a “good ecological status” of surface and groundwater bodies by 2015 (or at the latest by 2027) (CEC 2000). Thus, the River Basin Authority recently launched the Upper Guadiana Water Plan, a water management initiative which aims at restoring the area’s wetlands within the Water Framework Directive deadlines (CHG 2008). The plan is endorsed with 5500 M€ over the next 20 years, and incorporates actions such as the purchase of land and water rights from irrigators, or the closure (or legalization) of unlicensed wells. Complementary measures include reforestation and dryland farming. Implementation of the Upper Guadiana Water Plan is, however, subject to uncertainties, which mostly relate to likely budget cuts and to the irrigators’ willingness to part with their rights to extract water.

Thus, the main water stakeholders fall into four groups:

the Guadiana River Basin Authority, which depends on Spain’s Ministry for the Environment and is responsible for water policy;

the Castilla–La ManchaRegional Government, which is responsible for the implementation of agricultural policy;

farmers, who account for approximately 95% of the total groundwater withdrawals in the catchment; and

environmental conservation groups.

There are also other stakeholders, such as urban supply companies and industries, although their role in water conflicts is comparatively minor.

Several participation techniques have been used to incorporate knowledge, interests, and values into decision making (Rowe and Frewer 2000, Webler and Tuler 2001, CEC 2003, Lynam et al. 2007). Recognizing the strengths and weaknesses of the participatory approach is essential for the correct interpretation of results (Lynam et al. 2007). Moreover, experience shows that the choice of suitable techniques can be a matter of utmost importance (Hare et al. 2003). Group model building, also termed participatory modeling, is a specific subset of public participation methods. Group model building is essentially a method to structure participatory discussions, to try to improve the productivity of group processes (Andersen et al. 2007). This technique has been extensively used in system dynamics, and ultimately aims at obtaining a simplified representation of reality through the identification of key variables and their interrelations (Lynam et al. 2007).

Variables in environmental systems are often subject to cause and effect relations. These can be difficult to quantify due to the absence of appropriate data or to poor understanding. In such cases, links are best expressed as probabilities rather than exact values. The chief advantage of this approach is that probabilities convey the uncertainties involved in model inputs and outputs (Charniak 1991). Bayesian networks (BNs) are specifically designed for this purpose. Thus, BNs are best described as directed acyclic graphs which attempt to represent the main variables of a system and their quantified relationships using Bayes’ probability theory (Jensen 1996). In other words, BNs provide a probabilistic representation of the relationships between input parameters and their dependent variables.

Like other types of causal diagrams, BNs offer an excellent means to engage stakeholders, elicit knowledge and values, and produce outputs in various formats (Lynam et al. 2007). Furthermore, BN results can readily summarize information, and BNS can be useful for focusing the dialogue on key issues (Rieman et al. 2001). Another remarkable feature of BNs is that they allow the inclusion of both qualitative and quantitative knowledge. This, combined with the ability to carry out rapid updates and simulations, encourages communication and learning. As a result, some authors conclude that BNs are excellent for strategic considerations (Henriksen et al. 2007). On the other hand, BNs also present some drawbacks. Even if the tool is reasonably straightforward to use, time-intensive training is required to master it. Moreover, BNs are not particularly well suited to dealing with dynamic systems because the computational burden required to solve probabilistic relations increases exponentially with the number of variables (Castelletti and Soncini-Sessa 2007).

The first part of this section focuses on participatory Bayesian netword (BN) development in the Upper Guadiana Basin. A description of the evaluation methods ensues, leading to a discussion as to the potential of BNs to boost public participation.

In the Upper Guadiana Basin, a lengthy participatory process took place from 2005 to 2008 in the context of the NeWater project (Mysiak et al. 2010). This participatory process was divided into two parts: in the first two years (2005–2006), five stakeholder meetings were held to discuss water management issues, while meetings in the third and forth years (May 2007–April 2008) were devoted to the construction of BNs. This paper is focused on this second part of the participatory process; however, a brief explanation of previous meetings is also provided (Martínez-Santos et al. 2007).

The need for modeling approaches was established early on, in October 2005, during a meeting between researchers and stakeholders. This meeting followed a thorough stakeholder analysis and engagement process, and served the purpose of establishing the needs for research, tools, and capacity building. The following stakeholder meetings served the purpose of discussing water management issues from a general perspective, allowing stakeholders to discuss agro-economic, institutional, and hydrologic aspects of water management. They were also used to underpin the implementation of hydrological modeling approaches (Martínez-Santos et al. 2008a, 2009). Ensuring confidentiality helped reduce the conflictive atmosphere observed prior to the beginning of the project (Varela-Ortega et al. 2006).

In turn, meetings held over the second half of the project focused on the
development of BNs. This process was based on the implementation guidelines
produced by the MERIT project (Bromley 2005). Due to the existence of a
well-established stakeholder core, participants became involved from the
very outset, i.e., the problem definition stage. Early stakeholder involvement
was important to generate a sense of ownership, as well as to facilitate
the acceptance of eventual results (Rowe and Frewer 2000, 2004).

Two BNs were constructed using a specialized software package, HUGIN Researcher 6.9 (Hugin Expert 2008). The first one focused on farm-scale issues, whereas the second dealt with the aquifer scale. Both were built to simulate the outcome of different management actions on two key parameters, namely groundwater levels and the regional and local agrarian economy.

For practical reasons, the number of stakeholders was kept as small as possible, while also ensuring that participants were representative of all existing viewpoints (Rowe and Frewer 2004; Burguess and Chilvers 2006). A list of participants is given in Table 1.

Researchers adopted a variety of roles. These ranged from giving introductory talks to facilitation tasks and feedback collection. The research team attempted to ensure a fair participatory process through a two-way communication procedure (Webler et al 2001, Burgess and Chilvers 2006, Rowe and Frewer 2004). In addition, participants were consulted for their opinion at all stages of the process and were kept informed about each step or of any news concerning the process (Webler et al. 1995, Rowe and Frewer 2000, Von Korff 2006). Furthermore, researchers tried to conduct the process in an unbiased way (Rowe and Frewer 2004, Von Korff 2006).

The BN development process was structured in four steps, starting in May 2007 and finishing in April 2008. Table 2 illustrates the objectives and boundaries of each meeting (Webler et al. 1995, Burgess and Chilvers 2006, Von Korff 2006). For the first meeting, stakeholders were divided in two groups, each of which encompassed stakeholders with relatively similar viewpoints. This was deemed appropriate in order to avoid too much direct confrontation at the beginning (Cain 2001, Bromley 2005, Henriksen et al. 2007). A BN was thus constructed during each session. Researchers then merged both networks and continued filling in the states and the conditional probabilities. Subsequent interviews and meetings served to fine-tune the structure of the networks. Model inputs were also evaluated and updated by the stakeholders. This included several rounds of participation and evaluation activities, which helped to improve results and to incorporate diverse values, interests, and perspectives. These activities also constituted a validation exercise, as stakeholders observed the models to yield realistic results.

Process evaluation followed the methodology outlined by Abelson et al. (2003), Rowe and Frewer (2004), and Von Korff (2006). These authors emphasize the concept of “what works best when” (Fig. 1); in other words, how the specific results of a study can help evaluate the effectiveness of the participation tool in a way that can be useful in different contexts.

Identifying adequate benchmark criteria is crucial in evaluating participatory approaches. This is because criteria must draw useful insights to help other participatory initiatives. In this case, benchmarks were designed to evaluate the success of the participatory process itself, as well as the quality of the tool (i.e., how well BNs supported participation).

Criteria to evaluate the success of the participatory process

A number of goals for the participatory process were identified (see Objectives). These in turn helped to devise a series of criteria to evaluate the process (Table 3).

Conflict attenuation is also identified as an important criterion by most authors. However, this was not included in the evaluation because it had largely been achieved during the earlier stages of the project (prior to BN development).

Criteria used to evaluate BNs as a tool in participatory processes

The criteria used to evaluate the participatory tool belong in three categories (Lynam et al. 2007): capabilities, use, and products.

The capabilities of the tool refer to its potential to facilitate a successful participatory process. The use of the tool refers to the conditions or context in which the tool is applied, whereas products refer to the results. Table 4 provides more specific definitions.

According to these criteria, the participatory process may be evaluated
from different perspectives. An example is the use of existing theory, through
summarizing the opinions of researchers and their findings. Other approaches
include the use of surveys or interviews, or simply open discussions on
key aspects of the process (Rowe and Frewer 2004). In this research, a combination
of three methods was used: (1) stakeholder evaluation questionnaires, (2)
semi-structured telephone interviews, and (3) the researchers’ theoretical
assessment (Stewart et al. 1984, Einsidiel et al. 2001, and Henriksen et
al. 2007).

Evaluation questionnaires were distributed to stakeholders during a project meeting, straight after the main project results were presented. Fifteen completed questionnaires were returned, representing 75% of the stakeholders who had participated in the process. This sample was considered sufficiently representative. The questionnaire was kept deliberately simple, with three open questions and ten agree/disagree/don’t know questions.

The effects of two potential actions proposed in the Upper Guadiana Water Plan were simulated with farm and aquifer-scale Bayesian (BNs). These actions include: (1) purchase of irrigation rights by the River Basin Authority, and (2) pumping restrictions to improve farmer compliance (Zorrilla et al. 2007).

The farm-scale BN simulated the combined effects of the price of irrigation rights and the different levels of farmers’ compliance with water restrictions (based on percent water volume consumed over the allocated quotas). Outcomes were expressed in terms of water table recovery and farm income. Results show that both measures can make an important contribution to aquifer recovery. However, they also suggest that aquifer recovery needs widespread compliance with water restrictions and a high price for irrigation rights.

The aquifer-scale BN (see Fig. 2 and Appendix 1) reveals that the probability of achieving aquifer recovery by 2027 is relatively low (25–40%), even with full implementation of the Upper Guadiana Water Plan. Achieving recovery would reduce the current economic production of the agricultural sector by 50%. The number of farmers would decrease by a similar percentage. Aquifer recovery costs can therefore be estimated in economic and social terms and used by decision makers (Zorrilla 2009).

Earlier participatory processes held within the NeWater project enabled BNs to focus on technical aspects. The participatory process was evaluated by the stakeholders using the criteria established in the previous section. Table 5 presents an overview of the results. Overall stakeholder response was largely positive, as the majority of participants expressed confidence in the process.

The last objective, foster trust in institutions (criterion 7, Table 3), was not evaluated in the questionnaires. Nevertheless, this is a particularly important issue in the study area, where most stakeholders mistrust each other. It was therefore encouraging to witness the positive attitude of the River Basin Authority throughout the entire process, in which it was among the most active, collaborative, and enthusiastic of participants. This development helped other groups to bolster their confidence in the ability of the River Basin Authority to successfully address water management problems.

Three of the criteria used to evaluate the capability of BNs for participation purposes were included in the questionnaires. Results are shown in Table 6.

Most stakeholders acknowledge that BNs are useful for structuring meetings and encouraging communication and discussions (73% agree). One of them, however, argued that cultural peculiarities constrained the ability of BNs to drive the process. This is because participants often drifted off to side issues. He also pointed out that stakeholders complained occasionally, usually when they got upset about the debate.

Both the results of telephone interviews and the researchers´ judgments suggested that BNs are a useful means to identify the level of knowledge or uncertainty (criterion 9, Table 4). Some specific participant comments support this statement. Take for instance the following: “the process stressed the uncertainties concerning the exact extension of irrigated land, which can be, depending on the source, 130 000 ha or 260 000 ha”, or “BNs are a useful tool for combining data from different sources, and this allowed the integration of those data sources while helping to identify existing gaps in our knowledge”.

The present case supports the view that BNs require specific skills from the user (criterion 12, Table 4) (Lynam et al. 2007). An understanding of probabilistic models is thus necessary to supervise stakeholder inputs. It is also true, however, that a wide range of user-friendly BN software is available, both commercial and free. Concerning the resources (criterion 13, Table 4), time was a limiting factor. The literature highlights this as a recurring problem (Cain et al. 2003, Henriksen et al. 2007). In the present case, additional time would have allowed the participants to discuss specific issues. In particular, it would have been interesting to discuss what factors would lead farmers to sell water rights to the Guadiana River Basin Authority.

As far as model outputs are concerned, participants noted some advantages and some disadvantages in relation to the format of the results (criterion 14, Table 4). Probability distributions allow the level of uncertainty of any particular result to be specifically represented. They also emphasize that there are significant uncertainties in the predictions, and that these are aggravated by the existence of numerous drivers for change. On the other hand, people are more familiar with absolute numbers than probability distributions, a factor which constrains the effective communication and understanding of the results.

This section begins by comparing the results of this process to the existing literature. The focus then shifts to highlighting some of the strengths and weaknesses of the participatory approach. Finally, the limitations of the evaluation exercise are discussed.

Criterion 4, incorporate stakeholder values into the decision making,
received the highest score in the questionnaires (Table 5).
This implies that the use of Bayesian networds (BNs) achieved the goal, which is
obviously important in any participatory process (Bacon et al. 2002,
Lynam et al. 2002). While some authors question the ability of BNs to involve
stakeholders other than policy makers (Cain et al. 2003), this case shows
that informed stakeholders can effectively contribute to the process. Take
for instance the role of farmers and environmentalists, who actively participated
in the discussions while also contributing a huge amount of information.
Another remarkable outcome from the evaluation questionnaires is that the
participatory process increased the understanding of other stakeholder concerns
(criterion 2, Table 5) (Henriksen et al. 2007, and Cain et al. 2003).

One of the main advantages of BNs is that they contribute to structuring the participatory process. This is especially useful in contexts such as this, where a public participation tradition is absent (Hernández-Mora 2007, Espluga and Subirats 2008). Stakeholders confirmed that the approach helped give structure and focus to discussions (criterion 8, Table 6), and in this way helped to ease the atmosphere of conflict during the meetings.

The graphical interface of the BNs also proved useful (criterion 11, Table 6). Henriksen and Barlebo (2008) also showed that end-users perceived the graphical model, provided by the BNs, as a strong tool for visualizing complexity. This contributed to the understanding of the relationshipsbetween variables, and generated informed discussions at every meeting (criterion 3, Table 5). This is confirmed by stakeholder perception (87% of participants). Similar conclusions were drawn by Lynam et al. (2002), Cain et al. (2003), and Henriksen et al. (2007).

Scores given to criterion 4 confirm the ability of BNs to incorporate values into the decision making process. This is also true of the perceived usefulness of the process (87%, criterion 1, Table 5), the increased understanding (criterion 3, Table 5), and the BNs facility to help structure the process (criterion 8, Table 6). Thus, BNs are perceived as an effective means to communicate complex issues and the causes of uncertainty through a graphical interface, as well as to underpin stakeholder dialogue. This in turn implies that BNs can be used to support strategic planning. BNs can also contribute totransparency(73%, criterion 6, Table 5) by allowing the stakeholders to have access to background data and knowledge used during the development stage.

Limited confidence in the results (53%, criterion 5, Table 5) may be interpreted as an indicator of the situation in the Upper Guadiana Basin, where no win-win solutions appear possible. Broadening the participatory process could thus help identify new strategies. This score also emphasizes that power relations are important when stakeholders are involved, and that BNs never become a shared tool when viewed from the point of view of outcomes. However, a shared process seems to be more important for all parties than a shared outcome (or shared decision support tool). This is because stakeholder interests may lead to different interpretations of the outcomes (Lynam et al. 2002). In other words, the very process of developing the BNs was more important than the final results (Lynam et al. 2002).

BNs simplify reality. This is both an advantage and a disadvantage. Conceptualization helps stakeholders to better understand the system. Nevertheless, this can also cause some problems. In the case at hand, some stakeholders wanted the network to cater to complex side issues such as groundwater pollution. While perhaps desirable, this was simply not possible. Thus, the group eventually agreed that the network should only deal with the more pressing concerns.

BNs present a number of advantages over other participatory tools. For one, the graphical interface helps to focus discussions with non-expert stakeholders. BNs trigger multidisciplinary discussions and, due to the use of probability distributions, can explicitly represent uncertainty in the system (criterion 9, Table 4). However, dealing with probabilities presents some inherent disadvantages (criterion 14, Table 4). For instance, Lynam et al. (2007) noted that stakeholders may eventually get bored. In the case at hand, an individual stakeholder put it as follows: “arguing about numbers is tedious”. BNs allow this to be bypassed by talking in qualitative terms. These are later translated into numbers by researchers. Translation of single-event probabilities or percentages into frequencies with definitions of reference classes might be one way to help communicate results to the stakeholders, which is something that should be assessed in future studies (Anderson 1998).

Another drawback lies in the construction of conditional probability tables. Stakeholders find it difficult to assign values to some of the variables, particularly to those representing future actions (Cain et al. 2003, Croke et al. 2007). Such reluctance is not surprising. Croke et al. (2007) found that it was difficult to persuade experts to provide probabilities to problems they considered to be extremely complex and impossible to quantify without extensive research. Similarly, Rieman et al. (2001) observed that the use of conditional probabilities explicitly recognizes the limitations in the understanding of the system. Because of the partly subjective origin of some conditional probability tables, results from BNs can be misinterpreted or misused. This means that outputs should be viewed only as relative trends among alternatives, rather than as absolute numbers.

Considering the evaluation itself, perhaps the main limitation has been the short length of the questionnaire. For future studies, an early evaluation of the BN objectives should be included within the first meetings. Furthermore, an analysis of the efficiency of the tool as perceived from the point of view of different stakeholder positions (policy makers versus technicians, farmers, and environmentalists) should also be introduced.

When reality needs to be modeled, simplification contributes to a better understanding of the different elements within a given system and their interrelations. This is especially important for underpinning social learning in participatory processes, and in helping participants realize the uncertainties involved in the modeling process.

By their very nature, Bayesian networks (BNs) are perceived as effective tools for engaging stakeholders in decision making. BNs enable dialogical learning, while allowing for discussion and negotiation, and this also allows the sources of uncertainty to be explicitly addressed. The steps needed for the construction of a BN help to structure a participatory process. The use of BNs in water planning increases the probability that strategies will be endorsed by stakeholders. In the context of European legislation, BNs may contribute to achieving some of the goals of the EU Water Framework Directive, particularly those related to active involvement of stakeholders in water management.

Responses to this article are invited. If accepted for publication, your response will be hyperlinked to the article. To submit a response, follow this link. To read responses already accepted, follow this link.

This research was funded by the EU's NeWater project (Contract number: FP6-2003-GLOBAL-2-SUSTDEV-6.3.2-511179-2) (http://www.newater.everyone) and was carried out under the joint collaboration of the Complutense University of Madrid (UCM), the Technical University of Madrid (UPM), the Geological Survey of Spain (IGME), the Geological Survey of Denmark and Greenland (GEUS), and the Centre for Water Research of Oxford University. The authors wish to thank the reviewers for their useful comments, which have greatly improved the manuscript. Professor Ramón Llamas is acknowledged for his steady support and valuable contributions.

Bromley, J. 2005. MERIT Project guidelines: management of the
environment and resources using integrated techniques—guidelines for the
use of Bayesian networks as a participatory tool for water resource management.
Centre for Ecology and Hydrology, Wallingford, UK.

Burgess, J., and J. Chilvers. 2006. Upping the ante: a conceptual
framework for designing and evaluating participatory technology assessments.
Science and Public Policy33(10):713-728.

Cain, J. 2001. Planning improvements in natural resources management: guidelines for using Bayesian networks to support the planning and management of development programmes in the water sector and beyond. Centre for Ecology and Hydrology. Wallingford, UK.

CEC (Commission of the European Communities). 2000. Directive 2000/60/EC
of the European parliament and of the council of 23 October 2000 establishing
a framework for community action in the field of water policy. Official
Journal of the European CommunitiesL327:1-72.

CEC (Commission of the European Communities). 2003. Common implementation strategy for the water framework directive (2000/60/EC). Guidance Document No 8 Public Participation in Relation to the Water Framework Directive Produced by Working Group 2.9 – Public Participation. Office for Official Publications of the European Communities, Luxembourg.