According to the Development Grant Agreement (DGA, Schedule 2, p. 19) and the Project Appraisal Document (PAD, p.3), the objective of the Project was “to contribute to the Recipient’s efforts to achieve the Millennium Development Goals (MDGs) in water supply and sanitation by 2015, specifically by piloting innovative measures to scale up the provision of safe water supply free from arsenic and pathogens in rural areas and small towns.”

The outcome target was to provide safe water to at least 1.25 million persons in villages and towns; there was no outcome target related to sanitation.

Restructuring

The Board approved a restructuring of the project on October 6, 2008. The project development objective after restructuring was: “to contribute to Bangladesh's efforts to achieve the Millennium Development Goals in water supply and sanitation by 2015. Specifically, the project will (i) deliver increased access to the water and sanitation services in selected rural villages and small towns, focusing on areas affected by the 2007 floods and cyclones, and (ii) pilot some innovative service provision measures, including private sector participation in rural schemes” (Project Restructuring Paper, page 3).

When the project was restructured, disbursements stood at US$2.30.million (5.6% of the original and 14.5% of the disbursed IDA Grant).

The outcome target was reduced to the provision of safe water to 570,000 people in villages and towns.

The project’s development objective – “to contribute to Bangladesh's efforts to achieve the Millennium Development Goals in water supply and sanitation by 2015” – did not change as a result of the restructuring. The deliverables through which it was expected that the objective would be achieved were changed. This Review does not, therefore, undertake a split evaluation.

If yes, did the Board approve the revised objectives/key associated outcome targets?

Yes

Date of Board Approval:

10/06/2008

c. Components:

There were seven components:A. Piped water supply in villages and growth centers (Appraisal: US$23.1 million; Restructuring: US$4.7 million; Actual US$2.83 million). This component aimed at ensuring sustainable and safe water supply to select rural villages with more than 900 households by outsourcing design, construction, and management of piped water supply systems to private sponsors (entrepreneurs, cooperatives, NGOs etc.), who would meet 50% of the capital costs and would charge users for the water supplied and levy connection fees. The private sponsor would operate the system for a maximum of 15 years under a Service Agreement with the Water User Association or other local government authority. This component would also provide technical assistance and a partial capital cost grant (50%). Grant payments would be determined according to an Output-Based Aid approach, with disbursements made upon certified achievements of project milestones (such as functioning connections). The Government, through the Department of Public Health and Engineering and other institutions would be responsible for regulation and monitoring key aspects of operation such as water quality, technical standards and service levels.At the 2008 restructuring, the financing of this component was reduced to US$4.7 million from the appraisal amount of US$23.1 million. The number of schemes targeted was reduced from 300 to 21, and beneficiary households reduced from 100,000 to 25,000. In addition to this reduction in scope, the following revisions were made: (i) IDA would finance a 70% capital contribution per scheme rather than 50%; (ii) the maximum IDA grant contribution limit was raised from approximately U$69 to US$175 per household; (iii) the payment schedules were simplified and clarified; and (iv) the private sponsor would operate the system for a maximum of 18 rather than 15 years B. Water and sanitation services in medium-size towns (Appraisal: US$14.8 million; Restructuring: US$4.3 million; Actual US$3.60 million). This component was initially conceptualized to provide safe water in 3 to 5 mid-sized urban areas (pourashavas) through establishing ring-fenced Pourashava Water Supply Entities (PWSEs). The PWSEs would own the assets of the municipal water supply system and would outsource the design, construction and management of the piped water supply system to private operators in a contractual arrangement for a 10 year period. While rehabilitation and construction of water systems would be 100% financed through the IDA grant, the Pourashavas, through the PWSEs, would be obliged to repay 50% of the investment grant. This component would also provide technical assistance to finance pre-feasibility studies for urban sanitation. On restructuring, the financing of this component was reduced to US$4.3 million, and the establishment of PWSEs, the contracting of private operators, and investment in major works were cancelled. In addition the urban sanitation pre-feasibility study was dropped. Instead, immediate priority works in the five pourashavas and 19 other flood affected pourashavas would be undertaken. The Water Department in the pourashavaswould be accountable for the operation of the water systems, as well the financial accounts subject to annual audits.C. Non-piped arsenic mitigation activities for villages (Appraisal: US$4.4 million; Restructuring: US$5.61 million; Actual US$9.24 million). This component aimed to provide funding for non-piped supplies to serve villages of under 200 households located in three priority Upazilas Parishad (sub-districts), especially fringe areas of villages where piped water supply will be provided, but where it will not be feasible, for technical reasons to cover all households, and where more than 60% of the households’ tube wells were contaminated with arsenic. It was expected that around 200 villages would be covered in provision of arsenic and pathogen safe water sources by working with Support Organizations, Union Parishad’s and Ward Arsenic Committees, on a demand driven basis using the approach successfully piloted in the then ongoing Bank-supported Bangladesh Arsenic Mitigation Water Supply Project. Benefitting communities would contribute 10% of the investment cost and Ward Arsenic Mitigation Water User Groups would help develop arsenic mitigation action plans, handle contracting and operation and maintenance.On restructuring, the financing of this component increased by 28% from the appraisal amount of US$4.4 million to US$5.6 million, and it was accordingly revised to include, in addition to the original project scope , the construction by the Department of Public Health and Engineering of 5,000 water points at a cost of US$3.61 million, in cyclone-affected areas. This would be 100% financed by the project.D. Capacity building, regulation and training (Appraisal: US$3.2 million; Restructuring: US$1.36 million; Actual US$0.71 million). This component consisted of three sub-components: (i) Capacity Building and Training:

Technical advisory services, capacity building programs, and training workshops for potential stakeholders on issues related to rural piped water supply services; and provision of technical advisory services through experts in conversion of municipal water departments to commercially viable entities;

Capacity building programs for monitoring, planning, implementation and operations of investment.

Design and implementation of client satisfaction surveys in communities.

(ii) Regulatory Frameworks:

Technical advisory services for development/strengthening of regulatory frameworks in (a) involving the private sector in rural and urban piped water supply systems with community participation; and (b) ground water management and administration.

(iii) Water Monitoring:

Strengthening water quality monitoring routines and capacity nationally at the Department of Public Health and Engineering (DPHE), union parishad and pourashava levels through technical advisory services, equipment and logistical assistance.

Institutional capacity building support.

Ground water monitoring.

Development and testing of the application of risk-based approaches to water safety management.

At restructuring, this component was reduced by 58% from the appraisal amount of US$3.2 million to US$1.36 million. The Technical Assistance (TA) to develop the regulatory framework and benchmarking systems for the pourashava private operators, and activities to support the regulatory framework for groundwater management was cancelled. Water quality monitoring of all wells drilled under the Bangladesh Arsenic Mitigation Water Supply Project and this project would be undertaken and the findings entered into a national water quality database. In addition, ground water mapping, including the ongoing riverbank infiltration pilot activities would be completed; and a water safety plan for all schemes constructed under the project would be developed.E. Development of local credit markets and risk management mechanisms for village piped water supply (Appraisal:US$4.9 million; Restructuring: US$0 million; Actual: component cancelled) . This component aimed to develop local credit markets and insurance services for rural piped water supply provision in Bangladesh, initially through local private banks and financial mechanisms (for example, a Take-Out Facility, a Partial Credit Guarantee, and/or related financial mechanisms) interested in lending or through direct loans appraised and administered by the Infrastructure Development Company Ltd. This component was cancelled on restructuring as it had made almost no progress, was unlikely to be completed by closure, and was considered less relevant to the down-sized rural piped water supply component.F. Monitoring and Evaluation (Appraisal: US$1.4 million; Restructuring: US$0.5 million; Actual US$0.12 million). This was initially planned to draw lessons from implementation of water-supply schemes using innovative institutional and investment approahces, monitor compliance by sponsors and communities, develop systems that the Government could use for large-scale piped water programs, develop benchmarking system for piped water supply schemes, and continue evaluation of non-piped arsenic mitigation measures. After restructuring, this component was significantly scaled down, and relied mainly on data provided by private sponsors, proposals, progress, and TA reports. In addition one technical and performance audit was planned.G. Project management (Appraisal: US$3.3 million; Restructuring: US$2.19 million; Actual: US$2.42 million). This component financed the Project Management Unit (PMU) within Department of Public Health Engineering. The budget envelope of this component was reduced by 34% on restructuring. It financed provision of goods, equipment, technical advisory services and logistical assistance required by the PMU for the management of the project.

Project Costs:The cost of the project was initially estimated at US$55.11 million equivalent, on restructuring the projects costs were reduced by 66% to US$ 24.12 million equivalent. At closing project costs were US$18.91 million equivalent (ICR, p.19), or 34% of the original project costs and 78% of the restructured amount. Financing:At appraisal an IDA grant commitment of US$40.0 million was approved. On project restructuring this amount was reduced by almost 47% to US$18.67 million. According to the ICR (p. 19), the actual amount disbursed at project closure was US$15.89 million, 85% of the restructured amount. The balance of US$2.78 million was cancelled. There were no other external financing sources.At appraisal, local communities, sponsors and beneficiaries were expected to contribute US$9.2 million equivalent; on project restructuring this was revised to US$1.97 million equivalent; the actual contribution was US$1.16 million equivalent (59% of the restructured amount).Borrower Contribution:At appraisal the Borrower planned to contribute US$5.83 million; on project restructuring this was revised to US$3.48 million; the actual contribution was US$1.90 million (55% of the restructured amount).DatesTo enable the completion of works and achieve target outcomes, the project closing date was extended at the time of the restructuring by 8 months until December 31, 2010.

3. Relevance of Objectives & Design:

a. Relevance of Objectives:

High

The project’s objective is relevant to the Bank’s Country Assistance Strategy (CAS) for the period Fiscal Years 2011-2014. It responds directly to “Strategic Objective 3: Improve Social Service Delivery,” and “Outcome 3.2: Expanded Access to Safe Water and Sanitation Services,” in both urban and rural areas. It is also relevant to “Strategic Objective 4.3: Increased effectiveness of public service,” focusing on strengthening the institutional framework for government and public service delivery at the local level.

The project’s objective is relevant to government strategy for the water and sanitation sector in Bangladesh, which is focused on the attainment of the Millennium Development Goals (MDG), inter alia through the development of new approaches in the water sector (ICR, page 2).

The objective is relevant to resolving the issue of poor access to water supply and sanitation facilities in the rural areas of Bangladesh. 40% of pourashavas (rural municipalities) did not have piped water and used hand-pumps at the time of appraisal. Daily supply averaged between 3 to 12 hours. Almost 79 million people in Bangladesh were exposed to arsenic from well water with contamination levels significantly exceeding WHO guideline of 10ppb. Microbial contamination was equally serious. Despite some improvement in water supply and sanitation facilities, water related disease incidence is common and accounts for a large number of deaths every year.

b. Relevance of Design:

Modest.

Although the development objective was clear and measurable, the project was ill-designed to meet the objective. It involved both pilot testing of innovative approaches (for example, supporting private sponsors to design, construct and operate rural piped water supply schemes, and the contraction of private operators to design, construct and operate water supply schemes in the pourashavas) and the scaling up of such approaches.

A rigorous M&E framework should have been a cornerstone of such a design. However, despite the inclusion of a separate component for M&E, M&E design was so weak that no structure for monitoring the results of the pilot approaches was in place when implementation started, and was still not in place at the time of restructuring over four years later.

The likely interest of private sponsors in the schemes supported by the project does not appear to have been well researched. The only obstacles to such interest that are identified in project design are lack of credit and of insurance, and these are inadequately addressed. This is despite the fact that “given the large number of schemes, a large number of sponsors will be required” (PAD, page 53).

With regard to insurance, sponsors had indicated the need for support that would insure them against natural events like floods or tornadoes that could damage the infrastructure facilities, or against the appearance of arsenic or other contaminants despite detailed engineering analyses conducted during each project preparation” (PAD, page 7). The PAD acknowledges that there is no insurance industry in Bangladesh that would provide this kind of cover to private sponsors, and states that “this component would provide protection to private sponsors against well-defined conditions identified in the sub-project grant agreement" (PAD, page 37). No further details are provided.

Concerning credit, the intention was to develop a mechanism to interest local banks in long term maturity loans to sponsors or provide such loans through a special entity such as the Infrastructure Development Company. However, no such instrument was developed. As the ICR acknowledges, “preparatory work for this component appears limited. The Mid-Term Review Mission pointed to the fact that private sponsors could not come up with the required capital as one of the bottlenecks leading to restructuring” (ICR, page 9). The credit and insurance activities were cancelled at restructuring.

There was also an insufficient underpinning for the approach of using private operators to design, build and manage municipal water supplies in the pourashavas. The previous Bank-supported Bangladesh Arsenic Mitigation Water Supply Project “had done nothing in pourashavas, due to some early failures, and preparatory work for this component in [the project under review] was deferred to project implementation. The time required to do this preparation work (including changes in local government legislation, building political consensus in each identified pourashava, the selection and mobilization of the private operators) was significantly underestimated” (ICR, page 10).

The objectives specify safe water supply. However, the activities included in the design to monitor and regulate water quality and safety consisted of capacity-building and training and the development of water safety plans. These were insufficient to ensure the monitoring of water safety and the adoption of remedial measures where necessary.

The objectives (the MDGs) specify sustainability. Yet there were few project outputs to support technical and financial sustainability of the facilities constructed.

Equally, the objectives specify not only water supply, but also sanitation. There are practically no activities built into the project design to increase access to improved sanitary facilities.

There is no indication in the ICR that design took account of exogenous factors that might affect the project’s outcome.

At the October 2008 restructuring, some of the shortcomings in design were addressed. 75% of the Credit was canceled, and outputs were simplified as well as being reduced in number. 85 of the rural piped water schemes were canceled and 21 retained for which construction had already started. The scope and extent of innovative measures was also reduced. However, several shortcomings remained, including those related to the inadequacy of activities to ensure safe water supply, sustainability and access to improved sanitary facilities, all of which continued to be specified in the objectives.

4. Achievement of Objectives (Efficacy) :

The achievement of the project’s development objective -- to contribute to Bangladesh’s efforts to achieve the Millennium Development Goals (MDGs) in water supply and sanitation by 2015 – is rated substantial.Outputs

21 rural piped water schemes had been started by project closure. The original target was 300; the revised target after restructuring was 21. Each scheme included 2 production tube wells, one pump house, a pump and pumping machinery, storage tank, piped distribution network, and consumer connections. According to the ICR (page 20), none of the schemes was completed at project closure: six are described as “fundamentally completed," meaning that they were in operation but the consumer connections needed to be completed; twelve were “80-95% completed,” with seven of them awaiting electricity connection; and three were “65-75% completed."

Sixteen NGOs and 3 private enterprises served as sponsors for the 21 rural piped water schemes constructed. The ICR (page 21) reports that “all these sponsors are committed to cost recovery. However, they all have a rather relaxed attitude toward recovering their capital investment; they are as much concerned with providing a service to the communities. This is even true for the 3 private enterprises, all of which manage schemes in their owners’ home communities.”

The ICR (page 25) reports that, in the pourashavas, and according to the Department of Public Health and Engineering, 9,156 additional domestic and public standpost connections serving 21,974 households were delivered against a (revised) target of 20,000 households. No information on the actual percentage of physical work completed in each pourashava schemes is provided in the ICR or the Borrower’s ICR.

A total of 13,159 deep tube wells were constructed for the provision of non-piped water, against a (revised) target of 7,000. Of these 3,500 tube wells were drilled in heavily arsenic affected areas (target 2,000), and 9,659 shallow and deep tube wells were constructed in cyclone affected areas (target 5,000). A consultant survey at the end of the project found that all tube wells were working and providing water of “acceptable” quality. The rural non-piped schemes targeted 167,320 households in arsenic, and cyclone affected areas; however, in view of the weakness of the M&E framework, there is uncertainty about the number of outputs and their degree of completion at project closure.

The benchmarking and monitoring framework and the water safety plans for water quality control were not implemented.Work on the water quality regulatory framework was stopped at the restructuring. Water quality plans were supposed to be completed for each scheme, but this target was not achieved. The project did undertake water quality assessments of 81% of the wells constructed, but only 58% were entered in the national database.

Outcomes

Nowhere in either the PAD or the ICR are Bangladesh’s Millennium Development Goals (MDGs) in water and sanitation described. There is no evidence presented in the ICR showing the progress made towards attaining them, nor analysis indicating the extent to which progress made could be attributed to the project. According to the UNDP’s “The Millennium Development Goals: Bangladesh Progress Report, 2011,” the MDG for water and sanitation is “to halve, by 2015, the proportion of people without sustainable access to safe drinking water and basic sanitation.”

However, in its comments on a draft version of this Review, the Department of Public Health Engineering (DPHE) provided the following additional information: "As per JMP [an independent consultancy] estimate (JMP Report, 2012), people’s access to improved (safe drinking) water at national level was achieved 81% in Bangladesh by 2010 of which 6% achieved in piped water and 75% in non-piped water. Since 1995, 20% of the population gained access to safe drinking water. The estimate shows that population of Bangladesh was 148.69 Million in 2010 and accordingly 29.74 Million new people gained access to safe drinking water since 1995. [The project under review] implemented water schemes mostly during 2009-10 and provided safe water to about 996,514 population, which contributed to 3.35% increase in safe water coverage of the country and hence to MDG achievement." According to the DPHE figures, therefore, some 17% of the increase in improved water coverage since 1995 can be attributed to the project.

The DPHE statistics are somewhat different from, though not wholly inconsistent with, those of the UNDP Progress Report cited above, according to which the percentage of Bangladesh's population using an “improved drinking water source” rose from 78% to 86% between 1999 and 2009 (Annexure-1, page 99). The 2015 target is 100%.

The ICR states (page 13) that the project provided access to improved water sources for 996,514 people, against a (revised) target of 575,000. By far the largest increase was estimated in access to non-piped water – 836,642 people (661,642 in cyclone-affected, and 175,000 in arsenic-affected areas) as against a target of 350,000. The rural piped water schemes are estimated to have provided access to around 50,000 people by project closure against a target of 125,000, a 60% shortfall. According to subsequent information provided by the DPHE, dating from September-November 2012, “61,433 people [compared to about 50,000 at closure according to the ICR] were being served through [the] 21 [piped-water] schemes." The pourashava schemes are thought to have benefited 109,000 people against a target of 100,000.

There is no conclusive evidence presented in the ICR that the drinking water provided under the project is safe (though it may reasonably be assumed to be safer than that available prior to the project). According to subsequent information provided by DPHE, groundwater in Bangladesh (which is plentiful and available at shallow depths) is of much better microbiological quality than surface water. By the early 1990s, nearly 98% of the population had access to what was presumed to be relatively safe groundwater. The discovery of arsenic in groundwater in 1993 reduced this estimate to 74%. The project under review was one of several initiatives taken by the Government and its international partners to start systematic arsenic screening and mitigation programs. DPHE reports that the project's contribution was significant.

The DPHE also states that, "during implementation of the rural piped water supply schemes, a test tube well was installed for each scheme and water sample was tested for required parameters in [project-selected] laboratories. Based on the test results of the water quality, PMU allowed the sponsors to implement the schemes with or without water treatment facilities. Before going for official commissioning of the schemes, water quality tests were repeated in the laboratories.” For non-piped water options, "commissioning and payment of the contractors was dependent on the successful demonstration of water quality at the point source."

However, as noted above, the benchmarking and monitoring framework, and the water safety plans for quality control, were not implemented. Consequently, as DPHE acknowledges, “after the project was closed, there was no system or mechanism in the project to follow up the activities of the completed project. So routine monitoring of water quality of the schemes could not be continued any more, [although] DPHE local offices were instructed by the office of the Chief Engineer to continue monitoring ofthe scheme including water qualities of all the water supply options and provide technical assistance as and when required.”

The ICR (page 11) reports that the final supervision mission visited a random sample of rural piped and non-piped supplies, and found “no problems with construction impact, water quality, or health.” According to the Beneficiary Survey, conducted just prior to project closure, consumers served through non-piped supplies were satisfied with the water quality, access, and location of the hand-pumped sources, though pourashava and some piped water beneficiaries complained about water quality. The ICR states (page 37) that the Survey was conducted by a consultant whose field work “included discussions with beneficiaries and other stakeholders in the rural pipe, pourashava, and rural non-piped components.” The ICR expresses reservations about the Survey’s methodology (see Section 10 below).

There is no evidence of increased provision of improved sanitary facilities (specified in the objectives), since there were no related components or activities.

The MDGs specify sustainable access to safe drinking water and basic sanitation. Several schemes had not started operating by project closure, and most others had been in operation only a short while. At closure, therefore, evidence was still lacking about the willingness and ability of consumers to pay the necessary tariffs to cover the full cost of operation of the piped-water schemes, including interest and other financial charges, as well as about the capacity of the sponsors to operate the schemes in a sustainable manner.

According to the DPHE, "[Of the 21 piped water schemes], 9 schemes were operating at profit and 8 schemes were operating at break-even and the remaining 4 schemes needed some technical assistance to make these schemes commercially sustainable…..The monitoring findings also revealed that [non-revenue water] was [reduced] in 15 schemes (26.8%) and collection efficiency was 81.5%, which were quite satisfactory in consideration of innovative pilot nature of the schemes. It is evident from the above findings that people are willing to pay tariff for the piped water supply, if quality service is ensured.”

5. Efficiency:

Modest

Economic rates of return were estimated ex-ante and ex-post. The ex-ante cost benefit analysis covered three components (representing 77% of estimated project costs) -- rural piped water, the pourshava piped water supply, and non- piped improvements in rural water supply. Benefits of rural and pourshava piped water supply schemes consisted of the economic value of the water supplied plus estimated reductions in health costs. Benefits of the rural non-piped water supply were the potential time savings from reduced fetching and carrying, and estimated health cost reductions. Depending on assumptions about benefits, the economic rates of return (ERRs) ranged from 20% to 45% for the rural piped water supply, 16% to 34% for the pourshavas, and from 20% to 75% for the rural non-piped water supply. The overall, weighted average ERR was 30.4%.

The ex-post economic analysis used essentially the same methodology with updated information. The ERRs were estimated at 14.2% for the rural piped water supply component (including health benefits), 14.6% for the pourshavas (based on willingness to pay a flat rate tariff of US$ 2.3 per connection per month), and 61.3% for the rural non-piped water supply. Together, these accounted for 83% of actual project costs. The overall ERR was 42.1%. This was higher than the appraisal analysis because of the much greater weight of the non-piped water schemes (which, because of their low cost, had a higher rate of return) than at appraisal.

During the 33 months between effectiveness and restructuring, only 5.6% of the grant amount was disbursed, indicating low administrative efficiency. Even after restructuring, at which the scope of the project was reduced by 66%, 15% of the greatly diminished credit amount remained undisbursed at closure. There were significant operational inefficiencies. For example, the planning process for the rural piped water schemes took considerably longer to complete than anticipated. The number of months required to go from a Memorandum of Understanding with the sponsor to the completion of an approved design study ranged from 10 to 39 months, with a median of 20 months. From there to the grant agreement required a further 17 months on average. Thus, in most cases more than three years passed between the signature of the MOU and the breaking of ground. According to the ICR (page 21), the reasons for the long time lag included: weak performance on the part of the Project Management Unit; sponsors’ difficulty in making capital contribution, and cash flow problems; difficulties in procuring land; the need for additional test drilling; and price escalation. Inefficiencies also stemmed from the lack of an effective M&E framework throughout implementation.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:

Rate Available?

Point Value

Coverage/Scope*

Appraisal:

Yes

30.4%

77%

ICR estimate:

Yes

42.1%

83%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Although the project’s objective is highly relevant to Bangladesh’s needs in the water and sanitation sector, design relevance was modest. The degree of achievement of the development objective is rated substantial, based to a large extent on evidence subsequently provided by Bangladesh's Department of Public Health Engineering, especially concerning progress towards the MDG, safety of water supplies, sustainability, and greater confidence concerning the veracity of the recorded achievement of supplying close to a million people with access to improved water. Efficiency is rated modest. Overall project outcome is assessed as moderately satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

Risk to development outcome is high since:

Nearly three-quarters of the beneficiaries live in cyclone-prone areas. There is a high risk that future natural disasters will destroy many of the hand-pumps, despite the fact that project supported procedures were intended to improve responses to such events.

There is a significant risk that the private sponsor approach will not prove to be robust. Although there is now some evidence of the willingness and ability of consumers to pay the necessary tariffs, the capacity of the sponsors to operate the schemes sustainably was still in doubt in 25% of the cases at the end of 2012.

There is a significant risk that the considerable support services, which experience in other countries has shown to be necessary for privately managed rural water schemes, and which would assist not only operators, but also local government institutions and community organizations to perform their roles, may remain insufficient in Bangladesh.

a. Risk to Development Outcome Rating: High

8. Assessment of Bank Performance:

a. Quality at entry:

The project was designed in a way that would enable it to make an important contribution to resolving arsenic contamination of water in rural Bangladesh. Nevertheless, there were major shortcomings in Quality at Entry:

Design was overly ambitious and complex, with seven components, some of them difficult and innovative, in a single project that was intended to be a pilot exercise.

The difficulty of rapidly scaling up rural piped water schemes, when the pilot schemes were still experiencing delays, was underestimated.

The timetable for the pourashava private operator component was unrealistic, given the complexity of the prior actions and reforms that would be required (appointing the technical assistance consultancy to help with the formulation and presentation of the proposed reforms; building political consensus in each identified pourashava;establishing the Pourashava Water Supply Entities; selecting and mobilizing the private operators; and having the private operators design, build and manage the infrastructure).

The need for new legislation to be prepared and approved – for example, to enable the creation of Pourashava Water Supply Entities -- was not anticipated.

Procurement procedures were unusually demanding (the original project comprised more than 170 activities, some of which were themselves made up of multiple contracts. The rural piped water schemes component, for example, targeted 300 discrete contracts, each with multiple procurement stages).

Despite these onerous requirements, insufficient provision was made for strengthening the Project Management Unit, which had already demonstrated capacity weaknesses in two previous projects.

Despite the critical importance of a robust M&E framework, for a project focusing on learning from pilot schemes, no such framework was developed during preparation.

Despite what the PAD (page 75) describes as wide consultations with stakeholders, there was a lack of preparation, and even in some cases conceptualization, of the supporting institutional, technical and financial infrastructure that would be required, not only by private operators, but also local government, communities and NGOs.

Quality-at-Entry Rating: Unsatisfactory

b. Quality of supervision:

Supervision missions visited the project regularly and fairly frequently (15 Implementation Status Reports were filed during the implementation period of six-and-a-half years). Further support was provided by staff based in the Country Office (ICR, page 16).

The supervision team recognized within a year of effectiveness (February, 2005) that implementation was off-track and signaled this through down-grading the ratings in the supervision reports. By 2006, another ongoing operation, the Bangladesh Arsenic Mitigation Water Supply Project, following an analogous approach, had failed to construct and put into operation most of its pilot piped water schemes, raising the strong probability that the project under review would run into similar problems. From May 2006, until the restructuring about two-and-a-half years later, both progress towards attainment of development objectives and implementation progress were rated unsatisfactory or moderately unsatisfactory.

The missions regularly set clear milestones in an effort to focus on implementation efforts. Country and Sector Management met with the Secretary of Local Government Departments in an effort to address Project Management Unit weaknesses and other implementation issues.

At the October 2008 restructuring, some of the shortcomings in design were addressed. 75% of the Credit was canceled, and outputs were simplified as well as being reduced in number. 85 of the rural piped water schemes were canceled and 21 retained for which construction had already started. The scope and extent of innovative measures was also reduced. After restructuring, the pace of implementation accelerated considerably.

However, the ICR (page 16) reports that the decision to restructure the project was unnecessarily delayed by the project team. First, the decision was postponed until the mid-term review, which was planned for March 30, 2007.Then a change in Task Team Leader meant that the review did not take place until November 12, 2007. Restructuring finally took place almost a year later.

The weaknesses in M&E design were not addressed at the mid-term review, nor at any other time during implementation. This adversely affected the ability to assess project outputs and outcomes, and the views of beneficiaries.

Supervision did not establish reliable means of verifying if the water supplied through the rural piped water supply schemes was safe, nor were there water quality checks on all wells.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

There is limited information in the ICR on Borrower performance, and no separate analysis of the government’s and implementing agency’s performance. Footnote 13 on page17 states that “the ICR team was not in the position to unravel how the performance of the Department of Public Health Engineering (DPHE) was affected by factors in the Ministry and Government as a whole.” (The Department is under the Water Supply Wing of the Local Government Division, in the Ministry of Local Government, Rural Development, and Cooperatives. It was the government agency with principal responsibility for the project).

The ICR (page 10) reports lack of government commitment to the project. This was reflected by the fact that capable management and staffing were still not in place in the Project Management Unit (the Unit was housed in the DPHE) 18 months after project approval. The situation worsened when an individual with a very poor track record under the Bank-supported Bangladesh Arsenic Mitigation Water Supply Project was appointed to the position of Unit Director by the Government contrary to the Bank’s advice.

The project team subsequently informed IEG that “the Government was also not forthcoming in resolving many issues like provision of publicly-owned land for the construction of rural piped water supply schemes.”

The project team reported that the Government was very proactive in providing authorization and proposals to use project funds for post-disaster rehabilitation works in the wake of cyclone disasters.

Government Performance Rating: Unsatisfactory

b. Implementing Agency Performance:

The project was implemented by a Project Management Unit (PMU) housed in the Department of Public Health and Engineering.

Prior to restructuring, the performance of the PMU was characterized by delays, exacerbated by management indecision, staffing problems (in particular, financial management, procurement, and environmental specialists were not recruited for the PMU in a timely fashion), and the lack of an M&E system. The mid-term review noted: “The PMU continues to demonstrate an inability [to move towards] implementation at a reasonable rate” (by the time of the mid-term review, disbursements stood at 5.2%, with about 30 months to run until the project was due to close, and 33 months after it became effective.

After restructuring, the PMU management changed, and implementation, according to the ICR (page 11), improved considerably (the project team describes the new PMU Director as a dynamic individual who "was able to accomplish and enormous amount in the remaining two years of the project"). However there was still no M&E system in place at project closure. Also, water safety plans for rural piped water schemes were not prepared and water quality monitoring for all water point sources were not undertaken.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The project’s concept was based on the principle of learning and replication from pilot schemes. Recognizing M&E’s central importance for such an operation, it was given a dedicated component. However, preparation of the M&E framework was still incomplete at the time of Board approval. Some guidelines for preparing indicators, benchmarking systems to monitor performance of utilities, outputs and impacts of piped and non-piped schemes were provided in the PAD for Components 1, 2 and 3; however, many details, and the indicators themselves, were missing. The M&E function was to be carried out by a consulting firm under the supervision of a specialized sub-unit within the Project Management Unit. This firm was meant to develop the M&E monitoring framework, and assist in monitoring process, output and outcomes for each component, but did not in fact do so.

b. M&E Implementation:

The ICR reports (p.11) that, at the time of restructuring, the M&E framework for the project had still not been developed. The plans for the framework were, therefore, scaled back significantly, given the “the reality of the project,” and the very slow progress in implementing activities thus far.

After restructuring, a revised monitoring framework was planned, which would rely on data extracted from the proposals and progress reports of private sponsors and related technical assistance consultancies as a primary source for data and for a technical performance audit. No details or elaboration on the methodology or structure of the M&E system, or of reporting procedures, were provided in the Project Restructuring Paper. Staffing allocation to M&E related tasks, and of the allocation of institutional responsibility, in the wake of the consultancy firm’s removal, remained unclear.

A performance audit was undertaken just prior to closure, which included beneficiary and stakeholder surveys. According to the ICR (Annex 5, page 38) the audit produced a “positive but ambiguous picture of the post project situation...the methodology was generally weak and no firm conclusions can be drawn about the views of the ultimate beneficiaries of the project i.e. the water consumer.”

The ICR (page 11) points out that key design features of the project such as the effectiveness of Output-Based Aid, the business model based on single schemes rather than clusters, and water quality regulation, were never evaluated. This is a major shortcoming, given that the project was meant to test these innovations.

The ICR also confirms that “the beneficiary estimates are based on quite a number of assumption, rather than solid data.”

According to information subsequently provided by the Department of Public Health Engineering (DPHE), the lack of a functioning project M&E framework was to some extent compensated for by the use of progress reports from sponsors, consultants' technical reports, and contractors' completion reports. In addition, DPHE had its own M&E system at both central and field levels, which was applied to the project. DPHE reports that this, together with the completion report prepared by the Project Management Unit (PMU), enabled a reasonably accurate verification of the population served by non-piped water schemes. Monitoring of the piped water schemes was based on sponsors' progress reports to the PMU.

a. M&E Utilization:

The ICR contains no information on utilization of M&E data generated by the project.

According to the PAD (page 75), the Borrower had carried out an Environmental Assessment. The Environmental Management Framework, prepared prior to Appraisal, was to guide the selection, design, implementation, and operation of the sub-projects. It was planned that regular supervision and environmental audits would identify residual issues and recommend mitigation, if needed. After restructuring, the project still remained a Category “B” operation, and followed the same protocol as proposed in the original design.

The ICR provides little information on the assessment and monitoring of environmental safeguards compliance during implementation but notes (page12) that the Project Management Unit did not appoint an environmental specialist in a timely manner, and that no water safety plans were produced for the rural piped schemes.

In particular, there is no statement as to whether there was satisfactory implementation of the Environmental Management Framework for each sub-project with regard to environmental screening of the sub-project proposal, or environmental safeguard compliance for monitoring and implementation. It is equally unknown whether designated staff (Environment Coordinator, Environmental Officer, and consultants) were deployed during project implementation. There is no discussion of the independent environmental monitoring and audits mentioned in the PAD (page 75).

However, the final supervision mission visited a random sample of rural piped and non-piped schemes and found “no problems with construction impact, water quality or health” (ICR, page 12).

On the basis of the information provided, it is not possible to come to a conclusion as to whether OP 4.01 was complied with or not.

b. Fiduciary Compliance:

Procurement:

A procurement assessment was carried out during project preparation with the recommendation to place dedicated staff in the Project Management Unit (PMU). At appraisal, the overall procurement risk for the project was rated “High” due to inherent capacity limitations at the Department of Health and Engineering (DPHE) and because implementation would be highly decentralized.

Project design, with its seven components, translated into a major procurement burden for the PMU prior to restructuring. After restructuring, procurement arrangements and thresholds remained unchanged, but procurement packages were revised downwards in line with the reduced project scope.

The ICR reports (page 12) that the procurement specialist was not recruited by the PMU in a timely manner. The last Implementation Status Report of November 25, 2010, noted that all contracts had been awarded, and that there were no major issues to report from the post-procurement audit.Financial Management:

The Financial Management risk was rated “High” at appraisal due to: (i) the large number of entities participating in a project with limited financial management capacity; (ii) the weak capacity of the PMU; and (iii) handling of project funds by the community.

The ICR (p.12) mentions some procedural problems encountered during project implementation, such as not using computerized accounting systems during a certain time, but does not discuss any other shortcomings.

Almost 18 months after project approval, no Financial Management Specialist had been appointed to the PMU which led the supervision team to downgrade financial management performance to moderately unsatisfactory.

However, according to the last Implementation Status Report (ISR) of November 25, 2010, matters had improved to the point where the project had good budgetary control, and a qualitative enhancement in the presentation and submission of financial management reporting. Nonetheless, there was continued slippage on the updating of the computerized accounting system. The financial management rating was assessed as moderately satisfactory at project closure.

The ICR does not address training and capacity building of financial management staff, and does not discuss external audits (the Borrower’s ICR mentions on page 58 of the ICR that “[the] Foreign Aided Project Audit Department of [the Government of Bangladesh] accordingly audits the transaction[s] of the project of every financial year”).

c. Unintended Impacts (positive or negative):

None reported.

d. Other:

12. Ratings:

ICR

IEG Review

Reason for Disagreement/Comments

Outcome:

Moderately Satisfactory

Moderately Satisfactory

Risk to Development Outcome:

High

High

Bank Performance:

Moderately Satisfactory

Moderately Satisfactory

Borrower Performance:

Moderately Unsatisfactory

Moderately Satisfactory

After restructuring, the PMU management changed, and implementation, according to the ICR (page 11), improved considerably. This was subsequently confirmed by the project team, who described the new PMU Director as a dynamic individual who "was able to accomplish and enormous amount in the remaining two years of the project". During these two years, 87.5% of Credit disbursements took place.

Quality of ICR:

Satisfactory

NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons: The ICR derives the following lessons from the project's experience:

A complex project design that does not adequately respond to country and implementing agency experience, capacity level and ground reality can lead to major implementation problems – Project implementation was challenged by the complexity of the project, and the need to work with parallel local government institutions for rural and urban areas, compounded by weak PMU capacity. This could have been circumvented if project design had been kept simple and the number of institutional partners kept to a minimum.

If the procurement burden on the implementing agency is not adequately evaluated during project preparation, especially for projects piloting innovation, severe implementation bottlenecks are likely to materialize – At the time of restructuring, the project had a significant number (almost 170) of procurement packages pending, which involved undertaking multiple steps for processing, increasing the burden on the weak implementing agency.

The success of innovative components depends on paying due attention at the design stage to the pre-conditions for success and appropriate sequencing as well as to the capacity of the implementing agencies – Both the rural piped water scheme and pourashava components experienced problems because certain pre-requisites were not in place. Better sequencing of various activities could have avoided some of the implementation problems. Attempting to test the private operator approach in both rural piped and pourashava schemes was especially taxing.

IEG adds the following lesson:

Monitoring and Evaluation (M&E) are of particular importance in assessing projects involving innovation and learning from pilots, which require a robust M&E system designed during project preparation to include an appropriate results framework, methodology, institutional arrangements and indicators. In the case of this project, M&E design was incomplete and many critical features were left to the implementation phase, during which they were not addressed.

14. Assessment Recommended?

No

15. Comments on Quality of ICR:

The ICR’s most positive quality is the candor with which it addresses the weaknesses in project design and in Bank and Borrower performance during implementation. There are, however, a number of shortcomings:

The analysis of the achievement of objectives is output-oriented and the Millennium Development Goals for the water and sanitation sectors (the core of the development objective) are not discussed.

The discussion of Borrower performance is sparse and there is no separate consideration of government and implementing agency performance as the guidelines stipulate.

Although there is coverage of environmental safeguards, there is no clear statement as to whether OP 4.01 was complied with or not.

Similarly, there is no clear statement of compliance with Bank fiduciary policies, nor discussion of external financial audits.