Development objective in the Project Appraisal Document(PAD, page 5):"to improve the access, quality and use of telecommunications and ICT services to achieve socioeconomic developmentin the Organization of Eastern Caribbean States (OECS)."

Development objective in the Loan Agreements (for each participating country):"to improve public access to and the quality and use of telecommunications and information and communications technology (ICT) services, in support of socio-economic developmentamong the Participating Countries."

This review assesses the objective as stated in the Loan Agreements, since it is more monitorable.

1. Support for Legal and Regulatory Reforms (US$0.840 million at appraisal, US$0.840 million at closure)
This component was to strengthen the national and regional regulatory frameworks and promote additional competition in the telecommunications sector through the following outputs:

Updating the telecommunications act and tariff regulations of the participating countries in accordance with the new market and technology conditions.

Establishing a regional forum for consensus building and alternative dispute resolution. The forum would facilitate resolution of issues that have a commercial and public policy dimension and seek consensus on commercial issues among industry players.

Assisting the Eastern Caribbean Telecommunications Authority and National Telecommunication Regulatory Commissions in developing a modern cost methodology to determine interconnection and related consumer tariffs.

Providing a comprehensive review of telecommunications legislation, with a focus on interconnection and spectrum management, and implement numbering and dispute resolution procedures.

Providing a comprehensive regional program of related training to Eastern Caribbean Telecommunications Authority and National Telecommunication Regulatory Commission staff and government officials of the participating countries.

2. Universal Access (US$1.270 million at appraisal, US$1.270 million at closure)
This component was to review current universal access policy, create related guidelines and provide support to establish a Universal Service Fund. The component would produce the following outputs:

Assistance to Eastern Caribbean Telecommunications Authority member states in preparing/updating a comprehensive paper to define Universal Service Guidelines for improving access to broadband connectivity;

Establish the Universal Service Fund and provide a one-time subsidy to service providers.

3. Information and Communication Technologies (US$1.030 million at appraisal, US$1.030 million at closure)
The objective of this component was to improve growth and competitiveness in Information and Communication Technology enabled services through utilization of broadband infrastructure through the following outputs:

Extend the use of broadband applications by government and the private sector by (i) identifying appropriate policies and regulations; (ii) elaborating and recommending a common information systems architecture and standards for the member states and (iii) designing and implementing up to five pilot projects to encourage regional use of broadband technologies.

Establish an alternative telecommunications system to transfer data from the Radio Spectrum Monitoring System to Eastern Caribbean Telecommunications Authority and the five National Telecommunication Regulatory Commissions.

4. Project Management (US$0.245 million at appraisal, US$0.245 million at closure).

Project Cost: The actual project cost at closure was US$3.38 million, close to the appraisal estimate of US$ 3.39 million.

Financing: Of the total project cost, US$1.35 million was financed from the IDA credit, and US$1.36 million from the IBRD loan. Both amounts were the same as the appraisal estimates.

Borrower Cost: The Borrower contribution at appraisal and at closing was US$ 0.68 million (ICR, page 31).

Dates: The original closing date (December 31, 2009) was extended twice for a total of two years: the first extension was until December 31, 2010 due to delays in implementing the Universal Service and ICT pilot projects, and reallocation of loan and credit amounts to increase the consultancy category; and the second extension was until December 31, 2011 due to delays in implementing the Universal Service projects. The project closed on December 31, 2011.

3. Relevance of Objectives & Design:

a. Relevance of Objectives:

High

The objectives were relevant to country conditions, characterized by a lower teledensity than other upper middle income countries, higher unit charges than competitors, and relatively low internet access.

The objectives were relevant to the World Bank strategy for the OECS countries:According to the 2005 World Bank report on OECS - Towards a New Agenda for Growth,"... telecommunication services are critical to the development of service exports and in improving competitiveness in the OECS. Despite recent advances in this area, with the establishment of the Eastern Caribbean Telecommunications Authority and the successful dismantling of the sub-regional monopoly in some key areas, the OECS will have to make significant advances in lowering costs and expanding access. This issue is a matter of priority because of the broad impact that access to the internet can have on firms, households and the government."

The objectives were relevant to Borrower strategy: According to the Regional Partnership Strategy of OECS (2010-2014), "OECS Heads of Government....at the 50th meeting in Anguilla in November 2009, approved establishment of an ICT Ministerial Council to guide policy making in ICT - a major area for growth, diversification, regional integration and overall competitiveness.". According to the June 2001 Country Assistance Strategy (FY02-06), ".. the passage of coordinated, consistent telecommunications laws in five OECS countries and the steps taken to establish a single regulatory agency are already having a positive impact on telecommunications costs". In 1998, 5 country members of OECS (Dominica, Grenada, St. Kitts and Nevis, St. Lucia, and St. Vincent and the Grenadines) embarked on a liberalization process that passed new telecommunications acts, terminated monopoly rights, and established the first regional telecommunications regulatory authority in the world called the Eastern Caribbean Telecommunications Authority (ECTEL).

b. Relevance of Design:

Substantial.
The causal chain leading to the achievement of the development objectives of improved access, use and quality of Information and Communications Technology services is clear and convincing. For example, activities financed under the Information and Communication Technologies component could demonstrate the effective use of broadband applications which, in turn, could lead to improved access,qualityand use of Information and Communications Technology services.

The casual chain leading from the project activities to the achievement of improved access, use and quality of telecommunications servicesis also clear. For example,competition in the telecommunications sector can be resisted by the former monopolist, making network interconnection for new entrants difficult, expensive, slow, or inadequate. To address these problems, design included work on interconnection regulations, cost methodology for interconnection and consumer rates, and dispute resolution procedures (ICR, page 31). Similarly, Component 2, on universal access, addresses the issue that some groups may not have their needs met by the market, even in a generally competitive environment. In these cases universal access initiatives can help provide access beyond what would be achieved by the market, thereby extending the benefits of reform.

4. Achievement of Objectives (Efficacy) :

The development objective -- to improve public access to and the quality and use of telecommunications and information and communications technology (ICT) services, in support of socio-economic development among the Participating Countries -- is assessed below under the headings of its three sub-objectives (access, quality, and use).

Development Objective 1:Improved access to telecommunications and Information and Communication Technology services in support of socioeconomic development: Substantial

Outputs:

Universal Service guidelines and policy was prepared and implemented on October 15th, 2011,but with a delay of close to 2 years enabling the five Eastern Caribbean Telecommunications Authority member states to establish their Universal Service Funds (ICR page 33).

A procedures manual was prepared to assist Eastern Caribbean Telecommunications Authority and National Telecommunication Regulatory Commissions to implement Universal Service in Telecommunications, and on-site support was provided to assist Eastern Caribbean Telecommunications Authority and National Telecommunication Regulatory Commissions to implement Universal Service Fund component (ICR page 33).

The second Universal Service project in Dominica was successful in establishing broadband internet facilities in three under-served areas (Belles, Concorde and Sinekou). In addition, funding was provided to: (i) procure equipment and software to help National Telecommunication Regulatory Commission strengthen its capacity for Universal Service Fund (ii) procure equipment and software to implement programs for Dominica Association of Persons and Disabilities and (iii) provide computer skills training to youth population in six training centers.

Internet and broadband services were established in ten targeted rural and under served communities in Grenada.

A high speed wireless network for communication, collaboration, e-learning and research was established in eight high schools in St. Kitts and Nevis.

Information and Communication Technology equipment and software were procured for persons with special needs in five institutions in St. Lucia.

The activity to improve the Very High Frequency coverage for maritime areas in St. Vincent and the Grenadines was not carried out.

An update on Telecommunications Acts and Regulations of Eastern Caribbean Telecommunications Authority member states was drafted on October, 2010.

A procedures and training manual on Alternative Dispute Resolution in the telecommunications sector of the Eastern Caribbean Telecommunications Authority member states was prepared.

Technical assistance was provided by a consultancy on the development of Cost Models for Fixed and Mobile interconnection services.

A total of twenty two workshop and training courses were organized under the Regional Collaboration, Training and Resource and Information Sharing sub-component.

International Rates per minute to US (US$) remained unchanged (Baseline: 0.47; Target: 0.23; Actual: 0.46).

Additional information from the project team indicates that ICT use for St. Lucia increased from 5.48 (2010) to 5.72 (2011) and for St. Vincent and the Grenadines increased from 2.15 (2007) to 6.09 (2011). However, there is no information on ICT use for the remaining three member countries.

Additional information from the project team shows that mobile cellular subscriptions (per 100 people) increased from 69.4 in 2005 to 123.4 in 2010. While such penetration is likely to have been influenced by a wide range of factors as well as by the project, taken together with the increased access to the Internet and the enhanced affordability resulting from lower interconnection and internet tariffs, the efficacy of this sub-objective is rated substantial.

Development Objective 2:Improved quality of telecommunications and Information and Communication Technology services in support of socioeconomic development: SubstantialOutputs

The outputs for this objective are the same as those for objective 1.

Outcomes:

Usage of Broadband Services (subscribers per 100 people) increased from 2.88 percent in 2008 to 13.77 percent in 2011, slightly below the target of 15 percent. Broadband performance (in terms of down loading speed) more than doubled for Grenada, Dominica and St. Lucia, and increased by more than 50 percent for St. Kitts & Nevis and St. Vincent & Grenadines (see Table below). Taken together, this evidence indicates improved quality of services.

In the light of the evidence of improved quality of performance for broadband services, and related internet enhancements, which are applicable to telecommunications as well as to ICT more generally, the efficacy rating for this sub-objective is substantial.

However, the evidence related to improved quality of service could have been strengthened by conducting surveys and collecting information from community access points and schools in which the Universal Service projects have been implemented.

Development Objective 3:Improved use of telecommunications and Information and Communication Technology services in support of socioeconomic development: Modest

Outputs:

A model National Information and Communication Technology plan was developed and submitted to Eastern Caribbean Telecommunications Authority member states for adoption (ICR page 33).

A portal to facilitate interactive live streaming of classes and archived video of classes on the internet was made available to all schools in Dominica.

Lego Mindstroms Robotics Invention System which involved computers, robotic kits and related accessories enabled students of five participating secondary schools in Grenada to design and construct robots, were procured.

Establishment of a community technology center at Dieppe Bay village in St. Kitts and Nevis.

Establishment of a community based resource center to provide internet access for public use in the town of Soufriere in St. Lucia.

A Digital Classroom Live Portal which included 20 lessons was made available to all schools in St. Vincent.

According to the ICR (page 37), a recommendation was made in the consultant's final report on a voice/data communication solution to link Eastern Caribbean Telecommunications Authority and the National Telecommunication Regulatory Commissions in order to replace the existing data communication system linking these entities.

Outcomes:

The percentage of individuals using internet almost doubled for St. Lucia and St. Kitts & Nevis, increased between 2005 and 2011 by 10 percent for Dominica and Grenada, and by 4 times for St. Vincent & Grenadines (see Table below).

* 2011 data comes from 2010 report since latest available

Although this evidence to some extent points to improved use of Iservices, the ICR lacked evidence on some of the other important indicators of usage (such as, sending or receiving emails, e-commerce related activities, use of the internet in schools, internet banking, frequency of individual use of the internet, location of individual users of the internet).

5. Efficiency:

According to the ICR (Annex 3), there was a consumer surplus of about US$5 million from reduced telephone tariffs and a surplus of US$4 million from reduced internet charges between 2005 and 2011. However, there was no attempt made to relate this to the cost of the activities supported by the project. The surplus in and of itself, therefore, does not demonstrate efficiency in the use of project resources.
The implementation period was extended for a total of two of 2 years (about 44% of the original implementation schedule). Even then, US$ 146,870 had to be cancelled because equipment for Very High Frequency services in St. Vincent and the Grenadines had not been delivered by project closure. Capacity issues at the level of the National Telecommunication Regulatory Commissions constrained the speed at which the Universal Service projects were designed resulting in implementation delays. Also, implementation delays were caused because of several extensions of the bids due to limited private sector interest in the bidding process.

Efficiency is rated Modest

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:

Rate Available?

Point Value

Coverage/Scope*

Appraisal:

No

%

%

ICR estimate:

No

%

%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

Relevance of objectives was high and that of design substantial. The efficacy of the first and second sub-objectives - improved access and quality of telecommunications and Information and Communication Technology Services - is rated substantial and that of the third sub-objective - improved use of telecommunications and Information and Communication Technology Services - is rated modest due to lack of evidence.Efficiency is assessed as modest. Overall outcome is rated Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

There are a number of indications of a low risk to development outcomes:

There is a likelihood that the Eastern Caribbean Telecommunications Authority would continue to provide legal and technical advice to National Telecommunication Regulatory Commissions on the implementation of Universal Service projects. The Eastern Caribbean Telecommunications Authority would also advice National Telecommunication Regulatory Commissions on the processing of applications for Universal Service projects.

The National Telecommunication Regulatory Commissions of Dominica and St. Kitts and Nevis have signed separate Service Agreements with contractors (that implemented the first phase of their Universal Service Projects) to implement the second phase related to operation and maintenance of services. The Eastern Caribbean Telecommunications Authority would in turn support National Telecommunication Regulatory Commission in the implementation of these Service Agreements.

The Eastern Caribbean Telecommunications Authority has signed a Service Agreement with the operator (that established community access points in rural areas in Grenada) to maintain these services.

a. Risk to Development Outcome Rating: Negligible to Low

8. Assessment of Bank Performance:

a. Quality at entry:

Lessons learned from the previous OECS Telecommunications reform project, as well as those from international best practice and analytical work in the sector were incorporated in the project's design. These included (i) a regional approach for the development of the telecommunications and Information and Communication Technologies sector in order to ensure that telecommunication policies and regulations are updated and compatible across the region; (ii) continued support to the Eastern Caribbean Telecommunications Authority and the National Telecommunication Regulatory Commissions given that newly created regulatory authorities take a long time to consolidate and require continuous support in developing their institutional, human, and technical capacity; (iii) the vital importance of government commitment from all countries; and (iv) support for the development and implementation of universal access policies and regulations as well as setting up a Universal Service Fund to finance universal access initiatives based on experience in Latin America, South Asia and Africa.

The implementation period was unrealistically short given the need to engage in extensive consensus-building consultations involving several governments.

The risk of limited private sector participation in the bidding process for Universal Service projects was not identified in the PAD. This risk could have been foreseen given the small size of the market, and materialized during implementation resulting in several extensions to the closing dates of the bids causing implementation delays.

Important capacity building activities such as business outreach seminars and workshops were not incorporated into the project's design.

The PAD did not provide a detailed specification for the activity related to the alternative telecommunications system.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

17 Implementation Status Reports were filed, implying an average of almost two missions per year over the 6.7 year implementation period.

Business Outreach Seminars that were conducted during the last year of project implementation helped in increasing the participation rate of the bidders which in turn increased the disbursement rate of the project to 90 percent at closing.

Management provided detailed attention to Implementation Status Reports, pointing to the need to explain changes in performance ratings and to assess quality of service more carefully by going beyond the indicators.

There were four task team leaders during the 6.7 year life of the project which undermined continuity.

Although supervision reports identified problems, these were not always reflected in the Implementation Status Report ratings. For instance, despite significant delays in implementation the Universal Access component, the implementation progress rating and component level rating for Universal Access never dropped below moderately satisfactory throughout implementation.

An error in reallocating funds meant that the Eastern Caribbean Telecommunications Authority was unable to proceed with the purchase of the equipment for Information and Communication Technologies Pilot project.

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

The fact that this project was based on initiatives and requirements identified under a predecessor project (Regional Telecommunication Reforms Project) and that it involved considerable consultations with each of the five ECTELs member countries points to strong ownership by the Governments involved.

Commitment from the governments of respective member countries is also evident from their support towards organizing procurement and business outreach seminars along with the task team in order to raise the interest of Information and Communication Technology providers and vendors to participate in the bidding process.

Government Performance Rating: Satisfactory

b. Implementing Agency Performance:

The Eastern Caribbean Telecommunications Authority was the only implementing agency. It was obliged to undertake lengthy consensus-building procedures among the five member countries which contributed significantly to delays.

Further delays resulted from the implementation of the Universal Access component (representing close to 50 percent of the project's commitments). The National Telecommunication Regulatory Commissions had limited knowledge and experience of bidding processes and of defining the design and technical elements of the projects selected under this Universal Access component. There were, therefore, delays in obtaining information and feedback from the National Telecommunication Regulatory Commissions and other national agencies to the Eastern Caribbean Telecommunications Authority on identification of suitable Universal Access projects for implementation. It took longer than foreseen for the Eastern Caribbean Telecommunications Authority to reach agreements with the respective member countries on the implementation of the Information and Communication Technology pilot projects.

Implementing Agency Performance Rating: Unsatisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The outcome indicators in the PAD were inadequate for monitoring progress towards achieving the development objectives. For instance, the PAD did not include sufficient outcome indicators to measure improved quality and use of telecommunications and Information and Communication Technology services, two of the three sub-objectives in the PDO.
The Eastern Caribbean Telecommunications Authority was responsible for collecting the data to monitor and evaluate the outcomes and results of the project. The PAD did not provide baseline data for the following 4 indicators (i) number of community centers with broadband access; (ii) broadband ICT access availability in 100 community centers across (iii) increase in leverage of USF subsidies as ratio of private investment vs subsidies and (iv) increase in usage of broadband services. However, the intermediate and outcome indicators were measurable, and the data collection instruments and responsibility for data collection were specified in the PAD (Annex 3, page 29).

b. M&E Implementation:

During the mid-term review in March 2008, the Eastern Caribbean Telecommunications Authority raised the issue that the monitoring framework and some of the indicators that were being tracked were not valid and that changes were necessary to better reflect the project objectives. For example, the Authority wanted to use Public Access points to define accessibility in public places, such as schools, hospitals, libraries and post offices. Although, this indicator was added during the mid-term review, it was never monitored during implementation. No attempt seems to have been made to introduce indicators measuring improved quality and use during implementation despite the urgings from Bank management.

Data used to calculate the indicators was collected from telecommunications service providers using questionnaires and from publicly available information published by providers. The data received from providers were reviewed using quality control measures including manual audits of questionnaires, period over period comparison of data, and comparison of data to publically available information. Data were also collected through the Eastern Caribbean Central Bank's statistical database, which in 2010 carried out an exercise of rebasing National Accounts and CPI affecting some of the indicators such as Information and Communication Technology contribution to GDP and Internet penetration (ICR, page 18).

a. M&E Utilization:

No information is provided in the ICR concerning the utilization of the project’s M&E system.

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

No safeguard policies were triggered by this category C projects.

b. Fiduciary Compliance:

ECTEL was responsible for both Financial Management and procurement.

Financial Management was downgraded to moderately satisfactory in the Implementation Status Report (Dated: March 04th, 2010) due to recurrent findings on the internal control environment disclosed in last audit reports, late submission of periodic financial reports and audit reports.

There is no information in the ICR on timeliness of the project's external audits and the external auditors' opinions. According to the External Audit Report (dated March 8, 2013), all external auditor's opinions were unqualified.

According to the Implementation Status Report (Dated: December 09th, 2009), the Independent Procurement Review carried out in June 2009 indicated that the procurement process carried out by Eastern Caribbean Telecommunications Authority was well managed and compliant with Bank Procurement Guidelines.

c. Unintended Impacts (positive or negative):

None

d. Other:

None

12. Ratings:

ICR

IEG Review

Reason for Disagreement/Comments

Outcome:

Satisfactory

Moderately Satisfactory

Relevance of objectives was high and that of design substantial. The efficacy of two of the three sub-objectives is rated substantial and that of the third modest. Efficiency is assessed as modest.

Risk to Development Outcome:

Negligible to Low

Negligible to Low

Bank Performance:

Satisfactory

Moderately Satisfactory

There were significant shortcomings in quality at entry (see Section 8a above).

Borrower Performance:

Satisfactory

Moderately Satisfactory

There were major shortcomings in Implementing Agency performance (see Section 9b above).

Quality of ICR:

Unsatisfactory

NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons: The following lessons are based on analysis from this ICR Review:

Adequate time needs to be set aside for engaging in necessary consensus-building in a multi-government project.

If the interest and capacity of potential vendors is not built into the project's design and timetable, then there may be lack of private sector interest in small country projects.

The lack of appropriate indicators in the M&E system can make it difficult or impossible to monitor progress towards attainment of the development objectives.

14. Assessment Recommended?

Yes

Why? The review proposes that IEG undertake an independent, field-based Project Performance Assessment, which would provide the opportunity to reconsider the ratings as warranted by the evidence.

15. Comments on Quality of ICR:

The quality of the ICR is rated Unsatisfactory:

The information presented in the ICR is mostly output based. There is insufficient evidence on improved access to, quality and use of, telecommunicationsand Information and Communication Technology services.

The analysis of efficiency rests on a weak methodology, and made no reference to costs or to operational and administrative inefficiencies.

There are numerous internal inconsistencies for example: (i) actual cost for each component in the ICR (page 7) is the same as that in the PAD even though the project had partial cancellations; (ii) targets for the intermediate indicators "Leased lines rates are reduced by 30%" and "Monthly rates for International Leased lines" in the ICR were different from those in the PAD; (iii) data on Economic and Financial Analysis were poorly presented in the ICR (page 39-40); (iv) there is en error regarding the number of community centers in the ICR (page 6), which, according to the PAD, should be 100 and not 1000 as in the ICR; and (v) there are significant differences in the actual costs in the ICR (page 31) and those in the external audit (page 8).

The lessons learned section in the ICR (page 27) does not reflect the evidence from, or experience of, project implementation.

There is no information on the utilization of the M&E information or on external audits.

Although this project was assessed by the Quality assurance group, there were no ratings for Quality at Entry and Quality of Supervision under "QAG assessment (if any)" section in the ICR (page (ii)). Also, there was no reference to this assessment anywhere in the ICR.