The project’s Development Credit Agreement (p. 17) states that the project's objective was:

“to create an enabling environment for the evolution of autonomous and accountable public higher education institutions, and to develop effective support mechanisms for the improvement of the quality, relevance, efficiency and equity of higher education.”

The objective in the Project Appraisal Document (p. 5) has the same wording.

Component 1: Higher education system reform and oversight(originally estimated at US$8.8 million with actual disbursement of US$2.1 million, which was 24 percent of the planned amount) was to support the government in implementing its Higher Education Long Term Strategy 2003-2010 by promulgating a new higher education law, strengthening the Ministry of Education and Culture's (MoEC's) Directorate General of Higher Education management and administration, assisting with developing an institutional accreditation program, and developing a strategy to revitalize the Open University.

After a Level II restructuring (2009), several activities were added, including capacity building support for a Global Distance Learning Network to train and mentor technical staff of the Indonesia Higher Education Network (INHERENT), especially on video conferencing and on-line training; and scholarships to Open University staff to strengthen their distance learning and online training techniques.

Component 2: Grants to improve academic quality and institutional performance(originally estimated at US$97.9 million with actual disbursement of US$82.7 million, which was 85 percent of the planned amount) was to support improving the quality of higher education through targeted investments and by improving management capacity at Higher Education Institutions (HEls). Some of the component financing was cancelled due to government restrictions on implementing one of the grant programs as well as a lower than expected grant awards in another grant program (page 9 of the ICR).

Component 3: Project Management(originally estimated at US$7.8 million with actual disbursement of US$7.7 million, which was 98 percent of the planned amount.) was to provide financial support to the Implementation Unitto implement the project. After the Level II restructuring (2009), this support also efforts to expand the role of the Implementation Unit to build procurement, financial management, and human resource management capacity at HEls to run workshops, short training courses, and on-site mentoring, monitoring and supervision.

At the time of appraisal, the project was estimated to cost US$114.5 million. The actual cost of the project was US92.5 million, which represents 81 percent of the planned amount. All of the major activities added to Component 1 at the 2009 restructuring appear to have been carried out as planned, although often with alternative sources of financing (ICR, p. 26). This led to significant cost savings.

Financing

The Bank provided a US$30 million IDA credit, of which 99.9 percent was disbursed. It also provided a US$50 million IBRD loan, of which 80.3 percent was disbursed.

A total of US$6.9 million was formally cancelled from the IBRD loan, largely due to changes in Component 1. An additional US$1.9 million was left unspent from IBRD and cancelled at the end of the project. A small amount (US$ 45,000) was left unspent from the IDA credit.

Borrower Contribution

The government agreed to finance US$34.5 million equivalent to support the project. At the time of project closing, it had provided US$22.4 million equivalent, equal to 64.9% of the initial budgeted amount.

By the end of the project period, the government provided 84.5% of planned financing for Component 2. The shortfall was due to the Ministry of Finance's decision to stop block grants. The government also provided only 24.1% of planned financing for Component 1. This was largely driven by the decision to reduce the size of the component, from US$8.8 million to US$6.1 million. Actual expenditures on this component were only US$2.1 million, and the government actually contributed a larger share than originally planned (ICR, pp. 51-52). Dates

The project was approved by the World Bank Board of Directors on June 9, 2005 and became effective on December 20, 2005. The project was restructured (Level II restructuring) once, on September 24, 2009. The restructuring was largely based on observations made during the mid-term review, held in the same year, that led to changes in the project's implementation structure (ICR, pp. 8-9), and also in response to the government's 2009 decision to stop block grants to public HEIs and shift instead to line-item budgeting, as required by the Indonesia budget law.
The original closing date was June 30, 2011, but this was extended as part of the restructuring to December 31, 2012. The project closed on the latter date.

3. Relevance of Objectives & Design:

a. Relevance of Objectives:

The relevance of objectives is rated high.

At the time of approval, the project was based closely on the government's tertiary education strategy (Higher Education Long Term Strategy) as well as the Bank's Country Assistance Strategy (CAS), which focused both on improving human capital and strengthening public institutions. The project was designed to support the government's efforts to strengthen the country's tertiary education system (ICR, p. 5), alleviating shortages of higher-level skills that were slowing investment and the absorption of new technologies.

The project remained highly relevant at the time of project closing. The government continues to include the goals of the reform in its development strategies and passed a law in 2012 that codified many of the project's objectives, replacing a previous law that was ruled unconstitutional by the Constitutional Court (ICR, p. 27). While the government has decided to stop borrowing for education projects, the project remains relevant to current World Bank strategy in a number of areas, including: (i) institution building for the public sector; (ii) improving education and skills for the labor force; (iii) enhancing equality of education opportunities; and (iv) strengthening human resources in research and development. In many cases, the Bank plans to support these activities with technical assistance when lending is not an option (Country Partnership Strategy 2013-2014, pp. 20-21).

b. Relevance of Design:

The Relevance of Design is rated high.

The project had two different objectives: (i) improving the policy environment, and (ii) providing support to universities' mechanisms for improvements in the quality, relevance, efficiency and equity of higher education ("university" is used to indicate any higher education institution). While these goals are related and may be complementary in many cases, they require quite different sorts of interventions.

In order to improve the policy environment, the project focused on strengthening the legal framework for higher education as well as strengthening the Ministry's stewardship capacity (Component 1). Support for the legal framework included technical assistance to the Ministry. The project also supported the Ministry's financial management capacity and the autonomy of the accreditation system, as well as studies and data collection to monitor the higher education system and its impact on the life of graduates.

The project also providedsupport to universities. Central to this was the provision of grants to universities, through Component 2. The design of this component was based on policy recommendations for higher education and included competitive grants available to both public and private universities. These grants were expected to cover a wide range of areas, including the quality, relevance, and equityof higher education. The grants were designed to include support for laboratory equipment, staff development, and studies (quality and relevance) as well as support for community development and scholarships (equity).

In addition, the project also supported three grant programs to support university governance in different types of institutions, including autonomous public universities and non-autonomous public universities, using a mix of competitive grants and performance-based grants. This design showed appropriate flexibility in the face of different types of university structures. Most of the support focused on improving university administration, through investment in financial and procurement management as well as human resource management, which are intended to improve efficiency at the university level. The project also provided direct support to universities to strengthen their financial management capacity to manage the grant program, through Component 1.

Overall, all of the project's planned activities were closely aligned with the project's objectives or administration and represent careful attention to both aspects of the objectives.

4. Achievement of Objectives (Efficacy) :

Create an enabling environment for the evolution of autonomous and accountable public higher education institutions.

Achievement of this objective is rated substantial.

The project supported a number of policy initiatives that contributed to accountability in higher education in Indonesia. These are reflected both in outputs supported by the project as well as outcomes related to increased autonomy, accountability, and good governance of the institutions in the sector.

Outputs

The project supported the approval of Higher Education Laws in 2009 and 2012 that established a governance framework for universities. The project provided this support by financing seminars, study tours, and reports. The first law was overturned by the Constitutional Court and replaced in 2012 with a new law that had a similar objective and responded to concerns raised by the Court (the ICR does not describe the Court's specific concerns). The government reports that the project played an important role in supporting the development of both laws (ICR, p. 6).

The project provided training for university and ministry staff in financial management. It also supported training for procurement staff at the university level, which led to the certification of procurement staff in most of participating universities (28 of 29 participating universities had 50% or more of procurement staff certified in accordance with national certification policy, meeting the target).

The project supported a number of studies and surveys to provide accountability. This included a tracer study of graduates that was developed and implemented but has not been yet fully institutionalized. Tracer studies are often seen as important measures of accountability.

The project supported efforts to create an accreditation framework to monitor the quality of universities. In addition to the framework, the project provided technical assistance to universities to participate in the accreditation process.

Through Component 2, the project provided grants for good governance to a number of public universities (including autonomous universities). Although this is usually associated with the second objective (see below), these activities are also appropriately associated with the first objective. They included five results-based grants to autonomous universities (out of six proposals), proposal-based grants to all autonomous universities, as well as 29 grants (out of 115 proposals) for non-autonomous public universities. The ICR Annex Table 2.5 (pp. 43-44) outlines the activities supported by the grants. According to the ICR (p. 44), these activities were largely carried out as expected.

Outcomes

There was an increase in both the capacity and transparency of universities. This contributed to their autonomy and accountability. There are several indicators of this process:

Unqualified audit reports in seven universities, against a target of five by the end of the project.

86 percent of procurement by participating universities occurred during the bid validity period, compared to a target of 90 percent. Despite the fact that the project missed its target, this is an impressive achievement given that at the start of the project, most universities had no procurement capacity.

Likewise, 83 percent of participating universities published the results of their major procurement. While this does not meet the target of full disclosure by all universities, it represents an important advance from a situation where no procurement was publicized.

52 public and 42 private universities were accredited. This represents 55 percent of public universities and substantially exceeds the target of 5 percent of public universities accredited. Approximately 1.3 percent of private universities were accredited; it is not clear if the project had a target for private universities (ICR, p. 30). Given that accreditation is new to higher education in Indonesia, this represents a substantial achievement.

Performance-based contracts were awarded to five out of six autonomous universities. The creation of this type of contract is an important outcome to promote autonomy. The support to 29 other public universities may have also strengthened their capacity to act in a more autonomous manner.

Develop effective support mechanisms for the improvement of the quality, relevance, efficiency and equity of higher education.

Achievement of this objective is rated modest.

This objective was largely supported by the grant programs under Component 2. These programs were affected by a decision of the Ministry of Finance to prohibit the use of block grants in 2009 and a decision of the Constitutional Court to overturn the Higher Education law in 2010. By the end of the project, 84.5 percent of total amount of Component 2 was disbursed, and the project was able to continue with other forms of grants.

Outputs

Some of the outcomes related to the first objective can be considered outputs for this objective. They include: (i) unqualified audit opinions for seven higher education institutions; (ii) university procurement carried out in a timely fashion; (iii) training of procurement staff; and (iv) publishing of contracts financed under the grant. In addition:

38 competitive grants (averaging US$1.2 million) were awarded to mostly public universities. These grants covered a number of areas from seven cost categories: (a) staff development; (b) technical assistance; (c) research and studies; (d) community development; (e) procurement of goods for a variety of capital goods and educational inputs; (f) promotional activity aimed at secondary school students: and (g) scholarships for poorer applicants. ICR Annex Table 2.2 (p. 38) reports the allocation and actual expenditures under each of these cost categories.

The project awarded five performance-based grants to autonomous universities. While these grants contributed to the strengthening of autonomy (see above), they also contributed to the quality and equity of education in these universities.

Outcomes

The grade point average increased from 2.99 to 3.17 (compared to a target of 3.13) between 2009 and 2012 in universities participating in the grant program. While this is an important improvement, it is difficult to show causality with the project due to the lack of comparison data. It is also difficult to show that this reflects a real change as opposed to "grade inflation." The project's outputs (through grants) included activities that may plausibly have contributed to improving quality and student learning, including improved equipment, better-trained staff, and research.

The length of time necessary for graduates to find employment declined from 8.5 months to 5.8 months (compared to a target of 7.6 months). This is a measure of both relevance and efficiency. The use of a tracer study increases the credibility of this measure; however, the lack of data on the performance of graduates from non-participating universities as well as general changes in the labor market make it difficult to establish causality.

The number of students who received need based scholarships increased from 8 percent to 12 percent (compared to a target of 12 percent). Studies have shown that financial support plays a major role in increasing access to universities, and this was the likely outcome in Indonesia.

The average time to graduate dropped from 56.7 months to 51.6 months (compared to a target of 54 months). This represents an increase in internal efficiency. This increase is likely due to changes within the universities, which in turn may have been the result of the investments supported by the grants.

Overall, there was a significant improvement in all of the indicators associated with this objective. While the project's design included a causal chain that could plausibly have led to these outcomes, the available data make it difficult to compare the results with non-participating universities.

5. Efficiency:

The project's efficiency is rated substantial.

At the time of appraisal, the project team carried out a detailed economic analysis based on careful study of economic conditions, the labor market, and the project's design. The analysis clearly laid out its assumptions and carried out a sensitivity analysis. The economic analysis showed an estimated internal rate of return from 32 to 50 percent.

The ICR had an equally comprehensive economic analysis, focusing on the project's actual performance and its impacts given current economic conditions. These results show a similar rate of return (30 percent). These estimates are reasonable given the relatively high rates of return associated with higher education in Indonesia (PAD, p. 77, and ICR, p. 56) and the broad impact of the program on the higher education sector.

The program closed 18 months later than planned, which does not represent a serious shortcoming. Due to the change in the legal framework, it was not possible to carry out large parts of Component 2. The government and the Bank agreed to cancel parts of the loan rather than trying to find another area in which to invest the remaining resources. This was a prudent use of Bank resources.

a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return at appraisal and the re-estimated value at evaluation:

Rate Available?

Point Value

Coverage/Scope*

Appraisal:

Yes

32%

100%

ICR estimate:

Yes

30%

100%

* Refers to percent of total project cost for which ERR/FRR was calculated

6. Outcome:

The project was highly relevant, with objectives that were well connected to government strategy and the Bank's strategy in Indonesia, with a logical design that focused tightly and plausibly on achieving the objectives. The first achievement, improving the policy environment, was substantially achieved, The second objective, developing mechanisms to promote efficiency, equity, relevance, and quality, also appears to have been achieved, but there is not plausible evidence linking these achievements to the project's interventions. Project resources appears to have been invested efficiently. Taken together, these ratings are indicative of moderate shortcomings, resulting in an Outcome rating of Moderately Satisfactory.

a. Outcome Rating: Moderately Satisfactory

7. Rationale for Risk to Development Outcome Rating:

The reform initiated by the project is still in its initial phase, with a large number of public universities still without accreditation. Despite the government's continued support for the reform, as evidenced by the approval of a higher education law in 2012, the ICR (pp. 12, 32-33) notes that the government is currently giving less support to the reform process than it had in the past.

The government has stopped borrowing for education; however, the Bank continues to provide technical assistance and work with other development partners to promote the reform agenda. However, without clear support from the government, it is possible that many elements of the reform will not "stick."

a. Risk to Development Outcome Rating: Significant

8. Assessment of Bank Performance:

a. Quality at entry:

The project was based on extensive analytical work, supported by both by the Bank and other development partners (ICR, p. 5). This background work led to a project with a strong design that was well supported by the international literature and also well aligned with government priorities. This situation was enhanced by the World Bank's two internal reviews of the project during the design phase. These reviews led to a number of important innovations, including quality assurance and performance-based block grants. The design took into account the complicated structure of higher education and was cognizant of the politics in the sector.

Overall, the risk assessment matrix did not provide a realistic discussion of the risks that the project faced. None of the risks were rated as "high" (PAD, pp. 15-16). The risk analysis did not discuss the possibility that the grant program would be suspended, despite the fact that there already was some speculation that this might take place. The block grant mechanism was already prohibited under Ministry of Home Affairs regulations that were promulgated in 2004 and that were not being enforced at the time of project approval. The existence of these regulations was known before the project was approved and could have been better reflected in the risk matrix with more discussion of possible mitigation (ICR, p. 6). A similar approach was used in a previous Bank project, and it appears that this led the preparation team to underestimate the risk of this regulation being enforced.

Likewise, there was little discussion about the risk of university autonomy being restricted by the courts. Given that autonomy depended on a legal change and that courts often modify or invalidate new laws, this was a real risk at the time of preparation.

Quality-at-Entry Rating: Moderately Unsatisfactory

b. Quality of supervision:

The project had two TTLs over the seven-year implementation period. Both were based in Indonesia, and the ICR (p. 19) reports that the there was a smooth transition as both were part of the project's preparation and implementation. Since the project was supervised in the field, there was routine contact with the government and implementing agency as well as ten supervision missions and a mid-term review (ICR, p. 20).

The team was able to respond to a complicated political environment quickly and develop solutions as challenges arose. During the mid-term review, the project was restructured relatively efficiently to respond to changes, including adjusting the results framework and moving to reduce bottlenecks. The ICR (p. 20) also reports that the supervision team was frank in discussions with Bank management and in seeking solutions as required. The Bank showed flexibility in working with the implementing agency and universities to adjust to changes in policy, such as the enforcement of the regulation against block grants as well as bottlenecks in implementation, such as the low number of grant recipients in the first round (ICR, pp. 33-34).

Delays in procurement were a constant problem during implementation. The Bank was slow in giving its no objections to procurements, with delays as long as ten months. This affected implementation at both the central and university levels and was counterproductive to the project's objective of promoting autonomy (ICR, p. 20).

Quality of Supervision Rating: Moderately Satisfactory

Overall Bank Performance Rating: Moderately Satisfactory

9. Assessment of Borrower Performance:

a. Government Performance:

Prior to the project's approval, the government had already provided strong support to reform and gave it a central place in its development strategy (ICR, p. 20).

The government agreed to the terms of the project through the appraisal process and loan negotiations. After the project was approved, however, the Ministry of Finance enforced regulations that prohibited block grants for universities. This was done because these universities did not have autonomous status under the Indonesia budget law. This was a major feature of the project, and it became negatively affected by administrative decree. Ideally, the government could have worked with the implementing agency and the World Bank to identify this issue at the time of project design and find possible solutions to permit some form of the grant program to continue (ICR, pp. 21, 32).

Government Performance Rating: Moderately Unsatisfactory

b. Implementing Agency Performance:

The implementing agency was the Directorate General of Higher Education of the Ministry of Education and Culture, which was supported by an project unit.

The implementing unit was successful in managing a large and complicated project that involved interaction with both public and private universities. This included bi-annual meetings with participating institutions. The implementing unit was also active in working with the World Bank and improving project implementation. This required flexibility, as the Directorate General and implementing unit had to work to redesign the grant program after the Ministry of Finance's decision to stop block grants.

Implementing Agency Performance Rating: Moderately Satisfactory

Overall Borrower Performance Rating: Moderately Satisfactory

10. M&E Design, Implementation, & Utilization:

a. M&E Design:

The results framework was well designed to measure the program's impact on universities, with clear indicators that measured outcomes that were generally well aligned with the objectives and the project's design. However, the project's M&E framework did not include any sort of comparison groups or any way to compare the impact of the project with a control group of non-participating universities; this was a serious shortcoming that was not corrected during implementation.

The ICR (p. 9) reports that there was some confusion about what was a PDO-level indicator and what were intermediate indicators. The results framework in the PAD, for the most part, identifies the indicators at appropriate levels (PDO-level and component-specific intermediate indicators). One important element of the project was the use of performance-based contracts with the universities. Although the contracts were to be developed under the project, there was clear indication of the areas of performance that the project wished to promote (PAD, pp 50-51).

All of the indicators had baselines at the time of preparation. The process- and project-specific indicators were relatively straightforward to monitor by the Ministry, participating universities, and the project.

b. M&E Implementation:

The ICR does not report any problems in monitoring the indicators, and reporting took place in a timely fashion. The technical assistance for universities included support for M&E. Some of the universities did have problems with qualitative and process data, although generally quantitative data were provided as expected. According to the ICR (p. 9), the implementing unit and the Bank worked together to check the quality of the data and to aggregate as necessary for reports. M&E data were regularly included in aides-memoire prepared after missions.

One of the project's key indicators, the share of poorer students receiving scholarships, was not understood by the Ministry, and consequently the data were incorrectly reported. It was not possible to correct this problem by the end of the project period.

a. M&E Utilization:

Data played a major part in the project's implementation. For the universities that received performance-based contracts, M&E data were used to modify future contracts to develop better and more realistic targets. At the project level, data were used to inform supervision missions and were important for the 2009 restructuring. Data from the project's M&E system also flowed into the national higher education statistics (ICR, p. 10).

M&E Quality Rating: Modest

11. Other Issues:

a. Safeguards:

No safeguard policies were triggered during project preparation. No safeguard issues were reported during project implementation (ICR, p. 10).

b. Fiduciary Compliance:

The financial management risk assessment identified the project as high risk, and the project included a number of mitigation measures, including supporting the Ministry's financial management capacity. All of the supervision missions included a financial management specialist, and the project was rated satisfactory or moderately satisfactory on financial management performance. Initially, the project was slow to respond to audit observations, After the Bank provided additional technical assistance, Interim Financial Reports were submitted in a timely fashion. The audit reports were all submitted on time and were all unqualified (ICR, pp. 10-11).

Procurement was carried out both at the central level and by participating universities. Initially, there were some delays in procurement both due to the universities' lack of experience with Bank procedures and the Bank's slow processing of procurement review. This improved substantially as the project proceeded, both with additional training for implementing units and with changes in Bank policy. No major procurement issues were reported (ICR, p. 11).

c. Unintended Impacts (positive or negative):

None reported.

d. Other:

12. Ratings:

ICR

IEG Review

Reason for Disagreement/Comments

Outcome:

Moderately Satisfactory

Moderately Satisfactory

Risk to Development Outcome:

Significant

Significant

Bank Performance:

Moderately Satisfactory

Moderately Satisfactory

Borrower Performance:

Moderately Satisfactory

Moderately Satisfactory

Quality of ICR:

Satisfactory

NOTES:
- When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.
- The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate.

13. Lessons:

In addition to the lessons presented in the ICR (pp. 22-23), one additional lesson offered by IEG is the importance of doing a frank assessment of risk that provides not only ways to prevent the occurrence of the risk but to address the "damage" caused if the risk materializes. In the case of this project, there was a clear risk of the grant program facing legal issues, and it appears that this risk was not taken seriously by the preparation team. Another approach would have been to develop alternatives from the start in the case that the grant program was modified or cancelled by the government.

14. Assessment Recommended?

Yes

Why?

The project is recommended for further assessment for two reasons. First, the project is a good example of World Bank support to tertiary education. There have been no recent evaluations of higher education, and the project should be considered for assessment to fill this gap. Second, given the general improvement in tertiary education outcomes in Indonesia, it is important to show credibly if the project had an impact on these results.

15. Comments on Quality of ICR:

The ICR is well prepared, with detailed data on the project's activities, outputs, and outcomes. The ICR is also well written and organized. The economic section (Annex 3) is quite comprehensive and is a good example of ex-post cost-benefit analysis of similar quality to the PAD. Annex 2 is unusually detailed and presents data in an accessible fashion. The ICR would have benefited with more analysis of how the project may have been responsible for observed outcomes.