Evaluation

Introduction

Poland’s development assistance provided through the Polish MFA has been the subject of evaluation since 2012. The legal framework for the evaluation is set out in the Development Cooperation Act of 16 September 2011. Pursuant to its provisions, the objectives of development cooperation as defined in the Multiannual Development Cooperation Programme 2012-2015 (Art. 13.1.10) should become the subject of evaluation.

The MFA’s Department of Development Cooperation draws up an annual evaluation plan, which sets out the method applied to the organisation of the evaluation process and the main activities (the scope of evaluation, resource allocation, timetable). This document is later approved by an undersecretary of state for development. It is indicative and may be modified or adjusted to the changing conditions and needs.

The evaluation processes are focused on annual development cooperation programmes and projects or groups of projects. Initiatives implemented abroad and at home are analysed. Results of the evaluation are used mainly in decision-making processes, primarily to identify and programme the activities of Poland’s development cooperation in the short-term (annual) and medium-term (multiannual programme) perspectives.

Evaluation of programmes and projects implemented by Polish development cooperation is based on development evaluation standards and principles set out in DAC Quality Standards for Development Evaluation (2010), DAC Principles for Evaluation of Development Assistance (1991) and UNEG Norms and Standards for Evaluation (2016). Evaluation of Polish development cooperation also takes into account Europe Aid and European External Action Service guidelines, as set out in the document The Evaluation Policy for European Union Development Co-operation. Furthermore, it is consistent with the evaluation methodology drawn up by Evaluation Unit of the European Commission (DG DEVCO).

Evaluation is carried out ex post (after a project or a programme is completed). It is done by external companies selected through a public tender procedure, pursuant to the Public Procurement Act and the MFA’s internal laws and regulations.

Evaluation in 2017

In 2017, MFA assessed the activities of Polish aid, which supported system reforms, fire-fighting and police services in the Ukraine. Moreover, selected educational initiatives carried out in Poland as part of the development cooperation in favour of Ukraine and other Eastern Partnership countries were evaluated.

The research was carried out based on the tasks that have been accomplished under the Multiannual Development Cooperation Programmes 2012-2015 and 2016-2020.The main objective was to analyse the efficiency, sustainability, relevance, effectiveness, functionality and innovation of the undertaken development initiatives.

The research has led to a positive assessment of Polish development aid implemented in Ukraine and selected education initiatives carried out in the Eastern Partnership countries. Most of the social and economic objectives of the implemented development projects were met in all areas of support. Measures offered to Ukraine as part of Polish development cooperation are based on Polish transformation experiences and are readily adaptable to Ukrainian conditions.

The Polish experiences served as an inspiration for the Ukrainian party, especially during the initial period of work on decentralization. The effects of carried out activities show that a mechanism of change was triggered, covering competencies of the participants connected with activities and the implementation of new mechanisms of running the local government.

With regard to supporting the reform of the Ukrainian education system, the impact of Polish initiatives on comprehensive and systemic changes was much more visible. Moreover, a broad long-term nationwide impact could be expected.

The efficiency of initiatives supporting the education reform could have been greater if the needs of the recipients had been better identified and more emphasis had been placed on the practical aspects of operating of the Polish education system. Extending the projects to include the stage of implementation, monitoring and assessment of outcomes would be a big added value. The effectiveness could be increased if university lecturers specialising in education were included in the projects’ target group. Currently, they are regarded as reform hinderers.

The strengths of Polish development cooperation in this respect are: high level of aid relevance at both strategic and operational levels, good familiarity with Ukrainian realities coming from experiences and a history of contacts between Polish organisations and Ukrainian partners, and the practical dimension of projects and solutions offered.

Some recommendations:

adopting a more strategic approach to supporting reforms in Ukraine by concentrating initiatives;

continuing support in areas where ending projects would compromise their outcomes;

relying more on the existing channels for coordinating Polish activities with other donors;

strengthening the institutional presence of Polish development aid in Ukraine.

Projects in public security are characterised by high effectiveness in obtaining the assumed hard outcomes and outputs. In the case of some projects, indicators were exceeded.

Carried out projects are very sustainable. Ukrainian services benefit from the knowledge and skills acquired during specialist training. The projects combine education activities with “hard” infrastructural elements (modernisation and renovation of building structures) and equipment transfer. However, the projects are not being monitored for deferred effects.

Fire-fighting projects constitute a Polish “niche” and a “Polish export product.” The initiatives also have large horizontal synergy – subsequent initiatives are based on earlier experiences.

No evidence of complementarity between Polish projects and other donor initiatives was observed and no overlapping ventures in the area of security were identified. In comparison with other donors, Polish support provided to Ukrainian rescue services is systematic, complex and specialist.

Some recommendations:

cooperating with other donors on fire-fighting projects;

maintaining concentration and specialisation of rescue aid with a parallel inclusion of new recipients, subjects and areas in initiatives;

continuing cooperation with reliable partners.

Evaluation results – Component III (initiatives in the areas of education in the EaP countries)

The evaluated initiatives should be assessed very favourably both in terms of their outcomes and relevance. Their sustainability impact should also be pointed out. However, participants said what they appreciated most was the knowledge they had acquired and the possibility to exchange experience with people from other countries.

The sustainability of evaluated initiatives should be assessed positively. Most respondents still draw on the knowledge gained during training, studies and internships in their professional work. Project participants have also said that now they are better off financially.

The fact that some of Poland’s initiatives are unique is also significant which makes our country stand part from our donors and has a positive impact on its image.

Openness, sincerity and willingness to share experiences are some of the things that internship, training and scholarship participants appreciate most in Polish lecturers.

Some recommendations:

better coordinating initiatives and popularising good practices of knowledge sharing;

Evaluation in 2016

In 2016, evaluation was carried out based on the tasks that have been accomplished under the Multiannual Development Cooperation Programme 2012-2015.

The MFA assessed Polish development assistance activities in selected countries of East Africa and the Middle East. The objective was to analyse the efficiency,utility, effectiveness, sustainability and relevance of assistance financed from the budgetary reserve for development cooperation.

It was the last evaluation focusing on the initiatives stemming from the Multiannual Development CooperationProgramme 2012-2015, and the first exercise which covered the entire programming period. Conclusions and recommendations influenced the planning, implementing and managing of tasks in the 2016-20 period.

The objective of the evaluation research was to evaluate the Polish development aid provided to selected countries of East Africa and the Middle East in 2012-2015 and to make recommendations based on the results.

The research has led to a positive assessment of Polish development aid provided to selected countries of East Africa and Palestine. In most cases, the Polish aid projects have accomplished their planned social and economic objectives in all areas of support. Low financial outlays and the annual project cycle were identified as the main obstacles to achieving a broader (going beyond local) and long-lasting aid impact.

The most efficient types of support in the case of East Africa were simple and cost-efficient projects (e.g. projects realized in the field of environment) that bring about a fast and visible improvement of the beneficiaries’ situation. Polish development aid is relatively well recognized locally. Underdeveloped cooperation with Polish entrepreneurs was identified as a weakness of the Polish aid . Despite the projects’ complementarity in terms of their duration and geographical location, no significant synergy impact of the aid was evident. The key factors that determine project sustainability are: proper identification and training of local leaders who play the role of “knowledge multipliers” and advance planning of sustainability mechanisms.

The commonly applied principle of partnership (ownership/empowerment) and participatory aid model that allows to properly adjusting an intervention to the local context and needs are some of the strengths of Polish development aid.

As compared to other donors, the scale of Polish bilateral aid is small. Despite the professionalism of Polish institutions engaged in development aid, cooperation with other donors is impeded due to financial limitations and lack of multiannual financing development initiatives. The research results also show that the system of monitoring and evaluation of development aid needs to be developed further.

The analysis of the capacity of Polish institutions engaged in development aid in East Africa reveals that they possess a lot of experience, knowledge and skills, but they are now facing a barrier to further growth and need to be supported by means of relevantly targeted public policies.

Research results – Component II (the Middle East - Palestine)

The research shows that Polish Development Assistance (PDA) projects implemented in Palestine achieve the assumed impact. The projects and their impact are positively assessed by the beneficiaries and project participants, even though the impact is only local.

Polish development organizations/institutions are characterized by great engagement, flexibility and familiarity with the local situation. Moreover, aid initiatives implemented by the Polish diplomatic mission show that the political conditions have been properly diagnosed. Polish non-governmental organizations implementing projects in Palestine are prepared to engage in larger projects.

In most cases project sustainability is maintained, especially in infrastructural projects as a result of adjusting initiatives to the needs of the beneficiaries. Nevertheless, the factor impeding both the project implementation and sustainability is the complicated political situation. In view of this, it is necessary to maintain flexibility and at the same time to prepare for crisis management.

Good cooperation with proven local partners was a key factor of successful project implementation.

However, Polish development cooperation in Palestine has faced serious problems. First of all, outlays for the cooperation are very low, particularly as compared to other donors’. Consequently, the impact scope of the aid is small. Another problem is the lack of synergy with both other PDA projects and other donors’ projects. The process of monitoring and evaluating development aid in Palestine is not systemized and it lacks a comprehensive model of project impact assessment (the long-term results). Under the present institutional structure, the MFA and the diplomatic mission in Ramallah play both a strategic and an implementing role; these institutions cannot focus on strategic tasks due to their many commitments and a shortage of resources.

Research recommendations from the evaluation of Polish development aid in selected countries of East Africa and the Middle East

The key strategic recommendations refer to systemic issues:

gradual shift to a system for implementing large-budget projects,

strengthening the PDA strategic level by delegating tasks relating to implementation to an institution outside the MFA,

The operational recommendations focus on those elements of the PDA system that can be changed to make the work of project promoters easier and to increase the effectiveness of project initiatives:

developing projects that complement other donors’ activity,

introducing mechanisms of verifying the synergy impact of initiatives,

reviewing procedures with a view to earlier launching of projects during a budget year,

greater concentration on sustainability in preparing and assessing projects,

concentration of information and promotion initiatives in presenting the project impact by project promoters in different media.

Specific recommendations for East Africa concern strengthening the capacity of project promoters for diagnosis of the situation and adjusting initiatives to the local needs by using large modular projects for launching pilot versions (over the first year of the implementation of project initiatives). Others recommendations are: more frequent implementation of simple, cost-effective projects and introducing solutions that facilitate implementing projects by non-governmental organizations.

Specific recommendations for Palestine concern realization of a smaller number of projects but with higher budgets and in selected specialist areas. Furthermore, support for projects in the area of entrepreneurship should be increased. During the formulation of projects it is also essential to take into account the impact of the political conflict on the implemented projects and to design a model of risk management. The evaluator also recommends considering introducing additional incentives at the stage of submitting applications for funds with regard to such projects in which the implementer will require the ultimate beneficiaries to make their own financial contribution.

Evaluation in 2015

In 2015, the evaluation of the development cooperation coordinated by the Minister of Foreign Affairs concerned selected measures conducted by the Solidarity Fund PL (FSM) and programmes and projects benefiting Moldova, completed in the period 2012-2014.

The evaluation was conducted in Poland as well as in Georgia, Tunisia, Ukraine and Moldova.

The main goal of the evaluationwas to identify the factors that had the most influence on the efficiency of the Polish Development Assistance as well as to provide recommendations with respect to the measures planned for the upcoming years. The projects were analysed in terms of their appropriateness, usability, sustainability, efficiency and effectiveness.

The assessment was divided into two components. Component I covered the analysis of the initiatives conducted by the Fund under the tasks commissioned by the Minister of Foreign Affairs under the Act on Development Cooperation. Component II covered Polish Development Assistance in Moldova.

The analysis of the completed undertakings showed that:

Component I

The assistance initiatives related to supporting democracy were usually addressed to persons who, due to their profession, would share the newly-acquired knowledge and skills further. The appropriateness of the implemented measures is confirmed first and foremost by the discernible change in the attitudes and the working style that arises from gaining the new skills and learning new patterns and models.

Usability of the projects for the beneficiaries was also e.g. due to the high quality of the measures conducted, considering local character and the partner approach which was particularly appreciated by the beneficiaries. It is also important that due to the recent transformation experience, Poland is perceived as a more reliable country than other assistance donors.

For the sustainability of the projects, it is of key significance to continue the project measures after the external financing is no longer provided, which requires continued organisational and financial involvement of local partners. In the case of some analysed initiatives, these conditions were not met. The annual system of financing has also an unfavourable impact on the sustainability of many projects.

The institutional system within which the Fund realised the tasks commissioned by MFA within the territories of Georgia, Tunisia, and Ukraine facilitated the improvements in efficiency and effectiveness of Polish Development Assistance.

For the upcoming years, it is e.g. recommended as follows:

to introduce systemic solutions that facilitate the implementation of projects and cooperation with the project initiators, including the implementation of multi-annual projects and increasing their scale;

to put more emphasis to the comprehensive and methodical needs analysis at the stage of preparing project concepts;

to re-implement the supporting scheme for Polish entities in the area of obtaining financing for development projects from the sources other than the state budget of Poland by providing a part of the own contribution required by the grant donors;

to increase the involvement of diplomatic posts in the implementation of the Polish Assistance measures, e.g. by creating dedicated positions in priority countries;

to improve the presence of the Fund in priority countries of Polish Assistance, where the Fund’s operations will be carried out;

to introduce a clear division of tasks between MFA and the Fund, which will allow you to avoid overlapping of measures financed from different sources;

to improve the synergy among Polish Development Assistance projects implemented in the given country;

to improve the relationship between the effects of the projects with the business benefits of Polish entrepreneurs.

Component II

The implemented measures were consistent with the policy of introducing reforms by Moldova intended to bring the country closer to the European Union. The effects are most clearly visible in the projects related to the development of agriculture and entrepreneurship in rural areas. It concerns both the initiatives undertaken by the administration as well as by the Polish local governments and NGOs. The projects were mostly well-adapted to the needs of their beneficiaries, which resulted from a good knowledge of the conditions in Moldova and from the experiences of individual coordinators. The results are sustainable; which is mostly related to a high involvement of the Moldovan partners. From the viewpoint of the conducted measures, it was also important that Poland was perceived as a more reliable country than other assistance donors due to the recent transformation experience.

Detailed analysis was conducted with respect to the results of the project that involved establishing and running an Information Center for the local authorities in Ialoveni. The evaluators assessed the operation of this facility very highly.

The initiatives realised under the Small Grants Scheme were accurate, usable, effective and sustainable, yet consideration should be given in the future to improve their relation with the objectives of the Polish Development Policy.

For the upcoming years, it is e.g. recommended as follows:

as regards projects implemented by the administration, to put emphasis on granting support in drawing up acts of law or in implementing the existing law, while resigning from the projects that focus on improving the qualifications of the employee of the institution;

to continue the business support projects in rural areas and the implementation of ecological technologies;

to implement projects that facilitate combating corruption;

to introduce systemic solutions that facilitate the implementation of projects, including the introduction of multi-annual financing and strengthening the cooperation with the project initiators;

to improve the potential of Polish organisations operating in Moldova, e.g. by supporting them in the application process for external funds;

to increase the involvement of the Polish Embassy in Kishinev in conducting Polish Assistance activities;

to further support the operations of the Information Center for the local authorities in Ialoveni;

to undertake actions intended to involve Polish business in development projects;

to improve information-promotional activities concerning both the Polish Development Policy and individual projects;

to consider ending the implementation of projects in the field of democracy support in Moldova;

Evaluation in 2014

In the year 2014, the Ministry of Foreign Affairs has conducted an evaluation of tasks of Polish development cooperation covered by the Polish Aid programme 2011 and the plans for development cooperation 2012 and 2013. The evaluation concerned specific projects financed from the funds provided by the MFA under the scholarship programme of the Specialist Eastern Studies at the University of Warsaw, and also development aid programmes for small and medium sized enterprises in Georgia and Ukraine as well as a co-production agreement between Belarusian Radio Racja and the MFA for the year 2013.

The evaluation was performed in Poland, Ukraine and Georgia.

The main aim of the review was to formulate recommendation allowing, among other things, the increase of Polish aid (i.e. to increase the influence of projects on the beneficiary’s situation), to strengthen the image of Poland as a country active in measures for sustainable development worldwide, and also to support the decision-making processes at the MFA.

Developmental initiatives implemented as part of four Polish aid components in the years 2011, 2012, 2013 were positively evaluated. The actions met the beneficiaries’ actual needs. The Polish Development Aid offer was seen as attractive, specifically tailored to attain the goals. Competent execution of project tasks positively influenced the image of Poland as a development cooperation donor.

The analysis of realised ventures shows that:

The Scholarship Programme for Students of Specialist Eastern Studies at the University of Warsaw,managed by theCentreforEast EuropeanStudies at theUniversity of Warsaw, meets the needs and expectations of beneficiaries. Its graduates advance their scientific and job careers, thus proving the efficacy of the undertaking. The realization of the initiative contributes to the preparation of leaders of change in the participants’ home countries, and also brings mutual benefit for Polish image abroad. The programme, thanks to its long tradition and high quality (according to the students) of offered services, has become a recognised mark in participants’ home countries.

The Georgian component of SME ought to be continued in the future. The analysed projects were properly linked with the goals of Multiannual Development Cooperation Programme for 2012-2015 and the Millennium Development Goals. Priority of help went to particularly worse off target groups and areas in deep socio-economic hardship. The most permanent effects resulted from on-site visits to Polish enterprises and agro-touristic farms accompanied by training, which furthered the competences gained thanks to the project participation. The main reason behind limiting the permanence was a lack of cooperation from local authorities, and a more difficult access to credit financing the development of newly formed enterprises.

The Ukrainian component of SME was also highly valued by all the participating parties. The majority of organisations have been working in Ukraine for years, where they have their trusted partners. On the other hand, the problem of entrepreneurship development was not usually the main activity area for these organisations. The turning to SME stemmed from adjustment attempts by these organisations, per their statutory tasks, to adjust their actions to the priority within the framework of which it is possible to receive funding. At the same time, the projects were characterised by an accurate diagnosis of needs, while their execution by the professionalism and commitment of contractors.

Belarusian Radio Racja (BRR) undertakes a wide range of actions to improve access to objective information for target groups. The radio is realising its aim mainly by broadcasting programmes to the territory of Belarus. The radio is complemented in its mission by a website, profiles on Facebook, Twitter, Contact, YouTube and Google+. The radio broadcasts almost exclusively in Belarusian. The scheduling and specific programming blocks are, in spite of limited funds, worked on and modified based on sociological research conducted on the territory of Poland among Belarussian expats. BRR cooperates with many local correspondents and institutions operating in Belarus as well as abroad.

Concurrently, in the process of evaluation, conventions and recommendations were formulated, the implementation of which may result in a strengthened impact of the Polish Aid Programme and even better adjustment of projects to the beneficiaries’ needs.

Evaluation in 2013

In 2013, the MFA continued its evaluation of Polish development cooperation. Projects carried out in 2010-2012 focusing on regional development, public and local administration capacity building in Ukraine and systemicglobal education in Poland were evaluated in terms of their effectiveness. The horizontal effect of human rights measures that were carried out as part of these initiatives was also evaluated. The projects selected for evaluation had been carried out with the participation of Ukrainian partners in the territory of the Autonomous Republic of Crimea and in the Lviv and Vinnytsia Oblasts (regional development support) and in Poland (educational projects).

The main purpose of evaluation was, on the one hand, to define the role played by selected development initiatives in enhancing the economic and social potential of partner countries and, on the other, to identify the factors that affect the implementation of Polish projects. Its results can also be used by potential applicants and other participants of projects implemented under Polish development cooperation.

The evaluation has reaffirmed Polish development cooperation’s positive image, identified elements which after minor adjustment would additionally boost this positive image and outlined a possible reorientation in the programming and implementing of development initiatives over the next years. It also identified possible changes in assistance initiatives which could effectively improve Polish development activities.

The evaluation has shown specifically that:

Projects carried out in Ukraine

responded to genuine needs. Their beneficiaries gained knowledge about regional development and rural areas, the development of associations, activation and self-organisation of local communities, brownfield regeneration and internal audit. They have also improved their skills in drafting sustainable development strategies and regeneration/renewal plans, project applications, as well as conducting internal audits or setting up associations.

have blended well with measures implemented by the beneficiaries and also effectively complemented initiatives carried out by other donors. The applicants realized their planned objectives. They were able to do so because they were well prepared, had experience and tapped into the capacities of their Ukrainian partners. The implementation of projects has contributed to enhancing these capacities.

provided a development impetus at the local level and have helped to create an image of Poland as a modern and open state.

Global education projects implemented in Poland:

were diverse and were implemented on a wide scale and adjusted to the situation and the needs of the beneficiaries (teachers, students, academics). The projects achieved planned quality and quantity objectives. They have provided practical and useful knowledge that schools can apply. However, they did not bring about a structural change in the education system. It is up to the teachers to decide whether they want to introduce global education into the learning process.

have contributed to a change of attitudes on the part of the participants of the implemented projects. However, this change is clearly visible only in the people who directly benefitted from the assistance. One-off interventions that lack mechanisms that would make the changes permanent risk quickly losing people’s interest and do not leave lasting imprints in the form of changed attitudes and motivations to act.

have increasingly relied on modern technologies, but there is still room to tap into their full potential. E-learning trainings have proven the most popular. Thanks to such measures knowledge about global education can be developed on a wide scale.

The majority of evaluated projects accounted for human rights issues. The direct and indirect presence of this subject matter in implemented projects and its interpretation varied.

Evaluation in 2012

In 2012 the evaluation of programmes and projects implemented by Polish development cooperation focused on two areas: SENSE (Strategic Economic Needs and Security Exercise), a training programme implemented in 2010–2011 and organised for seven countries: Afghanistan, Armenia, Azerbaijan, Georgia, Moldova, Tunisia and Ukraine, and a group of projects that in 2009-2011 supported a programme that helped to change Georgia’s state-owned child care institutions into a system of foster family care and led to the creation of innovative institutions resembling family institutions.

The main aim of the evaluation was to assess the impact of development assistance projects financed through the Polish MFA on selected elements of the situation in the country of the beneficiary (including good governance and the development of the social sector). Recommendations were formulated on the basis of collected and analysed information that enabled increasing the impact of Polish development initiatives. The evaluation results can be used by different stakeholders, including those who implement development cooperation to draw up and carry out projects and to cooperate in partner countries.

Evaluation feedback relating to examined areas has shown a positive image of Poland’s development cooperation. At the same time it helped to identify elements that after relatively little adjustment could additionally boost this positive image. In addition, a possible reorientation in planning assistance initiatives in the coming years has been outlined. In particular:

SENSE has fulfilled its role as an effective assistance tool used to improve the quality of governance. However, due to the specific nature and the progress made by the EaP countries, the countries of this region will be excluded from the programme and will be included to a greater extent than before in initiatives carried out as part of the Eastern Partnership Academy of Public Administration.

The model of new forms of work with children and the family in childcare institutions that was suggested in Georgia has been accepted and represents a lasting outcome of partnership cooperation. This shows Poland’s potential in this area.

Both the SENSE programme and projects carried out in Georgia have significantly contributed to building a positive image of Poland, Poles and the Polish foreign aid programme.