Security Sector Development Programme for Burundi (SSDP)

Target Country(ies)

The Security Sector Development Prgramme is a three-year engagement for the Dutch in Burundi. Its objective is to deepen Netherlands-Burundian cooperation on the development of the Burundian security sector.

Target Organisation Type(s)

Primary Activity(ies)

Outcome Objectives

On the short term, the SSD Programme aims to relieve the Burundese army and police of some urgent operational needs through amongst others training, strategic advice, and the provision of infrastructure and material. Besides, the SSD Programme contributes to the strengthening of democratic oversight of the Burundese security forces and the inclusion of civil society herein. The long term objective concerns improvement of the security of Burundese civilians through strengthening the legitimacy, effectiveness and quality of the Burundese security sector.

Start Date

01.01.2009.

End Date

31.12.2011.

Support Mandates

Train the train course for Army officers who succeeded in completing the UNITAR on line SSR course and who will deliver SSR training to Army officers and civil officials living in the neighbourhood of the Army barracks.

This evaluation mandate, jointly commissioned by the Netherlands, the Government of Burundi and the BNUB, analyses the efforts made by the Burundians and their internationals partners to reform the security and justice sectors since the Arusha Peace Agreement signed in 2000. The main objectives of this evaluation were:

to provide an overview of the evolution of the reform process from 2000 to 2013;

to identify the results obtained and the remaining challenges ; and

to formulate recommendations to the Burundian Government and its international partners in order to improve the SSR process.

To carry out an analysis of the Inspector General of the Police in Burundi in cooperation with the Belgian Police in order to define recommendations on how the organisation, the structure and the functioning of the IG can be improved including defining the prioritised needs of the IG, while taking into account the limited budgetary means of Burundi as a country and its Police.

The Burundian Police agreed on an audit / evaluation of the IG and there was agreement reached that the recommendations of the audit will be taken into account while preparing and executing the 3rd
and 4th
phase of the SSD program which started mid 2014.

The Netherlands-Burundi cooperation was captured in a Memorandum of Understanding, signed in 2009. This MoU served as a framework for Netherlands-Burundi Security Sector Development programme (which included 3 pillars: Ministry of Public Security, Ministry of Defence and Security Sector Governance) for a period of eight years (2009-2017). This period was divided into four phases, each lasting two years. A strategic plan was developed for each phase.

The objective of this mandate was to carry out an evaluation of the Burundi SSD programme Phase II and formulate recommendations for the preparation of the next phases.

More specifically, ISSAT was required to:

Evaluate the relevance of the Phase II objectives and the activities implemented to reach those objectives, from an SSD perspective and within the Burundi context.

Evaluate the impact of Phase II activities with regard to the principles of SSR. Formulate recommendations to improve the relevance and design of Phase III objectives, activities, including measurable indicators.

Evaluate the process of the Defence review (taking into account the recent study on the Defence review commissioned by the DSS).

Define lessons learned in Phase II so the international community of SSR can profit from the successes and less successful parts of the SSD programme until now.

The evaluation will also contribute to the assessment of the functioning of the SSD programme structure (efficiency, ability of the existing structure design to achieve the desired objectives) and formulate recommendations to improve the programme structure for Phase III.

The target audience of this evaluation is the management units of the SSD, the responsible units of the programme within the Dutch MFA (Embassy and DSH). The report will also be made available to a wider external public in Burundi and abroad.

The Government of the Netherlands requested the assistance of ISSAT to support their Baseline Study of the Security Sector in Burundi as part of the Netherlands-Burundian Security Sector Development (SSD) Programme. ISSAT was requested to provide a military specialist with knowledge of the security/SSR context in Burundi.

Burundi and the Netherlands have cooperated on security sector development issues since 2004, which was formalised through the signing of a long term Memorandum of Understanding (MoU). As part of the MoU, the Netherlands and Burundi agreed to execute a baseline study of the security sector, to be undertaken by independent experts in the field of SSD, in close cooperation with Burundian and Dutch representatives.

The baseline assessment served two purposes:

To improve the understanding of the real and perceived of security needs, from an individual, institutional and political perspective. The study should also at the implications this will have for security provision, potential risks and challenges.

To provide a basis against which the results of the Dutch-BurundianSSDprogramme can be evaluated on an annual basis as well as at the end of the programme, and against which future activities can take shape.

The Government of the Netherlands requested the assistance of ISSAT to support their Scoping Mission for the Netherlands-Burundian support to Burundian Parliamentary Oversight over Security and Defence. This was a follow-on to the previous request for assistance for the baseline study. ISSAT was requested to provide an expert with knowledge of ethnicity and the security/SSR context in Burundi.

In addition to the ongoing baseline study, the Netherlands came up with a number of quick win/confidence building projects for implementation between 2009 and 2010. The proposals included responding to the request to provide technical assistance from the Parliament, the Civil Society and the Ministry of Foreign Affairs, by:

Supporting the Parliament in holding a hearing on the issues of ethnic balance within with security services;

Organising a seminar for civil society on civil involvement in/oversight of the security forces;

Organising a seminar on SSR for the Burundian Ministry of Foreign Affairs, to assist their role as coordinators of the national SSR process.

The Government of the Netherlands requested ISSAT assistance to support their Scoping Mission for the Netherlands-Burundian support to Burundian Parliamentary Oversight over Security and Defence. This is a follow-on to the previous request for assistance for the baseline study. ISSAT was requested to provide an expert with knowledge of ethnicity and the security/SSR context in Burundi.

In addition to the ongoing baseline study, the Netherlands came up with a number of quick win/confidence building projects to be implemented between 2009 and 2010. The proposals included responding to the request to provide technical assistance from the Parliament, the Civil Society and the Ministry of Foreign Affairs, by:

Supporting the Parliament in holding a hearing on the issues of ethnic balance within with security services;

Organising a seminar for civil society on civil involvement in/oversight of the security forces;

Organising a seminar on SSR for the Burundian Ministry of Foreign Affairs, to assist their role as coordinators of the national SSR process.

The Netherlands-Burundi cooperation was captured in a Memorandum of Understanding, which was signed in 2009. This MoU served as an umbrella for Netherlands-Burundi Security Sector Development Programme (SSDP) - which included 3 pillars: Ministry of Public Security, Ministry of Defence and Security Sector Governance, for a period of eight years (2009-2017). This period was divided into four phases, each lasting two years. A strategic plan was developed for each phase. The programme was currently concluding phase I, with phase II officially commencing on I January 2012.

The overall objective of the mission was to assist the Netherlands Embassy in Burundi to evaluate the impact and results of Phase I of the Security Sector Development Programme. This included an analysis of the efficiency of the project implementation structures. The secondary objective of the mission was to assess the project design and relevance of Phase I of the SSDP. This assessment will be done to provide recommendations on any necessary amendments or alterations to the project. Such recommendations would reflect contemporary field conditions and lessons learned from Phase I.

The mission was divided into three parts – which would reflect the original SSD design: support to police reform, support to the reform of the army reform, an evaluation of the programme structure and coherence with SSR processes.

The Netherlands and the Burundian government are developing a joint 8 year SSD programme, structured in four phases of 2 years each. The ongoing work deals with 3 different fields: defence, police, and governance. Both parties are currently working onthe second phase of the project. They have requested ISSAT advisory field support to help integrate SSD good practise into present discussion.

The Netherlands-Burundian cooperation was captured in a Memorandum of Understanding, signed in 2009. This MoU serves as an umbrella for the Netherlands-Burundian Security Sector Development programme that has 3 pillars: Ministry of Public Security (incl. the Burundi National Police), Ministry of Defence (incl. the Burundi Defence Forces) and Security Sector Governance and that covers a period of eight years (2009-2017). The programme was currently entering it phase II, for which strategic action plans are being developed.

The Dutch Embassy Office in Burundi sought advisory field and training support from ISSAT to run two trainings for military and police officers who were responsible to draft the strategic plans of the military and the police. The objective of the two trainings was to ensure that the participants understood the concept of SSD and its main principles, and that they would be able to apply the SSD principles in their daily activities, particularly when drafting the strategic plans.

ISSAT supported the Dutch SSD Programme in Burundi to design and implement two 2-day training workshops for the military and the police to achieve these objectives. The two workshop was structured as follows:

The participants included 20-25 military officers and around 30 police officers. The military participants included officers in charge to draft the strategic plans on the basis of the National Defence Review; some members of the MoD executive secretariat in charge of drafting the National Defence Review, and some national members of the management unit of the Dutch SSR programme. The police participants included officers in charge of implementing training projects and general staff officers in charge of planning.

The mandate was part of the on-going mentoring support that ISSAT is providing to the Burundi SSD Programme.

The overall objective of the mission was to assist the Netherlands Embassy in Burundi to evaluate the results and impact of Phase I of their Security Sector Development Programme (SSDP). This will include an analysis of the efficiency of the project implementation structures, its design and relevance to its designated beneficiaries.

Mandate

Summary

Informal Assessments: In order to fine tune a workshop to your audience, informal conversations with national actors and donors is a very good methodto understand the strengths and weaknesses of an audiences ability to absorb in concepts and willingness to accept those concepts.

Balanced Partnership: The Dutch programme overall is with an end goal of moving from an equal partnership in planning to eventually the Dutch taking only a technical advisory role of the programme, which puts local ownership and sustainability as a top priority for the programme.

New Concepts in Mid-Workstream: Be aware of the affects on local ownership trying to introduce a new concept can have that will alter work which has already been done. Be sure to have a mitigating strategy ready so that concepts can be integrated smoothly and a positive working relationship can be maintained.

Resources

OECD DAC handbook 2007, UN General Secretary report 2008, EU communication on SSR from Commission to the Council and Parliament 2006, “the SSR in a nutshell” from ISSAT.

Other Comments

The team noted a huge will to build a genuine SSD programme in line with all the SSR principles.The team also observed a true local commitment due to the balanced organisation of the teams in place. There were some fears however about the management of third pillar on governance, which is essential for the success of the whole programme.

Pre-deployment preparation for a team is crucial for the success of the mission and for the team to be flexible enough to deal with the challenges of the programme. Without a well-thought-through plan which is flexible enough to allow for adaptation, the team might lose precious time trying to make sense of all the information they are receiving without having an appropriate structure to absorb that into.

Key milestone during that phase are: desk research, identifying main challenges and issues to focus on, designing an overarching methodology and report structure.

Team working sessions at the end of each day for the period of at least one hour, are key to the success of a mission allowing the advisors to go through the information they have received in a more structured way and facilitating cross learning as well as ensuring that the team maintain common objectives in sight and move toward common objectives.

Translation of terms between different languages is tricky enough; this can be made even more complex if you add the technical terminology of an evaluation and that of the armed forces institution or the police institution. Advisors deployed to evaluation mission and need to conduct interviews with stakeholders should be very knowledgeable of the technical terminology, as well as the cultural context so as not to create confusion.

Involve ISSAT team more during missions: Under the guidance of team leader involve ISSAT resource people, before, during and after a mission through getting their input on mission design and asking for their support during implementation. After mission completion, informal debriefings, templates for reports, in-house comments on reports, and usage of the CoP for innovative solutions would be a positive way forward for ISSAT.

Other Comments

Bernard mentioned the possibility of doing a video with him tackling the differences between the Police institution and the military institution and how the latter is more open and transparent than the former. Aspects that he believes are very much universal to these institutions due to specific reasons he could detail in that video.

Lessons identified for improving the chances of success:

Recommendations can be discussed with local stakeholders in a way that allows for some knowledge transfer on programme design, in the case restricted to the elaboration of an action plan for ‘quick-wins’. Such an exercise can bridge recommendations with whichever projects that SSD decides to carry forward following the review.

SSD needs to act on the audit in terms of pushing to have some of the recommendations followed on; this implies that the Programme has to keep representing to the Burundian authorities that crucial decisions have to be taken and operationalised soon to have IGSP fulfilling its important role.

Lessons identified for future audits:

Civil society organisations are usually keen to engage in discussions about internal control mechanisms of police services. Such occasions offer entry-points to bring-in external stakeholders, otherwise kept fairly outside relevant processes and activities, as it happened this time, in spite of clear demands from ISSAT for such opening from the Burundian authorities.

Training in internal control and democratic oversight, linking up internal audit bodies with the role of external stakeholders (Parliament, Ombudsman, etc.), are the natural and crucial follow-up of this mandate. These activities can be a contribution from ISSAT to pursue support in areas identified as major capacity gaps, and keeping the momentum for a reinforcement of the SSD Governance pillar. It’s worth reminding that “Internal Control” goes on being one of the focus areas under the action plan for ‘Police’ pillar of SDD Phase III.

Dedicate the first day of the team in the field to set up interviews and organise agenda, since gaps in pre-arranging interviews are recurrent and almost unavoidable in such contexts.

Other Comments

Innovative approaches

The mandate implied an appraisal of actual mechanisms to build accountability of a police service in a post-conflict context; for ISSAT, this broadened and nuanced the focus areas that have to be considered when working in police and justice reform.

Workshops included in the mission, focusing on the use of analytical tools and project design, even if kept at an introductory level, can provide opportunities for knowledge transfer and build a sense of ownership from local stakeholders, while not substantially raising the overall budget of the mandate.

Specific Lessons Identified

Lessons Identified

What worked well

What was challenging

Recommendations

Presence of ISSAT PO who developed the ISSAT evaluation methodology on team ensured a active implementation of this methodology when designing the mandate

This was a big and complex assignment and the work load was consequently very important.

In PO’s opinion, this type of evaluations takes between a year and a year and a half’s time and costs around 300 000 - 400 000 EUR. In this case it was completed in 3 months time. As a result, team had to make decisions relating to what level of detail to get into.

Scoping mission should have narrowed down even more the scope of the evaluation.

Ensure there is always a PO from the newly formed “methodology cell” collaborating with mandate teams on methodology design for mandate.

ISSAT should better assess workload of such evaluations and plan accordingly.

There was enough time during the design phase. Team needed more time to integrate the necessary working sessions for the methodology design.

One more week in the field would have been useful for team which would have been used to appropriately plan meetings in the beginning and have sufficient time to go through all meetings without rushing through or skipping meetings.

Also there was not time for informal meetings which was a loss for the team since this is the time where fruitful discussions happen and networks and relations are built.

CD did not set up a schedule of meetings prior to team deployment.

Try to ensure that some preliminary version of meetings schedule is ready before deploying so as not to risk having to set up meetings while in field.

Methodology design

What worked well

What was challenging

Recommendations

After initial surprise by team members concerning the methodology’s scope, everybody was willing to work within this framework. Given the complex and wide nature of this evaluation, having an explicit, robust and detailed methodology the team about not omitting or overlooking certain issues.

The methodology was very useful for team members who had thematic skills on certain sectors but lacked or had limited evaluation technical skills. As a result, learning from this experience was very high within this team.

Having a clear and explicit methodology helped have every team member on the same page concerning what kind of information to look for, what are the gaps, which phase is the evaluation at, etc.

For team members who were not part of the evaluation methodology development process, it was more difficult to be up to speed on how and why the mandate’s methodology was designed.

Ideally, ISSAT should be able to organize a working session bringing together all team members (ISSAT, Roster and Mandator) to present the ISSAT evaluation methodology and explain how it will be adapted to the mandate so all team members feel ownership of the design phase.

This recommendation should be applied with additional emphasis regarding non-ISSAT core team staff.

All team members were aware of all of the evaluation questions even if they were just working on only one. This helped keep the big picture in everybody’s minds, including being aware of remaining gaps and highlighting overlaps and complementarities.

The methodology was very important to justify the result, in particular because this evaluation was very political. Recommendations had to be tied into robust data collection methods and objective processes to generate recommendations.

However, the fact that references to the methodology were made only in the annex, people could not instantly see the origin of a certain recommendations.

Reference to evaluation questions or other methodological guidelines should be made in the text if possible in addition to mentioning them in an annex so as to give the reader the possibility to instantly view the origin of recommendations without having to go back to the annex or the methodology section in the beginning.

This is even more significant knowing that most people might not read the methodology section in the beginning or the annexes and find it more useful to see the linkages between the methodology and the recommendations in the text.

Questions, sub-questions and indicators were approved by the Comité Directeur which gave the approach a lot of legitimacy.

However, these being signed by the CD made them less flexible in the team’s consideration. Once in the field, the team would have preferred to have some margin to adapt the sub-questions, add or take out certain indicators in view of the new information they had collected.

Going forward, evaluation team should be aware that sub-questions and indicators can be amended for valid reasons once those reasons are explained to Going forward, evaluation teams should be aware that sub-questions and indicators can be changes after data-collection starts. However, the main questions cannot be changed once in the field and have to remain as they are.

This aspect has to be clearly explained to partners and even included in ToRs if need be.

This is the first time an ISSAT mission uses the in-house methodology. Until this mission, each ISSAT evaluation was using a different methodology.

It is a risk for future mandates to design their own methodologies and not fully use the ISSAT inhouse one. This will mean that each evaluation mission will have its own methodology and encourage inconsistency in ISSAT’s approach.

Security-Justice-governance linkages were the focus of this evaluation

Scoping mission

What worked well

What was challenging

Recommendations

Scoping mission was very useful inorder to narrow down the scope of the methodology. Since Mandator was not clear on that level, a scoping mission was necessary to move forward.

The scoping mission was also very useful in terms of informing the partners of the objectives of the mission and ensuring they are aware of the big picture.

However, an activity beyond the mere informative scoping mission would have been very useful time-permitting. A workshop with the local partners presenting the methodology and how it will be used would have helped get more support and buy-in from the Commité Directeur concerning the methodology. They would have been better able to supply comments on the substance and would have been better able to understand where the recommendations come from and how they were formulated.

If time permits, plan for a working session with local partners to present the methodology approach, get their views and adapt as necessary. This should be more than just information sharing but a capacity building activity enabling the partners to contribute more meaningfully to the mandate’s methodology.

Implementation

What worked well

What was challenging

Recommendations

If team is under time presssure and is unable to type up notes from meetings, it is a good solution to fill in the evaluation grids directly after each interview

The desktop review and research phase was underestimated, the team should have given more time to that phase and could have even started filling in the evaluation grids and then used that to test and update it once in field. This would have saved a lot of time

Try to do an exhaustive research and desktop review phase before deploying. At the end of which, team should be able to fill up the grids on a tentative basis, after which those grids would be reevaluated and updated in view of information collected in the field.

Main findings from the evaluation should have been presented to the CD before the report was produced to get their views and so there are no surprises when the report is out

Present main findings to local stakeholders before typing up the report to avoid any surprises.

Team overcame the challenge here by grouping certain sub-questions and indicators when possible

The methodological framework was very big which resulted in a huge scope for the evaluation. This though could not have been avoided due to the nature of this evaluation. Had this been targeting a limited programme with known objectives and outputs, team would have been able to better select what is relevant and what is not.

If pressured by time, and only if applicable try to group certain sub-questions and indicators when possible

Working with Comité Directeur

What worked well

What was challenging

Recommendations

CD did not supply comments to the methodology approach once it was sent to them. This might be a result of several reasons, amongst others:

CD unable to fully comprehend this methodology and fully understand its links to the evaluation report and recommendations

OR

CD unwilling to provide comments for reasons we shall examine here.

Same recommendation as above pertaining to workshop with local stakeholders is applicable here.

Team should have done bigger effort to push the CD to react to the methodology and only deploy to the field once they have received the insights from CD.

The CD was set up to oversee the evaluation, was consulted by teh team on teh objectives of teh evaluation and endorsed the methodology.

The Evaluation report

What worked well

What was challenging

Recommendations

This evaluation targeted a local process but the report and methodology were drafted by internationals. The report should have been shared with Burundian stakeholders for a first read to get a sense of how the recommendations will be interpreted on the Burundian side. This step would have been very useful in terms of fine-tuning the language and making sure the recommendations are clear enough and cannot be mis-interpreted.

It might be useful in some cases to think about having the evaluation report read and commented by a local stakeholder; in particular if the evaluation is a highly political one. Language and presentation of certain ideas by internationals might be misinterpreted by nationals.

Team did not discuss thoroughly who will be the owner of this report. They had decided that the CD will be the owner but team would have benefitted from further discussion relating to what that fact would entail. Team should have clarified amongst each other certain boundaries or red lines beyond which they refuse to make concessions. There is a risk that the report is altered completely by the local stakeholders and team should be aware and ready to react to that.

It is useful for the team to engage in a detailed discussion relating to identifying the owner of the report and the consequences of that.

The team lacked clear vision as to the end usage of the report. Why was such a report drafted and to be used by whom and for which ends?

Going forward, evaluation team should try and clarify the usage of the end report: why is it being produced, to be used by whom and how?

Political Coverage of ISSAT on evaluations

What worked well

What was challenging

Recommendations

While the recommendations of the report were being presented, ISSAT did not benefit of full political coverage by Mandator as should be standard practice.

ISSAT should not take the political responsibility of justifying evaluation recommendations to national stakeholders, in particular in this highly political situation. This responsibility should lie with the mandator and this should be made explicit in the ToRs.

Tags

Outputs and Outcomes

This mandate fully delivered the expected outputs, including an audit report with recommendations in accordance to the mandator’s needs. The recommendations were taken already in consideration and two subsequent missions from Polfed to Burundi worked on how to implement some of them. The reaction to the recommendations was immediate and positive also from the IGSP, which already took suggested points of action in consideration and welcomes the clarification of roles and responsibilities between the Inspectorate and SSD. During the mandate, the position of Inspector-general was vacant but the new head of IGSP took office in the same day of the end of the field mission. A working group was set up just after the restitution session to discuss a set of activities aimed at obtaining quick-wins on the recommendations and that ISSAT proposed to be carried out until Q4 2014.

To this date, the IGSP still didn’t act on most of these actions and further steps depend on the completion of the new legal framework for the IGSP. The Organic Law and the Internal Statute of the IGSP have been drafted and provide finally a legal framework for organising the work of the Inspectorate, although a decree on the function of internal audit is still to be drafted. Until the legal framework is approved and fully in effect, the scope of activities of IGSP is nonetheless very limited, specially when it comes to investigations and filing of cases.

The extent to which recommendations are fully implemented will need to be checked in one or two years, since a previous audit to IGSP was basically left in the drawer in terms of acting on suggested points. The review report delivered by the team provides a strong base for the design of SSD projects in support of the IGSP but it also implies and needs that the mandator follows up on the results by pushing for tangible actions (see below).

Throughout the report, attention is given to technical and procedural notions that help unpack the concept of internal control and that lay out a comprehensive set of principles, mechanisms and tools. This corpus can be easily taken as part of training materials for IGSP staff and the PNB overall, as acknowledged by the SSD/MSP coach visiting ISSAT in October to discuss further areas of collaboration related to training in police integrity.

As part of the second/main field mission, in March 2014, a one-day workshop was organised by the team in Bujumbura to conduct a SWOT analysis of the IGSP with a group of fifteen staff of the Inspectorate. The SWOT exercise was considered to provide simple yet powerful analytical tools, with gains in terms of ownership of the review itself.

The evaluation team was assisted in Burundi by two members from IGSP that facilitated contacts and participated in several meetings. Future will tell what will result from this exposure of IGSP staff to ISSAT’s evaluation methodology and to intense discussions that articulated the principles of internal control with the realities on the ground and the political and social context in Burundi.

For ISSAT, the fact that the evaluation team included two experts working actually in police internal control in their own country provided an opportunity to gain a more technical view of the questions involved. It is considered a relevant gain of capacity for future reviews and an important enhancement of existing in-house knowledge on police reform, integrity and responsibility.

It is the perception of the IGSP that the mandate resulted in opening some doors to potential collaboration of the institution with civil society organisations.

Success factors

A good preparation of the mandate, starting with a one-day meeting in Geneva aligning approaches and planning work among Polfed and ISSAT experts on team, starting from scratch and designing the mission.

Scoping visit from the team leader to Burundi.

Wealth of local/contextual knowledge in the team.

Mixed team composition bringing complementary skills and approaches to democratic control mechanisms: technical/Polfed and SSR/ISSAT.

Trust build over time between ISSAT and SSD in Burundi.

Good local interaction between SSD, CTB (Belgian development agency) and GIZ (German aid agency).

ISSAT methodology.

Facilitation from IGSP staff.

Lessons identified for improving the chances of success:

Recommendations can be discussed with local stakeholders in a way that allows for some knowledge transfer on programme design, in the case restricted to the elaboration of an action plan for ‘quick-wins’. Such an exercise can bridge recommendations with whichever projects that SSD decides to carry forward following the review.

SSD needs to act on the audit in terms of pushing to have some of the recommendations followed on; this implies that the Programme has to keep representing to the Burundian authorities that crucial decisions have to be taken and operationalised soon to have IGSP fulfilling its important role.

Tags

Main challenges

Concepts and processes– Uneven understanding among different stakeholders, within the IGSP and across the MSP of the role, importance and mechanisms entailed by internal control in a functioning democracy.

Political engagement– As noticed in the AAR, even the most immediate quick-win activities recommended in the report need that IGSP has a legal framework in place, but political commitment from Burundian Government to move forward on this has been vague, in fact compromising IGSP critical role.

Programme support– The audit sets clear directions and suggests an action plan to address fundamental gaps in internal control, but any progress on this depends on a robust and immediate support from SSD to democratic oversight and accountability.

Programme overstretch– SSD is a complex, far-reaching SSR programme spanning for eight years and supporting the main security institutions in Burundi (Police and Defence Forces), but there is the risk that whatever is not included or prioritised and funded under SSD is not purposely sustained by the Burundian counterparts.

Uneven and unused capacity– IGSP badly lacks capacity across the board when it comes to main and secondary/procedural functions of an internal control body, from legal expertise to strategic management, while some of the actual existing capacity among staff is lost by the way IGSP functions.

Lessons identified:

Civil society organisations are usually keen to engage in discussions about internal control mechanisms of police services. Such occasions offer entry-points to bring-in external stakeholders, otherwise kept fairly outside relevant processes and activities, as it happened this time, in spite of clear demands from ISSAT for such opening from the Burundian authorities.

Training in internal control and democratic oversight, linking up internal audit bodies with the role of external stakeholders (Parliament, Ombudsman, etc.), are the natural and crucial follow-up of this mandate. These activities can be a contribution from ISSAT to pursue support in areas identified as major capacity gaps, and keeping the momentum for a reinforcement of the SSD Governance pillar. It’s worth reminding that “Internal Control” goes on being one of the focus areas under the action plan for ‘Police’ pillar of SDD Phase III.

Dedicate the first day of the team in the field to set up interviews and organise agenda, since gaps in pre-arranging interviews are recurrent and almost unavoidable in such contexts.

Challenges

Integrating Concepts Mid-Workflow: The workshops took place after the action plans for the military and the police had been written. For internationals external to the programme to come in after the fact to recommend new concepts for inclusion can come across as critical of national efforts and undermine local ownership. A very good communication strategy amongst the team, coupled with strong high level political support, is needed to mitigate the negative impacts of coming in late.

Successes

Partnering and Transferring Responsibility: The overall structure of the eight-year programme is designed to start with a shared responsibility between the Dutch and the Burundians, and progressively have the Dutch play more of a technical advisory role.

National Coordinator: The Burundian co-coordinator of the programme is very competent and has good relationships with both the Ministers of Interior and Defence. This allowed the programme to have substantive impact.

Tags

Challenges

Politics of SSR: Conveying the politics of SSR proved to be a concept that was difficult to transfer as not all uniformed participants viewed themselves as having a political role, rather they are neutral. While encouraging, there are still political aspects for all actors involved in SSR. This includes trainers wishing to convey the concept.

Successes

ISSAT Training: Portions of ISSAT’s Level 1 training were used to convey the key principles of SSR as well as fundamental components of supporting an SSR programme. The materials were well suited to be adapted to a specific audience.

Recommendations

New Concepts: There will always be a challenge in instilling a new concept or methodology in people unfamiliar with the SSR approach. Make sure to be aware of their willingness and ability to take on a new concept. These two indicators can help in planning an initial engagement as well as follow on actions.

Politics of SSR: Be aware that discussing the politics of SSR is in itself political and plan accordingly.

Tags

Successes

Clarity: There was a very good working relationship among the team as roles, responsibilities, and objectives were all made clear ahead of the mission.

Recommendations

Uniform Culture: When working with security actors, different uniforms (i.e. police, military) hold different traditions and institutional mentalities. If you can be aware of these factors well in advance, respect them, and work with them as opposed to working against them, then the collaboration with partners from these groups will greatly improve.

Tags

Challenges

Actions and words: The evaluation of impact is through a review of updated action plans for the military and police. The mission will be successful when we are able to observe evidence of SSR concepts from the workshop being incorporated into the updates. While this is one indicator that the concepts were learned, additional indicators need to be established to see if stakeholders can implement these concepts.

Tags

Successes

Informal assessment: Ahead of the workshops, the training team met with Dutch staff to discuss expected attitudes towards the SSR concepts among the Burundian team. The informal nature of this discussion was quite useful to get a quick assessment of training needs and absorbsion capacity, and enabled the training team to tailor the training material to the audience. Meeting informally, in this instance, as opposed to formal interviews also allowed the military and police officials to feel more relaxed and speak more openly about their thoughts.

Recommendations

Get to know the audience: As useful as the informal discussions with the Dutch team were, it would have been even more beneficial to have met informally with the Burundian counterparts. This would have given an even better idea of how to adapt the training, as well as established an initial rapport.

Tags

Challenges

Joint teams (international/national representatives): The Programme manager designed a theoretically very useful and positive approach for the evaluation which consists of deploying a national Security Sector representative to be associated to each advisor during the evaluation process. The objectives of such an approach were to –on the one hand- allow the Burundian counterparts to learn through actively participating in the exercise; on the other hand, to allow the evaluation to be anchored in solid knowledge of the context and the realities of the country’s security sector.

This however had varied implications in practice. Due to the absence of a clear structure for the national team members identifying their roles and deliverables, their contribution to the evaluation depended very much on variables such as: their relationship with the international advisor, personal dynamics, their knowledge of the process etc.

As a result in some cases they contributed by providing a local view and value to the evaluation and in other cases, they played the roles of logistics coordinators rather than contributing substantially to the evaluation.

This was due to the following factors:

The Terms of Reference for the mission, do include the designation of the Burundian team members and describe their role in a rather superficial way, not detailing what is to be expected from them after the desk review is done and the stakeholders identified;

After the three Burundian team members were designated by their respective institutions (police and armed forces), the team was not aware of any follow-up meeting with these individuals to brief them on their role and prepare them to proactively contribute to the design of the methodology.

The three Burundian team members were associated to an advisor in a way to allow the police officer to be on the pillar evaluating the army and vice versa. This did not work well in practice as the army officers were sometimes not comfortable speaking infront of the police representatives and neither were the police officers to the armed forces’ representatives. This led to instances where the Burundian team member was prevented to come to the meeting, or in some cases, to leave the room, while the international advisor carried on the interview regarding more sensitive issues.

The Burundian team members did not feel at ease or have the sufficient confidence in their role to be proactive in contributing to the evaluation. One of the Burundian officers did contribute slightly to asking questions but that was since he knew the international advisor well and felt comfortable enough to put questions forward.

Since the Burundians were not implicated in the process from the start, their learning during the interviews was very limited as they did not have the overall vision of the underlying approach and the long term objective of these questions.

Successes

Joint collaboration and consultations between ISSAT mission teams and In-house resource people: For this mission ISSAT M&E Expert supported directly the design of the evaluation’s methodology. This exercise was useful in two ways: It helped build the capacity of the ISSAT Advisor who had thematic expertise (Police Reform) but not the M&E knowledge and skills. The second benefit was related to the team acquiring an overarching guiding framework that facilitated their mission considerably. Without this preparatory exercise, the team could not have been able to deal with the complexities of the mission in a coherent manner which gave them the ability to respond to the challenges of the programme (absence of criteria for success, indicators, etc.) without losing sight of their objectives and without losing too much time in attempting to clarify how these considerations fit together.

Design of methodology and structure of the report before deployment: Having designed the structure of the report before deploying was also very helpful to the team as they could very rapidly and with ease collate the information they got out of their interviews and analyse them. This structure also facilitated the exchange of information amongst team members and their collaboration in terms of which piece of information fits where on the skeleton of the evaluation’s report.

In-depth exchanges with field programme leads and staff:The team leader for this mission reserved a considerable amount of time to work with the programme lead in the country. The objectives of these working sessions were to ensure that the mission team and the programme team had a coherent understanding and the same expectations from the evaluation. The draft ToRs were used as basis for discussion.

Recommendations

Joint collaboration and consultations between ISSAT mission teams and In-house resource people: Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:

Before missions: Working sessions and/or discussions with advisors as appropriate

During mission: Email interactions between mission team and Geneva team on targeted issues

After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). The purpose of such informal debriefing is to share the first observations and key lessons identified. An exchange and questions can follow during the debriefing. This exercise should be able to help the mission team to brainstorm on their findings and put their ideas together as they are writing their report.

It might be useful (depending on the missions) to consider spending one more week on every report and circulating among the ISSAT team for commen

Design of methodology and structure of the report before deployment: For evaluation missions, it is recommendable that the methodology and the report structure be designed before the team deploys to the field. However, depending on the programme, the methodology should be an overarching one proposing themes to be tackled and overarching questions to be asked without going too much into detail (such as designing indicators, or sub-questions). Necessary margins of flexibility should be left for the team to adjust the methodology as they arrive to the field and are briefed by the programme staff. Having a methodology before hand, helps the team conserve the guiding values of an evaluation (coherence, efficiency, etc.) and not lose sight of them.

Joint teams (international/national representatives): In terms of the initiative of creating evaluation teams that combine international advisors and national actors, it is the recommendation of this team to approach this matter in a much more structured manner through the following:

Once ToRs for the assignment are drafted, there should be a section dedicated to the international advisors role and the Burundian actors’ roles, detailing the responsibilities and roles of each group.

The ToRs should also include the value of including the national actors as a cross-learning process. The international advisors could pass on the comparative, international view and the modern technical know-how of evaluations and the Burundians could bring in the technical and cultural know-how of their security institutions and country. This would facilitate more realistic analysis and adapted methodologies to the reality of the country and programme.

In-depth exchanges with field programme leads and staff: Even if the ToRs are clear, the mission team should always dedicate time to discussing them with the programme team and other relevant stakeholders so expectations are harmonized amongst the group.

ISSAT templates:Disposing of an ISSAT templates for reports is very much needed. A report template per process would be very beneficial the teams (assessment report template, evaluation report template, etc.). This could include in each case a paragraph on what is ISSAT at the end which can serve as a useful reminder to mandators as well as help place ISSAT’s recommendations in their bigger picture (ISSAT’s mandate for reinforcing good practice).

Tags

Challenges

Newly formed teams: The challenge of newly formed teams is that members are not familiar with each other, their skills, competencies and methods of work.

See comments on Burundian team members’ contribution under methodology

Successes

The size and skills of the team: The team as it was defined worked very well in terms of its size and type of experience. Having 3 experts deployed for the evaluation of a big programme was the appropriate number as each advisor had one pillar to focus their efforts, while collaborating with the others on the other pillars. In terms of the skills and competencies deployed, this worked very well also. There was an expert on the thematic area (evaluation) and one expert on the technical questions (police and military) and one team leader who also dealt with governance aspects.

Pre-deployment preparation: All the team met in Brussels before deploying to Burundi to harmonize information, approaches and work plans.

Time dedicated for team work factored in daily: This team met once every day at the end of each dayto recap with the team main findings, challenges etc this helped harmonise the picture for all team members. These working sessions were of crucial value to ensure a certain coherence of work between members thus allowing for building on each other’s work as opposed to duplicating or collating information without knowledge of its value to the objective of the evaluation.

The Mandator’s role in evaluations: The mandators were very much hands-off when it came to the team’s work. They did not interfere with the design or planning of the mission and left the team entirely free in undertaking the evaluation.

Recommendations

Collaboration with the Burundian team members should have been managed better to get better results on both sides (see section on methodology and approach).

Time dedicated for team work factored in daily: This is absolutely key to the success of a mission. These meetings should be at least an hour long. These meetings were missing in a similar mission in Congo which had a big negative impact on the team’s work).

The Mandator’s role in evaluations: A hands-off approach by the mandator in terms of the substantive part of the evaluation and assuming the role of a partner resourceful of the historical background of the programme is a very positive framework to encourage.

Collaboration with the Burundian team members should have been managed better to get better results on both sides (see section on methodology and approach)

Time dedicated for team work factored in daily: This is absolutely key to the success of a mission. These meetings should be at least an hour long. These meetings were missing in a similar mission in Congo which had a big negative impact on the team’s work).

The Mandator’s role in evaluations: A hands-off approach by the mandator in terms of the substantive part of the evaluation and assuming the role of a partner resourceful of the historical background of the programme is a very positive framework to encourage.

Tags

Challenges

Ownership of programme staff: In theory, the Steering Committees of this programme (Political Committee and high-level Technical Committee) should have led this exercise. However, in reality, the Political Committee met only twice since the beginning of the programme (representation is at level of Ministers who are rarely available) and the Technical Committee (Burundians and programme staff) were not interested enough by this initiative. As a result the Dutch Embassy drove this evaluation. This fact is not however indicative of the national ownership levels for the programme. The Steering Committees and the level of representation were ineffective due to their flawed conception.

Ownership of local stakeholders: See comments on Burundian team members’ contribution under methodology.

Recommendations

Ownership of programme staff: Involve steering committees.

Ownership of local stakeholders: See recommendations on Burundian team members’ contribution under methodology

Recommendations

Necessity to evaluate ISSAT engagements in countries where there has been medium to long term engagement for ISSAT: ISSAT has been involved in Burundi for the past two years with 5 missions being deployed to support various donor programmes. An evaluation of ISSAT’s contribution and impact on those donors and their programmes would be a positive step to undertake.

Tags

Challenges

Local programme staff limited resources and time to undertake the additional workload of international mission preparation: The schedule of meetings and interviews was not prepared and shared with ISSAT experts before their field deployment despite the explicit requests from the –Geneva team. This is due in part to the knowledge of local programme staff of the flexibility of local stakeholders and the possibility of confirming interviews at the last minute. It was also due to the fact that the programme staff’s limited resources and time to dedicate to such an evaluation. They had to juggle their own working schedules with the additional responsibility of organizing such a mission.

Balance between planning imperatives and the context imperatives: This gap in planning meant that the team had to spend some of its time on logistics. However, this did not have a big effect on the mission’s effectiveness since local stakeholders were easily accessible and scheduling appointments for interviews was relatively rapid. The international team’s continued requests for a schedule of interviews as an important milestone for their planning led to certain moments of tension with the local counterparts. The latter interpreted the international team’s concern as lack of confidence in their capacities and knowledge. The international team however, was uncomfortable arriving to the country without a list of planned interviews in hand.

First engagements for experts from ISSAT roster: One of the experts on this mission was from the ISSAT Expert Roster embarking on a first mission with ISSAT. Substantive planning with this expert had already started when they discovered that the ISSAT operating procedures were not suitable for them. This was challenging for the team since the mission-planning process had already started and losing one team member at this stage would have problematic.

Security Briefings for ISSAT staff: Security briefings for ISSAT staff are standard imperatives for all missions. However, in the case of this country, the Dutch Embassy has not provided security briefings for its own staff members, who work and live in country with their families. Requesting security briefing for ISSAT inspite of this fact might cause exasperation as a result of unrealistic expectations on ISSAT’s side.

Successes

Flexibility of local stakeholders: The stakeholders implicated in the interviews, were very flexible in terms of their availability and the evaluation team could organise meetings with a fair degree of facility and relatively short notice.

Project Assistant supporting missions: The evaluation team received the support of a Project Assistant whose contribution to the mission was to perform the necessary desk review, collate reports and papers, review their relevance with the available advisor and then ensure that the team is able to access them through Basecamp. This worked very well as it ensured that everyone had a harmonized database of documents and all team members had access to the same documents saving them time and effort that they could then invest in the planning and methodology design. This good practice was a lesson learnt from previous missions where ISSAT Advisors used to undertake the above role at the expense of time that should be invested in the mission methodology design and other preparative activities.

Recommendations

Balance between planning imperatives and the context imperatives: e mission team tried to push for more rigorous planning regarding the interviews with stakeholders. However, when the situation in the field became clearer in terms of the programme staff’s resources and time, in addition to reassurances from them on the expected flexibility of stakeholders, the international team pushed less on their criteria and accepted the context’s imperatives. A balance needs to be struck between structured planning imperatives, the reality of the field, time limitations and the local programme staff’s capacity, resources and availability. This could be done in being realistic about the crucial issues to be ready before field deployment and the issues that permit more flexibility. This balance is usually facilitated by the team leader as in most cases team members have different visions of what should be urgent and what is not.

First engagements for experts from ISSAT roster: Having a consolidated power point presentation explaining ISSAT’s structure and functions should be made available to all Expert Roster Members on the CoP. Sharing ISSAT contracting procedures and conditions with first-time Expert Roster members before they embark on the mission planning is of capital importance.

Security Briefings for ISSAT staff: In cases where the mandator does not provide security briefings to its own staff members, higher flexibility from ISSAT might be required. It might need to look for alternatives whether online or through contacting relevant embassies.

Joint collaboration and consultations between ISSAT mission teams and In-house resource people: Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:

Before missions: Working sessions and/or discussions with advisors as appropriate

During mission: Email interactions between mission team and Geneva team on targeted issues

After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). This will help the process of thought of the team before the report typing phase.