Performance and quality analysis publications

2017 Quality Review of Aid Program Performance Reports

Each year ODE assesses the quality of Aid Program Performance Reports (APPRs) and other performance management instruments that DFAT uses to monitor and evaluate the aid program. The 2017 review focussed on full-length APPRs prepared by the 14 largest country and regional Australian aid programs.

The review found that half of the full-length APPRs were found to be of good quality when assessed against three ODE quality standards related to: aid objective definition, progress assessment robustness, and quality of management response. Compared to 2016 the review found that the overall quality of these APPRs has improved.

The review also found that APPRs are increasingly being used to strengthen program management, demonstrate accountability, and improve effectiveness. APPRs of more limited quality generally struggle to define what success looks like, with the shift to a greater policy influencing role by some programs making this more difficult. There are also ongoing challenges managing the tension between the role of APPRs in performance management and their role in public diplomacy. Good quality APPRs demonstrate that an appropriate balance can be reached.

2016 Quality Review of Aid Program Performance Reports

Aid Program Performance Reports (APPRs) are the principal mechanism in DFAT to assess the performance of country and regional aid programs. Each year ODE reviews APPR quality, and highlights strengths and opportunities for improvement. A more rigorous methodology was applied for the 2016 quality review to reflect the first full year that Australian Government's performance framework for the aid program, Making Performance Count and Aid Investment Plans were implemented.

The 2016 quality review found that two-thirds of the APPRs were of good quality with a number demonstrating high functioning performance management systems (clear objectives, robust evidence, and relevant, time-bound management actions). The remaining APPRs were assessed to be of more limited quality. The main concern was the inability of these programs to define what success looked like. Without that clarity, it was difficult to know whether expectations had been met and then have confidence that the right management actions have been identified.

Given the APPRs are of central importance to program management and decision-making, DFAT has agreed to implement the review recommendations designed to lift the quality and utility of APPRs. Examples of good practice reporting from the 2016 quality review have been shared with programs and included in revised DFAT guidance.

2015 Quality Review of Aid Program Performance Reports

Aid Program Performance Reports (APPRs) are the principal mechanism in DFAT to assess the performance of country and regional aid programs. Each year ODE reviews APPR quality, and highlights strengths and opportunities for improvement. The 2015 quality review found that APPR quality is largely adequate or good and consistent with the previous year. APPRs exhibit increased embeddedness of the aid policy, clear explanations of policy dialogue engagement and good quality objectives. The APPR narratives also performed well with the quality of their progress assessments.

While ratings in APPRs are mostly well justified, three reports were inadequate in justifying their traffic light ratings. There is also room for improvement in the quality of the management responses, mutual obligations, the integration of the performance benchmarks and reporting performance in addressing gender inequality.

Given the APPRs are of central importance to program management and decision-making, there is a question for DFAT as to whether it should accept the good standard of reporting it has achieved in the last two years or whether it should try to lift the bar.

2014 Quality Review of Aid Program Performance Reports

Aid Program Performance Reports (APPRs) are the principal mechanism in DFAT to assess the performance of country and regional aid programs. Each year ODE reviews APPR quality, and highlights strengths and opportunities for improvement. The 2014 quality review found that overall the 2013-14 APPRs are well written, informative descriptions of program activities and progress towards objectives. A highlight in these APPRs was strong content on gender equality.

The review identified two main issues facing some APPRs. Firstly, many do not clearly explain program strategy – though the introduction of Aid Investment Plans should assist with this. Secondly, a substantial number of program objectives, and/or indicators lack clarity. Without objectives that are clear and measurable it is difficult to explain and justify progress ratings.

Both this and the previous year's quality review demonstrate that the credibility of performance assessment in APPRs is enhanced when programs draw on high quality performance assessment frameworks (PAFs). ODE's quality review therefore recommends that PAFs should be required for all programs that produce APPRs.

2013 Quality Review of Aid Program Performance Reports

ODE assesses the quality of APPRs and other performance management instruments that DFAT uses to monitor and evaluate the aid program. This quality review confirms a gradual, clear improvement in the quality of APPRs since 2007-08 when they were first introduced. It found that the reports generally provide frank, well-written explanations of performance and serve their accountability role well.

To improve their clarity and credibility, ODE's quality review recommends that APPRs provide better explanations about the rationale underlying program objectives, broaden the use of credible evidence (especially that gathered from partners), and improve the utility of the reports as management tools. Whole of Government performance also needs to be better assessed in APPRs.