Impact Assessment of Mobile Phones on Development: Concepts, Methods and Lessons for Practice

Richard Duncombe

Abstract

The use of mobile phones is an increasingly important part of development across all sectors, which has led to rapidly increasing investment by the mobile phone industry and new demands on the resources of donors. Impact assessment (IA) is an increasingly important tool for evaluating whether or not mobiles-for-development (m-development) funds are being used effectively. This paper reviews 18 published studies, which are critically evaluated with reference to key building blocks for good practice in IA. These building blocks encompass: the extent to which the studies address the needs of defined audiences or beneficiaries, the choice of types of impact to assess and units of assessment, the application of suitable conceptual frameworks, and the rigour of the methodology followed.

The findings suggest that m-development IA to date falls into three main categories: purely quantitative approaches rooted in information economics, and purely qualitative approaches rooted in social impact assessment. A further mixed method category provides cost effective and timely findings, but with the result of diluting the rigour evident at the methodological poles.

Overall, the paper concludes there is a need for more and better guidance for the conduct of such studies, particularly in relation to identifying the rationale and objectives of IA, and linking the choice of concepts and methods with the requirements of key audiences and beneficiaries. Thus, the paper identifies m-development IA as a contested area, and by taking stock of experience thus far, seeks to raise the level of debate concerning the relative merit of alternative methodological and conceptual approaches.

Development questions

In what other ways could m-development IA be approached?

What is your own viewpoint on the relative merits of mixed method IA and purely participatory methods for assessing m-development interventions? How might your viewpoint affect any assessment you carry out?

Why do you think, at this stage, this area of study is skewed towards information economics on the one hand, and assessment of social capital indicators on the other?

From amongst the methodological approaches discussed in the paper, which can make the most useful contribution to: a) project level IA b) programme level IA; c) policy level IA?

To what extent can academic research inform the requirements of practitioners for IA in m-development?

What topics/approaches should be prioritised for m-development IA over the next five years?