There are now quite a number of experimental designs and RCTs (randomized controlled trials) of market information systems measuring revenue gained by rural agriculturalists (farmers, fishermen) and traders. While some show improvements to the prices farmers get for their crops, others disappointingly show a lack of impact. There are also studies of agricultural extension services delivered via mobile phone which suggest that crop productivity is boosted by such services. How might these studies guide what actions are taken (i.e. to provide such services) in the real world?

What is ‘ecological validity?’ It is the relationship between controlled, or lab-based experiments and what unfolds in a non-lab, non-controlled environment (aka the ‘real world’). Out in the world multiple variables interact in different ways. So what can we take away as actionable from an RCT that shows a market information system has resulted in boosted revenues for farmers who used it? How do we reconcile the successes with similarly designed studies that showed a failure of impact?

Recent RCTs of MIS have been designed to provide the ideal circumstances for their success. They do so in the following ways:

3) providing extensive training on the use of the system to overcome literacy limitations [Hildebrandt 2011]

Why do they do this? Because these studies intend to show what impact information (in isolation from other issues) has on markets, farmer profits, and revenues. That does NOT quite make these studies true evaluations of a particular system in real-world use.

So what does this mean for the real-world use of MIS systems such as Esoko, Reuters Market Light, etc?

For item (1) it means that a successful business model will most likely require that it be provided for free, (at least in the short to medium term) if it is to yield the effects measured by recent studies.

For (2) and (3) it means that service adoption and literacy issues remain significant and substantial barriers. Artificially eliminating them for the purpose of an RCT is problematic because it obscures this fact. Fortunately, many programs in this space are finally becoming more attuned to this reality and offering such services through more user-friendly and accessible IVR or call center models (as Esoko now does). For more literacy-demanding services, training (or infomediaries) will be needed. Where training is needed, the scalability of such services is an issue.

Susan Wyche and Chip Steinfield have a paper that’s just come out evaluating the user interface issues around MIS. They look at the affordances of such services and the challenges they pose for farmers. By contrast, RCTs championed by economists fail to address usability and literacy issues when they treat ‘information’ as the only important variable to isolate. It will likely require more interest and support for researchers with different kinds of training (HCI, social science) to make sure this important issue is not trivialized.