1.2 Methods in question: What were the main problems encountered and how to solve them?

It was the first time for most country teams that they applied the two methods of the research; it was also the first time that the two methods were used in such a large group of countries and within a comparative framework. It was therefore expected by the coordinators that problems mainly related to methodological issues such as sampling would arise during the conduct of the research, in spite of the training provided during the first workshop. Many of the problems were solved underway with the technical support of the coordinating institutions, some still remained unsolved for a few teams. This second workshop offered an occasion to review the main problems in order to improve the research results and to provide suggestions to improve the methods.

Methods in question: the NDP indicators

(a) Problems encountered

Five categories of problems were identified by the research teams

Methodological problems:A number of countries had difficulties in calculating the basket of drugs (India, Mali, the Philippines, Viet Nam) or did not follow the method proposed in the manual (Guinea). The main reason was the lack of access to data on disease prevalence and drug consumption. It was also felt that the ways to estimate the average price of a basket of food could be described in more detail in the manual (India). For two countries, structural indicators could be made more specific and qualitative (Thailand and Zimbabwe) whereas for one (the Philippines) there was a risk of bias depending on the position of the researcher for the structural indicators. Also the pharmacy surveys could be biased (Sri Lanka) as the pharmacists know that they are surveyed.

Availability of data:All the teams had problems in obtaining some data, mainly the financial ones. In addition data were often incomplete (Guinea) or aggregated at levels different than the ones required in the indicators (the Philippines, Thailand and Viet Nam). In some cases, the reliability of the data was difficult to assess (Guinea, India, Mali, Viet Nam and Zimbabwe).

Collection of data:Some countries faced problems in collecting the data because of geographical and logistical difficulties (Chad, Guinea, the Philippines), because places to be surveyed were closed (Chad) or received too few patients to collect enough prescriptions in two days (Guinea). The collection of data is also time consuming as it involves visiting many different services (Mali) and training surveyors. "Even good training is not a guarantee of the quality of the surveys" (the Philippines).

Data processing and analysis:This can be complicated, entries need to be checked before encoding (Guinea, the Philippines). Analyses need to be done by knowledgeable people. In addition, another important point was to ensure that the chosen indicators really did measure the implementation and achievements of the policy. To establish clear links between the indicators and the implementation and impact of the policy was sometimes difficult (Zimbabwe). Most of the teams emphasized the need for a sustainable team to carry out the task regularly and to have the process incorporated into the regular activities (Mali, the Philippines, Zimbabwe). However, this is not always easy when financial and human resources are scarce.

Data presentation:The presentation of the results to the various target groups is not always easy; too many badly explained data do not attract attention. It was however not always feasible for the teams to present their data in a user friendly format, this was due to lack of time, lack of knowledge of some of the opportunities offered by the computers, etc.

(b) Suggestions to improve the method

The participants classified the problems they faced during the implementation of the indicator exercise in three groups: problems linked with the indicators themselves, sampling issues and calculation of the basket of drugs. The paragraphs below are the summary of the conclusions of each working group.

Working group on indicators: The group reviewed all the indicators and provided detailed comments on each category of indicators described in the WHO manual. It was emphasized that before any move was made to use indicators, it was necessary to ask the question "What do we want to measure in relation to our policy? and why?".

For the background indicators, the group recommended to use data from the International Monetary Fund, the World Bank and national official publications as they provide reliability over time and facilitate comparison between countries. For data not available from these organizations, it is important to remember that it is not the "last" decimal which matters as the objective of these indicators is to provide the broad context and to identify trends. A number of suggestions to facilitate the applications of all the indicators were identified and presented at the plenary session.

Working group on sampling:The group reviewed the problems associated with sampling in the NDP surveys. It took the cases of the Philippines, Thailand, Viet Nam and Zambia as examples.

• Sampling procedures: The sampling procedures varied between the countries. In the Philippines, four regions of the country were selected. In each region, a sample of private and public facilities was drawn from lists available at the Ministry of Health. This sample seemed highly representative of the whole population. In Viet Nam, the sampling was slightly more complex, with selection of 6 provinces (3 in the North, 3 in the South; 2 including the two major cities, 2 representing the remote areas, and 2 representing the other areas). Again, the sample was most likely representative of the population. In Zambia, only two districts were selected, one in the North East, and the other in the South East, the most populated and economically active of the country. However, important parts of the country were omitted. This may introduce bias in the final estimates.

There is obviously a trade off between the representativeness of the sample and the cost of the survey, which has to be taken into account. The statistical analysis of the results of the survey could provide information on the potential bias involved in the sampling techniques.

• Choice of secondary units: A critical issue is the choice of the secondary units. In general, investigators have stratified their samples along the lines indicated in the manual, that is: public hospitals (sometimes distinguishing between primary, secondary and tertiary hospitals), private hospitals, public drug outlets, private drug outlets, and remote health facilities. Viet Nam selected public hospitals, public drug outlets at district level and private drug outlets at provincial level. Investigators noted that two other types of secondary units could have been included: black markets, and private medical doctors who are allowed to sell drugs to their clients.

• Choice of observation units (data sources): Another critical issue for the validity of the studies was the choice of the observation units. It was recommended to use a sample of prescriptions. However, in Thailand, investigators working on one of the samples in Bangkok had preferred to use "transactions" rather than "prescriptions". A transaction is a real negotiation between a client and a drug provider. In some places, transactions without a prescription may account for two-thirds of all drug purchases in the private drug outlets, even for antibiotics. Drug use in "transactions" (roughly self medication) may be quite different from drug use in "prescriptions" (normally with proper medical advice).

In one case (Zambia), the investigators added a household survey. This was considered important, since the most critical problem in this country was access to drugs. This seemed a valuable addition to the recommended strategy.

• Other issues: In Thailand, a "surrogate patient survey" was also conducted. A surrogate patient is a field worker or research assistant who visits health providers and presents them with fictitious case scenarios, then records the information. The method offers a chance to record non-self-conscious current practice from a point of view of the client in a first hand and standardized fashion. However, this approach may lead to bias if the field worker is not well prepared, and should not be recommended unless carefully planned. A participant also noted that this may raise serious ethical issues.

Working group on basket of drugs:Calculating the basket of drugs was a problem for some of the research teams. Bulgaria did it according to the manual and were helped by the existing statistical database. The group thought that without such a statistical backup (data on prevalence and consumption) most countries would find it difficult to calculate the basket of drugs. However it was agreed that the method proposed in the manual was the best one and should be used where possible. The group suggested alternative methods which may be used when the method proposed in the manual was not applicable. The alternative methods should be submitted to a limited group of experts for validation and if relevant should be included accordingly in the new edition of the manual. It was also thought that the explanations of the proposed method in Part IV of the manual could be improved in terms of clarity.

Methods in question: the political mapping

The political mapping analysis was applied in six countries: Colombia, India, the Philippines, Sri Lanka, Thailand, and Viet Nam. This lack of complete application in all the study countries suggests that problems existed either in the training provided for the method, or in the technical support provided by the coordinators. In addition, the six country teams encountered some problems inherent in the method of political mapping, as a form of applied political analysis for examining policy-making processes. The second workshop discussed several of these problems and limitations.

(a) Problems encountered

Technical support: The three countries that did not apply the method of political mapping had found that they did not have sufficient expertise within the country team to learn the new method. In some countries, the team may not have included persons with training in policy analysis or social science, which usually helps with a political mapping analysis. In other countries, the lack of adequate computing facilities may have contributed to lack of application. The political mapping method uses a Windows-based software program and requires a 486 level processor. Technical support from the coordinators could have assisted in applying the political mapping method in the three countries that did not complete this analysis.

Quality of data: The political mapping method requires excellent data and knowledge about the basic policy-making processes for a specific decision. In the six countries where an analysis was performed, these data were collected through interviews with stakeholders. An inherent limitation of the political mapping method is assuring the quality and validity of the qualitative data collected through interviews. For this purpose, the study team performs an important role in discussing and assessing key judgments about players, their positions, their power, and the likely impacts of different political strategies. As with other policy analysis methods, political mapping is subject to the limitation of "garbage in, garbage out." The quality of the data affects the reliability of the analysis.

Collection of data: The six countries that performed the political mapping analysis initially expected some difficulties in the collection of data. Most countries, however, did not encounter difficulty in obtaining interviews with key decision-makers involved in the policy. Government officials and industry executives agreed to participate in the study and were, according to country reports, relatively open in discussing the policy decisions and the related processes. Accordingly, the collection of data was not a major problem for the political mapping analysis.

Data processing and analysis: The six countries reported a number of challenges in carrying out the data processing analysis in the political mapping. Several countries noted that the data analysis requires excellent knowledge about the policy among team members, and also a "good balance" within the team. One way of assessing the reliability of qualitative data about policy-making processes is through group discussion and collective judgment. For these methods to work well, the group needs to be able to function well together and to contain different kinds of expertise, both technical and political. Finally, the country teams also reported that the method cannot eliminate some unpredictable elements in policy-making processes.

Data presentation: Overall, the six country teams found the computer software to be an effective method for presenting the results of the political mapping analysis. However, the user still needs to exercise judgment in deciding which elements of the analysis should be emphasized and which strategies should be recommended to decision-makers. In short, the method of political mapping does not replace analytic thinking or policy judgment. Finally, in a rapidly changing political environment, it may be necessary to update the analysis and the results on a regular basis, otherwise the analysis could be limited by specific circumstances. This time-specific dimension is an inherent limitation of the political mapping method.

(b) Suggestions to improve the method

Technical support: The participants agreed that it would have been possible to perform the political mapping analysis, if additional technical support had been provided to the three countries that did not carry out this part of the research project. Often, an external consultant, working with a national expert, can create an effective team of "outsider and insider," which is helpful both in collecting and assessing data related to policy-making processes. External consultants can also help in interviewing some government officials or industry executives, and in asking sensitive questions about policy-making processes.

Training: The participants suggested that additional training on the political mapping method, using the updated version of the computer software (Policy-maker 2.0) would help with future efforts to perform a political mapping analysis. New help texts and tutorials, which are now being prepared, are also expected to assist first-time users in applying the software and carrying out a political mapping analysis.

Language:It was also suggested that the political mapping software and manual could be translated into French, in order to make the method more accessible to French-speaking countries. The current materials are available only in English.

Selection of policy for analysis: The participants suggested that the selection of the issues for analysis is a critical factor in carrying out political mapping. The analysis seems to work better for well-defined issues and policies, in both data collection and data analysis phases.

Methods in question

NDP indicators: The main problems faced by research teams were related to issues such as sampling, basket of drugs and difficulty to obtain reliable data; the teams however agreed that the methods proposed in the WHO manual on indicators including the sampling ones were well described and practical. They recommended that the suggestions made during the workshop be taken into consideration when revising the manual. In addition, DAP will provide in a separate document or in the next edition of the manual, practical information based on the results of the application of the indicators in the 12 countries.

Political mapping: Main problems encountered by the teams were related to the quality of the data (as other policy analysis methods, political mapping is subject to the limitation of "garbage in, garbage out"); and the data analysis. The data collection was easier than expected and the computer software was an effective method for presenting the results. The participants suggested additional training on the political mapping method.