The publications in this collection do
not reflect current scientific knowledge
or recommendations. These texts
represent the historic publishing
record of the Institute for Food and
Agricultural Sciences and should be
used only to trace the historic work of
the Institute and its staff. Current IFAS
research may be found on the
Electronic Data Information Source
(EDIS)
site maintained by the Florida
Cooperative Extension Service.

Copyright 2005, Board of Trustees, University
of Florida

STEPS IN THE ANALYSIS AND INTERPRETATION
OF ON-FARM RESEARCH-EXTENSION DATA
BASED ON
MODIFIED STABILITY ANALYSIS:
A TRAINING GUIDE

BY

PETER E. HILDEBRAND1

STAFF PAPER SP93-11

MAY 1993

Staff papers are circulated without formal review by the Food and Resource Economics
Department. Contents are the sole responsibility of the author.

STEPS IN THE ANALYSIS AND INTERPRETATION
OF ON-FARM RESEARCH-EXTENSION
BASED ON
MODIFIED STABILITY ANALYSIS:
A TRAINING GUIDE

ABSTRACT

Modified Stability Analysis or MSA is a procedure for designing,
analyzing and interpreting on-farm research-extension activities
conducted to assess new technologies or management practices.
The guide provides the steps to follow to make technology
recommendations for specific bio-physical and socioeconomically-
created environments and tailored to the desires, needs and
resource constraints of specific farmers. It also provides a
basic understanding of the design requirements for on-farm
research to make it amenable to analysis by MSA. The example
used throughout the guide comes from a real on-farm research-
extension project conducted near Manaus in the Brazilian Amazon
region.

On-farm research can have various functions and be managed by
researchers, extension workers and/or farmers (Hildebrand and
Poey, 1985). The most appropriate for incorporating farmer
participation is a simple (few treatments), non-replicated design
which has one to three treatments to be compared with the
farmers' own technologies. For purposes of demonstration, a
trial conducted in the Amazon basin of Brazil (Singh, 1990) with
four treatments and on eight environments, without replication,
will be used. Additional information on design will follow
discussion of the steps in analysis.

RESPONSE OF TREATMENTS TO DIFFERENT ENVIRONMENTS

The term "environment" is used here rather than "farm" or
"location" because on a single farm, or even in a single field,
more than one environment can exist for the production of the
livestock or crops grown. Similar farmers do manage different
environments differently just as different farmers may manage
similar environments differently. Furthermore, making technology
conform to varying environments, rather than the contrary, is
more in keeping with sustainable agriculture.

Measure of environments: Environmental Index. El Step 1.

The factors which influence the environment for raising crops or
livestock are many and complex and are very difficult to assess.
A convenient substitute measure of the quality of each
environment where a trial has been conducted is the average
"yield" of all the treatments included when, and only when, the
same treatments are included in all the sampled environments.
The first step is to calculate this index, EI, which provides an
effective measure of the environmental differences in the
research domain represented by the range of Els.

To facilitate further analyses, it is convenient to sort the data
by descending (or ascending) values of this index. The data in
Table 1 are in descending order of the environmental index EI.

ANALYSIS AND INTERPRETATION OF ON-FARM RESEARCH DATA

INTRODUCTION

On-farm research can have various functions and be managed by
researchers, extension workers and/or farmers (Hildebrand and
Poey, 1985). The most appropriate for incorporating farmer
participation is a simple (few treatments), non-replicated design
which has one to three treatments to be compared with the
farmers' own technologies. For purposes of demonstration, a
trial conducted in the Amazon basin of Brazil (Singh, 1990) with
four treatments and on eight environments, without replication,
will be used. Additional information on design will follow
discussion of the steps in analysis.

RESPONSE OF TREATMENTS TO DIFFERENT ENVIRONMENTS

The term "environment" is used here rather than "farm" or
"location" because on a single farm, or even in a single field,
more than one environment can exist for the production of the
livestock or crops grown. Similar farmers do manage different
environments differently just as different farmers may manage
similar environments differently. Furthermore, making technology
conform to varying environments, rather than the contrary, is
more in keeping with sustainable agriculture.

Measure of environments: Environmental Index. El Step 1.

The factors which influence the environment for raising crops or
livestock are many and complex and are very difficult to assess.
A convenient substitute measure of the quality of each
environment where a trial has been conducted is the average
"yield" of all the treatments included when, and only when, the
same treatments are included in all the sampled environments.
The first step is to calculate this index, EI, which provides an
effective measure of the environmental differences in the
research domain represented by the range of Els.

To facilitate further analyses, it is convenient to sort the data
by descending (or ascending) values of this index. The data in
Table 1 are in descending order of the environmental index EI.

ANALYSIS AND INTERPRETATION OF ON-FARM RESEARCH DATA

INTRODUCTION

On-farm research can have various functions and be managed by
researchers, extension workers and/or farmers (Hildebrand and
Poey, 1985). The most appropriate for incorporating farmer
participation is a simple (few treatments), non-replicated design
which has one to three treatments to be compared with the
farmers' own technologies. For purposes of demonstration, a
trial conducted in the Amazon basin of Brazil (Singh, 1990) with
four treatments and on eight environments, without replication,
will be used. Additional information on design will follow
discussion of the steps in analysis.

RESPONSE OF TREATMENTS TO DIFFERENT ENVIRONMENTS

The term "environment" is used here rather than "farm" or
"location" because on a single farm, or even in a single field,
more than one environment can exist for the production of the
livestock or crops grown. Similar farmers do manage different
environments differently just as different farmers may manage
similar environments differently. Furthermore, making technology
conform to varying environments, rather than the contrary, is
more in keeping with sustainable agriculture.

Measure of environments: Environmental Index. El Step 1.

The factors which influence the environment for raising crops or
livestock are many and complex and are very difficult to assess.
A convenient substitute measure of the quality of each
environment where a trial has been conducted is the average
"yield" of all the treatments included when, and only when, the
same treatments are included in all the sampled environments.
The first step is to calculate this index, EI, which provides an
effective measure of the environmental differences in the
research domain represented by the range of Els.

To facilitate further analyses, it is convenient to sort the data
by descending (or ascending) values of this index. The data in
Table 1 are in descending order of the environmental index EI.

Table 1. Response of maize (Mg ha'-) to three soil amendments and
the farmers' practices from on-farm research results in Amazonas,
Brazil (Singh, 1990). FP is farmers' practices, PCW is processed
city waste (from Manaus), CM is chicken manure, and TSP is triple
super phosphate. For more details see Singh, 1990.

The yield data for each treatment should be related to the
environmental index. The second step is to view the observations
by graphing the results of one treatment against El as in Figure
1.2 It is necessary to decide if the relationship is linear or
curvilinear, and some estimate of the relationship must be made.
It is satisfactory simply to draw a line or a curve through the
data. With practice, this can become fairly precise. Linear
regression can be accomplished easily with many inexpensive
calculators, and linear or curvilinear regression can be
estimated with a computer. The estimated relationship shown in
Figure 2a is from linear regression. Figures 2b and 2c show
comparisons of linear and curvilinear regression for FP and CM.
For treatments FP and CM it is fairly evident that curves
represent the nature of the data better than straight lines.
Therefore, for the remainder of this analysis, curves will be
used for these two treatments. For PCW and TSP straight lines
are adequate.

2 For this step all graphs should have identical axes so they
can be compared easily by placing one graph over the other. This
will also facilitate the comparison of treatment responses to
environment in the next step.

Figure 2c. Comparison of linear and quadratic response in Mg ha"1
of the CM treatment to environment (EI) for maize in Amazonas,
Brazil (Singh, 1990).

Interaction of treatments with environment

When all treatments have been related to, or regressed on EI, the
third step is to compare the response of the treatments to
environment, as in Figure 3. No interaction exists if all the
lines are parallel. If no treatment by environment interaction
exists (in practice this seldom occurs), the treatment which is
greatest over all environments is the best for the criterion
used, here Mg ha"1. However, if the lines are not parallel, such
as in this case and is most usual in practice, treatment by
environment interaction exists and different treatments may be
best for different environments. Notice that the values of the
El are shown in the lower part of Figure 3. This will be useful
in assessing risk and characterizing recommendation domains.
They also help in deciding upon the confidence that can be placed
in the results of the trial based on available data (often data
from only one year).

Figure 3. Estimated responses in Mg ha'' of the four treatments
to environment (EI) for maize in Amazonas, Brazil (Singh, 1990).

Step 3.

Assessing confidence of relative responses to environment

Three criteria help assess the confidence which can be placed in
the relative responses of the treatments to environment. The
first relates to the range of environments sampled, the second to
the general conditions of the year in the research domain, and
the third to distribution of the environments in the research
domain.

1) The range of the environmental index, EI, should be at least
as large as the overall mean El. If this criterion is violated,
it usually means the research domain included only the best
environments (perhaps only "progressive farmers" were involved),
or else that the year was exceptional and resulted in high yields
throughout the research domain.

2) The range of treatment yields used to calculate the Els
should be similar to expected yields over a series of years. If
the year was particularly good or bad overall, or only very good
or very poor sites were chosen, this criterion could be violated.

3) The distribution of the Els should be reasonable across the
range of environments sampled. That is, the environments should
not be grouped with only one or two lying outside the grouping.

The data in Table 1 fairly well satisfy the three criteria. The
range of Els (3.1 0.2 = 2.9) is greater than the overall mean
El (1.9), easily satisfying the first criterion. The range of
Els also easily represents the normally expected range of yields
under these conditions, satisfying the second criterion. The
distribution of Els, shown in Figures 1 to 3, is also quite
reasonable, satisfying the third criterion. Therefore, even
though the number of environments is quite low, given the number
of treatments (see the discussion of trial design in a later
section), it should be expected that the relationships among the
treatments over various environments represented in Figure 3 will
be stable over time, should the trial be repeated in this
research domain (not necessarily the same farms or sites). It
also means that the persons involved in the trial can have
confidence in making recommendations to farmers in the specified
recommendation domains (see step 5) based on only this one year's
data.

Multiple evaluation criteria Step 4.

The evaluation criterion used to calculate the environmental
index EI, above, is Mg ha"1, the most common criterion used by
agronomists in crop trials and appropriate in most cases as the
basis for calculating the EI. However, few farmers use this
criterion when making production decisions. If seed, labor or
cash are most scarce, more appropriate evaluation criteria are
kg/kg seed, kg/day of labor in a critical period, or kg/dollar of
cash cost, respectively. MSA easily lends itself to analysis
using multiple criteria. The fourth step is to compare
alternative evaluation criteria. Figure 4 is based on analysis
of the usual farmers' criterion of kg/$ cash cost, Table 2.

Notice that the same El is used regardless of the criterion being
evaluated. The El values used to form the X-axis do not change.
The criteria used on the Y-axis do change. The same procedures
were used to obtain these relationships as were used to obtain
the relationships based on the researchers' criterion, Mg ha1.
Cash costs of the treatments were FP = $12, PCW = $208, TSP = $98
and CM = $127. Notice that very different conclusions result
when the evaluation criteria change. This is important because
it relates to the recommendations that will be made.

Table 2. Response of maize (kg $-1 cash cost) to three soil
amendments and the farmers' practices from on-farm research
results in Amazonas, Brazil (Singh, 1990). FP is farmers'
practices, PCW is processed city waste (from Manaus), CM is
chicken manure, and TSP is triple super phosphate. For more
details see Singh, 1990.

Figure 4. Estimated responses in kg $-1 of the four treatments to
environment (EI) for maize in Amazonas, Brazil (Singh, 1990).

DEFINING RECOMMENDATION DOMAINS FOR DIFFUSION Step 5.

The fifth step is to interpret the results, define recommendation
domains and create extension messages. This involves five sub-
steps: 1) interpret multiple evaluation criteria, 2) assess
risk, 3) characterize the environments, 4) define the
recommendation domains, and 5) create the extension messages.
Recommendation domains, the situations for which specific
treatments or technologies will be recommended, depend upon both
the characteristics of the environments and the choice of
evaluation criteria.

Interpret alternative evaluation criteria Step 5a.

Many evaluation criteria may apply to the same set of on-farm
research data. In the example, two evaluation criteria have been
demonstrated: Mg ha1 in Figure 3 and kg $-1 in Figure 4. In the
case of the former, using the researchers' criterion, TSP would
be recommended for the two highest Els and CM would be
recommended for the remaining environments. The two top Els are
PFi. But the fourth highest is also PFi. The difference is that
the top two have a pH higher than 5.0 and phosphorus levels above
7.0 ppm. But farmers will not have this kind of information, so
it may be necessary to group all PF, in one recommendation domain
and all other classes in the other. The persons involved in the
trial, including the farmers, should make this judgement to help
facilitate dissemination of the results.

For the farmers' criterion, kg $', none of the amendments are
superior to the farmers' practices for land cleared from either
secondary or primary forest in the first year of crop production.
Thus, based on this criterion, none of the amendments would be
recommended for farmers producing maize on land being used the
first year after clearing. Beyond the first year of production,
either CM or TSP would be recommended, the choice, perhaps
depending on risk considerations.

Risk considerations Step 5b.

The probability of low values (a measure of risk) of the selected
criterion for any of the technologies being assessed in the trial
can be estimated by means of a distribution of confidence
intervals based on the treatment results in the specific
recommendation domains. As an example, consider the choice
between CM and TSP in the second or third year of use, and based
on the farmers' criterion, kg $-.

The equation:

S (t. s/n) (1)

gives the confidence interval for the a level of probability from
a two-tailed "t" table for n-1 degrees of freedom and where s =
sample standard deviation for the observations in the potential
recommendation domain. In the two-tailed "t" table, a
probability level a = 0.4 means that 40% of the values lie
outside the interval and 60% lie inside the interval defined by
the equation. The lower value of this equation:

S- (t, s/Vn) (2)

provides information on the level of risk associated with the
technology in this recommendation domain. In this case, the a
level of probability from a one-tailed "t" table is the
probability that yield or other evaluation criterion values would
fall below the value represented by the second equation. Table 3
shows the calculations and Figure 5 graphically shows the risk
levels for CM and TSP for the farmers' criterion, kg $-, and for
maize cropping after the first year of use, using equation (2).
In this case, CM is less risky (has a lower probability of low
values) than TSP so would be recommended for the relevant
environment.

Figure 5. Risk levels for the criterion kg $-1 and the treatments
CM and TSP from the maize trials in Amazonas, Brazil (Singh,
1990).

0.25
0.20
0.15
0.10
0.05
0.025
0.01
0.005
0.0005

Characterizing the environments Step 5c.

Environments can be characterized using both biophysical and
socioeconomic factors that may, at the same time, be both
quantitative and qualitative in nature. Data obtained for the
environments in the Amazon example include soils characteristics
and a category called "land class", Table 4. The soils
characteristics are self explanatory. Land class refers to the
kind of forest that was cleared (P = primary, S = secondary) and
the number of years it has been cropped (1 = first year, etc.).
The term WL refers to land that had been cleared by bulldozer at
the time of colonization and is, essentially, waste land.

Because the data in Table 4 have been sorted by EI, it is easy to
assess the relationship between El and these characteristics.
Lower Els are associated with lower pHs, lower phosphorus levels,
lower ECECs and higher aluminum saturation. If desired, these
relationships can also be graphed and/or estimated by regression
with El being the dependent variable as was done with pH in
Figure 6.

Perhaps the most useful for farmers and extension agents is the
land class characteristic, because farmers in these conditions
seldom, if ever, have detailed soil information on their fields.
It can be seen that both the nature of the forest that was
cleared for the field and the number of years in use are closely
associated with EI.

Based on the above analyses, interpretations and judgements,
several recommendation domains can be specified, based on this
single on-farm trial. The treatment to be recommended depends on
the environment and on the evaluation criterion of the farmer.
The former (environment) requires information about the kind of
field the farmer is going to plant, when it will be planted,
and/or other environmental factors, not included as treatments,
which may have appeared in the analysis to be important. The
latter (evaluation criteria) depend on the scarcity of resources
available to the specific farmer and what that farmer would like
to maximize for the specific crop or livestock in question. For
the example used in this guide, Table 5 and Figure 7 summarize
the recommendation domains and messages available to an extension
agent for different farmers. If Mg ha"' is a relevant criterion,
TSP would be recommended for land taken from primary forest and
in first year of production. CM would be recommended for all
other land. If kg $- is relevant, as it would most likely be to
these farmers, what they already do (FP) is the best on all land
in first year of use. If farmers want or need to produce maize a
second year, CM would be recommended.

16

Table 5. Summary of the recommendation domains and the
technology recommended for maize, Rio Preto da Eva, Amazonas,
Brazil, based on environmental factors and evaluation criteria.

Land Type
PFI SF1 PF2 SF2 WL
Criterion

Mg ha-' TSP CM CM CM CM

kg $- FP FP CM CM None

Source: Singh, 1990

Multiple
Criteria
Mg/ha

kg/$
Mg/ha
kg/$
Mg/ha
kg/$

Mg/ha

kg/$

Mg/ha

Not recommended kga/$

On-Farm
Research-Extension
in
Multiple Environments

PF,

Level of Adoption High
L None

Figure 7. Extension messages for multiple environments and
several evaluation criteria with MSA.

Population
of Farms
TSP

FP
CM

FP
CM

CM

CM

CM
CM

DESIGN OF ON-FARM TRIALS

It is now time to return to consideration of the design of on-
farm trials appropriate to generate the kind of data used in this
example and amenable to interpretation using Modified Stability
Analysis.

Treatments. Treatments should be few in number to facilitate
participation of farmers in the trial. This enhances diffusion
of acceptable results but also helps research and extension
workers understand the farmers' evaluation criteria, needed for
analysis of the data. All environments should have the same
treatment, or at least a common sub set of treatments. Note that
differences in farmer management become factors affecting
environment and have a positive rather than a negative effect on
trial design. The farmers' practices, as treatments, must be
included and may differ from farm to farm to reflect each
farmer's individual management practices. These differences, of
course, must be carefully documented to serve in characterizing
the environments of each farm.

Replications. No replications are needed for analysis by MSA.
If replications are desired by the research or extension worker
to help assure that a trial will not be lost at a specific
location, two blocks is a sufficient number.

Environments. The number of environments is more important than
the number of replications in each environment. Based on Stroup
et al. (pp.13-14) a simple rule can be devised:

The rule of 48 The number of treatments times the number of
environments should equal approximately 48. Thus, for 8
treatments (a large number for this type of trial, but not
excessive), 6 environments would be sufficient. For 6
treatments, 8 environments; for 4 treatments, 12
environments; for 3 treatments, 16 environments; and for 2
treatments, 24 environments.

Finally, in order to increase the probability that the first of
the three criteria for confidence will be met, the design should
include a wide range of environments, including different kinds
of farmers and physical settings, and these should be distributed
as well as possible to help satisfy the third of these criteria.
The second criterion depends largely on natural conditions beyond
the control of the persons doing the on-farm research.

Table 1. Response of maize (Mg ha"1) to three soil amendments and
the farmers' practices from on-farm research results in Amazonas,
Brazil (Singh, 1990). FP is farmers' practices, PCW is processed
city waste (from Manaus), CM is chicken manure, and TSP is triple
super phosphate. For more details see Singh, 1990.

Table 2. Response of maize (kg $-' cash cost) to three soil
amendments and the farmers' practices from on-farm research
results in Amazonas, Brazil (Singh, 1990). FP is farmers'
practices, PCW is processed city waste (from Manaus), CM is
chicken manure, and TSP is triple super phosphate. For more
details see Singh, 1990.