The Definition of Some Estimative Expressions

Survey shows general agreement on the meaning of "probable" and some equivalents, elsewhere much disagreement.

THE DEFINITION OF SOME ESTIMATIVE EXPRESSIONS

David L. Wark

Finished intelligence, particularly in making estimative statements, uses a number of modifiers like "highly probable," "unlikely," "possible" that can be thought of as expressing a range of odds or a mathematical probability, and these are supplemented by various other expressions, especially verb forms, conveying the sense of probability less directly "may," "could," "we believe." Certain other words express not probability but quantity, imprecisely but perhaps within definable ranges -- "few," "several," "considerable." Some people object to any effort to define the odds or quantities meant by such words. They argue that context always modifies the meaning of words and, more broadly, that rigid definitions deprive language of the freedom to adapt to changing needs.

It is possible, however, to state the definitions in quantitative terms without making them artificially precise. And if two-thirds of the users and readers of the word probably, for example, feel it conveys a range of odds between 6 and 8 out of 10, then it is more useful to give it this definition than to define it more or less tautologically in terms of other words of probability. This would not deny to context its proper role as the arbiter of value, but only limit the range of its influence. Nor would it freeze the language in perpetuity; as the meanings of the words evolved the quantitative ranges could be changed.

This article describes the results of a survey undertaken to determine if such words are indeed understood as measurable quantities and if so to ascertain the extent to which there is a consensus about the quantitative range of each. A three-part questionnaire on the subject was distributed in the intelligence community -- to INR/State, the DIA Office of Estimates, and five CIA offices -- and a simplified version of it was sent to policy staffs in the White House, State, and the Pentagon. Responses were received from 240 intelligence analysts and 63 policy officers.

The responses showed a satisfactory consensus with respect to various usages of likely and probable, phrases expressing greater certainty than these, and modifications of chance -- good, better-than-even, slight. There was no satisfactory agreement on the meaning of possible or a wide variety of verb forms such as we believe and might. There was also little agreement on the non-odds quantitative words such as few and many. The policy offices consistently assigned lower probabilities than intelligence analysts did. Correlation between values assigned in and out of context was good.

The Questionnaire

Part One of the questionnaire listed 41 expressions that might be thought of as indicating odds and offered the choice of 0, 10, 20, etc. through 100 as the percentage probability or chances out of 100 signified by each. If the respondent believed that no quantitative answer was satisfactory he could mark "Not Applicable" instead. These expressions of course had to be judged without benefit of context, but in order to check on the validity of such judgments some of them were repeated in Part Two, where they were included in 17 sentences taken from intelligence documents which had been produced in six different offices of the community. The names of all persons and countries in the sentences were changed to sterilize them against bias. Part Three then listed nine expressions of magnitude not referring to probability and offered an assortment of ranges for each.

The idea of a consensus is relative, but for purposes of Parts One and Two it was defined as requiring 70% or more of respondents to name odds within 10 points, plus or minus, of the most frequent response. If the odds or chances most frequently specified for possibly were 50 out of a hundred (as they were) and 70% of all the responses had fallen within the range 40 to 60, the requirements for a consensus on this word would have been satisfied. Only one figure was recorded for each question: when an answer was ranged by marking several adjacent figures, it was recorded as the mean. Mr. Kent's range of 10 to 90 for possible would thus have been recorded as 50. Definitions were also considered invalidated by 20% or more of "Not Applicable" responses rejecting the question.

The replies were tabulated in four categories in descending order of valid definition, as follows:

Category D -- no consensus, and 20% or more of respondents marked "Not Applicable."

Findings

The following tables summarize the findings of the survey. After each expression from Parts One and Two are shown the odds most frequently specified and the percentage of respondents within 10 points of that. For questions submitted to policy officers as well as analysts, their responses are shown separately. The expressions of magnitude in Part Three are listed with the percentage of "Not Applicable" responses and the most frequent response for each.

Of the 41 expressions in Part One three fell into Category A (superconsensus), thirteen into Category B (consensus), seventeen into Category C (no consensus), and eight into Category D (rejected as indefinable). From Part Two five expressions in context fell into Category B, twelve into Category C, and three into Category D. All the quantitative phrases in Part Three were rejected as not measurable by 20% or more of the respondents except for next few years and next year or so. Though rejected by only 7%, next few years found no consensus: 19% marked 2 to 3 years, 30% 2 to 4 years, and 34% 2 to 5 years. Next year or so meant 1 to 2 years to two-thirds of the respondents, 1 to 3 years to the rest.

The difference between the good consensus on a set of odds for one expression and no consensus on another shows up clearly when the odds are graphed according to how frequently each set was specified in the responses to a question. When 70 % of all responses fall within 10 points of the most frequent one, the graph has a steep curve and a narrow base. The high, narrow peak indicates a clearly defined consensus, whereas a broad-based curve with a single peak shows less agreement and a curve with several peaks reflects clear differences about what the word means.

Steady retrogression from consensus can be seen in graphs of sample responses from successive categories. Following are these seven from Parts One and Two.

Out of Context

Category

Almost Certainly

A

Probably

B

Possibly

C

Serious Possibility

D

Seems

E

In context:

"The North Koreans have thus far shown marked respect for US power, and we do not expect them to change this basic attitude" expresses what probability that the North Koreans will continue provocations against South Korea? ... D

"At the same time, the reservations conveyed in the military comment suggest that the practical military changes resulting from the new line may well be less dramatic than the tone of de Gaulle's speech might indicate and that in any event, his government is not committing itself to a one -- weapon system of defense" expresses what probability that the military will have a one -- weapon system? ... D

The red line in each graph traces the response pattern of 239 analysts, the black line in the first four that of 63 policy officers. The dotted black line is the latter adjusted to scale. "Mode" designates the peaks of most frequent response.

Conclusion

Of the 303 questionnaires returned, only one indicated that no quantitative equivalent was suitable for any of the probabilistic expressions. All others selected sets of odds for at least half of those listed in Part One, and 80% did so for two-thirds of them. Even though a number who disapprove of quantitative definitions probably just did not bother to return their questionnaires, the results appear to indicate that the vast majority in the intelligence community consider it legitimate to think of such expressions in quantitative terms.

On the other hand, although more than 70% of both analysts and policy officers agreed within a 20-point range on the expressions in Categories A and B, the results for some offices on the analytical side did not agree with the consensus for all analysts, and there were similar exceptions among the policy offices. So when an analyst in one office uses the word probably, policy officers and analysts in other offices do not necessarily interpret the word to mean the same thing. In Categories A and B, however, the differences are usually not great. There follows the quantitative definition-most frequent plus and minus 10-of expressions on which there was found to be a satisfactory consensus.

Chances Out of 100

Are

90 - 100

Will

90 - 100

Almost Certainly

80 - 100

Undoubtedly

80 - 100

Highly Likely

75 - 95

Highly Probable

75 - 95

Probably Will

70 - 90

Probably

60 - 80

Likely

60 - 80

Good Chance

60 - 80

Seems Likely

60 - 80

Better Than Even Chance

50 - 70

May

40 - 60

Probably Not

10 - 30

Unlikely

10 - 30

Some Slight Chance

0 - 20

The out-of-context definitions in Part One were spot-checked by the sentence questions of Part Two. The results are not conclusive: only one sentence was provided for context, and there was no way of telling if respondents were influenced by personal knowledge of the subject matter. But despite these limitations, because the most frequent definitions in and out of context agreed within 10 points, it appears that nearly the same meanings were conveyed either way. The comparison appears below.

Most Frequent Response

In Context

Alone

Analyst

Policy

Analyst

Policy

Undoubtedly

100

--

Undoubtedly

100

90

Believe

80

80

Believe

70

70

Estimate

80

80

Estimate

75

70

Apparently

70

--

Apparently

70

--

Indicates that

70

--

Indicates that

70

--

Believe the Chances are Good

70

--

Good Chance

70

70

Possible

50

50

Possibly

50

50

Might

50

--

Might

50

50

Most Frequent Response

Analyst

Policy

Undoubtedly

100

90

Highly Probable

90

85

Highly Likely

90

80

Probably

75

70

Estimate

75

70

The results from Part Three showed there is little consensus on the common expressions of vague magnitude, at least without the guidance of context.

An effort was made to keep the questionnaire as simple to understand and as short as possible. In Parts One and Three the effort was generally successful, but Part Two was neither simple nor short. Most of the questions in the latter related to specific people and places, and there was danger that respondents would permit their opinions and knowledge of the subject to influence their answers. In addition, several of the estimative sentences were long and involved, carrying the hazard of confusion about what they meant and what was wanted in evaluation of them.

For pragmatic reasons, administration of the survey had to be informal. It is possible that such things as attitudes of supervisors, office collusion, or misunderstanding of the purpose of the survey could have introduced bias. A careful perusal of each of the questionnaires failed to turn up any obvious evidence that such factors influenced the findings. But if it were done again the questionnaire should be modified in Part Two and the conditions under which it is filled out should be controlled and standardized.

1 The full context on these questions was the sentence, "Although lacking the drama of visits by top leaders, the travel of these delegations to Albania indicates that the momentum of the Albanian-Polish rapprochement is being maintained and suggests that some progress is being made in reducing the area of remaining ideological differences." Respondents were asked to specify the probability that Albania and Poland were headed toward a rapprochement and the probability that the ideological differences would be settled.

2 Respondents were asked for the probability that the speaker believed what he conveyed.

3 Respondents were asked for the probability that changes would be minor.

4 Respondents were asked for the probability of that to which the "government is not committed." The full context is given on page 73.

5 This question was a non-sequitur. The full context is given on page 73.

* Required plugins

Footer Navigation

Contact CIA

The Office of Public Affairs (OPA) is the single point of contact for all inquiries about the Central Intelligence Agency (CIA).

We read every letter, fax, or e-mail we receive, and we will convey your comments to CIA officials outside OPA as appropriate. However, with limited staff and resources, we simply cannot respond to all who write to us.

Please check our site map, search feature, or our site navigation on the left to locate the information you seek. We do not routinely respond to questions for which answers are found within this Web site.

Employment: We do not routinely answer questions about employment beyond the information on this Web site, and we do not routinely answer inquiries about the status of job applications. Recruiting will contact applicants within 45 days if their qualifications meet our needs.

Because of safety concerns for the prospective applicant, as well as security and communication issues, the CIA Recruitment Center does not accept resumes, nor can we return phone calls, e-mails or other forms of communication, from US citizens living outside of the US. When you return permanently to the US (not on vacation or leave), please visit the CIA Careers page and apply online for the position of interest.

Solicitations to transfer large sums of money to your bank account: If you receive a solicitation to transfer a large amount of money from an African nation to your bank account in exchange for a payment of millions of dollars, go to the US Secret Service Web site for information about the Nigerian Advance Fee Fraud or "4-1-9" Fraud scheme.

If you have information which you believe might be of interest to the CIA in pursuit of the CIA's foreign intelligence mission, you may use our e-mail form. We will carefully protect all information you provide, including your identity. The CIA, as a foreign intelligence agency, does not engage in US domestic law enforcement.

If you have information relating to Iraq which you believe might be of interest to the US Government, please contact us through the Iraqi Rewards Program —

Contact CIA

Contact Us Form

Message:*

Email:*

Name:

Phone Number:

Mobile/Cell:

* = required

Contact CIA

Report Threats

The United States and its partners continue to face a
growing number of global threats and challenges. The CIA’s mission
includes collecting and analyzing information about high priority
national security issues such as international terrorism, the
proliferation of weapons of mass destruction, cyber attacks,
international organized crime and narcotics trafficking, regional
conflicts, counterintelligence threats, and the effects of environmental
and natural disasters.

These challenges are international in scope and are priorities for
the Central Intelligence Agency. If you have information about these or
other national security challenges, please provide it through our secure
online form. The information you provide will be protected and
confidential. The CIA is particularly interested in information about
imminent or planned terrorist attacks. In cases where an imminent
threat exists, immediately contact your local law enforcement agencies
and provide them with the threat information.