13
Implementing Change: Organizational Challenges

Amy Zegart

Improving organizational performance is never easy. As sociologist Jim March has noted, success requires that organizations balance exploration—the search for new ways of doing things—with exploitation, the ability to harness new practices and jettison older, less effective ones (March, 1991). These challenges confront all organizations, but two factors make them more acute for intelligence agencies. The first is bounded rationality (Simon, 1976). In the theoretical world, individuals have the luxury of perfect rationality, seeing all of the relevant options, assessing trade-offs with clarity, and making the best decisions. The real world is not as nice. There, rationality is inherently limited or bounded by uncertainty, imperfect information, and cognitive constraints that lead individuals to make decisions that appear to be “good enough”—but may turn out to be nowhere close (Simon, 1976). Intelligence officials have the toughest time of all, confronting bounded rationality problems in spades. Their job is to give policy-making customers decision advantage amidst swirling uncertainty, missing information, enemy deception and denial, and fast-changing events that are often unforeseeable, even to the participants themselves.

The second acute intelligence challenge is secrecy. As I discuss below, the more specialized any organization becomes, the harder it is for any one part of the organization to understand or improve what another part is doing, a phenomenon that sociologists call “structural secrecy” (Vaughan, 1996). In the classified universe, of course, this structural secrecy is compounded by actual secrecy, which protects vital information from adversaries, but also compartmentalizes information, ideas, organizations, and practices to a much greater extent.

The National Academies of Sciences, Engineering, and Medicine 500 Fifth St. N.W. | Washington, D.C. 20001

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 309
13
Implementing Change:
Organizational Challenges
Amy Zegart
Improving organizational performance is never easy. As sociologist Jim
March has noted, success requires that organizations balance exploration—
the search for new ways of doing things—with exploitation, the ability
to harness new practices and jettison older, less effective ones (March,
1991). These challenges confront all organizations, but two factors make
them more acute for intelligence agencies. The first is bounded rationality
(Simon, 1976). In the theoretical world, individuals have the luxury of
perfect rationality, seeing all of the relevant options, assessing trade-offs
with clarity, and making the best decisions. The real world is not as nice.
There, rationality is inherently limited or bounded by uncertainty, imperfect
information, and cognitive constraints that lead individuals to make deci-
sions that appear to be “good enough”—but may turn out to be nowhere
close (Simon, 1976). Intelligence officials have the toughest time of all,
confronting bounded rationality problems in spades. Their job is to give
policy-making customers decision advantage amidst swirling uncertainty,
missing information, enemy deception and denial, and fast-changing events
that are often unforeseeable, even to the participants themselves.
The second acute intelligence challenge is secrecy. As I discuss below,
the more specialized any organization becomes, the harder it is for any one
part of the organization to understand or improve what another part is
doing, a phenomenon that sociologists call “structural secrecy” (Vaughan,
1996). In the classified universe, of course, this structural secrecy is com-
pounded by actual secrecy, which protects vital information from adver-
saries, but also compartmentalizes information, ideas, organizations, and
practices to a much greater extent.
309

OCR for page 309
310 INTELLIGENCE ANALYSIS: FOUNDATIONS
Despite the intelligence community’s (IC’s) unique challenges, the fields
of organization theory and political science offer useful insights and cau-
tionary warnings about the organizational side of improving intelligence
analysis. The chapters in Part II (Analytic Methods) of this volume mine an
array of relevant literature for the best analytic tools to improve intelligence
analysis. Here, we turn to a different task: Examining a broad sweep of
relevant social science research with an eye to identifying which organiza-
tional factors impede or facilitate effective analysis. Worth underscoring,
though, is the fact that social science does not offer ready-made instructions
about how to make intelligence analytic improvements stick. However, it
does offer some useful generalizations that can illuminate the trade-offs and
challenges involved to guide more effective implementation.
INSIgHTS AND LIMITATIONS OF ORgANIZATION THEORY
Organization theory is a wide-ranging, multidisciplinary field that
includes sociology, psychology, political science, economics, and profes-
sional school fields such as urban planning and management. Although
organization theorists tackle vastly different questions using a multitude of
methodologies, they all share an interest in understanding how organiza-
tions behave, and why. In general, the field’s research is animated by three
central issues: (1) how internal organizational structures and features affect
organizational outcomes (particularly efficiency and survival); (2) how
external factors influence what goes on inside an organization; and (3) how
the interaction between internal and external forces shapes an organiza-
tion’s prospects for survival.
For our purposes, the field offers three insights for improving intel-
ligence analysis, described in the following pages.
Insight #1: Adopting New Practices Is Difficult Even for Firms
This idea is more important than it sounds. Critics frequently bemoan
that government is not run more like a business, and recommend export-
ing private-sector practices into public-sector bureaucracies (Osborne and
Gaebler, 1993; Osborne and Plastrik, 1998). The data show, however, that
most businesses are not run like businesses. Consider survival, which is the
most rudimentary indicator of firm adaptation (Aldrich, 1999).1 Accord-
ing to the U.S. Census Bureau, nearly a third of the 5.5 million American
businesses that existed in 1990 failed within four years (Aldrich, 1999).
1 As Aldrich points out, such findings most likely understate adaptation failure because they
focus only on surviving populations, excluding all of the organizations that never made it past
the start-up phase, when survival rates are considerably lower.

OCR for page 309
311
IMPLEMENTING CHANGE
Every year, more than half a million American businesses go bust. That’s
about 1,500 per day or 1 business every minute (U.S. Census Bureau, 2009,
Table 739).2 What’s more, social science research suggests that corporate
fads often flop. Pfeffer and Sutton (2006), for example, note that studies
repeatedly find that the majority of corporate mergers (some estimates are
70 percent or more) fail to deliver promised benefits and actually end up
destroying value. Analysis of 93 studies covering more than 200,000 merg-
ers published in peer-reviewed journals found that on average, the negative
effects of a merger on shareholder value appeared within days after the
merger was announced (Pfeffer and Sutton, 2006).
Even top-performing firms struggle to sustain their performance.
Between 1955 and 2005, for example, nearly 2,000 companies made For-
tune magazine’s list of the largest 500 U.S. corporations. Of these, only
three held the number one spot for more than a single year; 27 made the
list once without ever appearing again; and just 71, or 3.8 percent, man-
aged to stay on the list for the entire 50-year span (Schlosser and Florian,
2004).3 Between 2000 and 2003, more than 400 public companies went
bankrupt, including Enron, which rose to seventh on the Fortune 500 list,
and Bethlehem Steel, one of the great industrial giants of the 20th century
(Loomis, 2004; Serwer, 2002). Their combined liabilities reached more than
$500 billion, a figure 10 times greater than the annual budget for all U.S.
intelligence agencies combined (Office of the Director of National Intelli-
gence, 2007).4 As Lewin and colleagues (2004) conclude, the empirical data
clearly support the observation that “most firms are selected out” (p. 108).
These findings describe organizational adaptation prospects in the best
of circumstances; adaptation challenges are likely to be far greater in
public-sector agencies. As Allison (1980), Moe (1989), Wilson (2000),
Zegart (2007), and others have noted, private-sector firms enjoy key adap-
tation advantages that government agencies lack. Four are paramount.
First, market competition incentivizes firms to adapt or die. Indeed, popu-
lation ecology theorists argue that private-sector innovation arises between
organizations, not within them: Newer, fitter firms are constantly replac-
ing older, outdated ones through a Darwinian process of natural selection
(Hannan and Freeman, 1977, 1984). But this degree of organizational
churn does not exist in government. As many have observed, government
agencies are notoriously hard to kill because some interest groups and
2 Note that these figures cover firm deaths each year from 1990 to 2005. Because they pre-
date the current economic recession, they are likely to underestimate current firm death rates.
3 The three firms that remained at number one for more than a single year are General Mo-
tors, Exxon Mobil, and Walmart. Rankings are based on previous year’s revenues.
4 Intelligence budget calculations based on an Office of the Director of National Intelligence
press release, which reported the first post-9/11 declassified National Intelligence Program
budget: $43.7 billion for FY 2007 (Office of the Director of National Intelligence, 2007).

OCR for page 309
312 INTELLIGENCE ANALYSIS: FOUNDATIONS
elected officials out there will always resist (Downs, 1967; Stinchecombe,
1965; Lowi, 1979; Kaufman, 1976; Lewis, 2003).5 Public-sector agencies—
especially intelligence agencies—rarely fear they will go out of business.6
Instead, history has shown that policy makers usually respond to perceived
government failures by creating new agencies, not eliminating existing ones.
Although intelligence agencies may have other incentives to adapt, the mar-
ket’s powerful imperative to change or close up shop is not one of them.
These realities suggest that the benefits of competition are naturally
more limited in the IC than in the private sector. On the one hand, compe-
tition can stimulate ideas, sharpen analysis, guard against groupthink and
other pitfalls, and generate new ways of doing things. Yet because intelli-
gence agencies compete without the shadow of organizational death, weak
practices in one agency are likely to linger alongside better ones elsewhere.
The second advantage that firms enjoy in the adaptation struggle
is that their creators and employees want them to succeed (Moe, 1990;
Zegart, 1999, 2007). In the business world, no one foists a new company
on reluctant owners and no employee cheers silently for the day when
company profits plummet. Instead, businesses are filled with organiza-
tional well-wishers who have vested interests in the organization’s con-
tinued success. Government agencies, by contrast, are created by many
who want them to fail. In politics, new agencies are forced into existence
by winning political coalitions who impose their will on the losers. This
means that losers have a say in the new organization’s design and opera-
tion. The fragmented structure of the American political system ensures
that political opponents have many opportunities to sabotage the creation
of a new agency at the outset, hobbling it with all sorts of structures,
rules, and requirements that hinder its performance over time (Moe, 1989;
Zegart, 1999, 2007). As Terry Moe writes, “American public bureaucracy
is not designed to be effective” (Moe, 1989, p. 267). Whether it’s the Envi-
ronmental Protection Agency or the Office of the Director of National
Intelligence (ODNI), government agencies are constrained from the start
by the politics of their own creation.
The third advantage businesses have when it comes to driving orga-
nizational change is managerial discretion. Subject only to minimal legal
requirements, managers in private firms can determine or change their
5 David Lewis has questioned the immortality thesis, finding that 438 new agencies were
created between 1946 and 1997. But to put those numbers into perspective, more businesses
are born in a single day before lunch.
6 Indeed, congressional scholars have made much of Congress’s oversight powers. See
McCubbins (1985); Weingast and Moran (1983); and Epstein et al. (1999). But Congress’s
oversight weapons are much weaker than they appear and at times create perverse incentives,
rewarding failures by granting bigger budgets, more personnel, and other corrective measures
that bureaucracies value (see Moe, 1987).

OCR for page 309
313
IMPLEMENTING CHANGE
organization’s mission; hire and fire whomever they choose; institute
whatever procedures, policies, and customs they believe are necessary;
and attract capital from a multitude of sources. As James Q. Wilson
shows in detailed case studies that range from prisons to schools to the
Central Intelligence Agency (CIA), public-sector managers are far more
constrained. They can only dream of exercising this kind of discretion to
shape the organization’s mission and match resources against priorities
(Wilson, 2000).
Fourth and finally, businesses typically have an easier time instituting
major change because chief executive officers (CEOs) usually stay on the
job longer than their public-sector counterparts. Although CEO tenure
has declined in recent years, it still averages 7 years (Kaplan and Minton,
2006; Kelman and Myers, 2009). That’s more than twice as long as the
3.3-year median tenure of Senate-confirmed Cabinet secretaries and three
times longer than the median service of deputy-secretary–level appointees
in the first Bush and Clinton Administrations (Dull and Roberts, 2008).
Average tenure of top intelligence officials is even shorter: Since 9/11, CIA
director tenure has averaged 2 years, and directors of national intelligence
(a position created in April 2005) have averaged 1.47 years. Although the
Federal Bureau of Investigation (FBI) director holds a 10-year fixed term,
the Bureau’s top counterterrorism position has been held by eight people
since 9/11, averaging just 1 year each (Stein, 2006).7 These figures are par-
ticularly noteworthy given the fact that organization theorists consistently
have found that frequent leadership turnover hurts firm performance.8
In sum, organization theory tells us that adaptation is difficult under
the best of circumstances. Businesses are fortunate. They are fueled by
market competition and its shadow of death, focused by a unified mission,
filled with stakeholders seeking success, armed with broad managerial dis-
cretion to match resources against organizational needs, and led by senior
executives who stay long enough to see major changes implemented. But
even these blessings lead to failure more frequently than one might expect.
7 Since the article was printed, Arthur Cummings became executive assistant director of the
National Security Branch, making him the eighth top counterterrorism official.
8 Classic early work in the 1960s and 1970s examined sports teams and found that frequent
coaching turnover was correlated with poor team performance. Since the 1990s, a robust
literature has found the relationship between executive tenure and firm performance to be
curvilinear. Organizational performance typically rises with CEO tenure to a point, then falls
as executives and organizations get stuck in outmoded thinking and practices. Importantly,
Kelman and Myers (2009) note that the CEO tenure inflection point (when performance starts
to diminish) is 5 years or more. This is substantially longer than the tenure of most intelli-
gence agency heads. For turnover literature, see Kesner and Sebora (1994); Dull and Roberts
(2008); Rainey and Steinbauer (1999); and Kelman and Myers (2009). For related work on
institutional change and the survival of political leaders, see Bueno de Mesquita et al. (2003).

OCR for page 309
314 INTELLIGENCE ANALYSIS: FOUNDATIONS
Insight #2: Organizational Structure Matters More Than We Think
The second insight focuses on the relationship between an organiza-
tion’s structure and its ability to learn. Cyert and March’s 1963 classic,
A Behavioral Theory of the Firm, first introduced the idea that organiza-
tions were not fixed and rigid, but adaptive learning systems. Subsequent
research was quite diffuse, but generally agreed on four important points:
(1) organizational learning involves acquiring, processing, and integrating
information important to the functioning of the organization; (2) organiza-
tional learning positively affects future performance (Fiol and Lyles, 1985;
Levitt and March, 1988); (3) organizations learn in a host of directed and
spontaneous ways; and (4) organizational structure can influence learning
in profound and often hidden ways.
This last point is particularly important for intelligence agencies
because they are in the information learning business, confront extreme
levels of uncertainty, and have faced persistent calls for structural overhaul
since World War II. The list of reorganization efforts is long, including the
CIA’s creation in 1947; the National Security Agency’s establishment in
1952; the consolidation of imagery into the National Imagery and Mapping
Agency in 1996; the creation of the Terrorist Threat Integration Center in
2003 and its successor, the National Counterterrorism Center, in 2004; the
creation of the ODNI in 2005; and repeated counterterrorism, intelligence,
and national security reorganizations inside the FBI from the 1990s to the
present. In each case, reformers sought to improve the IC’s performance
by restructuring the organizations within it. As Hammond (2009) writes,
“while many prescriptions for intelligence community ‘reform’ have proved
difficult to implement, structure seems to have been subjected to reforms
and reorganizations fairly often, perhaps because structural problems are
seen, whether correctly or not, as more easily solved” (p. 4).
Organization theorists have not settled the question of which structural
arrangements are best, even in private industry. However, they have illumi-
nated more clearly why no one best structure exists.
Briefly put, organization theorists have found neutral design to be
impossible; the structure of the organization itself—its hierarchy, its
arrangement of subunits—affects how information is organized and what
decisions result (Simon, 1976; Hammond and Thomas, 1989; Seidman,
1998). A hypothetical example illustrates the point. Imagine for a moment
that you are the head of an agency, and you possess magical powers to
eliminate all conceivable sources of bias so that your decisions are based
solely on the information provided by your subordinates. Waving your
wand, you eliminate the personal and cognitive biases of everyone in the
organizational chain of command, including yourself. You neutralize the
pressures of political interests and external stakeholders seeking a particular

OCR for page 309
315
IMPLEMENTING CHANGE
outcome. You eliminate the pathologies of small-group decision making.
You ensure that information does not get filtered or altered in the commu-
nications process, so each subordinate unit passes along all the information
it has. You align incentives so that everyone has every reason in the world
to provide “just the facts,” information that is unvarnished, untainted, and
unconnected to personal or career objectives. Furthermore, let’s assume
that all the information you receive is highly credible. Even in these ideal
circumstances, your decision will be biased, and it may turn out to be
wholly inconsistent with the data. Why? Because how you organize units
in the bureaucracy determines whether the same pieces of information get
concentrated as signals or dispersed as noise (Wohlstetter, 1962).
Bendor and Hammond (2010) provide two simple examples that show
these structural forces at work. In the first, an intelligence agency director
has three bureaus monitoring terrorist groups. The director will alert the
President about a possible impending attack only if at least two of the three
bureau chiefs report that they are concerned about terrorist activity patterns
in their domains. Bureau chiefs, in turn, operate with the same decision
rule: A bureau chief will send a report expressing concern to the director
only if at least two of his three subordinates raise a red flag. Reporting
is determined by answering the following question: “Do you believe that
the groups in your jurisdiction are intensifying their terrorist activity?” A
“0” means “no,” and a “1” means “yes.” Table 13-1 shows the same data
aggregated in two different structures.
The first structure organizes bureaus by geography: Regions A, B,
and C. Inside each regional bureau, subordinates are responsible for track-
ing the activities of al Qaeda-affiliated, Iran-affiliated, and unaffiliated ter-
rorist groups. The bureau chief from region A gets signals of concern from
all three subordinates (1,1,1), so he sends a report to the agency director.
Region B’s bureau chief gets only one signal of concern (1,0,0), so he does
not send a report to the director. Region C also has only one signal (0,1,0),
so does not report a concern. In this structure, because only one of the three
regional bureaus raises a red flag, the director does not alert the President.
Now consider the second structure, which organizes bureaus by the
TABLE 13-1 The CIA Reporting Problem
Region A Region B Region C
AQ-affiliated groups 1 1 0
Iran-affiliated groups 1 0 1
Unaffiliated groups 1 0 0
NOTE: AQ = al Qaeda.
SOURCE: Bendor and Hammond (2010, p. 651:Table 27.2). Reprinted by permission of
Oxford University Press, see http://www.oup.com.

OCR for page 309
316 INTELLIGENCE ANALYSIS: FOUNDATIONS
type of terrorist groups they monitor. Within each bureau, subordinates
track activities in different geographic regions. The bureau that monitors
al Qaeda-affiliated groups receives two reports from regional subordinates
(1,1,0), so it reports concern to the director. The Iran-affiliated group
bureau also receives two signals from different regions (1,0,1), so it reports
concern. Because two of three bureaus have reported concern, the director
alerts the President. The data and decision rules are exactly the same in
both structures. But because these two structures aggregate the information
differently, the director warns in one case, but not the other.
In the second example, Bendor and Hammond (2010) also show how
hierarchies can produce counterintuitive judgments. Now an agency director
wants to know whether al Qaeda-affiliated groups are more or less likely
than Iran-affiliated terrorist groups to commit attacks in the near future (see
Table 13-2). There are two bureaus. Bureau A’s information suggests that 20
percent of al Qaeda-affiliated groups (10 of 50) are planning terrorist attacks,
while no Iran-affiliated groups are planning attacks. Bureau A therefore
concludes that al Qaeda-affiliated groups are more likely to commit terrorist
activities in the near future. Bureau B has different data showing that 100
percent of al Qaeda-affiliated groups (10 of 10) are planning terrorist attacks,
while 80 percent (40 of 50) of Iran-affiliated groups are planning attacks.
Based on these data, Bureau B also reports to the director that al Qaeda-
affiliated groups are more likely to commit near-term attacks.
However, when the director aggregates the data from both bureaus, she
finds a very different picture: One-third of al Qaeda-affiliated groups (20
of 60) are planning near-term attacks, while two-thirds of Iran-affiliated
groups (40 of 60) are planning attacks. Using the same metrics (percentage
TABLE 13-2 Terrorist Activities Reports
Bureau A Bureau B
Terrorist No Terrorist Terrorist No Terrorist
Activities Activities Activities Activities
Planned Planned Planned Planned
AQ-affiliated 10 40 AQ-affiliated 10 0
groups groups
Iran-affiliated 0 10 Iran-affiliated 40 10
groups groups
NOTE: AQ = al Qaeda.
SOURCE: Bendor and Hammond (2010). Original publication included two tables: Origi-
nal table: 27.3 Terrorist Activities Reports—Bureau A (p. 652) and 27.4 Terrorist Activities
Reports—Bureau B (p. 652). Reprinted by permission of Oxford University Press, see http://
www.oup.com.

OCR for page 309
317
IMPLEMENTING CHANGE
TABLE 13-3 Director’s Aggregated Data from Bureaus A and B
Terrorist Activities No Terrorist Activities
Planned Planned
AQ-affiliated groups 20 40
Iran-affiliated groups 40 20
NOTE: AQ = al Qaeda.
SOURCE: Table derived from Bendor and Hammond (2010).
of affiliated groups planning attacks) and the same decision rule (select the
group type with the higher percentage of member organizations planning
attacks), the director reaches the opposite conclusion of Bureaus A and B.
She judges that Iran-affiliated groups are more likely to commit near-term
attacks (see Table 13-3).
As this example illustrates, data collected in subunits can lead every
subunit to the same evidence-based hypothesis, even when the aggregation
of data across subunits suggests the exact opposite belief. Called Simpson’s
paradox, this problem is well known among statisticians and occurs when
associations between variables in smaller datasets become inverted once the
data are combined (Simpson, 1951). One of the more popular examples
of Simpson’s paradox involves the batting averages of baseball stars Dave
Justice and Derek Jeter. Although Justice had a higher batting average than
Jeter in 1995 and 1996, Jeter had a higher batting average when data from
both years were totaled. The reason: large differences in the number of at-
bats each year (Ross, 2004).
Intelligence experts, of course, have long been aware of structural
dilemmas. In 1949, Sherman Kent explicitly contemplated the trade-offs
between a centralized versus decentralized intelligence system as well as
the relative costs and benefits of organizing units by geography or function
(Kent, 1949). No arrangement, he concluded, was ideal.9 But more recent
organization theory suggests that these structural problems may be even
more pernicious than many realize. The Bendor and Hammond examples
provide a cautionary warning: Robust analytic techniques are not enough.
Organizational structures can exert enormous, unseen, and unexpected
influence over how information is aggregated and what hypotheses emerge
(Bendor and Hammond, 2010).
Organizational structure also affects an organization’s ability to learn
9 Kent (1949) came down in favor of “the regional breakdown as far as possible,” but ac-
knowledged that such a structure posed two problems: “how to handle matters which defy
regionalization” and “how to handle those problems of a multinational nature for which the
organization provides no full-time functional supervisor or coordinator” (pp. 122–123). See
also Hammond (2009).

OCR for page 309
318 INTELLIGENCE ANALYSIS: FOUNDATIONS
and improve its own performance. As Vaughan (1996) and Zegart (2007)
have noted, the structure of an organization can impede its ability to adapt,
even when the need to adapt is clear. The key here is specialization. In their
quest for efficiency, organizations create subunits to break down large tasks
into smaller ones. Each subunit becomes specialized, using particular skills,
employing particular people, and developing particular knowledge so each
part of the organization does what it does best. But these pockets of spe-
cialization make it difficult for one part of the organization to understand
the work of another, complicate coordination by creating distance between
managers and operators, and foster standardized ways of communicat-
ing and operating across organizational divisions. Although March and
Simon’s (1958) classic work finds many benefits to standard operating
procedures,10 more recent research finds that standard operating procedures
are a double-edged sword, increasing organizational reliability but ham-
pering innovation.11 Standard forms, automated computer systems, and
reporting procedures help managers across an organization to perform the
same tasks in the same ways each time. These measures, however, also weed
out new ideas and stifle improvements that do not fit easily into existing
forms, channels, or procedures—a phenomenon Vaughan calls “structural
secrecy” (1996).
Two examples show the powerful effects of structural secrecy at work.
First, Vaughan’s case study of the Space Shuttle Challenger disaster finds
that Morton Thiokol engineers were gravely concerned about the resilience
of the shuttle’s O-ring joints in cold weather. They turned out to be right:
In 1986, Challenger exploded shortly after launch because abnormally
cold weather had caused the O-rings on the solid rocket boosters to fail.
The night before the disaster, Thiokol’s engineers desperately tried to abort
the launch. But their warnings were muted and ultimately disregarded in
large part because of the National Aeronautics and Space Administration’s
(NASA’s) own standard operating processes, structures, and norms. No
minimum temperature launch criterion had been established, so NASA
managers did not see the urgency of creating one the night before a launch.
Thiokol’s crucial presentation relied on qualitative judgments from previous
flights (the putty damage between the O-rings looked different in colder
weather flights than others) rather than NASA’s standard “engineering-
supported” technical positions that were based on quantitative analysis.
Because the Shuttle program’s division of labor physically separated key
10 March and Simon argued that standard operating procedures help organizations cope with
two problems: too much information and too little information. Standard procedures, they
noted, simplify the task of management and provide useful feedback loops that enable manag-
ers to identify trends early enough to take corrective action before problems turn into crises.
11 For problems with standard operating procedures, see Allison (1971); Vaughan (1996);
and Sagan (1993).

OCR for page 309
319
IMPLEMENTING CHANGE
participants in different locations, the pivotal communication occurred in
a three-way teleconference, with no video transmission. As Vaughan notes,
“many visual cues that normally aid interpretation—such as gestures, facial
expressions, body posture, activity—were unavailable.” Instead, “commu-
nication depended on individual willingness to speak to an unseen audi-
ence” (Vaughan, 1996, p. 357). Paradoxically, the very structures, rules,
and technologies designed to improve organizational efficiency sabotaged
NASA’s ability to learn.
Zegart finds that structural secrecy also hindered the FBI’s ability to
penetrate the 9/11 plot. In a 7-week period during the summer of 2001,
three FBI field offices uncovered what turned out to be key clues. In Phoe-
nix, Special Agent Kenneth Williams identified a disturbing trend, wrote
a memo warning that Osama bin Laden might be sending terrorists to
train in U.S. flight schools, and recommended several specific steps, includ-
ing notifying other intelligence agencies. As FBI Director Robert Mueller
later reflected, “You are not going to have a better intelligence product
than the Phoenix memo.”12 During the same period, FBI agents in Min-
neapolis detained a suspicious foreign flight school student named Zacarias
Moussaoui, a self-proclaimed Jihadist who wanted to fly 747s and later
became the only person convicted in the United States in connection with
the attacks. Third and finally, the FBI’s New York office began search-
ing for Khalid al-Mihdhar and Nawaf al-Hazmi, two suspected al Qaeda
operatives who later hijacked American Airlines Flight 77 and flew it into
the Pentagon. But because the FBI was divided into 56 largely independent
and autonomous field offices (one longstanding joke at the Bureau was that
the FBI consisted of 56 field offices with a headquarters attached), none of
the agents working these cases knew about the others. On three separate
occasions in that 7-week period, the threat of a domestic terrorist attack
caught the attention of someone in the FBI, but failed to trigger a broader
effort to collect information, share information, or take stock of what the
FBI already knew. The Bureau’s field office structure enhanced specializa-
tion—enabling individual field offices to address local law enforcement
priorities—but prevented officials in one part of the organization from
learning what others in the organization already knew (Zegart, 2007).
In sum, organizational learning research suggests that structure mat-
ters much more than most people believe, that organizational reliability
and innovation are often mutually exclusive, that managers must work
outside standard operating procedures to identify obsolete practices and
foster innovation, and that officials must be vigilant about monitoring
how structural arrangements aggregate, or fail to aggregate, information
to guard against misleading analytic judgments.
12 Personal communication, Robert Mueller, FBI, January 2007.

OCR for page 309
320 INTELLIGENCE ANALYSIS: FOUNDATIONS
Insight #3: Internal Barriers to Organizational Change Are Powerful
Social science research finds what intelligence insiders already know
to be true: employees become wedded to organizational routines, thinking,
norms, ideas, and identities and these attachments make change difficult
(see Tinsley, this volume, Chapter 9, for discussion of these issues in greater
depth). Here, a point worth underscoring is that resistance to innovation
stems more from the everyday aspects of organizational life than from a
few old-timers or old-thinkers. Levitt and March argue that organizational
performance often falls victim to “competency traps,” which are routines
that were once beneficial, but have become obsolete over time (Levitt
and March, 1988; March, 1981). Avoiding competency traps requires
systemic and careful work to identify and exploit “old knowledge” that
still works (March, 1991; Crossan et al., 1999), “unlearn” routines that
do not (Hedberg, 1981), and explore new approaches that might work
better (March, 1991; Levinthal and March, 1993). For intelligence, this
research suggests that improving analysis requires more than hiring talent
or generating good ideas and new tools. It requires an explicit management
program to identify and shed maladaptive practices, encourage the search
for new and better ones, foster supportive cultures and habits, and erode
counterproductive ones.
Limitations
The most serious limitation of organization theory is its focus on firms.
As Steve Kelman (2007, p. 226) writes, “Improving government perfor-
mance is a topic worthy of significant research attention, yet dramatically
insufficient scholarly firepower is directed at it.” The result is that organiza-
tion theory pays relatively little attention to political incentives, institutions,
and power, forces that are crucial for understanding adaptation challenges
in government agencies (Zegart, 2007).
INSIgHTS AND LIMITATIONS OF POLITICAL SCIENCE
The political science literature offers different insights and limitations
for improving intelligence analysis, as described in the paragraphs below.
Insight #1: Institutional Incentives Drive Behavior
Although the political science literature is vast, the discipline’s domi-
nant approach for the past 20 or 30 years has been rational choice. See
Chapter 3, this volume, by Bruce Bueno de Mesquita for discussion of
rational choice analysis in much greater depth and for an examination of

OCR for page 309
321
IMPLEMENTING CHANGE
how game theoretic models offer useful analytic tools. But rational choice
also illuminates the “how to make good analytic practices stick” side of
the equation.
Put simply, theories of rational choice focus on what makes individu-
als alike, not what makes them different. Rational choice theorists argue
that all individuals, whatever their personalities, wants, and needs, act in
predictable and systematic ways for predictable and systematic reasons:
Namely, they select alternatives and conduct activities that maximize net
benefits to themselves. In politics, individuals are driven by the incentives
of office to maximize their political advantages. No normative judgment is
implied; rational choice describes the way the political world works, not
the way reformers wish it to be.13
Legislators, for example, select committee assignments that deliver
benefits to folks back home because they prefer winning reelection to losing
(Mayhew, 1974). Similar dynamics explain Presidential behavior. Although
no two Presidents are alike, all of them wield the same powers, confront the
same institutional players, seek to secure their place in history, and make
decisions based on which policies produce the greatest advantages for their
administration at the lowest political cost. For political scientists, outcomes
stem less from the idiosyncratic personalities or beliefs of individuals, and
more from the forces that transcend them (Moe, 1985, 2009).
For intelligence analysis, rational choice theories remind us that leader-
ship is not a panacea; institutional incentives frequently explain why people
and organizations behave in the ways they do—for example, why constitu-
ent elements of the IC historically resisted centralization under the CIA,
and why they are likely to continue resisting centralization under the new
ODNI, including efforts to improve analytic practices, even now.14
At the ground level, rational choice theory suggests that bad incentives
often prevent good people from improving organizational performance. A
new analytic technique, for example, may produce better judgments. But
getting analysts to use it requires convincing them, and their managers,
that the costs of learning and using something new are worth it. Although
charismatic leadership can help foster change, institutionalizing these kinds
of improvements requires structuring incentives and communicating them
clearly. Net career benefits—for each person involved—matter a great deal.
In short, the literature suggests that making improvements stick means
relying less on the force of individual personalities and more on harnessing
the incentives that motivate us all.
13 Three seminal works in rational choice are Arrow (1951); Downs (1957); and Olson
(1965). For an important critique, see Green and Shapiro (1996).
14 I do not mean to suggest that rational self-interest is the only reason intelligence elements
might resist centralization. But it is an important and often underappreciated one.

OCR for page 309
322 INTELLIGENCE ANALYSIS: FOUNDATIONS
Insight #2: Individual Rational Decisions, Collective Suboptimal Results
Political science research also cautions that individually rational deci-
sions can produce collectively suboptimal results. The classic example is
the tragedy of the commons, where individual farmers seek to gain advan-
tage by allowing their sheep to graze as much as possible on public lands.
Yet, because every farmer has the same cost–benefit calculation, they all
make the same choice. Overgrazing ensues, the fields become fallow, and
everyone suffers. Current examples of tragedy of the commons problems
abound. Nobody likes wasteful government spending, but every member of
Congress has strong incentives to draft legislative earmarks to fund his or
her district’s pet projects, leading to wasteful earmark proliferation. When
the stock market starts falling dramatically, the natural reaction among
nonprofessional investors (and some professional ones) is often to avoid
bigger losses by selling fast. But when many respond to these incentives in
the same way, the market plummets even more and losses grow. Rational
behavior for one becomes detrimental for all. This same basic logic explains
in part why intelligence agencies in the Pentagon and other parts of the
IC historically have fought against centralized control by the Director of
Central Intelligence (DCI) and its ODNI successor, even though doing so
hinders the coordination and collaboration essential to intelligence success.
One reason agency employees circumvent or resist central directives is that
they see personal or organizational benefits to protecting their own agency’s
turf and costs to ceding it. The result, however, is that the entire intelligence
system suffers.15
Limitations
Political science has been hampered by two key weaknesses. The first
is that the field rarely treats agencies as dependent variables. Organizations
are inputs to policy outcomes, not phenomena to be studied in their own
right. Most political scientists are uninterested in internal organizational
forces such as norms, routines, and cultures, precisely the forces that fuel
bureaucratic resistance to change. Indeed, “culture” is something of a dirty
word in the discipline, denoting a residual, “squishy” variable that can-
not be measured clearly and that is usually employed only when all other
explanations fall short.
The second limitation stems from the first: Political science pays little
attention to the nuts and bolts of how agencies actually work. Although
public administration and political science used to be closely aligned fields,
15 Of course, there are also pathological and psychological reasons for resisting centraliza-
tion, including rigid adherence to outdated agency cultures, traditions, and identities, and
more general aversion to change.

OCR for page 309
323
IMPLEMENTING CHANGE
they split decades ago. For years now, political science has considered pub-
lic administration to be too practically oriented, too atheoretical, and too
methodologically weak. The claims are not entirely without merit (Kelman,
2007). But the effect has been to create a yawning gap between theory and
practice, and a dearth of policy-relevant political science work to inform
public management. It is no coincidence that the “reinventing government”
movement, which gave rise to the Clinton–Gore National Performance
Review, came from practitioners instead of scholars (Kettl, 1998, 2005;
Aberbach and Rockman, 2000).
A WORD ABOUT THE BUSINESS MANAgEMENT LITERATURE
A separate and growing body of research concerns the practical inter-
ests of managers. In the early days, the debate focused mostly on how to
improve firm efficiency. Taylor’s seminal work in 1911 argued that man-
agers’ core challenge was to institute practices that increased managerial
control, reduced worker discretion, and broke down tasks into smaller and
smaller pieces. Rejecting the aphorism that “Captains of industry are born,
not made,” Taylor sought, as he put it, “to try to convince the reader that
the remedy for . . . inefficiency lies in systematic management, rather than
in searching for some unusual or extraordinary man” (Taylor, 1911, p. 7).
Starting in the 1930s and 1940s, Harvard Business School produced an
alternative “human relations” approach that found workers also needed
to be motivated to be productive.16
After World War II, business programs skyrocketed, producing major
changes and a growing popular orientation. In 1956, fewer than 4,000 stu-
dents received a Master’s in Business Administration (MBA). By 2003, that
number had topped 100,000 (U.S. Department of Education, 2005).17 In a
20-year period alone—from 1974 to 1994—the number of American uni-
versities offering MBA degrees doubled, from 389 to nearly 800 (Deutsch,
1993). Because business schools are in the business of training managers,
they have an incentive to produce research that highlights the importance
of leadership and the role of managers inside organizations (Kelman, 2007).
Although important social science research has continued to be developed
inside business schools, a cottage industry of best-selling leadership and
management books has also arisen, dispensing advice to business leaders
16 I n
a series of famous experiments at the Hawthorne Western Electric Plant, Roethlisberger
and Dickson (1949) found that worker productivity increased under any form of attention.
One of the seminal theoretical works in the field is Barnard (1938). For an excellent overview
of the literature, see Charles Perrow (1986).
17 That’s five times the number of students studying for Master’s Degrees in Public Policy or
Public Administration (Kelman, 2007).

OCR for page 309
324 INTELLIGENCE ANALYSIS: FOUNDATIONS
and general audiences alike.18 Despite its popularity, however, this literature
has substantial limitations in improving intelligence analysis. Two reasons
explain why.
First, the literature assumes away nearly all of the most important
constraints on government agencies. Wallace Sayre’s oft-quoted law that
public and private management are fundamentally alike in all unimportant
respects has fallen by the wayside (Allison, 1980). To be clear, this litera-
ture does not assert that its lessons apply well to government agencies; it
neglects government agencies altogether (Kelman, 2007). General rules of
thumb are drawn almost entirely from private-sector cases and are intended
for private-sector audiences. Grafting these ideas from firms to intelligence
agencies is difficult. For example, Jim Collins’s (2001) book, Good to
Great, examines the factors that distinguish high-performing firms from
average ones in the same industry. One of his key findings is personnel, or
as he puts it, “getting the right people on the bus and getting the wrong
people off the bus.” This advice makes good sense for companies, but
overlooks important intelligence realities. In the intelligence world, antici-
pating who the “right people” are and how many of them you’ll need is
riddled with uncertainty. The right people at one point in time (say, Warsaw
Pact experts) may turn out to be the wrong people later. Conversely, some
employees (e.g., Pashtu speakers) may seem relatively insignificant one day
and indispensable the next. Aligning the workforce will always lag substan-
tially behind an intelligence agency’s needs because hiring people entails
undergoing a lengthy security clearance process and firing them requires
dealing with onerous civil service procedures and regulations. Selecting
the “right people” hinges as much on identifying intangible qualities—a
willingness to embrace change and take intellectual risks, a drive to get
things done, an aptitude for working well with intelligence customers and
colleagues—as substantive knowledge or other measurable skills. Finally,
for decades intelligence agency cultures have prized lifetime service to the
mission and country, not “here today, gone tomorrow” labor markets
where organizations and employees alike expect to move on as conditions
warrant.19 Getting on and off the intelligence bus is not so fast or easy.20
The second limitation of this work is methodological. With some
important exceptions (Collins, 2001), the popular management literature
18 Some of the best known examples are Collins (2001); Useem (1999); and Kotter (1996).
19 The demographics of today’s IC workforce raise new questions about career tenure—spe-
cifically, whether the post-9/11 generation of analysts expects more fluid career paths into
and out of government and, if so, how the IC can harness top talent either through modify-
ing retention practices or developing career paths that enable analysts to move in and out of
government more easily.
20 Collins defends the applicability business practices to nonprofits and government agencies
in Good to Great and the Social Sectors: A Monograph to Accompany Good to Great (2005).

OCR for page 309
325
IMPLEMENTING CHANGE
commits many of the selection bias errors discussed by Bueno de Mesquita
(this volume, Chapter 3). In general, the literature presents sweeping con-
clusions, nostrums, and top-10 lists based on illustrative case studies and
weak causal reasoning rather than more rigorous experimental testing,
surveys, or systematic research methods. Peters and Waterman’s In Search
of Excellence (1981) is a classic example. The authors examine several top-
performing companies, find a few things these companies have in common,
and conclude that the commonalities must be the keys to success. Peters
and Waterman might be right. Or they could be terribly wrong, identifying
traits that are shared by most companies—successes and failures alike—and
that have little or no bearing on performance.
These methodological weaknesses have created a great deal of con-
ventional management wisdom with questionable results. In 1996, John
Kotter published one of the best known change-management books ever
written, Leading Change. Kotter’s book contained no references, footnotes,
or rigorous empirical research unless one counts occasional references to
“that reminds me of a story” illustrative examples. Nevertheless, Leading
Change spawned a huge change-management movement that produced
thousands of articles and books. Yet in 2008, a McKinsey and Company
survey of 3,199 executives around the world reported that only a third of
all transformations succeeded, the same percentage that Kotter found 12
years earlier. The McKinsey study concluded, “It seems that, despite pro-
lific output, the field of change management hasn’t led to more successful
change programs” (Aiken and Keller, 2009 p. 100).21
The point here is not to criticize for the sake of criticizing. It is to shine
a light on which social science research paths offer dead ends and which
offer promising avenues to improve the implementation of analytic prac-
tices. In the final analysis, organization theory and political science offer
some important, relevant insights. The popular management literature,
however, appears far less promising for improving intelligence analysis.
REFERENCES
Aberbach, J. D., and B. A. Rockman. 2000. In the web of politics: Three decades of the U.S.
federal executive. Washington, DC: Brookings Institution Press.
Aiken, C., and S. Keller. 2009. The irrational side of change management. McKinsey Quarterly
2:100–109.
Aldrich, H. 1999. Organizations evolving. Englewood Cliffs, NJ: Prentice-Hall.
Allison, G. 1971. Essence of decision: Explaining the Cuban missile crisis, 1st ed. Boston,
MA: Little Brown.
21 These authors go on to commit the same methodological mistakes, proffering advice about
how to handle “the irrational side of change management” based largely on their own profes-
sional experiences working with companies attempting transformations.