Below is a raw (and likely hideous) rendition of the
original report.
(PDF)

United States Government Accountability Office
GAO Report to Congressional Committees
September 2012
ORGANIZATIONAL
TRANSFORMATION
Enterprise
Architecture Value
Needs to Be Measured
and Reported
GAO-12-791
September 2012
ORGANIZATIONAL TRANSFORMATION
Enterprise Architecture Value Needs to Be
Measured and Reported
Highlights of GAO-12-791, a report to
congressional committees
Why GAO Did This Study What GAO Found
According to OMB, the federal Among the 27 agencies that GAO studied, all have fully or partially defined goals or
executive branch plans to spend at purposes for their architectures, 11 have fully or partially established a method or
least $75 billion on information metrics for measuring outcomes resulting from the use of their architectures, while
technology (IT) investments in fiscal 5 have fully or partially measured and reported outcomes and benefits
year 2012. In response to a statute (see table).
which mandates that GAO identify
duplicative activities within federal Outcomes and
Goals or Metrics and benefits periodically
agencies, GAO previously identified purpose method measured and
enterprise architecture as a Agency defined established reported
mechanism for reducing duplication Agriculture ● ○ ○
and overlap in investments. An Air Force ● ○ ○
architecture is a “blueprint” that
Army ● ◐ ○
describes how an organization
Commerce ● ◐ ○
operates in terms of business
processes and technology, how it Defense—Business Enterprise Architecture ● ○ ○
intends to operate in the future, and Defense—Enterprise Architecture ● ○ ○
how it plans to transition to the future Education ● ◐ ◐
state. Knowing whether architecture Energy ● ○ ○
outcomes are being achieved requires Health and Human Services ● ● ◐
defining the architecture’s goals,
Homeland Security ● ○ ○
establishing a method and metrics to
Housing and Urban Development ● ● ◐
measure architecture outcomes, and
periodically measuring and reporting Interior ● ○ ○
these outcomes. To assess agencies’ Justice ● ○ ○
use of architecture as a mechanism for Labor ● ○ ○
reducing duplication and overlap, GAO Navy ● ○ ○
committed to determine the extent to State ● ○ ○
which agencies are measuring and Transportation ● ◐ ○
reporting architecture outcomes and
Treasury ● ◐ ◐
benefits. To do this, GAO reviewed
Veterans Affairs ◐ ○ ○
relevant documentation from 27 major
federal agencies, reviewed the results Environmental Protection Agency ● ○ ○
of a GAO survey on the benefits of General Services Administration ● ◐ ○
using architecture, and interviewed National Aeronautics and Space
● ○ ○
agency officials. Administration
National Science Foundation ● ○ ○
Nuclear Regulatory Commission ● ◐ ○
What GAO Recommends
Officer of Personnel Management ● ◐ ○
GAO is making recommendations to Small Business Administration ● ○ ○
the agencies and OMB to improve Social Security Administration ● ○ ○
measurement and reporting of United States Agency for International
architecture outcomes. In commenting ● ● ●
Development
on a draft of this report, OMB and most Source: GAO analysis of agency data.
of the agencies generally agreed with
the findings and recommendations. Agencies cited a lack of guidance as a key reason why they have not established
methods and metrics for measuring outcomes and benefits. Although the Office
of Management and Budget (OMB) has issued recent enterprise architecture
View GAO-12-791. For more information, guidance to agencies, OMB has not yet provided sufficient details on the method
contact Valerie C. Melvin at (202) 512-6304 or and metrics that could be used to measure architecture program outcomes.
melvinv@gao.gov.
United States Government Accountability Office
Contents
Letter 1
Background 3
Almost All Agencies Had Defined the Purpose of Their
Architectures, but Had Yet to Fully Measure and Report
Outcomes and Benefits 12
Conclusions 21
Recommendations for Executive Action 22
Agency Comments and Our Evaluation 23
Appendix I Objective, Scope, and Methodology 29
Appendix II Detailed Assessments of Individual Departments and
Agencies against Relevant Elements of Our Enterprise
Architecture Management Maturity Framework 32
Appendix III Comments from the Department of Labor 63
Appendix IV Comments from the Department of the Treasury 64
Appendix V Comments from the Department of Agriculture 65
Appendix VI Comments from the Department of Commerce 67
Appendix VII Comments from the Department of Defense 68
Appendix VIII Comments from the Department of Education 70
Page i GAO-12-791 Organizational Transformation
Appendix IX Comments from the Department of Homeland Security 72
Appendix X Comments from the Department of the Interior 74
Appendix XI Comments from the Department of State 75
Appendix XII Comments from the Department of Veterans Affairs 79
Appendix XIII Comments from the National Aeronautics and Space Administration 82
Appendix XIV Comments from the Social Security Administration 84
Appendix XV Comments from the Environmental Protection Agency 86
Appendix XVI Comments from the Department of Health and Human Services 87
Appendix XVII Comments from the Department of Energy 89
Appendix XVIII Comments from the Department of Housing and Urban Development 93
Appendix XIX GAO Contact and Staff Acknowledgments 96
Page ii GAO-12-791 Organizational Transformation
Tables
Table 1: Summary of Agencies’ Progress in Measuring and
Reporting Enterprise Architecture Outcomes and Benefits
Relative to Elements of GAO’s EAMMF (Version 2.0) 13
Table 2: Agencies Included in our Study 29
Table 3: Department of Agriculture Satisfaction of EAMMF
Elements 33
Table 4: Department of the Air Force Satisfaction of EAMMF
Elements 34
Table 5: Department of the Army Satisfaction of EAMMF Elements 35
Table 6: Department of Commerce Satisfaction of EAMMF
Elements 37
Table 7: DOD Business Enterprise Architecture Satisfaction of
EAMMF Elements 38
Table 8: Department of Defense Satisfaction of EAMMF Elements 39
Table 9: Department of Education Satisfaction of GAO EAMMF
Elements 40
Table 10: Department of Energy Satisfaction of EAMMF Elements 41
Table 11: Department of Health and Human Services Satisfaction of
EAMMF Elements 42
Table 12: Department of Homeland Security Satisfaction of
EAMMF Elements 43
Table 13: Department of Housing and Urban Development
Satisfaction of EAMMF Elements 44
Table 14: Department of the Interior Satisfaction of EAMMF
Elements 46
Table 15: Department of Justice Satisfaction of EAMMF Elements 47
Table 16: Department of Labor Satisfaction of EAMMF Elements 48
Table 17: Department of the Navy Satisfaction of EAMMF Elements 49
Table 18: Department of State Satisfaction of EAMMF Elements 50
Table 19: Department of Transportation Satisfaction of EAMMF
Elements 51
Table 20: Department of the Treasury Satisfaction of EAMMF
Elements 52
Table 21: Department of Veterans Affairs Satisfaction of EAMMF
Elements 53
Table 22: Environmental Protection Agency Satisfaction of EAMMF
Elements 54
Table 23: General Services Administration Satisfaction of EAMMF
Elements 55
Table 24: National Aeronautics and Space Administration
Satisfaction of EAMMF Elements 56
Page iii GAO-12-791 Organizational Transformation
Table 25: The National Science Foundation Satisfaction of EAMMF
Elements 57
Table 26: Nuclear Regulatory Commission Satisfaction of EAMMF
Elements 58
Table 27: Office of Personnel Management Satisfaction of EAMMF
Elements 59
Table 28: Small Business Administration Satisfaction of EAMMF
Elements 60
Table 29: Social Security Administration Satisfaction of EAMMF
Elements 61
Table 30: U.S. Agency for International Development Satisfaction
of EAMMF Elements 62
Page iv GAO-12-791 Organizational Transformation
Abbreviations
CIO chief information officer
Commerce Department of Commerce
DHS Department of Homeland Security
DOD Department of Defense
EAMMF Enterprise Architecture Management Maturity
Framework
Education Department of Education
Energy Department of Energy
EPA Environmental Protection Agency
GSA General Services Administration
HHS Department of Health and Human Services
HUD Department of Housing and Urban Development
Interior Department of the Interior
IT information technology
Justice Department of Justice
Labor Department of Labor
NASA National Aeronautics and Space Administration
NRC Nuclear Regulatory Commission
NSF National Science Foundation
OMB Office of Management and Budget
OPM Office of Personnel Management
SBA Small Business Administration
SSA Social Security Administration
State Department of State
Transportation Department of Transportation
Treasury Department of the Treasury
USAID United States Agency for International Development
USDA United States Department of Agriculture
VA Department of Veterans Affairs
This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.
Page v GAO-12-791 Organizational Transformation
United States Government Accountability Office
Washington, DC 20548
September 26, 2012
The Honorable Joseph Lieberman
Chairman
The Honorable Susan Collins
Ranking Member
Committee on Homeland Security and Governmental Affairs
United States Senate
The Honorable Darrell Issa
Chairman
The Honorable Elijah Cummings
Ranking Member
Committee on Oversight and Government Reform
House of Representatives
Billions of taxpayer dollars are spent on information technology (IT)
investments each year; according to the Office of Management and
Budget (OMB), the executive branch plans to spend at least $75 billion in
fiscal year 2012. We have previously reported that federal expenditures
on IT could be reduced by, among other things, using enterprise
architecture as a tool for organizational transformation. 1
An enterprise architecture is a blueprint for organizational change defined
in models that describe (in both business and technology terms) how the
entity operates today and how it intends to operate in the future; it also
includes a plan for transitioning to this future state. Effective use of an
enterprise architecture is a hallmark of successful organizations and can
be important to achieving operations and technology environments that
maximize institutional mission performance and outcomes. Among other
things, this includes realizing cost savings through consolidation and
reuse of shared services and elimination of antiquated and redundant
mission operations, enhancing information sharing through data
standardization and system integration, and optimizing service delivery
through streamlining and normalization of business processes and
1
GAO, Opportunities to Reduce Potential Duplication in Government Programs, Save Tax
Dollars, and Enhance Revenue, GAO-11-318SP (Washington, D.C.: Mar. 1, 2011). An
interactive, web-based version of the report is available at
http://www.gao.gov/ereport/gao-11-318SP.
Page 1 GAO-12-791 Organizational Transformation
mission operations. Moreover, the use of architectures is required by the
Clinger-Cohen Act of 1996 and by OMB. 2
In our March 2011 report on opportunities to reduce potential duplication
in government programs, 3 we identified enterprise architecture as a
mechanism for identifying potential overlap and duplication. We noted
that realizing this potential and knowing whether benefits are in fact being
achieved from the use of an architecture requires associated measures
and metrics. Accordingly, under the statutory requirement which
mandates that GAO identify federal programs, agencies, offices, and
initiatives with duplicative goals and activities within departments and
government-wide, 4 we committed to study the extent to which federal
departments and agencies are measuring and reporting enterprise
architecture outcomes and benefits.
To address our objective, we analyzed 27 major departments’ and
agencies’ 5 documents describing their enterprise architecture goals and
purposes and their approaches to measuring and reporting architecture
outcomes and benefits, such as their IT Strategic Plan, Enterprise
Architecture Program Management Plan, and Enterprise Architecture
Value Measurement Plan. We compared the agencies’ approaches to
relevant elements of our Enterprise Architecture Management Maturity
Framework (EAMMF). 6 Further, we reviewed outcomes reported to
agency enterprise architecture oversight officials and analyzed responses
to a 2011 GAO survey about the benefits associated with agencies’
architecture programs. We also discussed our analyses with and obtained
testimonial evidence from cognizant agency officials. A more detailed
2
40 U.S.C. § 11315; The E-Government Act of 2002 also provided a more detailed
definition of the concept and elements of enterprise architecture. See 44 U.S.C. §§
3601(4) and 3602; OMB Circular A-130 (Nov. 30, 2000); and Chief Information Officers
Council, A Practical Guide to Federal Enterprise Architecture, Version 1.0
(February 2001).
3
GAO-11-318SP.
4
Pub. L. No. 111-139, § 21, 124 Stat. 29 (2010), 31 U.S.C. § 712 Note.
5
These 27 major departments and agencies are the 24 Chief Financial Officer Act entities
identified in 31 U.S.C. § 901(b), as well as the Departments of the Air Force, Army, and
Navy.
6
GAO, Organizational Transformation: A Framework for Assessing and Improving
Enterprise Architecture Management (Version 2.0), GAO-10-846G (Washington, D.C.:
August 2010).
Page 2 GAO-12-791 Organizational Transformation
discussion of our objective, scope, and methodology is provided in
appendix I.
We conducted this performance audit from November 2011 to September
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objective. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objective.
An enterprise architecture is a blueprint that describes the current and
Background desired states of an organization or functional area in both logical and
technical terms, as well as a plan for transitioning between the two states.
An enterprise can be viewed as either a single organization or a
functional area that transcends more than one organization. An
architecture can be viewed as the structure (or structural description) of
any activity. Thus, enterprise architectures are systematically derived and
captured descriptions depicted in models, diagrams, and narratives. More
specifically, an architecture describes the enterprise in logical terms (such
as interrelated business processes and business rules, information needs
and flows, and work locations and users) as well as in technical terms
(such as hardware, software, data, communications, security attributes,
and performance standards). It provides these perspectives both for the
enterprise’s current environment and for its target environment, and it
provides a transition plan for moving from the current to the target
environment. Enterprise architectures are a recognized tenet of
organizational transformation and IT management in public and private
organizations.
When employed in concert with other institutional management
disciplines, such as strategic planning, portfolio-based capital planning
and investment control, and human capital management, an enterprise
architecture can greatly increase the chances of configuring an
organization to promote agility and responsiveness, optimize mission
performance and strategic outcomes, and address new federal initiatives
like promoting open and participatory government and leveraging cloud
computing.
Page 3 GAO-12-791 Organizational Transformation
Federal Legislation and The Clinger-Cohen Act of 1996, among other things, requires federal
OMB Guidance Pertaining agency chief information officers (CIO) to develop, maintain, and facilitate
to Establishment of an the implementation of IT architectures. 7 Subsequent OMB guidance more
broadly interpreted IT architecture as an enterprise architecture. 8 In
Enterprise Architecture
September 1999, the federal CIO Council published the Federal
Enterprise Architecture Framework, 9 which provided federal agencies
with a common construct for their architectures to facilitate the
coordination of common business processes, technology insertion,
information flows, and system investments among federal agencies. The
framework defined a collection of interrelated models for describing
multiorganizational functional segments of the federal government. 10
Further, in 2000 and 2001, the federal CIO Council developed enterprise
architecture guidance focused on assessing an IT investment’s
compliance with an architecture 11 as well as guidance that addressed the
end-to-end steps associated with developing, maintaining, and
implementing an architecture program. 12
OMB is responsible for overseeing the development of enterprise
architectures within and across federal agencies. 13 In February 2002, it
7
40 U.S.C. § 11315. According to GAO’s EAMMF, such architectures provide an important
means of integrating business processes and agency goals with IT.
8
See for example OMB, Information Technology Architectures, Memorandum M-97-16
(June 18, 1997), rescinded with the update of OMB Circular A-130 (Nov. 30, 2000) , which
requires that agencies document and submit their enterprise architecture to OMB. Chief
Information Officers Council, Architecture Alignment and Assessment Guide (October
2000). Chief Information Officers Council, A Practical Guide to Federal Enterprise
Architecture, Version 1.0 (February 2001).
9
Federal Enterprise Architecture Framework, Version 1.1 (September 1999).
10
The most recent revision to the Federal Enterprise Architecture Framework (Version 2.0)
is included in OMB’s Common Approach to Federal Enterprise Architecture, which is
discussed subsequently in this report.
11
Chief Information Officers Council, Architecture Alignment and Assessment Guide
(October 2000).
12
Chief Information Officers Council, A Practical Guide to Federal Enterprise Architecture,
Version 1.0 (February 2001).
13
The E-Government Act of 2002 provided a more detailed definition of the concept and
elements of enterprise architecture and established the OMB Office of Electronic
Government and assigned it, among other things, responsibilities for overseeing the
development of enterprise architectures within and across federal agencies. See 44
U.S.C. § 3601(4) and 44 U.S.C § 3602(f)(14).
Page 4 GAO-12-791 Organizational Transformation
established the Federal Enterprise Architecture program. According to
OMB, the program is intended to facilitate government-wide improvement
through cross-agency analysis and identification of duplicative
investments, gaps, and opportunities for collaboration, interoperability,
and integration within and across agency programs. Federal enterprise
architecture reference models are intended to inform agency efforts to
develop their agency-specific enterprise architectures and enable
agencies to ensure that their proposed investments are not duplicative
with those of other agencies and to pursue, where appropriate, joint
projects. In 2007, OMB issued the Federal Enterprise Architecture
Practice Guidance 14 to provide high-level overviews of architecture
concepts, descriptions of the content included in architecture work
products, and direction on developing and using architectures, including
measuring enterprise architecture program value.
According to the latest version of OMB’s Enterprise Architecture
Assessment Framework (version 3.1, dated June 2009), 15 its purpose is
to provide the measurement areas and criteria for federal agencies to use
in realizing architecture-driven performance improvements and outcomes
(e.g., improving mission performance; saving money and avoiding costs;
enhancing the quality of agency investment portfolios; improving the
quality, availability, and sharing of data and information; and increasing
the transparency of government operations). To accomplish this, the
framework uses key performance indicators to assess architecture
maturity or effectiveness relative to three capability areas—completion,
use, and results. Each capability area contains a set of key performance
indicators and associated outcomes, as well as criteria for gauging
progress in meeting the outcomes. In particular, according to the
framework, as part of the results capability area, agencies should
measure actual results attributed to the architecture, and therefore the
effectiveness and value of architecture activities. However, to reduce the
reporting burden on agencies, in August 2009, OMB issued a
memorandum that stated that agencies were no longer required to
provide self-assessments of enterprise architecture completion, use, and
results to OMB.
14
OMB, Federal Enterprise Architecture Practice Guidance (November 2007).
15
OMB, Improving Agency Performance Using Information and Information Technology
(Enterprise Architecture Assessment Framework v3.1) (June 2009).
Page 5 GAO-12-791 Organizational Transformation
In May 2012, OMB released the Common Approach to Federal Enterprise
Architecture 16 to promote increased levels of mission effectiveness by
standardizing the development and use of architectures within and
between federal agencies. The approach stresses that enterprise
architecture can enable service delivery, functional integration, and
resource optimization, and can be an authoritative reference for the
design and documentation of systems and services. According to a
memorandum accompanying the Common Approach, each agency is to
submit to OMB by August 31, 2012, an enterprise roadmap that covers
fiscal years 2012 to 2015, to serve as an authoritative reference for IT
portfolio reviews. 17 The roadmap is to map the organization’s strategic
goals to business services and integrate technology solutions across the
agency’s lines of business. It is to discuss the overall architecture and
identify performance gaps, resource requirements, planned solutions,
transition plans, and a summary of the current and future architectures. It
is also to describe the enterprise architecture governance process,
implementation methodology, and documentation framework.
As one of the elements intended to ensure that agency enterprise
architecture programs can be effective in developing solutions that
support planning and decision making, the guidance begins to lay out a
Collaborative Planning Methodology. The methodology entails defining
what benefits will be achieved, when those benefits will be achieved, and
how those benefits will be measured, as well as measuring performance
outcomes against identified metrics. The guidance emphasizes the
importance of measuring the attainment of outcomes, so that the positive
effects (added value) of the architecture program can be identified.
Specifically, each agency’s roadmap is to document how the
effectiveness and efficiency of the program will be measured. The
guidance discusses the difference between outcome and output
measures, and notes that while output measures are important for
indicating an initiative’s progress, outcome measures are needed to
indicate the attainment of goals. According to the Federal Chief
16
OMB, The Common Approach To Federal Enterprise Architecture (May 2012).
17
OMB, Memorandum for Federal Agency Chief Information Officers, Increasing Shared
Approaches to Information Technology Services (Washington, D.C., May 2, 2012).
Agencies will be required to submit an updated enterprise roadmap to OMB by April 1st
each year, beginning April 1, 2013.
Page 6 GAO-12-791 Organizational Transformation
Enterprise Architect, OMB plans to provide agencies more detailed
guidance on measuring enterprise architecture value by December 2012.
To assist in developing this guidance, the Architecture Subcommittee of
the CIO Council’s Strategy and Planning Committee has established a
working group to develop an approach for measuring enterprise
architecture value through identifying best practices from the public and
private sectors. 18 The group plans to draw upon research to create a
value measurement program that aligns with the Collaborative Planning
Methodology discussed in the Common Approach, and deliver a white
paper to OMB on value measurement by the end of fiscal year 2012.
According to the Federal Chief Enterprise Architect, the working group’s
recommendations will be considered for incorporation into the Common
Approach.
GAO’s Enterprise In August 2010, we issued an Enterprise Architecture Management
Architecture Management Maturity Framework that provides federal agencies with a common
Maturity Framework benchmarking tool for assessing the management of their enterprise
architecture efforts and developing improvement plans. 19 The framework
includes 59 core elements, or building blocks, of enterprise architecture
management. The core elements represent practices, structures,
activities, and conditions that, when properly employed based on the
unique facts and circumstances of each organization and the stated
purpose of its enterprise architecture program, can permit that
organization to maximize its chances of realizing an architecture’s
institutional value. The core elements are categorized into seven
hierarchical stages of management maturity and four critical success
attribute representations.
In particular, core element 41 describes the practice of measuring and
reporting enterprise architecture outcomes. The architecture is a strategic
asset that represents an investment in the organization’s future and is
intended to produce strategic mission value (results and outcomes).
Measuring the extent to which this expected value is actually being
realized is important to identifying what, if any, enterprise architecture
18
The CIO Council includes CIOs and Deputy CIOs from 28 federal agencies and is
chaired by the Office of Management and Budget Deputy Director for Management.
19
GAO-10-846G.
Page 7 GAO-12-791 Organizational Transformation
program changes are warranted. Such value can be derived from
realizing cost savings through consolidation and reuse of shared services
and elimination of antiquated and redundant mission operations,
enhancing information sharing through data standardization and system
integration, and optimizing service delivery through streamlining and
normalization of business processes and mission operations.
In addition, core element 58 specifies that enterprise architecture quality-
and outcomes-measurement methods should be continuously improved.
Organizations should periodically reevaluate their methods for assessing
corporate and subordinate architecture quality and program outcomes
and address the extent to which program measures and metrics are
sufficiently measurable, meaningful, repeatable, consistent, actionable,
and aligned with the architecture program’s strategic goals and intended
purpose.
Prior GAO Work Has In 2002 and 2003, we reported on the status of enterprise architectures
Highlighted Federal government-wide. 20 We found that some federal agencies had begun to
Agency Enterprise establish the management foundation needed to successfully develop,
implement, and maintain an enterprise architecture, but that executive
Architecture Challenges
leadership was key to addressing management challenges identified by
enterprise architecture programs: (1) overcoming limited executive
understanding, (2) inadequate funding, (3) insufficient number of skilled
staff, and (4) organizational parochialism. Accordingly, we made
recommendations to OMB to improve enterprise architecture leadership
and oversight. OMB responded to these recommendations by
establishing its Chief Architects Forum to, among other things, share
enterprise architecture best practices among federal agencies, and by
developing an assessment tool, which it used to periodically evaluate
enterprise architecture programs at federal agencies.
20
GAO, Information Technology: Enterprise Architecture Use across the Federal
Government Can Be Improved, GAO-02-6 (Washington, D.C.: Feb. 19, 2002); Information
Technology: Leadership Remains Key to Agencies Making Progress on Enterprise
Architecture Efforts, GAO-04-40 (Washington, D.C.: Nov. 17, 2003).
Page 8 GAO-12-791 Organizational Transformation
In 2006, we reviewed enterprise architecture management at 27 major
federal departments and agencies. 21 Our work showed that the state of
architecture development and implementation varied considerably across
departments and agencies, with some having more mature programs
than others. Overall, most agencies had not reached a sufficient level of
maturity in their enterprise architecture development, particularly with
regard to their approaches to assessing each investment’s alignment with
the architecture and measuring and reporting on architecture results and
outcomes. Our 2006 report also noted that challenges we identified in our
earlier reviews continued to present hurdles to effective implementation of
enterprise architecture.
We have also reported on enterprise architecture management and
development at several individual departments and agencies, including
agencies that have demonstrated improvements to their architectures:
• In 2009, we reported that recent versions of the Department of
Homeland Security’s (DHS) enterprise architecture had largely
addressed our prior recommendations aimed at adding needed
architectural depth and breadth. 22 Nonetheless, we concluded that
important content, such as prioritized segments and information
exchanges between critical business processes, was still missing from
its architecture.
• Between 2009 and 2012, we conducted several reviews of the
Department of Housing and Urban Development’s (HUD) enterprise
architecture and made a number of recommendations for
21
GAO, Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging
Architectures for Organizational Transformation, GAO-06-831 (Washington, D.C.: Aug. 14,
2006).
22
GAO, Homeland Security: Despite Progress, DHS Continues to Be Challenged in
Managing Its Multi-Billion Dollar Annual Investment in Large-Scale Information
Technology Systems, GAO-09-1002T (Washington, D.C.: Sept. 15, 2009).
Page 9 GAO-12-791 Organizational Transformation
improvement. 23 Over the course of these reviews, we found that HUD
had made progress in establishing its architecture, although as of
September 2012, the department had not yet finalized its updated
architecture policy, as we had recommended.
• In September 2011, we reported on the status of the three military
departments’ (Air Force, Army, and Navy) architecture programs. 24
We reported that while each of the military departments had long-
standing efforts to develop and use enterprise architectures, they had
much to do before their efforts could be considered mature.
Accordingly, we recommended that the military departments each
develop a plan for fully satisfying the elements of our framework. The
Department of Defense (DOD) and the Army concurred with these
recommendations, but the Air Force and Navy did not. In this regard,
DOD stated that the Air Force and Navy did not have a valid business
case that would justify the implementation of all the elements.
However, we maintained that the recommendation was warranted. To
date, none of the military departments have addressed our
recommendations.
• In April 2012, we reported that the Social Security Administration
(SSA) had developed an enterprise architecture for years 2011
through 2016 that captured certain foundational information about the
current and target environments to assist in evolving existing
information systems and developing new systems; however, the
architecture lacked important content that would allow the agency to
more effectively plan its investments to reach its vision of modernized
systems and operations. 25 We recommended that SSA develop an
23
GAO, Information Technology: HUD Needs to Strengthen Its Capacity to Manage and
Modernize Its Environment, GAO-09-675 (Washington, D.C.: July 31, 2009); Information
Technology: HUD Needs to Better Define Commitments and Disclose Risk for
Modernization Projects in Future Expenditure Plans, GAO-11-72 (Washington, D.C.: Nov.
23, 2010); Information Technology: HUD’s Expenditure Plan Satisfies Statutory
Conditions, and Implementation of Management Controls Is Under Way, GAO-11-762
(Washington, D.C.: Sept. 7, 2011); and HUD Information Technology: More Work
Remains to Implement Necessary Management Controls, GAO-12-580T (Washington,
D.C.: Mar. 29, 2012).
24
GAO, Organizational Transformation: Military Departments Can Improve their Enterprise
Architecture Programs, GAO-11-902 (Washington, D.C.: Sept. 26, 2011).
25
GAO, Social Security Administration: Improved Planning and Performance Measures
Are Needed to Help Ensure Successful Technology Modernization, GAO-12-495
(Washington, D.C: Apr. 26, 2012).
Page 10 GAO-12-791 Organizational Transformation
enterprise architecture plan that included certain key elements. The
agency responded that it would comply with recent direction from the
Federal Chief Architect to deliver an enterprise architecture roadmap
that meets OMB standards.
In addition to our evaluation of agency-specific enterprise architectures, we
have reported on the need for federal agencies to measure and report
architecture outcomes. Specifically, in March 2011, we reported that while
some progress had been made in improving the content and use of
departments’ and agencies’ architectures, more time was needed for
agencies to fully realize the value of having well-defined and implemented
architectures. 26 We noted that some agencies had reported that they were
addressing the EAMMF core element associated with measuring and
reporting enterprise architecture results and outcomes and had realized
significant financial benefits. For example, we reported that the Department
of the Interior had demonstrated that it was using its enterprise architecture
to modernize agency IT operations and avoid costs through enterprise
software license agreements and hardware procurement consolidation,
which had resulted in reported financial benefits of at least $80 million.
However, we concluded that over 50 percent of the departments and
agencies had yet to fully address this element.
In February 2012, we again reviewed the extent to which major federal
agencies had reported financial benefits from the use of enterprise
architecture. We found that four agencies (in addition to the Department
of Interior) had done so. These four agencies were the Department of
Health and Human Services (HHS), which, facilitated by its architecture
program, moved to a new telecommunications contract, resulting in a
savings of about $21 million; the Nuclear Regulatory Commission (NRC),
which avoided an estimated $1.3 million cost in 2011 by eliminating
duplicative staff planning systems; DOD, which reported saving $179
million between fiscal years 2008 and 2010 by streamlining Navy
business operations, retiring legacy systems, and moving toward a real-
time paperless business environment for processing vendor payments;
and the Department of Agriculture, which reported savings of $27 million
over 5 years (2011 through 2015) by moving 120,000 e-mail users to a
cloud-based solution. 27 We also noted that 12 agencies had reported
26
GAO-11-318SP.
27
Cloud computing is a form of computing that relies on Internet-based services and
resources to provide computing services to customers.
Page 11 GAO-12-791 Organizational Transformation
financial benefits but had not reliably measured them (i.e., they did not
provide supporting documentation), and an additional 10 agencies had
not reported financial benefits, although 8 of these agencies reported that
they had established or expected to establish a process to measure
benefits in the future. 28
Most of the 27 major agencies in our current study had yet to periodically
Almost All Agencies (i.e., regularly and repeatedly, such as monthly, quarterly or annually)
Had Defined the measure and report enterprise architecture outcomes and benefits. Our
framework 29 recognizes that knowing whether architecture outcomes are
Purpose of Their being achieved requires an approach to measuring the value of
Architectures, but architecture activities that includes defining the architecture’s intended
Had Yet to Fully purpose or strategic goals; establishing metrics along with a method to
measure architecture outcomes and benefits; and periodically measuring
Measure and Report and reporting to the agency’s architecture executive committee
Outcomes and (executive-level representatives from each line of business, who have the
authority to commit resources) these outcomes and benefits. While all
Benefits agencies had fully or partially defined their architecture’s strategic goals
or intended purpose, only 3 had fully and 8 had partially established
metrics and a method to measure outcomes and benefits. Of the
agencies that fully or partially established a method and metrics, 4 had
measured and reported outcomes only once, and 1 had periodically (e.g.,
monthly) reported on outcomes and benefits. A summary of the 27
agencies’ progress in measuring and reporting architecture outcomes and
benefits is presented in table 1. For detailed assessments of individual
departments and agencies against relevant elements of our framework,
see appendix II.
28
GAO, Follow-up on 2011 Report: Status of Actions Taken to Reduce Duplication,
Overlap, and Fragmentation, Save Tax Dollars, and Enhance Revenue, GAO-12-453SP
(Washington, D.C.: Feb. 28, 2012).
29
GAO-10-846G.
Page 12 GAO-12-791 Organizational Transformation
Table 1: Summary of Agencies’ Progress in Measuring and Reporting Enterprise Architecture Outcomes and Benefits
Relative to Elements of GAO’s EAMMF (Version 2.0)
The enterprise Metrics and a method Enterprise architecture
architecture’s have been established to outcomes and benefits
strategic goals measure enterprise are periodically
or intended architecture strategic measured and reported
purpose are mission value (outcomes to the architecture
Agency defined and benefits) executive committee
Department of Agriculture ● ○ ○
Department of the Air Force ● ○ ○
Department of the Army ● ◐ ○
Department of Commerce ● ◐ ○
Department of Defense – Business Enterprise Architecture ● ○ ○
Department of Defense - Enterprise Architecture
a
● ○ ○
Department of Education ● ◐ ◐
Department of Energy ● ○ ○
Department of Health and Human Services ● ● ◐
Department of Homeland Security ● ○ ○
Department of Housing and Urban Development ● ● ◐
Department of the Interior ● ○ ○
Department of Justice ● ○ ○
Department of Labor ● ○ ○
Department of the Navy ● ○ ○
Department of State ● ○ ○
Department of Transportation ● ◐ ○
Department of the Treasury ● ◐ ◐
Department of Veterans Affairs ◐ ○ ○
Environmental Protection Agency ● ○ ○
General Services Administration ● ◐ ○
National Aeronautics and Space Administration ● ○ ○
National Science Foundation ● ○ ○
Nuclear Regulatory Commission ● ◐ ○
Office of Personnel Management ● ◐ ○
Small Business Administration ● ○ ○
Social Security Administration ● ○ ○
US Agency for International Development ● ● ●
Source: GAO analysis of agency data.
● Satisfied ◐ Partially Satisfied ○ Not Satisfied
Page 13 GAO-12-791 Organizational Transformation
Note: Agencies fully satisfied the relevant elements of our framework by providing sufficient
documentation to verify that all aspects of the elements were met. Agencies partially satisfied the
elements by providing documentation to verify that at least some aspect of the elements was
satisfied. Elements that were neither fully nor partially satisfied were judged to be not satisfied.
a
According to DOD, the DOD Enterprise Architecture is a federation of descriptions that provide
context and rules for accomplishing the mission of the department, and the DOD Business Enterprise
Architecture is the blueprint to guide and constrain investments as they relate to or impact business
operations.
Almost All Agencies Had Before an agency knows what outcomes it should measure, it needs to
Defined the Goals or define the purpose or expected value (i.e., goals) of its architecture. The
Purposes of Their purpose can include, among other things, consolidating the organization’s
IT infrastructure, normalizing and integrating its data and promoting
Enterprise Architecture information sharing, reengineering core business or mission functions and
processes, modernizing applications and sharing services, modernizing
the entire IT environment, and transforming how the organization
operates. Expected value from implementation of enterprise architecture
can include, for example, reduced operating costs, enhanced ability to
quickly and less expensively change to meet shifting external
environment and new business demands or opportunities, or improved
alignment between operations and strategic goals. Twenty-six of the
agencies we reviewed had fully defined their architecture goals or
purposes. The following are examples of the goals or purposes defined
by these agencies:
• The Department of Energy’s goals include identifying, reusing, and
leveraging, where possible, existing and planned technology and
infrastructure components across the department and identifying
areas, through capital planning and investment control and enterprise
architecture integration analysis, to reduce costs, identify redundancy,
and increase system and process effectiveness.
• HHS’s goals include enabling improved mission and business
outcomes by providing products to support sound decisions, business
processes, and effective solutions; enabling the optimized use of
resources; and increasing interoperability and information sharing
within HHS and between HHS and external stakeholders.
• HUD’s goals include improving the efficiency and effectiveness of the
department’s programs; simplifying its IT environments by promoting
standards and sharing and reusing common technologies; improving
interoperability by establishing enterprise-wide standards; and
reducing system development and operation and maintenance costs
Page 14 GAO-12-791 Organizational Transformation
by eliminating duplicative investments, promoting sharing of common
services, and establishing department-wide standards.
• The Department of Justice’s purpose includes identifying redundant
legacy programs to either retire or migrate to an enterprise solution,
thereby reducing the complexity and cost of the IT environment.
• The Department of Labor’s purpose is to use its enterprise
architecture process with its capital planning and investment
management process to ensure that investments support strategic
goals and are not duplicative of existing business solutions. Using this
approach, the department plans to identify duplicative resources and
investments, gaps, and opportunities for internal and external
collaboration resulting in operational improvements and cost-effective
solutions to business requirements.
• The Department of Transportation’s goal is to use its architecture as a
decision-making tool to support business plan development and
identify areas of duplication and inefficiencies in the department.
• The General Services Administration’s (GSA) goals are to increase
system interoperability and cost efficiencies, reduce duplication, and
increase innovation.
• The National Science Foundation’s goals include improving utilization
of IT resources by eliminating duplicative investments and promoting
the sharing of common services and standards.
• The U.S. Agency for International Development’s (USAID) goals
include facilitating analysis of the agency’s IT environment, including
IT hardware, software, and enterprise applications, to promote the
effective and efficient deployment of IT services.
However, one agency (Department of Veterans Affairs) had only partially
defined its architecture’s purpose. The Department of Veterans Affairs
(VA) is in the process of developing an architecture program overview
statement and guiding principles. Specifically, according to draft
documentation, the department’s architecture is to guide efficient,
effective, and interoperable implementation of the department’s vision of
providing seamless delivery of benefits and services to veterans.
According to department officials, these architecture principles are
expected to be finalized and formally released by September 30, 2012.
Page 15 GAO-12-791 Organizational Transformation
Because they had defined the goals or purposes of their architectures,
almost all of the agencies had taken an important first step toward
establishing metrics and a method for measuring architecture outcomes
and benefits.
More than Half of the Measuring the extent to which the expected value is actually being
Agencies Had Not Yet realized is important to identifying what, if any, architecture program
Established Metrics and a changes are warranted. According to our framework, agencies should
establish measurable, meaningful, repeatable, consistent, and actionable
Method for Measuring metrics that align with the architecture’s intended purpose or strategic
Enterprise Architecture goals and document a methodology that provides the steps to be followed
Value to consistently and repeatedly measure architecture outcomes and
benefits. Further, according to OMB guidance, metrics should measure
outcomes (i.e., results of products and services such as benefits to
Congress and the American taxpayer), or expected value, rather than
output (i.e., direct products and services).
Of the federal agencies that we reviewed, three had fully established
metrics and a method to measure architecture outcomes and benefits,
while eight had partially done so. Specifically, HHS, HUD, and USAID had
fully established metrics and a method for measuring and reporting
enterprise outcomes and benefits.
• HHS had established a metric to measure the extent to which it
increases the number of services that are reused based on its
enterprise architecture service component reference model. 30 The
department had also established a method for how the metrics are to
be measured, including how they are to be calculated, the data
sources to be used, and targets to be achieved.
• HUD had established a method and metrics to measure the extent to
which the department decreases the number of technology products
that duplicate existing capabilities and the extent to which it has
decreased the number of obsolete systems in its IT inventory, using
its enterprise architecture. The department had also established the
steps to measure results and outcomes, including identifying
appropriate sources, and determining baseline, target, and actual
value measurements.
30
A service component reference model identifies and classifies IT service components.
Page 16 GAO-12-791 Organizational Transformation
• USAID had established metrics and guidance for measuring
enterprise architecture outcomes, including cost savings and
avoidance due to process efficiency, technology standardization,
retirement, and consolidation.
Partial steps had been taken by the other eight agencies. Specifically,
one agency (Army) had established a metric and method—for measuring
the extent to which it reduces the number of applications within data
centers— but only for one of its three segment architectures. The other
seven agencies had established metrics but not a method for measuring
and reporting architecture outcomes and benefits. The metrics for each of
the seven agencies are described below.
• Commerce established as a metric the IT cost reduction associated
with adopting enterprise-wide standards.
• Education established a metric to measure spending on development,
modernization, and enhancement relative to steady-state spending
(i.e., the cost to maintain current systems and technologies).
• Transportation established an expected architecture outcome of
reduced total cost of ownership of IT investments, and planned to
measure cost savings and/or cost avoidance identified through
reviews of business processes, data, applications, and technology.
• Treasury established architecture metrics associated with its data
center consolidation initiative, including the extent to which it
decreases the number of servers, increases the percentage of
operating systems that are virtual, and decreases the demand for data
center square footage.
• GSA established as a metric the extent to which the agency is
increasing its use of IT standards.
• NRC established a metric to measure progress toward having
common access controls by measuring the reduction in passwords
and sign-ons.
• Office of Personnel Management (OPM) has established cost savings
as a metric to measure architecture outcomes.
The remaining 16 agencies in our study had not established metrics or a
method for measuring architecture outcomes. While some of these
Page 17 GAO-12-791 Organizational Transformation
agencies had established metrics that measure output, such as the
percentage or number of segments and solution architectures or
architecture artifacts that have been reviewed and approved by the
enterprise architecture program, these metrics do not measure outcomes
(i.e., results of enterprise architecture products and services such as
benefits to Congress and the American taxpayer) of the program.
Without established metrics and a method to measure architecture
outcomes, agencies cannot ensure that they are able to consistently and
repeatedly measure outcomes.
Five Agencies Had Fully or Using established metrics and a documented method, architecture
Partially Measured and outcomes should be periodically measured and reported to senior
Reported Architecture executives. We have previously found that executive leadership was key to
addressing management challenges identified by enterprise architecture
Value programs, such as overcoming limited executive understanding and
inadequate funding. As such, architecture outcomes and benefits should be
periodically reported to senior agency executives who are responsible for
making decisions about the architecture program and whether to invest
additional resources or make changes to the program.
Of the 27 agencies in our review, 1 had consistently and repeatedly
measured and reported, using established metrics, outcomes of its
architecture program. Specifically, USAID had reported monthly the
measured outcomes to its CIO and through an internal agency website
established for CIO staff. Outcomes reported include cost savings of
$12.3 million and cost avoidance of $9.5 million as a result of transitioning
disparate human resource systems to a human resource shared services
center using enterprise architecture. The agency also reported estimated
savings of $15.7 million from moving its e-mail service to a cloud-based
solution, which was recommended by the architecture team to replace
multiple installations of the current e-mail solution.
Two other agencies had measured and reported outcomes with an
established method, but did so only once. Specifically,
• HHS determined, based on its enterprise architecture service
component reference model, and reported to the CIO in November
2010 that 16 percent of its services were reused. However, the
department had not measured the metric again and thus did not know
the extent to which it had increased its reuse of services since 2010.
Page 18 GAO-12-791 Organizational Transformation
• HUD measured and submitted its architecture value measurement
report for fiscal year 2011 to a department executive committee in
August 2012, and highlighted areas, based on measurements, where
additional focus and improvement are needed. For example, the
report noted that HUD had not decreased the number of technology
products in its enterprise architecture technical reference model 31 that
duplicate existing capabilities.
While two additional agencies, as described below, had also measured
and reported architecture outcomes once, they did so without an
established method for measuring outcomes, but rather in an ad hoc
manner.
• Education reported in its October 2011 Office of the CIO Organization
Performance Review report that development, modernization, and
enhancement funding in the IT portfolio increased from 10 percent of
total IT spending in fiscal year 2011, to 13 percent of total IT spending
in fiscal year 2012 through use of the department’s architecture
segment modernization planning process. However, Education had
not established a method for measuring and reporting architecture
outcomes and benefits. As a result, it cannot ensure that it will be able
to consistently and repeatedly measure architecture outcomes over
time.
• Treasury reported in its E-Government Act Report for fiscal year 2011
that its enterprise architecture plans focused on reducing duplication
through its data center consolidation initiative. Accordingly, it reported
through its CIO to OMB a reduction of 1,283 in the number of servers,
an increase from 25 percent to 36 percent of operating systems that
were virtualized, and a reduction in data center square footage of
15,896 between 2010 and 2011. However, Treasury had not
established a method for measuring and reporting architecture
outcomes and benefits. As a result, it cannot ensure that it will be able
to consistently and repeatedly measure outcomes over time.
The remaining agencies (22) had not yet measured and reported
architecture outcomes to senior executives. Agencies generally cited two
reasons why they had not done so. Specifically, agencies had not
31
A Technical Reference Model describes the standards, specifications, and technologies
that support the delivery of service components.
Page 19 GAO-12-791 Organizational Transformation
determined how to attribute discrete outcomes to enterprise architecture
when other activities, such as strategic planning, capital planning, and
project management may have contributed to the outcomes. In addition,
agencies cited an absence of guidance and best practices for how to
measure enterprise architecture outcomes. As discussed in the next
section, OMB has issued recent enterprise architecture guidance to
agencies, but has not yet provided sufficient details on the method and
metrics that could be used to measure architecture program outcomes.
Collectively, this means that while efforts are underway, without the use
of associated measures and metrics by the majority of agencies, the 27
major departments and agencies are not positioned to know whether
outcomes and benefits are in fact being achieved. Until agencies
establish an approach for measuring enterprise architecture outcomes,
including a documented method (i.e., steps to be followed) and metrics
that are measurable, meaningful, repeatable, consistent, actionable, and
aligned with the agency’s enterprise architecture’s strategic goals and
intended purpose; and measure and report enterprise architecture
outcomes and benefits to top agency officials and to OMB, agency senior
executives are less likely to be sufficiently informed about whether to
invest additional resources or make changes to the enterprise
architecture program.
OMB’s Guidance to The E-Government Act of 2002 assigned OMB the responsibilities for
Agencies Lacks Sufficient overseeing the development of enterprise architectures within and across
Details on Measuring the federal agencies. Since then, OMB has issued guidance and
frameworks for developing and using architectures, including a May 2012
Enterprise Architecture policy and guidance on establishing a common approach to developing
Value and using enterprise architectures within and between federal agencies.
The policy required each federal agency to submit by August 31, 2012, an
enterprise roadmap that reports, among other things, how architecture
program effectiveness and efficiency will be measured. However, while
this guidance begins to describe an approach for collaboratively
identifying, planning for, achieving, and measuring needed organizational
outcomes (called the Collaborative Planning Methodology) and discusses
the difference between outcome and output measures, it does not provide
sufficient details on the method and metrics that could be used to
measure architecture program outcomes.
As we noted earlier, according to our framework a methodology should
provide the steps to be followed to consistently and repeatedly measure
architecture outcomes and benefits. While OMB’s collaborative planning
Page 20 GAO-12-791 Organizational Transformation
methodology emphasizes the importance of measuring benefits and
describing how they will be measured when planning for and executing
collaborative projects, it does not call for specific metrics to be used or
identify steps to be followed to consistently and repeatedly measure
outcomes and benefits. Further, OMB does not call for agency roadmaps
to include measurement methods and metrics, and reports on specific
outcomes and benefits that an agency has achieved or plans to achieve.
In discussing this matter, the Federal Chief Enterprise Architect agreed with
our assessment but stated that the methodology was not intended to be
guidance on measuring architecture value and that more detailed guidance
was being developed. According to the Federal Chief Architect, the detailed
guidance on measuring enterprise architecture value is expected to be
provided to agencies by December 2012, in time to facilitate the
development of their next roadmap submissions, due in April 2013.
With the development of clear and sufficiently detailed guidance on
measuring outcomes by OMB, agencies may be better positioned to
develop methods and metrics for measuring and reporting the strategic
value produced by their enterprise architecture programs. Moreover, with
reports about architecture outcomes and benefits, agency executives
could increase their understanding of the architecture programs, such that
warranted changes could be addressed, or the need for expanded
architecture development and use may be able to be economically
justified. An established method and metrics to measure outcomes and
benefits will enable agencies to repeatedly and consistently measure and
report the extent to which they are achieving value.
Enterprise architecture value has yet to be measured and reported across
Conclusions the majority of the federal agencies. While most of the agencies reviewed
have defined their architecture’s goals or purpose, the majority had yet to
establish metrics and a method for measuring and reporting architecture
value. This means that while efforts are underway, the majority of the
agencies do not know the extent to which they are realizing benefits that
they have set out to achieve, such as cost savings or avoidance through
eliminating duplicative investments. Furthermore, most of the agencies
had not measured and reported outcomes to stakeholders or agency
executives. Without measurable, meaningful, repeatable, consistent, and
actionable metrics that align with the architecture’s strategic goals or
intended purpose and a documented methodology that provides the steps
to be followed to consistently and repeatedly measure outcomes and
benefits, senior agency executives may not have the information needed
Page 21 GAO-12-791 Organizational Transformation
to determine whether to invest additional resources or make changes to
the program. OMB’s forthcoming guidance is an opportunity to overcome
the absence of detailed directions to agencies on how they can measure
and report enterprise architecture strategic value.
To enhance federal agencies’ ability to realize enterprise architecture
Recommendations for benefits, we recommend the following actions.
Executive Action
We recommend that the Secretaries of the Departments of Agriculture,
the Air Force, the Army, Commerce, Defense, Education, Energy,
Homeland Security, the Interior, Labor, the Navy, State, Transportation,
the Treasury, and Veterans Affairs; the Attorney General; the
Administrators of the Environmental Protection Agency, General Services
Administration, National Aeronautics and Space Administration, and
Small Business Administration; the Commissioners of the Nuclear
Regulatory Commission and Social Security Administration; and the
Directors of the National Science Foundation and the Office of Personnel
Management ensure the following two actions are taken:
• fully establish an approach for measuring enterprise architecture
outcomes, including a documented method (i.e., steps to be followed)
and metrics that are measurable, meaningful, repeatable, consistent,
actionable, and aligned with the agency’s enterprise architecture’s
strategic goals and intended purpose; and
• periodically measure and report enterprise architecture outcomes and
benefits to top agency officials (i.e., executives with authority to
commit resources or make changes to the program) and to OMB.
In addition, we recommend that the Secretaries of the Departments of
Health and Human Services and Housing and Urban Development
ensure that enterprise architecture outcomes are periodically measured
and reported to top agency officials.
To assist agencies in measuring and reporting outcomes achieved
through enterprise architecture, we recommend that the Director of OMB
ensure that the planned December 2012 guidance for enterprise
architecture value measurement and reporting includes
• sufficient details on the method and metrics that agencies could use
to measure their architecture program’s value and
Page 22 GAO-12-791 Organizational Transformation
• a requirement for agencies to include in their April 2013 enterprise
roadmap submissions a measurement method (i.e., steps to be
followed) and metrics, and report on the outcomes and benefits
achieved through enterprise architecture.
We received comments on a draft of this report from OMB and the 24 32
Agency Comments agencies in our study. OMB’s Federal Chief Enterprise Architect stated in
and Our Evaluation oral comments and via e-mail that OMB agreed with the report and the
recommendations. Among the agencies in our study, 5 responded via e-
mail that they had no comments on our draft report. 33 One of these
agencies—USAID—provided technical comments, which we incorporated
as appropriate. An additional 2 agencies provided letters stating that they
had no comments on our draft report. Specifically, Labor’s Assistant
Secretary for Administration and Management stated in a written
response (reproduced in appendix III) that the department had no
comments, and Treasury’s Deputy Assistant Secretary for Information
Systems and Chief Information Officer stated in a written response
(reproduced in appendix IV) that the department had no comments on the
draft report but appreciated GAO’s efforts in its development.
Among the remaining agencies, 13 agreed with our results. These
comments are summarized below.
• USDA’s Acting Chief Information Officer stated in written comments
that the department concurred with our findings and recommendations
and plans to develop metrics and guidance to comply with OMB
guidance on measuring enterprise architecture, when it is provided.
USDA’s written comments are reproduced in appendix V.
• Commerce’s Acting Secretary stated in written comments that the
department agreed with the general findings and specific
recommendations as they relate to the department. Commerce’s
written comments are reproduced in appendix VI.
• DOD’s Deputy CIO for Information stated in written comments that the
department concurred with our recommendations and is developing
32
DOD included comments from the departments of the Air Force, Army, and Navy.
33
Transportation, GSA, NSF, and NRC and USAID.
Page 23 GAO-12-791 Organizational Transformation
an enterprise architecture management plan that provides high-level
processes, including to measure architecture outcomes. DOD’s
written comments are reproduced in appendix VII.
• Education’s CIO stated in written comments that the department
concurred with our recommendations and described steps the
department plans to take to address the recommendations. For
example, the department plans to develop, document, and implement
a measurement and reporting method that will be used to periodically
monitor its progress toward achieving goals, desired outcomes, and
benefits. The department also provided technical comments that we
have incorporated, as appropriate, in the report. Education’s written
comments are reproduced in appendix VIII.
• DHS’s Director, Departmental GAO-OIG Liaison Office, stated in
written comments that the department concurred with our
recommendations and described actions it plans to take to address
them. For example, DHS stated that it plans to brief architecture
outcomes for the goals and objectives outlined in the strategic plan to
the CIO by October 31, 2012. DHS’s written comments are
reproduced in appendix IX.
• Interior’s Assistant Secretary for Policy Management and Budget
stated in written comments that the department concurred with our
recommendations. Interior’s written comments are reproduced in
appendix X.
• A Management Analyst in Justice’s audit liaison group commented via
e-mail that the department agreed with our recommendations. The
official also provided technical comments that we have incorporated,
as appropriate.
• State’s Comptroller provided written comments which noted that the
department concurred with our conclusions and recommendations,
and described steps being taken or planned to address the
recommendations. For example, the department plans to implement,
in fiscal year 2013, a metric to measure reduction in the percentage of
information exchange elements between critical management
systems through use of its enterprise architecture. State’s written
comments are reproduced in appendix XI.
• VA’s Chief of Staff stated in written comments that the department
generally agreed with our conclusions and concurred with the
recommendations. The department also described actions it had
Page 24 GAO-12-791 Organizational Transformation
taken in fiscal year 2012 to re-establish its enterprise architecture
program and actions it plans to take to continue to mature the
program in fiscal year 2013 that would begin to address our
recommendations. VA’s written comments are reproduced in
appendix XII.
• NASA’s CIO stated in written comments that the agency concurred
with the recommendations, and described steps the agency plans to
take to address them. For example, NASA plans to revise, by June
2013, its procedural requirements to better align architecture metrics
and methods to measure outcomes. NASA’s written comments are
reproduced in appendix XIII.
• SSA’s Deputy Chief of Staff stated in written comments that the
agency agreed with the recommendations. SSA’s written comments
are reproduced in appendix XIV.
• EPA’s Assistant Administrator and Chief Information Officer stated in
written comments that the agency agrees with our findings and
described steps it plans to take to address our recommendations.
For example, it plans to develop a performance measurement plan,
which will identify processes to measure enterprise architecture
outcomes. EPA’s written comments are reproduced in appendix XV.
• HHS’s Assistant Secretary for Legislation in written comments stated
that the department concurred with our findings and described actions
it is taking, and plans to take, to improve architecture value
measurement. HHS’s written comments are reproduced in appendix
XVI.
The remaining four agencies provided comments that expressed
concerns with certain aspects of our results. These comments are
summarized below.
• Energy’s Chief Architect provided written comments in which the
department stated that it had established metrics and a method for
measuring architecture value and that its efforts justify a partially-
satisfied rating. However, our study found that, although the
department has submitted its Enterprise Modernization Roadmap to
OMB, the roadmap includes potential architecture program metrics
that are still being defined and have yet to be finalized and approved.
The department also stated that it had achieved several
accomplishments which justified a partially-satisfied rating for
measuring and reporting architecture outcomes and benefits. In this
Page 25 GAO-12-791 Organizational Transformation
regard, the department highlighted accomplishments such as
collecting and reporting architecture success stories to a working-sub
group of its Information Technology Council, which includes senior-
level IT management from the offices of the Chief Financial Officer
and the CIO. However, we found that these accomplishments and
success stories are not based on an established set of metrics and a
documented, consistently applied methodology for measuring and
reporting architecture outcomes. As a result, the department cannot
ensure that it will be able to consistently and repeatedly measure
outcomes over time. Thus, we stand by our findings. Energy’s written
comments are reproduced in appendix XVII.
• HUD’s CIO provided written comments on the report stating that the
department has complied with our recommendation that enterprise
architecture outcomes be periodically measured and reported to top
agency officials. Specifically, the department stated the fiscal year
2011 report on outcomes was submitted to an executive committee in
August 2012. It also stated that an EA Value Measurement Plan will
be issued annually and results of the measures in the plan will be
documented in an annual report for the fiscal year. However, while the
department has completed and submitted its first report (i.e., for fiscal
year 2011) to an executive committee, it has yet to measure and
report on the metrics again, and therefore does not know the extent to
which it has achieved its target outcomes. In addition, the official
commented on the statement in the background of our report that the
department had not yet finalized its architecture policy, as we had
previously recommended. The official commented that a policy has
been in place since April 2002. However, as the official stated, its
updated policy has yet to be approved. As a result, we stand by the
statement. HUD’s written comments are reproduced in appendix
XVIII.
• A Senior Analyst, e-mailing on behalf of OPM’s Office of the CIO,
provided comments in which the agency stated that savings through
enterprise architecture are being measured and reported. However, it
provided no evidence to support this statement and stated that more
information will be available once the department implements a
revised enterprise architecture roadmap, expected by the end of
December 2012. As a result, we did not change our finding.
• A Program Manager in SBA’s Office of Congressional and Legislative
Affairs provided e-mail comments. Specifically, in comments on our
finding that stated the agency had not defined its enterprise
architecture strategic goals or intended purpose, SBA stated that its
Page 26 GAO-12-791 Organizational Transformation
architecture’s purpose and goals are defined and it provided
supporting documentation in this regard. In response, we updated the
finding and recommendation accordingly. In comments on our finding
that the agency had not established a method or metrics to measure
outcomes and benefits, SBA provided its Capital Planning and
Investment Control Policy Guide and its fiscal year 2011 Summary of
Performance and Financial Information. However, neither of these
documents demonstrated a method and metrics for measuring
architecture outcomes. In comments on our finding that the agency is
not periodically measuring architecture outcomes and benefits, the
agency stated that outcomes are measured and reported as part of
the integrated enterprise architecture-capital planning and investment
control effort through the Business Technology Investment Advisory
Committee and the Business Technology Investment Council. While
the agency provided some documentation, it did not provide
requested examples of reports submitted to the officials. The agency
also added that outcomes are reported in its annual performance
report and provided documentation. While we agree that some
outcomes are documented, the report does not highlight architecture-
related outcomes, and SBA did not provide documentation linking the
outcomes to its enterprise architecture. Therefore, we did not change
our findings relative to establishing a method and metrics and
measuring and reporting architecture outcomes.
Page 27 GAO-12-791 Organizational Transformation
We are sending copies of this report to other interested congressional
committees; the Director of the Office of Management and Budget; the
Secretaries of Agriculture, the Air Force, the Army, Commerce, Defense,
Education, Energy, Health and Human Services, Homeland Security,
Housing and Urban Development, the Interior, Labor, the Navy, State,
Transportation, the Treasury, and Veterans Affairs; the Attorney General;
the Administrators of the Environmental Protection Agency, General
Services Administration, National Aeronautics and Space Administration,
Small Business Administration, and U.S. Agency for International
Development; the Commissioners of the Nuclear Regulatory Commission
and the Social Security Administration; and the Directors of the National
Science Foundation and Office of Personnel Management. In addition,
the report will be available at no charge on the GAO website at
http://www.gao.gov.
If you or your staffs have questions on matters discussed in this report,
please contact me at (202) 512-6304 or melvinv@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. GAO staff who made contributions to
this report are listed in appendix XIX.
Valerie C. Melvin
Director
Information Management and
Technology Resources Issues
Page 28 GAO-12-791 Organizational Transformation
Appendix I: Objective, Scope, and
Appendix I: Objective, Scope, and
Methodology
Methodology
Our objective was to determine the extent to which federal agencies are
measuring and reporting enterprise architecture outcomes and benefits.
To accomplish the objective, we focused on 28 enterprise architecture
programs relating to 27 major departments and agencies. These 27
included the 24 departments and agencies identified in the Chief
Financial Officers Act, 1 as well as the Departments of the Air Force,
Army, and Navy. At the Department of Defense (DOD), we reviewed two
department-wide architecture programs—the Business Enterprise
Architecture and the DOD Enterprise Architecture. Table 2 identifies the
agencies included in our study. These agencies were also included in our
2006 review of agencies’ management maturity. 2
Table 2: Agencies Included in Our Study
Agency
Department of Agriculture
Department of the Air Force
Department of the Army
Department of Commerce
Department of Defense (Business Enterprise Architecture and Enterprise Architecture)
Department of Education
Department of Energy
Department of Health and Human Services
Department of Homeland Security
Department of Housing and Urban Development
Department of the Interior
Department of Justice
Department of Labor
Department of the Navy
Department of State
Department of Transportation
Department of the Treasury
Department of Veterans Affairs
1
See 31 U.S.C. § 901(b).
2
GAO, Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging
Architectures for Organizational Transformation, GAO-06-831 (Washington, D.C.:
Aug. 14, 2006).
Page 29 GAO-12-791 Organizational Transformation
Appendix I: Objective, Scope, and
Methodology
Agency
Environmental Protection Agency
General Services Administration
National Aeronautics and Space Administration
National Science Foundation
Nuclear Regulatory Commission
Office of Personnel Management
Small Business Administration
Social Security Administration
United States Agency for International Development
Source: GAO.
We reviewed the responses to a survey we administered in May 2011, of
federal agencies’ efforts to measure and report enterprise architecture
results and outcomes. The purpose of the survey was to follow up with
the agencies we reviewed in 2006, about the costs and benefits
associated with their enterprise architecture programs. 3 In addition, we
requested and reviewed documents describing each agency’s enterprise
architecture program, focusing on the purpose and goals of the programs
and the methods and metrics used to measure outcomes, such as IT
strategic plans, program management plans, enterprise transition plans,
enterprise modernization roadmaps, and value measurement plans. We
analyzed the extent to which the documentation satisfied elements
related to outcomes measurement and reporting in version 2.0 of our
Enterprise Architecture Management Maturity Framework (EAMMF). 4
Specifically, we assessed agencies against elements of the framework
related to defining the architecture’s intended purpose or strategic goals,
establishing a method and metrics to measure architecture strategic
mission value (outcomes and benefits), and periodically measuring and
reporting outcomes and benefits to an architecture executive committee.
We also reviewed outcomes and benefits reported to agency architecture
oversight officials, for example, in value measurement reports or
performance measurement reports. We assessed the reliability of the
reported outcomes and benefits by discussing with agency officials the
3
GAO-06-831.
4
GAO, Organizational Transformation: A Framework for Assessing and Improving
Enterprise Architecture Management (Version 2.0), GAO-10-846G (Washington, D.C.:
August 2010).
Page 30 GAO-12-791 Organizational Transformation
Appendix I: Objective, Scope, and
Methodology
method and data used to determine them, and by reviewing relevant
documents, such as business cases and return on investment analyses.
To guide our analysis, we defined detailed evaluation criteria for
determining whether a given element was fully satisfied, partially satisfied,
or not satisfied. To fully satisfy an element, sufficient documentation had
to be provided to permit us to verify that all aspects of the element were
met. To partially satisfy an element, sufficient documentation had to be
provided to permit us to verify that at least some aspects of the element
were met. Elements that were neither fully nor partially satisfied were
judged to be not satisfied.
Our evaluation included independently analyzing the extent to which each
agency had satisfied the elements using the survey responses and
supporting documentation as a starting point. We then corroborated the
analyses with supporting documentation, sought additional information as
necessary through interviews with the agencies’ architecture officials,
obtained and reviewed additional documentation as appropriate, and
refined our determinations about the degree to which each element was
satisfied. Finally, we shared with agencies preliminary versions of the
analyses that appear in this report as appendix II, and made further
adjustments, as appropriate, based on additional discussions and
supporting documentation. We also met with the Federal Chief Enterprise
Architect to discuss current efforts and plans to guide federal agencies’
efforts to measure and report enterprise architecture outcomes and
benefits.
We conducted our work from November 2011 to September 2012 in
accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objective. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objective.
Page 31 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of Individual
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Departments and Agencies against Relevant Elements
Architecture Management Maturity Framework
of Our Enterprise Architecture Management Maturity
Framework
The following sections summarize the extent to which each of the 27
departments and agencies addressed elements in GAO’s Enterprise
Architecture Management Maturity Framework (EAMMF) that pertain to
measuring and reporting enterprise architecture outcomes and benefits.
The assessments given for each element are defined as follows:
● The agency or department fully satisfied the element.
◐ The agency or department satisfied some, but not all, aspects of the
element.
○ The agency or department did not satisfy any aspect of the element.
Page 32 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Agriculture Table 3 shows the Department of Agriculture’s (USDA) satisfaction of
relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 3: Department of Agriculture Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● USDA has defined its architecture’s purpose and goals. Specifically,
according to the department’s IT strategic plan for 2012 through 2016,
enterprise architecture and portfolio management practices are to be used
to address mission needs in a cost-effective and efficient manner. In
addition, according to the plan, the enterprise architecture program is to be
used as a strategic enabler to drive planning activities, provide insights,
and identify improvement opportunities for consolidation and reuse.
A method and metrics have been established
to measure enterprise architecture strategic
○ The department has not established a method and metrics for measuring
enterprise architecture outcomes; however, according to agency officials,
mission value (outcomes and benefits). it plans to do so. In particular, officials said the department is planning to
integrate the capital planning, budget, and enterprise architecture
processes and is working with the Federal Chief Information Officer (CIO)
in the area of shared services using architecture. In addition, according to
the department’s IT strategic plan, the department had planned to
establish and periodically report on enterprise architecture program
metrics by the end of fiscal year 2012. Officials explained that they now
expect to complete this effort by the end of fiscal year 2013 because they
are waiting for the Office of Management and Budget to provide additional
guidance.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department has not periodically measured and reported enterprise
architecture outcomes.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 33 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of the Air Table 4 shows the Department of the Air Force’s satisfaction of relevant
Force framework elements in version 2.0 of GAO’s EAMMF.
Table 4: Department of the Air Force Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● The December 2009 Air Force Architecting Concept of Operations defines
the vision and goal for the department’s architecture as follows:
• Vision: to enable the delivery of timely, relevant, unambiguous
information to support informed decision making by Air Force leaders to
maximize military capabilities while optimizing allocation of resources.
• Goal: to use architecture to unravel the complexity of systems,
processes, and programs to reveal their interdependent relationships to
decision makers, in an easily understandable format, so they may be
adequately considered as decisions are made.
Further, according to the Concept of Operations, the architecture is to be
used as a tool to eliminate redundancy, build efficiency, and maximize
resource distribution to ultimately increase the combat effectiveness of the
Air Force.
A method and metrics have been established
to measure enterprise architecture strategic
○ Air Force officials reported that the department has not yet established a
method or metrics to measure and report enterprise architecture outcomes
mission value (outcomes and benefits). and benefits. Officials stated that they have had a 60 percent architecting
division personnel turnover rate since June 2011, and have not been able
to identify industry-recognized enterprise architecture results metrics.
Nonetheless, officials stated that they anticipate documenting potential
metrics in October 2013.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The Air Force has yet to measure and report enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 34 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of the Army Table 5 shows the Department of the Army’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 5: Department of the Army Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Army has defined the purpose and goals for each of its three segment
architectures. According to Army officials, the collective purpose of the
segment architectures is to make performance-based and cost-informed
decisions that lead to the optimization of operations and technical
environments. The purpose and goals for each of the three segment
architectures has been defined as follows:
• Generating Force. According to Army’s 2011 Business Transformation
Plan, the purpose of this architecture is to drive integration across
functional domains, ensure integration between the Generating and
Operating Forces, and inform stakeholders on acquisition decisions
pertaining to the migration of legacy functionality to Army’s Enterprise
Resource Planning solution. Expected benefits are the streamlining of
end-to-end business processes aligned to the business enterprise
architecture and the elimination or reduction of the need to tailor
commercial-off-the-shelf systems.
• Operating Force. According to Army’s 2004 Architecture Approval and
Development memorandum, the purpose of this architecture is to assist
in managing systems that support the current and future Army and to
become a critical component in prioritizing and synchronizing Army-
wide efforts. According to Army’s 2011 Network Integration Roles,
Responsibilities, and Functions memorandum, architecture analysis will
be used to identify duplicative systems and incompatible
implementations and to integrate requirements, platforms, and network
capabilities across program offices, among other things.
• Network. According to Army’s January 2011 Network Enterprise
Architecture Foundation document, the purpose of this architecture is
to provide relevant, trusted, affordable, and timely information to
decision makers that support Army development and transformation,
and help sustain the Army’s transformation by facilitating an end-to-end
alignment of capabilities and investments in support of Army planning
and prioritization documents. Also, according to Army’s 2010 Global
Network Enterprise Construct Implementation Plan, the network
architecture program’s strategic initiatives include federating and
integrating networks, enforcing standards, and aligning Army and
federal data center consolidation initiative goals and objectives.
Page 35 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Element Satisfied? Summary
A method and metrics have been established
to measure enterprise architecture strategic
◐ Army has established metrics to measure outcomes and benefits of its
Network segment architecture related to its data center consolidation
mission value (outcomes and benefits). initiative. The metrics measure the extent to which the number of data
centers are closed each year, and the extent to which the number of
servers, the amount of floor space, and energy usage and its associated
costs are reduced. In addition, the Army has established a metric to
measure the extent to which it reduces the number of applications on data
servers and within data centers. It has also established, in its January
2012 Performance Plan for Reducing the Resources Required for Data
Servers and Centers, a method for measuring the reduction in
applications, which includes using an automated tool to collect, rationalize,
and track the migration of its applications.
However, Army has not established a method and metrics to measure
outcomes and benefits for its Generating Force and Operating Force
segment architectures. Although officials reported that they are tracking
the status of architecture artifact development, artifacts are architecture
program outputs rather than outcomes resulting from the use of an
architecture.
Army officials reported that the department faces a challenge that directly
relates to the lack of a centralized enterprise architecture office that can
provide oversight and guidance for architecture activities. A regulation
intended to address this challenge, with measures for assessing whether
the Army architecture is meeting the department’s needs, has been
drafted but has not been approved.
Enterprise architecture outcomes and
benefits are periodically measured and
○ Although the Network segment has established architecture method and
metrics related to data center consolidation, it has yet to measure and
reported to the agency’s enterprise report the outcomes and benefits. According to the Army’s performance
architecture executive committee. plan, an annual application reduction report will be provided to the DOD
CIO starting in fiscal year 2013. With regard to the Generating Force and
Operating Force segment architectures, the Army has yet to measure and
report enterprise architecture outcomes and benefits.
Source: GAO analysis of agency-provided data.
Page 36 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Commerce Table 6 shows the Department of Commerce’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 6: Department of Commerce Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Commerce defined goals for its enterprise architecture program in its
September 2010 Strategic Information Technology Plan for 2011-2015.
These include using the department’s enterprise architecture to continually
improve its business processes, align resources with Commerce’s top-
level strategic goals, and identify and support key IT management
decisions. According to the plan, Commerce plans to leverage its
architecture to reduce redundancy in its IT portfolio, combine capabilities,
utilize already-existing resources, and ensure that available IT resources
are documented and visible for all potential users.
According to the department’s Chief Enterprise Architect, the department
is reevaluating the goals and objectives of the enterprise architecture
program to make it more responsive to management requirements and to
place less emphasis on report and document generation. The department
established an enterprise architecture objective to adopt enterprise-wide
standards for enterprise architecture, purchasing, and cost savings in its
balanced scorecarda for the first quarter of fiscal year 2012.
A method and metrics have been established
to measure enterprise architecture strategic
◐ Commerce has established metrics and associated targets to measure
achievement of the objective to adopt enterprise-wide standards for
mission value (outcomes and benefits). enterprise architecture, purchasing, and cost savings. These include the
number of IT product standards adopted (target is two) and IT cost
reduction (target is $50,000 in annual savings).
However, the department has yet to establish a methodology that provides
the steps to be followed to measure enterprise architecture strategic
mission value. According to the Chief Enterprise Architect, the department
has documented a methodology to be used to demonstrate potential cost
savings. However, officials did not provide supporting documentation.
According to the Chief Enterprise Architect, measuring and reporting
enterprise architecture outcomes is a challenge because it is difficult to
attribute outcomes directly to architecture since outcomes are achieved
through a larger process that includes strategic and capital planning.
Enterprise architecture outcomes and
benefits are periodically measured and
○ According to the Chief Enterprise Architect, the department achieved
savings in the first quarter of 2012 by switching from a decentralized
reported to the agency’s enterprise approach to procuring computers, software, and computer services to a
architecture executive committee. single, department-wide vehicle. However, Commerce did not provide
documentation to support the measurement and reporting of these cost
savings.
Source: GAO analysis of agency-provided data.
a
The balanced scorecard is a private-sector concept introduced by Robert Kaplan and David Norton
in 1992 to assess organizational performance and is used by several government agencies. The
balanced scorecard is a form of performance plan that is used to help measure performance, make
improvements, and assess how well organizations are positioned to perform in the future.
Page 37 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Defense– Table 7 shows the Department of Defense (DOD) Business Enterprise
Business Enterprise Architecture’s (BEA) satisfaction of relevant framework elements in
Architecture version 2.0 of GAO’s EAMMF.
Table 7: DOD Business Enterprise Architecture Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s
intended purpose or strategic goals
● According to DOD’s March 2012 BEA Overview and Summary Information, the
purpose of the architecture is to (1) serve as a blueprint for business
are defined. transformation that helps to ensure that the right capabilities, resources, and
materiel are rapidly delivered to warfighters; (2) guide and constrain
implementation of interoperable defense business system solutions; (3) guide
information technology investments to align with strategic business capabilities;
and (4) support portfolio management during the investment review process.
DOD’s September 2011 Strategic Management Plan for fiscal years 2012-2013,
which establishes management goals for business operations, includes the goal
to reengineer/use end-to-end business processes to reduce transaction times,
drive down costs, and improve service. Associated with the goal is an initiative to
improve business operations through optimal use of defense business systems
and the BEA.
A method and metrics have been
established to measure enterprise
○ While the department has established metrics for measuring achievement of its
goal to improve business operations through optimal use of defense business
architecture strategic mission value systems and the BEA, the metrics do not measure BEA outcomes and benefits.
(outcomes and benefits). Specifically, DOD established metrics in its Strategic Management Plan for fiscal
years 2012-2013 which include percentage of defense business
systems/services represented in both the Defense Information Technology
Portfolio Repository (the department’s authoritative business systems inventory)
and the BEA, percentage of defense business systems/services represented in
both the Select and Native Programming Data Input System—Information
Technology (the department’s system used to prepare its budget submission)
and the BEA, and percentage of defense business systems/services reporting to
OMB through the BEA. However, these metrics measure output (i.e., direct
products and services), rather than outcomes (i.e., results of enterprise
architecture products and services such as benefits to Congress and the
American taxpayer) of the enterprise architecture program.
In addition, department officials stated that the department’s process to measure
and report architecture outcomes includes requiring components to submit
examples of business system improvement for inclusion in the department’s
annual report to Congress on Defense Business Operations. These are to be
substantiated with quantifiable measures that demonstrate desired business
outcomes and benefits. However, the guidance provided to program offices for
submitting these examples does not include the steps to be followed and metrics
for measuring BEA outcomes.
Enterprise architecture outcomes and
benefits are periodically measured
○ DOD has not periodically measured and reported enterprise architecture
outcomes and benefits. Specifically, while the March 2012 Congressional Report
and reported to the agency’s on Defense Business Systems describes enhancements to the BEA related to
enterprise architecture executive business process modeling and standardizing business data, and reports the
committee. number of legacy systems that are not part of the target architecture (based on
DOD IT Portfolio Repository data), the report does not include any additional
examples.
Source: GAO analysis of agency-provided data.
Page 38 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Defense– Table 8 shows DOD’s Enterprise Architecture satisfaction of relevant
Enterprise Architecture framework elements in version 2.0 of GAO’s EAMMF.
Table 8: Department of Defense Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● DOD has defined goals for its DOD enterprise architecture. Specifically,
according to the department’s February 2009 Directive on Management of
the DOD Information Enterprise, the DOD enterprise architecture, which is
composed of DOD enterprise and component levels, is to be maintained
and applied to guide investment portfolio strategies and decisions, define
capability and interoperability requirements, establish and enforce
standards, guide security and information assurance requirements across
DOD, and provide a sound basis for transition from the existing
environment to the future.
A method and metrics have been established
to measure enterprise architecture strategic
○ DOD has yet to establish a method and metrics for measuring DOD
enterprise architecture outcomes and benefits. According to officials,
mission value (outcomes and benefits). DOD’s approach to establishing a method and metrics for measuring DOD
enterprise architecture strategic mission value (outcomes and benefits) will
be accomplished through the development and publication of a DOD
instruction and an enterprise architecture management plan. In particular,
the draft instruction on enterprise architecture calls for establishing metrics
for assessing the effectiveness of the enterprise architecture to provide
information that contributes to mission effectiveness and efficiency. In
addition, the draft Enterprise Architecture Management Plan calls for the
development of metrics to assess the use of enterprise architecture,
provides examples of potential metrics, including reduction in
redundancies in DOD’s portfolio, and calls for the development of baseline
and target threshold values for each selected metric. The plan also states
that the DOD CIO and architecture organization are to determine the final
set of metrics and threshold values based on the resources available to
assess such metrics.
However, the department has not yet issued the instruction or the plan or
determined the specific method or final set of metrics to be used in
measuring enterprise architecture outcomes and benefits. According to
officials, the department expects the plan to be approved in December
2012 and the instruction to be approved in April 2013.
Enterprise architecture outcomes and
benefits are periodically measured and
○ DOD has yet to measure and report DOD enterprise architecture
outcomes and benefits. According to DOD officials, the implementation of
reported to the agency’s enterprise the instruction on enterprise architecture and the enterprise architecture
architecture executive committee. management plan will allow the benefits of architecture to be measured
and reported. However, the department has not yet issued the instruction
or the plan.
Source: GAO analysis of agency-provided data.
Page 39 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Education Table 9 shows the Department of Education’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 9: Department of Education Satisfaction of GAO EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Education has defined the purpose of its architecture program. Specifically,
according to its July 2011 Enterprise Transition Plan, enterprise architecture
provides for
• a priority-driven approach to planning and executing the activities needed
to transition from the baseline architecture to the target architecture,
• improved strategic decision-making and communication to achieve the
enterprise vision for technology at the department,
• increased control mechanisms for technology planning and investment,
• improved responsiveness to the enterprise technology needs of the
department’s business, and
• the ability to leverage technology to create a more effective and efficient
department.
A method and metrics have been
established to measure enterprise
◐ The department has established a metric to measure enterprise architecture
outcomes, but has yet to establish a method. Specifically, according to the
architecture strategic mission value October 2011 Office of the CIO Organization Performance Review report, a
(outcomes and benefits). key performance indicator for the Office of the CIO is to increase
development, modernization, and enhancement (DME) spending through use
of its enterprise architecture segment modernization planning process. To that
end, the department has established a metric to measure the extent to which
the ratio of the increase in spending on DME is increased relative to steady-
state spending. According to the department’s June 2010 IT Portfolio
Analysis, increasing spending in DME leads to a decrease in spending to
maintain current systems and technologies (i.e., steady-state spending). The
department explained that its architecture program works with line-of-business
segment owners to develop modernization plans that include achieving
operational efficiencies.
In June 2012, the department finalized an IT Portfolio Management Value
Measurement methodology that describes the process for determining the
value of an investment relative to the department’s IT portfolio. The
information is to be used to set priorities for funding decisions or selecting
investments to be included in the department’s IT portfolio. However, the
process is not a method for measuring architecture strategic mission value.
Enterprise architecture outcomes and
benefits are periodically measured and
◐ In October 2011, the department reported the ratio of DME versus steady-
state spending increased as a result of its enterprise architecture activities.
reported to the agency’s enterprise Specifically, according to the department’s October 2011 Office of the CIO
architecture executive committee. Organization Performance Review report, DME funding in the IT Portfolio
increased from 10 percent of total IT spending in fiscal year 2011, to 13
percent of total IT spending in fiscal year 2012 through use of the
department’s architecture segment modernization planning process. However,
this metric has yet to be periodically measured and reported as an
architecture outcome.
Source: GAO analysis of agency-provided data.
Page 40 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Energy Table 10 shows the Department of Energy’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 10: Department of Energy Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Energy has defined its enterprise architecture goals, which include:
• Maintain alignment between technology solutions and department
mission and goals.
• Provide enterprise architecture training and outreach opportunities,
thereby promoting enterprise architecture value and transparency to
support better business decisions department-wide.
• Work in conjunction with program and staff/support offices to further
define, elaborate, and identify areas for additional development in line
with the department’s mission.
• Continue to identify, reuse, and leverage, where possible, existing and
planned technology and infrastructure components across the
department.
• Identify areas, through capital planning and investment
control/enterprise architecture integration analysis, to reduce costs,
identify redundancy, and increase system and process effectiveness.
• Foster the organization and presentation of enterprise architecture to
support decision making, program analysis, and efficient achievement
of mission goals, utilizing an upgraded enterprise architecture data
repository.
A method and metrics have been established
to measure enterprise architecture strategic
○ Energy has not established metrics and a method for measuring
enterprise architecture strategic mission value. Specifically, although the
mission value (outcomes and benefits). department’s August 2012 Enterprise Modernization Roadmap includes
potential enterprise architecture program metrics (e.g., cost savings
through retiring legacy systems and cost avoidance by leveraging existing
solutions over procuring new ones through the use of enterprise
architecture), the metrics are still being defined and have yet to be
finalized and approved. Regarding a methodology, the roadmap states
that appropriate processes will be developed once the metrics are
developed and approved.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department has yet to measure and report enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 41 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Health and Table 11 shows the Department of Health and Human Services’ (HHS)
Human Services satisfaction of relevant framework elements in version 2.0 of GAO’s
EAMMF.
Table 11: Department of Health and Human Services Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise
architecture’s intended
● HHS has established the following enterprise architecture goals and objectives:
• Enable improved mission and business outcomes by providing products to support sound
purpose or strategic decisions, business processes, and effective solutions; providing structured methods and
goals are defined. guidance; supporting the development of transformation plans for addressing business
needs and priorities; enabling the optimized use of resources; and increasing
interoperability and information sharing within HHS and between HHS and external
stakeholders.
• Provide a consolidated view of HHS’s enterprise by providing a consolidated view of HHS’s
current and future business, information, and technologies; providing relevant, reliable, and
timely information analytics capabilities to support sound business decisions; increasing the
level of enterprise program integration and enterprise data sharing; facilitating the federated
management and maintenance of enterprise architecture information through the use of a
common framework; and facilitating the development of a consolidated view of information
about systems and investments.
• Strengthen the enterprise architecture program foundation by demonstrating the utility of
enterprise architecture to support program and business needs and priorities, aligning the
architecture program to HHS and federal enterprise business needs and priorities, and
fostering effective enterprise architecture practices at the operating division level.
A method and metrics
have been established to
● HHS has established a method and metrics to measure enterprise architecture outcomes. The
department developed an Enterprise Architecture Value Measurement Plan in December 2009
measure enterprise which includes measuring the extent to which the department increases the number of service
architecture strategic components that are reused. Specifically, it includes measuring the extent to which the
mission value (outcomes department increases the percentage of applicable service components in its service
and benefits). component reference model that are provided by one IT system and used by another. The
Enterprise Architecture Value Measurement Plan also identifies a method for how the metrics
are to be measured, including how they are to be calculated, the data sources to be used, and
targets to be achieved.
In addition, the Enterprise Architecture Value Measurement Plan includes measuring potential
cost avoidance based on recommendations made by the enterprise architecture program,
such as for business process reengineering; elimination of redundant IT systems and services;
and consolidation and reuse of IT systems, services, and data. However, agency officials said
they have yet to develop a methodology for measuring cost avoidance. The Chief Enterprise
Architect stated that it is a challenge to capture cost information, which is important to
establishing a baseline, because investments cut across a number of systems.
Enterprise architecture
outcomes and benefits
◐ Enterprise architecture results were measured and reported to the Chief Information Officer in
November 2010. Among other measures, the briefing reported that 16 percent of service
are periodically components were reused. However, the department did not measure the metric again and
measured and reported therefore, does not know the extent to which it increased its reuse of service components.
to the agency’s According to the Chief Enterprise Architect, the department is in the process of establishing
enterprise architecture new IT and enterprise architecture priorities and intends to establish a new enterprise
executive committee. architecture results measurement and reporting approach by the end of fiscal year 2012.
Source: GAO analysis of agency-provided data.
Page 42 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Homeland Table 12 shows the Department of Homeland Security’s (DHS)
Security satisfaction of relevant framework elements in version 2.0 of GAO’s
EAMMF.
Table 12: Department of Homeland Security Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● The DHS Enterprise Architecture Strategic Plan for Fiscal Years 2012-
2016 identifies the vision, mission, goals, and objectives of the
department’s enterprise architecture:
• Vision: Improving mission and performance, optimizing resources, and
unifying DHS.
• Mission: To optimize DHS resources and provide a framework for
strategic improvement and investment decisions.
• Goals: Plan and execute segment architecture, enhance operating
effectiveness, mature enterprise architecture practices, and strengthen
enterprise architecture program management.
• Objectives: To achieve each of the four goals, the plan outlines five
corresponding objectives, including establishing performance metrics to
drive standardization and accountability and identifying cost savings
and avoidance through efficient and effective use of resources.
A method and metrics have been established
to measure enterprise architecture strategic
○ DHS has not established metrics and a method for measuring enterprise
architecture outcomes and benefits. The department identified examples
mission value (outcomes and benefits). of enterprise architecture benefits (e.g., streamlined processes, increased
tool reuse, cost avoidance, increased sharing, increased process
improvements, and increased information sharing) and categorized them
(direct user/customer benefits, operational/mission performance benefits,
financial benefits, strategic/political benefits, and non-user/public benefits),
and according to DHS officials, the examples and categories are being
used to define enterprise architecture metrics. In addition, the department
has developed a tool for documenting and reporting enterprise
architecture outcomes. However, it has not finalized metrics and a method
with detailed steps to ensure that outcomes are consistently and
repeatedly measured.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department has yet to measure and report enterprise architecture
outcomes and benefits. DHS officials stated that they expect to report
reported to the agency’s enterprise architecture outcomes to a department executive body by October 1,
architecture executive committee. 2012.
Source: GAO analysis of agency-provided data.
Page 43 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Housing Table 13 shows the Department of Housing and Urban Development’s
and Urban Development (HUD) satisfaction of relevant framework elements in version 2.0 of
GAO’s EAMMF.
Table 13: Department of Housing and Urban Development Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● According to agency documentation, the primary purpose of the enterprise
architecture is to capture the information required to effectively plan a
course for achieving HUD’s strategic vision and goals. It is to be one
element of interrelated planning activities that are to enable HUD
managers and staff to define a vision, develop strategies and plans for
achieving the vision, make resource decisions, implement strategies, and
evaluate performance.
HUD’s enterprise architecture goals are to improve the efficiency and
effectiveness of the department’s programs; simplify HUD’s IT
environment by promoting standards and sharing and reusing common
technologies; improve interoperability by establishing enterprise-wide
standards; and reduce system development and operation and
maintenance costs by eliminating duplicative investments, promoting
sharing of common services, and establishing department-wide standards.
The department also defined enterprise architecture goals in its draft 2011
Enterprise Architecture Value Measurement Plan, including the goal of
enabling the use of enterprise IT technologies for reuse and to reduce
infrastructure complexity. Associated with this goal are objectives to
leverage existing IT technology products to meet business and functional
requirements, standardize enterprise technologies where it is cost
effective, and decommission obsolete systems that are no longer in use.
A method and metrics have been established
to measure enterprise architecture strategic
● HUD has established a method and metrics to measure its enterprise
architecture outcomes and benefits. Specifically, the department’s fiscal
mission value (outcomes and benefits). year 2011 draft Enterprise Architecture Value Measurement Plan includes
measuring the extent to which the department had decreased the number
of technology products added to its Technical Reference Model (TRM) that
duplicate existing capabilities, the extent to which it had increased the
number of standardized enterprise technologies across the department
that replace legacy products and do not duplicate existing capabilities, and
the extent to which it had decreased the number of obsolete systems in its
IT inventory. The plan also included steps to measure results and
outcomes, including identifying appropriate sources, and determining
baseline, target, and actual value measures.
Page 44 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Element Satisfied? Summary
Enterprise architecture outcomes and
benefits are periodically measured and
◐ The department measured and reported enterprise architecture outcomes
for fiscal year 2011 to a department executive committee in August 2012.
reported to the agency’s enterprise The December 2011 Enterprise Architecture Value Measurement Report
architecture executive committee. highlights areas, based on measurements, where additional focus and
improvement are needed. For example, according to the report, 12
technology products were added to its TRM that duplicate existing
capabilities versus a target of 6; 0 enterprise-licensed technologies
replaced legacy products versus a target of 3; and 0 obsolete systems
were decommissioned versus a target of 13. According to officials, the
department had not been able to retire these systems because their
maintenance costs were included in fixed-price contracts that included
systems that were currently being used. However, the department has yet
to measure and report the metrics again, and therefore, does not know the
extent to which it met its targets. According to the department, an
architecture Value Measurement Plan will be issued annually and results
of the measures in the plan will be documented in an annual report for the
fiscal year.
Source: GAO analysis of agency-provided data.
Page 45 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of the Interior Table 14 shows the Department of the Interior’s (DOI) satisfaction of
relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 14: Department of the Interior Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● The Department of the Interior established a purpose and goals for its
enterprise architecture program in 2009. Specifically, the purpose is to
develop, maintain, and oversee the implementation of an enterprise
architecture that helps the department achieve its strategic goals. Goals
for the enterprise architecture program are to (1) improve the
implementation of architectural plans and (2) increase the portion of the
enterprise architected. Department officials stated that an Enterprise
Modernization Roadmap with an updated enterprise architecture purpose
and goals is expected to be completed by September 2012.
A method and metrics have been established
to measure enterprise architecture strategic
○ While the department has established metrics for measuring achievement
of its goals, the metrics do not measure enterprise architecture outcomes
mission value (outcomes and benefits). and benefits. According to the department’s 2009 Enterprise Architecture
Program Management Plan, the key performance measures for the
program are the percentage of segments with completed architectures and
the percentage of development/modernization/enhancement funding
associated with completed and in-progress segment architectures.
However, these are not measures of enterprise architecture program
outcomes, but rather, measures of enterprise architecture development
and implementation. Officials reported that an assessment of the
department’s enterprise architecture program was recently conducted and
a new enterprise architecture program management plan is being
developed; however, they have not established a time frame for when the
plan will be completed.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department is not periodically measuring enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 46 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Justice Table 15 shows the Department of Justice’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 15: Department of Justice Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Justice has defined its architecture’s intended purpose. Specifically,
according to its IT Strategic Plan for 2010 through 2015, the Enterprise
Architecture Program Management Office is to review all IT investments to
identify enterprise solutions that address the needs of a core mission area
or multiple components. According to the strategy, enterprise solutions
help to eliminate redundant IT investments, increase information sharing,
and make use of shared infrastructure services, thus reducing the cost
and complexity of managing the department’s IT environment. Also,
according to the strategy, enterprise architecture analysis is to support
identifying redundant legacy programs to either retire or migrate to an
enterprise solution, thereby further reducing the complexity and the cost of
the IT environment.
A method and metrics have been established
to measure enterprise architecture strategic
○ The department has not established a method or metrics to measure
enterprise architecture outcomes and benefits. According to the
mission value (outcomes and benefits). department’s IT strategy, the enterprise architecture program is to help
identify and eliminate redundant programs, thus reducing costs. However,
the department stated that it is difficult to associate these cost savings
specifically within the department-level enterprise architecture because a
number of factors and groups contribute to the results.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department does not measure and report enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 47 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Labor Table 16 shows the Department of Labor’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 16: Department of Labor Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Labor has defined the purpose of its enterprise architecture program.
Specifically, according to its April 2011 Enterprise Transition Plan, the
department uses its enterprise architecture process with its capital
planning and investment management process to ensure that investments
support strategic goals and are not duplicative of existing business
solutions. Through use of this approach, according to the plan, the
department is able to identify duplicative resources/investments, gaps, and
opportunities for internal and external collaboration, resulting in
operational improvements and cost-effective solutions to business
requirements. In addition, the plan states that the department’s enterprise
architecture framework promotes interoperability and information sharing
and provides benefits such as
• enterprise target architecture definitions that support the department’s
mission objectives and strategic business plans,
• identification of redundancy and consolidation opportunities, and
• realization of cost savings and cost avoidance through improved
performance.
A method and metrics have been established
to measure enterprise architecture strategic
○ The department has not established a method or metrics to measure its
enterprise architecture outcomes and benefits. According to the April 2011
mission value (outcomes and benefits). Enterprise Transition Plan, Labor will establish enterprise architecture
program metrics to evaluate outcomes of the use of enterprise architecture
in investment decision making. However, department officials reported that
they have general measures related to capital planning that they use
across the Office of the Chief Information Officer and they do not
associate the measures specifically with enterprise architecture because a
number of factors contribute to the results.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department does not measure and report enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 48 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of the Navy Table 17 shows the Department of the Navy’s (DON) satisfaction of
relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 17: Department of the Navy Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● DON has defined the purpose of its enterprise architecture program.
According to the program’s 2010 All-View document, which provides an
overview and summary information of the DON enterprise architecture, the
purpose is to:
• Guide the department’s investments towards achieving departmental
goals and objectives.
• Assist DON program managers in the development of their “solution
architectures”–as mandated by the Joint Capabilities Integration
Development System and Acquisition processes.
More specifically, the enterprise architecture is to
• promote interoperability;
• delineate existing and future programs and projects;
• establish uniform and standard models for business processes and IT
systems that are common across DON;
• document all aspects of the enterprise including the functional
activities, business processes, information, participants, systems,
applications, and supporting technology infrastructure;
• support oversight and governance of IT investments;
• enable and align business and IT investments through improved
portfolio management, capital planning and investment control, and
other acquisition and budgeting processes;
• enable decision makers to identify capability gaps and overlaps; and
• provide insight into Doctrine, Organization, Training, Materiel,
Leadership, Personnel, and Facilities domains as they relate to the
business, information, systems, applications, and information
technology required for decision support at all levels of DON.
A method and metrics have been established
to measure enterprise architecture strategic
○ DON has not established a method or metrics to measure enterprise
architecture outcomes and benefits. Officials stated that they anticipate
mission value (outcomes and benefits). establishing a method in the second half of 2013. Officials reported that a
lack of best practices for measuring enterprise architecture value
continues to inhibit their ability to demonstrate enterprise architecture
return on investment.
Enterprise architecture outcomes and
benefits are periodically measured and
○ DON has yet to measure and report enterprise architecture outcomes and
benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 49 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of State Table 18 shows the Department of State’s satisfaction of relevant
framework elements in version 2.0 of GAO’s EAMMF.
Table 18: Department of State Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● According to State’s IT Strategic and Tactical Plans for fiscal years 2011
to 2013, the purpose of the enterprise architecture is to focus on
interoperability and application services.
A method and metrics have been established
to measure enterprise architecture strategic
○ The department has yet to establish a method or metrics for measuring
enterprise architecture outcomes and benefits. Agency officials stated that
mission value (outcomes and benefits). while their objective is to have good IT investments, it is difficult to
measure enterprise architecture’s contribution because IT investment
results are due to many factors, including good project management, and
adequate funding, as well as enterprise architecture. While, according to
the agency’s IT Tactical Plan, a key performance indicator for its
enterprise architecture program is evidence of increased use and value of
enterprise architecture products and services in providing consistent and
effective IT solutions, promoting interoperability, information sharing, and
collaboration, State has yet to establish metrics and a method for
measuring the value of its enterprise architecture products. Department
officials reported that they expect to create metrics by December 2012.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department has yet to periodically measure and report enterprise
architecture outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 50 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Table 19 shows Department of Transportation’s satisfaction of relevant
Transportation framework elements in version 2.0 of GAO’s EAMMF.
Table 19: Department of Transportation Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● The department’s Information Resources Management Strategic Plan for
fiscal years 2007-2012 includes the goal to establish its enterprise
architecture as the authoritative decision tool for IT investments. According
to the plan, enterprise architecture is to be used as a decision-making tool
to support business plan development, identify areas of duplication and
inefficiencies in the department, and select top priorities for department-
wide implementation and management.
A method and metrics have been established
to measure enterprise architecture strategic
◐ The department has established a metric but not a method to measure
enterprise architecture strategic mission value. Specifically, according to
mission value (outcomes and benefits). Transportation’s Information Resources Management Strategic Plan, an
expected enterprise architecture outcome is reduced total cost of
ownership, indicated by cost savings and/or cost avoidance identified
through review of business processes, data, applications, and technology,
and by the number/percentage of eliminated duplicative systems.
In addition, in December 2008, the department developed an enterprise
architecture Performance Measurement Guide, which included an
enterprise architecture performance objective to support investment
decisions for approved segment and solution architectures with an
outcome of solutions that foster transparency, increase mission
effectiveness, reduce redundancies, and minimize costs. However, a
method to measure the objective was not established. Specifically, the
guide included steps to finalize enterprise architecture performance
measures and indicators, such as conducting outreach with stakeholders
to disseminate information about performance measures and indicators,
and capturing baselines and defining targets. In addition, the guide
included a plan to collect and analyze data and to measure results on a
quarterly basis, starting the first quarter of calendar year 2009. However,
performance measures and indicators were never finalized and the guide
did not include the steps to be followed to collect and analyze the data.
According to the department’s Chief Architect, enterprise architecture was
a high priority in 2008 and 2009, but in mid-2010 the department shifted
priorities and limited resources, which has constrained enterprise
architecture efforts.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The department has not measured and reported enterprise architecture
outcomes and benefits. Though the department’s 2008 Enterprise
reported to the agency’s enterprise Architecture Performance Measurement Guide states that the department
architecture executive committee. planned to collect and analyze data and to measure and report results on
a quarterly basis starting the first quarter of calendar year 2009, it has yet
to do so. In addition, while the department reported in its response to our
survey on enterprise architecture results and outcomes that its
architecture program contributed to an estimated $83 million in cost
savings in fiscal year 2009, it did not provide documentation to support the
cost savings estimate or evidence that the outcome was reported to
agency executives.
Source: GAO analysis of agency-provided data.
Page 51 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of the Table 20 shows the Department of the Treasury’s satisfaction of relevant
Treasury framework elements in version 2.0 of GAO’s EAMMF.
Table 20: Department of the Treasury Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● Treasury has defined its enterprise architecture goals. Specifically,
according to Treasury’s E-Government Act Report for fiscal year 2011, its
enterprise architecture plans have focused on reducing duplication
through its data center consolidation initiative because infrastructure
reflects the majority of the department’s IT spending. Accordingly,
Treasury has defined goals related to data center consolidation.
Specifically, its goals include increased cost efficiency through
consolidation of facilities and infrastructure, increased economies of scale
and associated buying power, and reduced overhead associated with
operating multiple instances of common facilities and services. According
to officials, the department is in the process of developing broader goals
that will integrate its capital planning and investment control and
architecture processes.
A method and metrics have been established
to measure enterprise architecture strategic
◐ Treasury has established metrics for measuring enterprise architecture
outcomes and benefits but has yet to document a method. Specifically, the
mission value (outcomes and benefits). department plans to measure a decrease in the number of servers, an
increase in the percentage of operating systems that are virtual, and a
decrease in demand for data center square footage. According to the
department’s September 2011 Data Center Consolidation Plan, reductions
in physical assets should produce increases in capacity and cost
efficiencies for management of space and IT services. However, the
department has not established a methodology for measuring its
architecture outcomes with detailed steps to be followed (including
sources of information). Further, the Chief Enterprise Architect stated that
metrics corresponding to broader goals that integrate capital planning and
investment control and enterprise architecture are under development.
Enterprise architecture outcomes and
benefits are periodically measured and
◐ The department has measured and reported architecture outcomes
associated with its data center consolidation. Specifically, its September
reported to the agency’s enterprise 2011 Data Center Consolidation Plan, which was approved by the
architecture executive committee. department’s Deputy Assistant Secretary for Information Systems and
Chief Information Officer, reported a reduction in the number of servers,
an increase in the percentage of operating systems that were virtualized,
and a reduction in data center square footage between 2010 and 2011.
However, this metric has yet to be periodically measured and reported as
an architecture outcome. As noted above, going forward, Treasury is
developing new goals and metrics.
Source: GAO analysis of agency-provided data.
Page 52 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Department of Veterans Table 21 shows the Department of Veterans Affairs’ (VA) satisfaction of
Affairs relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 21: Department of Veterans Affairs Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
◐ According to the Chief Architect, VA is in the process of developing an
enterprise architecture program overview statement and guiding enterprise
architecture principles. Specifically, according to draft documentation, VA’s
enterprise architecture is to provide tools, rules, principles, and standards
to guide efficient, effective, and interoperable implementation of the
department’s vision of providing seamless delivery of benefits and
services to veterans. Global Enterprise Architecture principles include that
all VA solutions are to utilize enterprise-wide standards, services, and
approaches to deliver seamless capabilities to veterans, facilitate IT
consolidations through reuse, and simplify the use of VA functions.
According to department officials, enterprise architecture principles have
been finalized, but are not planned to be formally released until September
30, 2012.
A method and metrics have been established
to measure enterprise architecture strategic
○ According to the department’s Chief Architect, VA has not established a
method and metrics for measuring enterprise architecture outcomes and
mission value (outcomes and benefits). benefits because the department’s approach to enterprise architecture and
enterprise architecture governance is being revised. The Chief Architect
said that once the approach is updated, an enterprise architecture value
measurement plan will be developed. However, the official noted that it is
a challenge to know how much to attribute outcomes to enterprise
architecture relative to other factors in the decision-making process.
Enterprise architecture outcomes and
benefits are periodically measured and
○ VA has not periodically measured and reported enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 53 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Environmental Protection Table 22 shows the Environmental Protection Agency’s (EPA) satisfaction
Agency of relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 22: Environmental Protection Agency Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● EPA has defined its architecture goals and objectives. Specifically,
according to the agency’s February 2011 Modernization Blueprint, a major
goal is to use the architecture to identify segments within the organization
that could serve as candidates for service sharing and reuse. Additional
goals and objectives are described in the agency’s Office of Technology
Operation and Planning Mission Investments Solution Division April 2011
draft strategic plan for fiscal years 2011 to 2016. These include, among
other things, providing architecture services to enable stakeholders to
mature and increase value from their architectures, and developing a
common standard to be used in evaluating segments and solutions across
the enterprise.
A method and metrics have been established
to measure enterprise architecture strategic
○ The agency has not established a method and metrics for measuring
enterprise architecture outcomes and benefits. Specifically, agency officials
mission value (outcomes and benefits). reported that the results from an enterprise architecture management
maturity self-assessment, completed in April 2012, will be analyzed to
determine areas for improvement and reflected in the agency’s enterprise
architecture performance measurement plan. According to agency officials,
the enterprise architecture program performance measurement plan is
expected to be completed in fiscal year 2013.
The agency has included metrics in its Office of Technology Operation and
Planning Mission Investments Solution Division draft strategic plan and
Enterprise Architecture Value Measures project charter approved in May
2012. These metrics include the percentage of segments and solution
architectures that have been reviewed by the enterprise architecture
program, the percentage of investments that identify future use of
enterprise services, and the percentage of complete mandatory data fields
in the agency’s enterprise architecture repository. However, these metrics
measure outputs (i.e., direct products and services) of the program rather
than outcomes (i.e., results of enterprise architecture products and
services such as benefits to Congress and the American taxpayer).
According to officials, the metrics to measure cost savings and efficiencies
from enterprise architecture will be identified in fiscal years 2013 and
2014. Officials also added that it has been challenging to measure
performance because baselines have changed from year to year due to
changing OMB reporting requirements.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The agency is not measuring and reporting architecture outcomes and
benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 54 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
General Services Table 23 shows the General Services Administration’s (GSA) satisfaction
Administration of relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 23: General Services Administration Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● According to GSA’s March 2011 briefing to OMB on its enterprise
architecture modernization plan, its enterprise architecture provides
services to increase interoperability between systems; increase reuse of
systems, information, and services; increase agility and flexibility in
building and operating systems; and facilitate achievement of agency
goals. Further, according to GSA’s March 2012 Enterprise Modernization
Roadmap, enterprise architecture is to, among other things, increase
system interoperability and cost efficiencies, reduce duplication, and
increase innovation.
A method and metrics have been established
to measure enterprise architecture strategic
◐ GSA has established metrics but not a method for measuring enterprise
architecture outcomes and benefits. Specifically, GSA’s March 2012
mission value (outcomes and benefits). Enterprise Modernization Roadmap includes enterprise architecture
program work plans and corresponding output and outcome metrics. For
example, GSA has identified the percentage of applications complying
with IT standards as a metric for measuring the extent to which the agency
is increasing its use of IT standards, which is one of its desired outcomes.
The roadmap also includes a desired outcome of increasing development,
modernization, and enhancement spending by 25 percent per year
beginning in fiscal year 2014. According to the agency, the ratio of
development, modernization, and enhancement to steady-state spending
allows GSA’s enterprise architecture program to highlight the allocation of
IT spending and opportunities to reduce operating costs.
However, GSA has not established a method for measuring and reporting
enterprise architecture outcomes and benefits. According to GSA officials,
the benefits of architecture are achieved early in system development and
are difficult to relate to future return on investment measured later in a
system’s life cycle. Nonetheless, officials said they were beginning to
develop a method for measuring enterprise architecture outcomes, but did
not expect the plan to be completed for 2 to 5 years.
Enterprise architecture outcomes and
benefits are periodically measured and
○ The agency has not measured and reported enterprise architecture
outcomes and benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 55 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
National Aeronautics and Table 24 shows the National Aeronautics and Space Administration’s
Space Administration (NASA) satisfaction of relevant framework elements in version 2.0 of
GAO’s EAMMF.
Table 24: National Aeronautics and Space Administration Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise
architecture’s
● NASA’s November 2011 Enterprise Architecture Policy identifies the purposes of the agency’s
enterprise architecture, which include
intended purpose or • being a composition of architectures and set of integrated reference models that map all IT
strategic goals are initiatives, capabilities, and services to agency needs;
defined. • serving to guide executive decision making, establishing a clear linkage between present
capabilities and future NASA mission needs, including identifying potential shortfalls and
redundancies in IT capabilities, the time frame in which the shortfall or redundancy exists, and
an analysis of industry alternatives and remedial solutions/approaches;
• providing a foundation for further development, modernization or modification, and
enhancements of integrated architectures;
• identifying mission IT dependencies;
• being used as a tool to integrate strategic planning efforts and to select, guide, manage,
rationalize, and prioritize agency investments;
• establishing the framework for agency interoperability by providing the standard, rigorous
construct for horizontal and vertical integration of mission needs and business processes
through architecture;
• being integral to the budget life cycle, enabling informed and timely procurement decision
making to influence capital and strategic sourcing investments; and
• promoting transparency and accountability by aligning functions/capabilities, services,
systems, components, and related standards to agency strategy.
A method and metrics
have been established
○ NASA has yet to establish a method or metrics for measuring enterprise architecture outcomes.
However, according to NASA’s Chief Enterprise Architect, an approach for measuring enterprise
to measure enterprise architecture performance is being developed. Specifically, draft enterprise architecture procedural
architecture strategic requirements include metrics to measure the number of approved architecture artifacts. However,
mission value these metrics measure output (i.e., direct products and services) of the enterprise architecture
(outcomes and program, rather than outcomes (i.e., results of enterprise architecture products and services such
benefits). as benefits to Congress and the American taxpayer). Moreover, according to the Chief Architect,
the agency does not yet have a mature enterprise architecture program and establishing one is a
challenge because the agency’s IT environment is not structured in a way that readily accepts an
enterprise-wide architecture. Specifically, the official stated that much of NASA’s funding is
provided to the agency’s centers, which invest the money to meet their specific needs with little
regard for the agency’s overall needs or existing capabilities. In July 2012, the NASA Chief
Enterprise Architect stated that a NASA policy is expected to be issued by 2013, requiring that a
method and metrics for measuring enterprise architecture value be established.
Enterprise
architecture outcomes
○ NASA has yet to measure and report its architecture outcomes and benefits.
and benefits are
periodically measured
and reported to the
agency’s enterprise
architecture executive
committee.
Source: GAO analysis of agency-provided data.
Page 56 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
National Science Table 25 shows the National Science Foundation’s (NSF) satisfaction of
Foundation relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 25: National Science Foundation Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● NSF has identified enterprise architecture goals in its September 2008
Information Resource Management Plan. These goals are:
• Improve utilization of IT resources by eliminating duplicative
investments, and promoting sharing of common services and
standards.
• Improve program performance by ensuring business functions support
strategic goals and priorities, data are optimized in support of the
business, and applications and technology solutions are driven by
business needs.
• Simplify IT investment decisions by providing a line of sight from
strategy to business function to technology, which enables decision
makers to select investments that support NSF’s core mission, and to
identify duplicative or misaligned initiatives.
• Reduce IT diversity and complexity within NSF by promoting standards
and the sharing and reuse of common technologies.
• Improve interoperability through the establishment of enterprise-wide
standards that promote platform and vendor independence, enabling
greater interoperability across disparate applications, both internal and
external.
A method and metrics have been established
to measure enterprise architecture strategic
○ NSF has yet to establish a method and metrics to measure enterprise
architecture strategic mission value. Specifically, the agency has
mission value (outcomes and benefits). established metrics for measuring the percent of IT investments that
comply with the agency’s transition strategy, the percent of IT projects that
comply with the agency’s Enterprise Architecture Modernization Roadmap,
the percent of IT services associated with an appropriate segment
architecture, and the percent of approved software and technical
architectures fulfilling opportunities to reuse shared services and IT
infrastructure (i.e., ensuring that solution architectures in development
reuse common infrastructure components or develop components for
future reuse where possible). However, these metrics measure output
(i.e., direct products and services) of the enterprise architecture program,
rather than outcomes (e.g., cost avoidance, improved mission
performance from reengineered business processes and modernizing
systems, or benefits to Congress and the American taxpayer).
NSF officials stated that the agency is exploring opportunities to mature its
process for measuring enterprise architecture outcomes, and plans to
revise its Enterprise Architecture Program Management Plan to align with
new OMB enterprise architecture guidance by the end of fiscal year 2012.
Enterprise architecture outcomes and
benefits are periodically measured and
○ NSF has yet to measure and report enterprise architecture outcomes and
benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 57 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Nuclear Regulatory Table 26 shows the Nuclear Regulatory Commission’s (NRC) satisfaction
Commission of relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 26: Nuclear Regulatory Commission Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● NRC has defined the purpose and goals for its architecture. Specifically,
the purpose is to support IT goals that were established in its IT strategic
plan for fiscal years 2012 through 2016. These include:
• NRC staff and stakeholders can quickly and easily access the
information they need.
• IT business solutions are easy to use, cost effective, and strengthen
agency performance, which according to agency officials, is focused on
avoiding duplication and cost savings.
• IT infrastructure is available, cost effective, and responsive to agency
needs.
A method and metrics have been established
to measure enterprise architecture strategic
◐ NRC has established a metric for measuring enterprise architecture
outcomes, but has yet to establish a method. Specifically, for the goal that
mission value (outcomes and benefits). the agency’s IT infrastructure is available, cost effective, and responsive to
agency business needs, NRC plans to measure progress toward having
common access controls by measuring the reduction in passwords and/or
sign-ons. However, a methodology with detailed steps for measuring
enterprise architecture outcomes has not yet been established.
Enterprise architecture outcomes and
benefits are periodically measured and
○ NRC has not periodically measured and reported enterprise architecture
outcomes.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 58 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Office of Personnel Table 27 shows the Office of Personnel Management’s (OPM)
Management satisfaction of relevant framework elements in version 2.0 of GAO’s
EAMMF.
Table 27: Office of Personnel Management Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● According to OPM’s IT Strategic Plan for 2010-2013, the agency’s
enterprise architecture defines IT management principals, goals, and
objectives and establishes a roadmap to achieve the enterprise
architecture vision of centralizing and managing OPM’s IT infrastructure
for the benefits and efficiencies that can be realized through technology.
The IT strategic plan also includes the objective to utilize the enterprise
architecture as a management and governance tool to strengthen decision
making and standard setting, coordinating with OPM business lines to
ensure technology decisions and implementations for new systems align
with the agency’s as well as the federal government’s enterprise
architecture.
A method and metrics have been
established to measure enterprise
◐ OPM has established cost savings as a metric to measure enterprise
architecture results and outcomes, and developed an Enterprise
architecture strategic mission value Architecture Return on Investment Framework. According to the
(outcomes and benefits). framework, return on investment is calculated over a period of time and
relates the value contributed in dollars to the cost in dollars of the
enterprise achitecture program. The framework identifies steps the
enterprise architecture office plans to follow to determine the architecture’s
role in cost savings or improving mission, including defining enterprise
architecture’s role (strategic partner, collaborator, change agent etc.) in
improving business and IT and the percentage to attribute to enterprise
architecture for each role; collaboratively identifying the role enterprise
architecture is supposed to play before each major effort; evaluating the
actual role of enterprise architecture after each major effort; and soliciting
feedback on how well the role was performed. However, the framework
does not include steps to determine the cost savings or mission
improvement to which enterprise architecture contributes or to calculate in
dollars architecture’s return on investment. According to OPM officials,
one of the agency’s challenges in developing a method is quantifying
value based on the contributions of enterprise architecture to
business/mission improvement.
Enterprise architecture outcomes and
benefits are periodically measured and
○ According to OPM officials, the agency has measured enterprise
architecture cost savings and reported them to the Chief Information
reported to the agency’s enterprise Officer. However, officials have not provided documentation to support
architecture executive committee. that the cost savings have been reliably measured.
Source: GAO analysis of agency-provided data.
Page 59 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Small Business Table 28 shows the Small Business Administration’s (SBA) satisfaction of
Administration relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 28: Small Business Administration Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s intended
purpose or strategic goals are defined.
● According to SBA’s 2009 Capital Planning and Investment Control Policy
Guide, the agency’s enterprise architecture process is a management
practice that is to support strategic planning; capital planning and
investment control; system development; and IT asset management
activities to optimize the agency’s resources and achieve its performance
goals. Specifically, according to the guide, decision makers are to
leverage the agency’s architecture to help ensure that investments
• support the business needs,
• address specific and measurable performance gaps,
• align with the agency’s mission and goals,
• comply with the agency’s standards, and
• reduce or eliminate spending on unneeded, redundant, and/or
duplicative IT assets.
A method and metrics have been
established to measure enterprise
○ SBA has not established a method or metrics to measure enterprise
architecture outcomes and benefits. Nonetheless, according to agency
architecture strategic mission value officials, SBA measures the extent to which proposed IT investments align
(outcomes and benefits). with the enterprise architecture during its capital planning and investment
control process. However, such a metric measures output (i.e., direct
products and services) of the program rather than outcomes (i.e., results
of enterprise architecture products and services such as benefits to
Congress and the American taxpayer) of the program. Agency officials
also stated that the architecture program maintained Financial Assistance
and Disaster Assistance segment architectures, which were used to
facilitate planning and decision making, and that the agency achieved
high- level performance goals in fiscal year 2011 associated with the
segments. However, the agency did not provide documentation showing
that a method and metrics had been established for measuring
architecture outcomes, or that the high-level performance outcomes were
linked to enterprise architecture.
Enterprise architecture outcomes and
benefits are periodically measured and
○ SBA is not periodically measuring enterprise architecture outcomes and
benefits.
reported to the agency’s enterprise
architecture executive committee.
Source: GAO analysis of agency-provided data.
Page 60 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
Social Security Table 29 shows the Social Security Administration’s (SSA) satisfaction of
Administration relevant framework elements in version 2.0 of GAO’s EAMMF.
Table 29: Social Security Administration Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s
intended purpose or strategic
● SSA has defined its enterprise architecture goals in its 2010 Enterprise Architecture
Program Plan. Specifically, the architecture is to provide visibility for IT initiatives
goals are defined. and support alignment with SSA’s strategic business plans; support design and
configuration management decisions and alignment of IT initiatives with SSA’s
infrastructure; and support decisions regarding operations, maintenance, and the
development of IT resources and services.
In addition, as we previously reported, SSA’s enterprise architecture for years 2011
through 2016 described a vision that includes eliminating existing stove-piped
application software, and reusing services to develop service-oriented architecture
applications to replace aging online and back-office desktop applications.a
According to the SSA’s Enterprise Architecture Transition Strategy, these efforts are
expected to help reduce costs and increase productivity.
A method and metrics have been
established to measure
○ SSA has yet to establish a method and metrics to measure enterprise architecture
outcomes. SSA has established a metric to measure the number of IT projects
enterprise architecture strategic compliant with agency architecture standards. In addition, SSA officials stated that
mission value (outcomes and they measure the percent of IT investments aligned to the agency’s strategic
benefits). portfolios. However, these metrics measure outputs (i.e., direct products and
services) of the program, rather than outcomes (e.g., benefits to Congress and the
American taxpayer). SSA officials stated that they are considering developing
additional metrics to measure enterprise architecture value, for example, a metric to
measure the extent to which architecture helps identify opportunities to reuse
services and related software modules. However, they noted that the lack of
guidelines and best practices contribute to the difficulty in measuring outcomes. To
address this challenge, SSA officials said that they will participate in the Enterprise
Architecture Value Measurement workgroup of the federal CIO Council Strategy and
Planning Committee’s Architecture Subcommittee. They added that as new metrics
are identified they will be documented and communicated throughout the agency.
Enterprise architecture outcomes
and benefits are periodically
○ SSA has not periodically measured and reported enterprise architecture outcomes
and benefits.
measured and reported to the
agency’s enterprise architecture
executive committee.
Source: GAO analysis of agency-provided data.
a
GAO, Social Security Administration: Improved Planning and Performance Measures are Needed to
Help Ensure Successful Technology Modernization, GAO-12-495 (Washington, D.C.: Apr. 26, 2012).
Page 61 GAO-12-791 Organizational Transformation
Appendix II: Detailed Assessments of
Individual Departments and Agencies against
Relevant Elements of Our Enterprise
Architecture Management Maturity Framework
U.S. Agency for Table 30 shows the U.S. Agency for International Development’s (USAID)
International Development satisfaction of relevant framework elements in version 2.0 of GAO’s
EAMMF.
Table 30: U.S. Agency for International Development Satisfaction of EAMMF Elements
Element Satisfied? Summary
The enterprise architecture’s
intended purpose or strategic
● USAID has developed architecture goals. Specifically, the agency’s December 2011
Enterprise Architecture Program Charter identifies the following goals:
goals are defined. • Support improvement of mission-critical business processes through business
process analysis, and identification and application of enterprise architecture
standards.
• Guide analytical efforts to locate, validate, and promote the strategic use of agency
information.
• Facilitate analysis of the agency’s IT environment, including IT hardware, software,
and enterprise applications, to promote the effective and efficient deployment of IT
services.
• Provide governance for USAID technology efforts by designing and supporting the
implementation of enterprise architecture models and standards.
A method and metrics have been
established to measure
● USAID has established two metrics for measuring enterprise architecture outcomes:
(1) cost savings and avoidance due to process efficiency, technology standardization,
enterprise architecture strategic retirement, and consolidation; and (2) client satisfaction based on client survey
mission value (outcomes and responses on architecture’s value to business (such as facilitating decision making on
benefit). technology and processes using enterprise architecture tools). The agency has also
established guidance for measuring cost savings and avoidance and an approach to
measure client satisfaction by developing and implementing surveys.
Enterprise architecture outcomes
and benefits are periodically
● The agency has periodically measured enterprise architecture outcomes and,
according to agency officials, these outcomes are reported to the Deputy CIO and CIO.
measured and reported to the In addition, the agency has established a website for CIO staff, including the Deputy
agency’s enterprise architecture CIO and CIO, to review monthly architecture outcomes.
executive committee. According to its February 2012 Enterprise Architecture Performance Results report, the
agency achieved $12.3 million in savings and $9.5 million in cost avoidance by
transitioning disparate human resource systems to a human resource shared services
center. The Federal Enterprise Architecture and the agency’s enterprise architecture
were used to select a shared services center.
In addition, the agency reported estimated savings of $15.7 million (not including $4
million in migration costs) over the next 5 years, beginning in fiscal year 2013, by
moving its e-mail service to a cloud-based service. According to agency officials, the
cloud-based solution was recommended by the architecture team because it can
replace multiple installations of the current e-mail solution. According to the agency’s
return on investment analysis, this will reduce hardware and software maintenance,
and labor and other expenses. However, as of September 2012, the new service had
yet to be approved. Officials explained that delay in approval was causing a reduction
in the return on investment and that the cost savings were being updated, accordingly.
Source: GAO analysis of agency-provided data.
Page 62 GAO-12-791 Organizational Transformation
Appendix III: Comments from the
Appendix III: Comments from the Department
of Labor
Department of Labor
Page 63 GAO-12-791 Organizational Transformation
Appendix IV: Comments from the
Appendix IV: Comments from the Department
of the Treasury
Department of the Treasury
Page 64 GAO-12-791 Organizational Transformation
Appendix V: Comments from the Department
Appendix V: Comments from the Department
of Agriculture
of Agriculture
Page 65 GAO-12-791 Organizational Transformation
Appendix V: Comments from the Department
of Agriculture
Page 66 GAO-12-791 Organizational Transformation
Appendix VI: Comments from the
Appendix VI: Comments from the Department
of Commerce
Department of Commerce
Page 67 GAO-12-791 Organizational Transformation
Appendix VII: Comments from the
Appendix VII: Comments from the Department
of Defense
Department of Defense
Page 68 GAO-12-791 Organizational Transformation
Appendix VII: Comments from the Department
of Defense
Page 69 GAO-12-791 Organizational Transformation
Appendix VIII: Comments from the
Appendix VIII: Comments from the Department
of Education
Department of Education
Page 70 GAO-12-791 Organizational Transformation
Appendix VIII: Comments from the Department
of Education
Page 71 GAO-12-791 Organizational Transformation
Appendix IX: Comments from the
Appendix IX: Comments from the Department
of Homeland Security
Department of Homeland Security
Page 72 GAO-12-791 Organizational Transformation
Appendix IX: Comments from the Department
of Homeland Security
Page 73 GAO-12-791 Organizational Transformation
Appendix X: Comments from the Department
Appendix X: Comments from the Department
of the Interior
of the Interior
Page 74 GAO-12-791 Organizational Transformation
Appendix XI: Comments from the
Appendix XI: Comments from the Department
of State
Department of State
Page 75 GAO-12-791 Organizational Transformation
Appendix XI: Comments from the Department
of State
Page 76 GAO-12-791 Organizational Transformation
Appendix XI: Comments from the Department
of State
Page 77 GAO-12-791 Organizational Transformation
Appendix XI: Comments from the Department
of State
Page 78 GAO-12-791 Organizational Transformation
Appendix XII: Comments from the
Appendix XII: Comments from the Department
of Veterans Affairs
Department of Veterans Affairs
Page 79 GAO-12-791 Organizational Transformation
Appendix XII: Comments from the Department
of Veterans Affairs
Page 80 GAO-12-791 Organizational Transformation
Appendix XII: Comments from the Department
of Veterans Affairs
Page 81 GAO-12-791 Organizational Transformation
Appendix XIII: Comments from the National
Appendix XIII: Comments from the National
Aeronautics and Space Administration
Aeronautics and Space Administration
Page 82 GAO-12-791 Organizational Transformation
Appendix XIII: Comments from the National
Aeronautics and Space Administration
Page 83 GAO-12-791 Organizational Transformation
Appendix XIV: Comments from the Social
Appendix XIV: Comments from the Social
Security Administration
Security Administration
Page 84 GAO-12-791 Organizational Transformation
Appendix XIV: Comments from the Social
Security Administration
Page 85 GAO-12-791 Organizational Transformation
Appendix XV: Comments from the
Appendix XV: Comments from the
Environmental Protection Agency
Environmental Protection Agency
Page 86 GAO-12-791 Organizational Transformation
Appendix XVI: Comments from the
Appendix XVI: Comments from the Department
of Health and Human Services
Department of Health and Human Services
Page 87 GAO-12-791 Organizational Transformation
Appendix XVI: Comments from the Department
of Health and Human Services
Page 88 GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Appendix XVII: Comments from the
Department of Energy
Department of Energy
Page 89 GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Department of Energy
Page 90 GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Department of Energy
Page 91 GAO-12-791 Organizational Transformation
Appendix XVII: Comments from the
Department of Energy
Page 92 GAO-12-791 Organizational Transformation
Appendix XVIII: Comments from the
Appendix XVIII: Comments from the
Department of Housing and Urban
Development
Department of Housing and Urban
Development
Page 93 GAO-12-791 Organizational Transformation
Appendix XVIII: Comments from the
Department of Housing and Urban
Development
Page 94 GAO-12-791 Organizational Transformation
Appendix XVIII: Comments from the
Department of Housing and Urban
Development
Page 95 GAO-12-791 Organizational Transformation
Appendix XIX: GAO Contact and Staff
Appendix XIX: GAO Contact and Staff
Acknowledgments
Acknowledgments
Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov
GAO Contact
In addition to the contact named above, Neelaxi Lakhmani (Assistant
Staff Director), Mark Bird (Assistant Director), Virginia Chanley, Kelly Dodson,
Acknowledgments Cheryl Dottermusch, Michael Holland, James Houtz, Catherine Hurley,
Stuart Kaufman, Lee McCracken, Tyler Mountjoy, Donald Sebers,
Jennifer Stavros-Turner, and Merry Woo made key contributions to this
report.
(310964)
Page 96 GAO-12-791 Organizational Transformation
GAO’s Mission The Government Accountability Office, the audit, evaluation, and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding decisions.
GAO’s commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and afternoon, GAO posts on its website newly released reports, testimony,
and correspondence. To have GAO e-mail you a list of newly posted
Testimony products, go to http://www.gao.gov and select “E-mail Updates.”
Order by Phone The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s website,
http://www.gao.gov/ordering.htm.
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional information.
Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
Visit GAO on the web at www.gao.gov.
Contact:
To Report Fraud,
Waste, and Abuse in Website: http://www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470
Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional 4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations 7125, Washington, DC 20548
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548
Please Print on Recycled Paper.