The National Nanotechnology Initiative (NNI) is a component of the overall federal investment in science and technology (S&T), and the NNI goals fit within the broad S&T goals and priorities of the federal government as a whole. To assess the NNI and progress toward its goals, it is important to understand the framework within which it operates and how the definitions of success of federal initiatives are established.

The federal research and development (R&D) enterprise comprises programs in individual agencies focused on specific needs and missions and activities, such as the NNI, that involve a number of agencies and address national goals and priorities. The highest-level goals of federal S&T investment appear in various budget documents, such as the June 2012 memorandum on S&T priorities for the FY 2014 budget from Jeffrey Zients, acting director of the Office of Management and Budget (OMB), and John Holdren, director of the Office of Science and Technology Policy (OSTP):

Scientific discovery, technological breakthroughs, and innovation are the primary engines for expanding the frontiers of human knowledge and are vital for responding to the challenges and opportunities of the 21st century. We look to scientific innovation to

Citation Manager

"
2 Measuring Progress and Defining Success in the Context of Federal Research Initiatives ."
Triennial Review of the National Nanotechnology Initiative . Washington, DC: The National Academies Press,
2013 .

Please select a format:

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
2
Measuring Progress and
Defining Success in the Context
of Federal Research Initiatives
The National Nanotechnology Initiative (NNI) is a component of the overall
federal investment in science and technology (S&T), and the NNI goals fit within
the broad S&T goals and priorities of the federal government as a whole. To assess
the NNI and progress toward its goals, it is important to understand the framework
within which it operates and how the definitions of success of federal initiatives
are established.
The federal research and development (R&D) enterprise comprises programs
in individual agencies focused on specific needs and missions and activities, such
as the NNI, that involve a number of agencies and address national goals and
priorities. The highest-level goals of federal S&T investment appear in various
budget documents, such as the June 2012 memorandum on S&T priorities for
the FY 2014 budget from Jeffrey Zients, acting director of the Office of Manage-
ment and Budget (OMB), and John Holdren, director of the Office of Science and
Technology Policy (OSTP):
Scientific discovery, technological breakthroughs, and innovation are the primary engines
for expanding the frontiers of human knowledge and are vital for responding to the chal-
lenges and opportunities of the 21st century. We look to scientific innovation to
• Promote sustainable economic growth and job creation,
• Improve the health of the population,
• Move toward a clean energy future,
• Address global climate change,
• Manage competing demands on environmental resources, and
• Ensure the security of the Nation.
23

OCR for page 23
24 Triennial Review of the N at i o n a l N a n o t e c h n o l o g y I n i t i at i v e
That memorandum1 and its predecessors2 articulated the continuing role of
S&T in providing benefit to the nation and identified specific outcomes that the
U.S. government and taxpayers expect from federal investment in S&T (Box 2.1).
The priorities memorandums also identified S&T priorities that required
invest­ ent by and cooperation among multiple federal agencies and departments
m
for success. In responding to those priorities, the heads of the executive depart-
ments and agencies were instructed to
Balance priorities to ensure resources are adequately allocated for agency-specific, mission-
driven research while focusing resources, where appropriate, on addressing the (following)
multi-agency research activities that cannot be addressed effectively by a single agency.
The 2014 R&D priorities document3 lists nine multiagency R&D priorities: ad-
vanced manufacturing; clean energy; global climate change; R&D for informed
policy making and management; information technology R&D; nanotechnology
(the NNI); biologic innovation; science, technology, and mathematics education;
and innovation and commercialization.
Nanotechnology has been highlighted as a multiagency R&D priority in the
annual priorities memorandums almost since the NNI was created. The memo-
randums call for agencies to “strengthen interagency coordination,” to find “novel
approaches to collaboration,” and to support “joint programs using shared re-
sources.” Whereas the NNI is emphasized as a priority of the current and past
administrations, it also has been described as “a governmental initiative, represent-
ing a priority area for investment and activity, but not a distinct funding program
with separate budget authority or central management.” The direction from OSTP
and OMB for the agencies to focus resources in the NNI “where appropriate” and
the lack of a clear management or budget authority for the Nanoscale Science,
Engineering, and Technology (NSET) Subcommittee of the National Science and
Technology Council Committee on Technology has led to an agency-mission focus
1 emorandum
M for the Heads of Executive Departments and Agencies, “Science and Technology
Priorities for the FY 2014 Budget,” M-12-15, June 6, 2012, available at http://www.whitehouse.gov/
sites/default/files/m-12-15.pdf, accessed November 13, 2012.
2 or a summary, see R.M. Jones, “Making hard choices: OMB and OSTP issue guidance to
F
agencies on formulation of FY 2014 budget requests,” FYI: The AIP Bulletin of Science Policy News,
Number 88, June 19, 2012, available at http://www.aip.org/fyi/2012/088.html, accessed November
13, 2012; and Memorandum for the Heads of Executive Departments and Agencies, “FY 2009
Admin­stration Research and Development Budget Priorities,” M-07-22, August 14, 2007, avail-
i
able at http://m.­ hitehouse.gov/sites/default/files/omb/assets/omb/memoranda/ fy2007/m07-22.pdf,
w
a
­ ccessed Novem­ er 13, 2012.
b
3 emorandum for the Heads of Executive Departments and Agencies, “Science and Technology
M
Priorities for the FY 2014 Budget,” M-12-15, June 6, 2012, available at http://www.whitehouse.gov/
sites/default/files/m-12-15.pdf, accessed November 13, 2012.

OCR for page 23
Measuring Progress and Defining Success 25
BOX 2.1
Nanotechnology-Specific Excerpts from Representative
OSTP-OMB Priorities Memoranda
2005 Priorities Document1
Nanoscale R&D priority areas continue to include material science and research relevant to medical
care and homeland security. Though research at the nanoscale offers natural bridges to interdisciplinary
collaboration, especially at the intersection of the life and physical sciences, the administration encour-
ages novel approaches to accelerating interdisciplinary and interagency collaborations. Activities such as
joint programs utilizing shared resources, as well as support for interdisciplinary activities at centers and
user facilities, are encouraged.
2009 Priorities Document2
Robust federal investment in the agency programs that make up the NNI will expedite realization
of the potential of nanotechnology to address national priorities in areas such as energy, security, health
care, and the environment and will maintain U.S. scientific and technological leadership in this field.
Agencies should strengthen interagency coordination and support research on potential risks to
h
­ uman health and the environment, consistent with the National Science and Tech­ ology Council’s 2006
n
report, Environmental, Health, and Safety Research Needs for Engineered Nanoscale Materials.3
More broadly, the NNI should support both basic and applied R&D in nanoscience, develop instru­
mentation and methods for nanoscale characterization and metrology, and disseminate new technical
capabilities to help industry advance nanofabrication and nanomanufacturing. Nanoscale research offers
a natural bridge to collaboration between the life and physical sciences; therefore, agencies are encour-
aged to use approaches that accelerate interdisciplinary and interagency collaboration. Agencies are
encouraged to participate in activities such as joint programs utilizing shared resources or leveraging
complementary assets, as well as support for interdisciplinary activities at centers and user facilities.
2014 Priorities Document4
Within the interagency NNI, agencies should give priority to implementation of the 2011 Environ-
mental, Health, and Safety Research Strategy,5 presenting an approach to ensuring the safe, effective, and
responsible development and use of nanotechnology; and support for the Nanotechnology signature ini-
tiatives, which spotlight topical areas that represent key opportunities and can be more rapidly advanced
through focused interagency R&D efforts.
1 FY 2005 Interagency Research and Development Priorities, available at http://www.­ hitehouse.gov/
w
sites/ default/files/omb/assets/omb/memoranda/m03-15.pdf, accessed November 13, 2012.
2 Memorandum for the Heads of Executive Departments and Agencies, August 14, 2007, available at
http://m.whitehouse.gov/sites/default/files/omb/assets/omb/memoranda/fy2007/m07-22.pdf.
3 National Science and Technology Council (NSTC), Environmental, Health, and Safety Research Needs
for Engineered Nanoscale Materials, Nanoscale Science, Engineering, and Technology Subcommittee,
Committee on Technology, Executive Office of the President, September 2006, http://www.whitehouse.
gov/files/documents/ostp/NSTC%20Reports/NNI_EHS_research_needs%202006.pdf, accessed October 1,
2013.
4 Memorandum for the Heads of Executive Departments and Agencies, June 6, 2012, available at http://
www.whitehouse.gov/sites/default/files/m-12-15.pdf, accessed November 13, 2012.
5 NSTC, Environmental, Health, and Safety Research Strategy, Subcommittee on Nanoscale Science,
Engineering, and Technology, Committee on Technology, Executive Office of the President, October
2011, available at http://www.nano.gov/sites/default/files/pub_resource/nni_2011_ehs_research_strategy.
pdf, accessed October 1, 2013.

OCR for page 23
26 Triennial Review of the N at i o n a l N a n o t e c h n o l o g y I n i t i at i v e
within the NNI for many agencies. All the member agencies see nanotechnology
as enabling and the NNI as an important means of nurturing nanotechnology in
the agencies, throughout federal R&D, and in R&D throughout the nation, but the
agencies have used the NNI and its interagency bodies (the NSET Subcommittee
and the National Nanotechnology Coordination Office [NNCO]) primarily as a
vehicle for information-sharing and coordination of nanotechnology R&D activi-
ties. Program coordination and joint programs with shared resources have been
planned and implemented by agencies “as appropriate” only when they support
the primary missions of the agencies involved.
The conflict between the guidance from OSTP and OMB to the agencies to
strengthen collaboration to meet the NNI goals and the agencies’ interpretation of
“as appropriate” is a continuing source of tension between the NNI and those who
review it. Entities that support oversight activities, such as the President’s ­Council of
Advisors on Science and Technology (PCAST), which currently serves as the National
Nanotechnology Advisory Panel called for by law, the Government Account­ bility a
Office (GAO), and the NRC panels that have advised and reviewed the NNI, have
called upon the NNI agencies, through the NSET Subcommittee and the NNCO, to
create long-term collaborations with a shared vision, supported by joint planning,
coordination, and management. This sentiment dates back to the 2002 report of
the NRC entitled Small Wonders, Endless Frontiers, which made recommendations
for the initial organization and management of the NNI. That report included rec-
ommendations for the NSET Subcommittee to increase multiagency investments
in research, in particular at the intersection of nanoscale technology and biology.
This report recognizes the challenges to interagency programming and therefore
calls for a special fund for Presidential grants, under OSTP management, to sup-
port interagency research programs relevant to nanoscale science and technology.
The expectation of interagency collaboration also is reflected, for example, in the
2012 PCAST evaluation of NNI strategic planning: “While the NSET Subcommittee
in 2011 produced a ‘National Nanotechnology Initiative Strategic Plan,’ individual
agency contributions lack the cohesion of an overarching framework, and there is
no clear connection between the goals and objectives of the NNI strategic plan with
those of individual agencies.” That observation led the PCAST to recommend, as a
first step, clarifying how the NNI fits into agency priorities and programs: “NNCO
in partnership with OSTP should work with the agencies to develop implementation
plans for achieving the goals and objectives outlined in the 2011 NNI strategic plan.”
The conflict is also reflected in the 2012 GAO report Nanotechnology: Improved
Performance Information Needed for Environmental, Health, and Safety Research, in
which GAO evaluated NNI environmental, health, and safety (EHS) documents
according to its six desirable characteristics for a national strategy, as seen in
Table 2.1. The report stated:

OCR for page 23
Measuring Progress and Defining Success 27
TABLE 2.1 Summary of Desirable Characteristics of a National Strategy
Desirable Characteristic Brief Description
Purpose, scope, and Addresses why the strategy was produced, the scope of its coverage, and the process
methods by which it was developed.
Problem definition and Addresses the particular national problems and threats at which the strategy is
risk assessment directed.
Goals, subordinate Addresses what the strategy is trying to achieve, steps to achieve the results, and
objectives, activities, and priorities, milestones, and performance measures for gauging results.
performance measures
Resources, investments, Addresses what the strategy will cost, the sources and types of resources and
and risk management investments needed, and where resources and investments should be targeted by
balancing risk reductions and costs.
Organizational roles, Addresses who will be implementing the strategy, what their and others’ roles will be,
responsibilities, and and mechanisms for them to coordinate their efforts.
coordination
Integration and Addresses how a national strategy is related to other strategies’ goals, objectives, and
implementation activities and to subordinate levels of government and their plans for implementing
the strategy.
SOURCE: GAO, Report to the Chairman, Committee on Environment and Public Works, U.S. Senate, Nanotechnology Im-
proved Performance Information Needed for Environmental, Health, and Safety Research, Appendix I, accessed August 8, 2012.
NNI strategy documents for EHS research issued by the NSTC address two and partially
address the other four of the six desirable characteristics of national strategies identified by
GAO that offer a management tool to help ensure accountability and more effective results.
For example, the NNI strategy documents provide a clear statement of purpose, define key
terms, and discuss the quality of currently available data, among others. However, they do
not include performance information—such as performance measures, targets, and time
frames for meeting those measures—that would allow stakeholders to evaluate progress
towards the goals and research needs of the NNI. In addition, the documents do not in-
clude, or sufficiently describe, estimates of the costs and resources needed for the strategy.
Without this information, it may be difficult for agencies and stakeholders to implement
the strategy and report on progress toward achieving the research needs and assess if invest-
ments are commensurate with costs of the identified needs.4
The concerns raised by the GAO report with respect to NNI EHS documents
apply in general to the initiative as a whole. When NNI agencies do not share long-
term goals as evidenced by joint planning, resource allocation, coordination, and
4 Government Accountability Office, Nanotechnology: Improved Performance Information Needed for
­
Environmental, Health, and Safety Research, GAO-12-427, 2012, p. 1, available at http://www.gao.gov/
assets/600/ 591007.pdf, accessed August 8, 2012.

OCR for page 23
28 Triennial Review of the N at i o n a l N a n o t e c h n o l o g y I n i t i at i v e
management to reach the goals, it becomes difficult at best to define meaningful
“performance measures, targets, and time frames for meeting those measures” or
to assess “if investments are commensurate with costs of the identified needs.” As
noted in Table 2.1, performance measures (“metrics”) follow directly from defining
“what the strategy is trying to achieve, steps to achieve the results, and priorities,
milestones, and performance measurers for gauging results.” The question is then to
what extent the NNI agencies and the interagency NSET Sub­ ommittee should be
c
expected to create, implement, and assess a formal, long-term, interagency ­national
strategy for the NNI that is based on the GAO model.
The present committee’s interim report assessed the suitability of current
procedures and criteria for determining progress toward NNI goals and laid out
possible definitions of success and associated metrics for the NNI on the basis of
its four stated goals. (The ­nterim report is Appendix E.) The committee found
i
that the current procedures and criteria that the NSET Subcommittee and OSTP
use for assessing progress toward NNI goals are embodied in annual supplements
to the President’s budgets and the NNI strategic plans. As noted above, the an-
nual supplements describe current and proposed investments in nanotechnology
by agency; quantify investment by agency, program component area, and Small
Business Innovation Research and Small Business Technology Transfer programs;
and provide examples of specific accomplishments by individual agencies and by
groups with respect to the four broad NNI goals5 and the objectives in the stra-
tegic plan. The NNI strategic plan provides “the framework that underpins the
nanotechnology work of the NNI member agencies,” describes the interests of the
individual agencies in nanotechnology R&D and the NNI, and identifies objectives
and actions to reach the four goals.6 The implicit criteria for progress toward the
NNI goals that the NSET Subcommittee and OSTP formally use are thus based on
the quality of the specific examples of research or activities that meet the four goals
in the programs of the individual agencies or in collaboration with other groups
and on whether objectives in the strategic plan have been met.
Finding: The committee found that although the four NNI goals establish the
scope of the NNI and the annual supplements are useful for communicating
the breadth of NNI activities in the agencies, they are insufficient for measuring
progress and guiding the management and coordination of the NNI.
5 ee
S Box S.1 in the Summary.
6 ational
N Science and Technology Council (NSTC), National Nanotechnology Initiative ­ trategic
S
Plan, Committee on Technology, Subcommittee on Nanoscale Science, Engineering, and Technology,
February 2011, available at http://www.nano.gov/sites/default/files/pub_resource/2011_­ trategic_plan.
s
pdf, accessed December 19, 2012, p. 23.

OCR for page 23
Measuring Progress and Defining Success 29
In its interim report, the committee stated the need for an explicit frame-
work for managing and coordinating the NNI that links investment, outputs, and
short-term outcomes to specific long-term goals and links outcomes in the indi-
vidual agencies and of interagency collaborations. Such a framework would include
­essential performance-management concepts, such as having a vision, goals, clearly
articulated desired outcomes (long-term, medium-term, and short-term), accurate
data on the resources and funding available and how they are allocated, agreed-on
milestones and timelines, and models that link all these with specific agreed-
on performance measures (metrics) chosen to relate short-term and medium-term
outcomes and outputs to specific longer-term goals and outcomes.
That stated need is in line with the PCAST and GAO findings and recommen-
dations. As an interim step, PCAST recommended that all the individual NNI agen-
cies develop NNI implementation plans and articulate how their plans are related
to the NNI strategic plan. Some of the participating agencies, such as the National
Science Foundation and the National Institutes of Health, already have publicly
articulated strategic plans or implementation strategies for nanotechnology, but
most do not; see Box 2.2. Without goals that are more specific than the four general
NNI goals, a public commitment to specific outcomes, and a publicly articulated
performance-management framework, it is not possible to measure NNI progress
toward its four goals. The committee’s interim report also noted that the suggestion
for a framework is especially timely in light of the rapid development of compu-
tational tools and methods for collecting and analyzing data that may be useful
for tracking and measuring the progress of the NNI and guiding its management.
Recommendation 2-1: An overarching definition of success for the NNI as
a federal initiative should be evidence that NNI agencies are establishing
and implementing an effective, explicit framework for planning, manag-
ing, and coordinating publicly identified NNI interagency programs, such
as the signature initiatives. Such a framework should be based on essential
performance-management concepts, and plans for and progress toward spe-
cific outcomes should be reported annually in the NNI supplement to the
President’s budget.
In the following chapter, Chapter 3, the committee lays out some of the key
factors to consider when developing a framework for planning, management, and
coordination of NNI programs:
•• Who benefits, and how? The NNI stakeholders are discussed in light of how
they benefit from the NNI and what their definitions of success would be.
•• What financial resources are available within the NNI, and for what pur-
pose? The funding for NNI, which is all provided by the agencies, is

OCR for page 23
30 Triennial Review of the N at i o n a l N a n o t e c h n o l o g y I n i t i at i v e
BOX 2.2
Current Processes and Procedures for Evaluating Progress Toward NNI Goals
Each year in the annual NNI Supplements to the President’s budget, the NSET and the NNCO
r
­ eport lists (1) budget and expenditures data allocated to the program component areas (PCAs) from each
agency’s budget, (2) highlights from each agency programs, and (3) areas in which multiple agencies or
external organizations have been active. Progress of the NNI against its four stated goals is reported in
largely anecdotal form and is generally agency-centric. Several agencies have reported examples of suc-
cessful nanotechnology-relevant programs and projects; some provide numerical data, and some have
presented short summaries without many details. Interagency activities are reported in the same manner.
This clearly reflects the priorities of the NNI agencies: NNI agencies manage their overall portfolios to
focus on their primary missions, with nanotechnology being secondary.
For example, according to agency websites, the mission of the Department of Energy is to “ensure
America’s security and prosperity by addressing its energy, environmental and nuclear challenges through
transformative science and technology solutions;”1 the mission of the Department of Agriculture is to “pro-
vide leadership on food, agriculture, natural resources, rural development, nutrition, and related issues
based on sound public policy, the best available science, and efficient management”;2 and the mission
of the Department of Defense is to “provide the military forces needed to deter war and to protect the
security of our country.”3
Programs and individual projects are monitored and evaluated within each agency with respect to
its agreed-on mission-based deliverables by using processes and metrics developed by the agencies. But
the evaluations typically are program-specific and agency-specific, and the deliverables and outcomes
are generally reported in forms that cannot be easily aggregated and analyzed for their nanotechnology-
related content. The committee found that no method or system is common to the NNI agencies for
measuring and tracking progress toward NNI goals. Broad generalizations can be made, but there is little
granularity except for specific examples of successful projects, discoveries, and products related to the
agencies’ missions, which are mapped onto the four goals.
1 U.S. Department of Energy, About Us, available at http://energy.gov/about-us, accessed October 1,
2013.
2 U.S. Department of Agriculture, Mission Statement, available at http://www.usda.gov/fundinglapse.
htm?navid=MISSION_STATEMENT, accessed October 1, 2013.
3 U.S. Department of Defense, About the Department of Defense (DOD), available at http://www.
defense.gov/about/, accessed October 1, 2013.
analyzed in terms of the primary goals of the funding and the level of
interagency interaction.
•• What subjects in the NNI could benefit most from the framework? The
NNI signature initiatives and the NSET Subcommittee working groups are
examined in some detail.

Bookmark this page

Important Notice

As of 2013, the National Science Education Standards have been replaced by the Next Generation Science Standards (NGSS), available as a print book, free PDF download, and online with our OpenBook platform.