2
Study Methodology

Reflecting the two core questions (How many firms have actually been excluded by the ruling from participation in the NIH SBIR program? And what is the likely effect of this exclusion on these firms and on the NIH SBIR program) the methodology for this study is divided into two areas.

Identifying Venture-funded Firms

Identifying firms that have received venture funding is a challenge. SBIR-funded firms, which are in most cases privately held, are not required to reveal whether they have received third-party investment. As a result this information is not collected and stored by SBIR-funding agencies or SBA.

In order to establish the distribution of venture funding prior to the SBA ruling,1 this research initially focused on firms winning Phase II awards from 1992-2002 inclusive.

Phase II awards were selected for study because they account for the largest amount of NIH SBIR funding. As Phase II awards continue to grow in size, and are extended in length, the proportion of funding allocated for post-Phase I continues to grow. What is more, commercial success almost always comes after

1

The years prior to the ruling are years when a “natural” level of participation for venture-funded companies might be established.

Citation Manager

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 22
2
Study Methodology
Reflecting the two core questions (How many firms have actually been ex -
cluded by the ruling from participation in the NIH SBIR program? And what is
the likely effect of this exclusion on these firms and on the NIH SBIR program)
the methodology for this study is divided into two areas.
2.1 IDENTIFYING VENTURE-FUNDED FIRMS AND
ESTIMATING THE “EXCLUSION EFFECT”
Identifying Venture-funded Firms
Identifying firms that have received venture funding is a challenge. SBIR-
funded firms, which are in most cases privately held, are not required to reveal
whether they have received third-party investment. As a result this information is
not collected and stored by SBIR-funding agencies or SBA.
In order to establish the distribution of venture funding prior to the SBA ruling,1
this research initially focused on firms winning Phase II awards from 1992-2002
inclusive.
Phase II awards were selected for study because they account for the largest
amount of NIH SBIR funding. As Phase II awards continue to grow in size,
and are extended in length, the proportion of funding allocated for post-Phase I
continues to grow. What is more, commercial success almost always comes after
1The years prior to the ruling are years when a “natural” level of participation for venture-funded
companies might be established.
22

OCR for page 22
23
STUDY METHODOLOGY
Phase II rather than just Phase I—so a focus on the latter includes almost all
commercial successes.
The focus on 1992-2002 coincides with the rapid development and matu-
ration of the biotechnology industry. It also reflects the reauthorization of the
program in 1992, which led to prioritization of the legislative goals of SBIR with
an emphasis on commercialization. Finally, and not least, the focus on data from
1992 to 2002 is driven by the availability of data.
The firms winning Phase II awards also constitute the universe of firms ad -
dressed by the NRC’s Phase II Survey,2 and the NIH Phase II Survey,3 which
provide the best available data on outcomes from the SBIR program. These data
will be critical to the second phase of the analysis, identifying impacts.
The standard database on venture funding—also used by GAO in its 2006
study4—is the Thomson VentureSource database, and this is the primary source of
venture funding data utilized in this National Research Council study. 5 The NRC
compiled a list of firms that received at least one Phase II award at NIH between
1992 and 2002 inclusive and Thomson VentureSource ran that list against its own
database of firms that had received venture funding as of the end of 2006.
Because VentureSource requires that firms be identified based on company
name, a process was developed to broaden the net, identifying all firms that could
conceivably be matched to firms in the NIH awards database (using wildcards in
the database search6). These possible matches were then tested manually against
known company addresses to eliminate false positives from the results dataset. 7
The final list from VentureSource provides considerable detail on venture
capital investments: It indicates the name of the company, the date of the round
of funding, the type of funding, and the amount.8 This appears to be the most
2 For details on the NRC Phase II Survey, which included at least one questionnaire to every Phase II
winner 1992-2002 inclusive, see National Research Council, An Assessment of the SBIR Program at
the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies
Press, 2009.
3 For details on the methodology and scope of the NIH Phase II Survey, see National Institutes of
Health, National Surey to Ealuate the NIH SBIR Program: Final Report, July 2003, available at
.
4 U.S. Government Accountability Office, Small Business Innoation Research: Information on
Awards Made by NIH and DoD in Fiscal Years 2002 through 200, GAO-06-565, Washington, DC:
U.S. Government Accountability Office, April 2006.
5A secondary database owned by was also used by GAO but was not made
available to the NRC.
6We supplied a list of firm names; VentureSource replaced part of the name with wildcards and
then searched the database.
7The VentureSource search results included information on the city and state of the firm, and these
data were cross-checked against city and state data from the SBIR awards database.
8 “Type of funding” refers to terminology within the venture capital community. Thomson and
others distinguish between seed funding, A round funding, B round funding and C round funding.

OCR for page 22
2 VENTURE FUNDING AND THE NIH SBIR PROGRAM
definitive list of venture-funded firms available and it is used in the first phase
of this study.9
Estimating the “Exclusion Effect”
It is important to keep in mind that simply receiving some venture funding
is in itself not disqualifying. To be disqualified from participating in the SBIR
program, firms must be owned or controlled by firms that themselves fail one of
the two tests—breaching the size requirement and/or the individual ownership
requirement—outlined above in Box 1-1.10 Thus, the list of venture-funded firms
was then analyzed to determine whether it was likely that these firms would in
fact be excluded by the SBA ruling.
Unfortunately, privately owned firms are often very reluctant to provide in-
formation about their ownership structure. It is therefore not possible to determine
directly which venture-funded firms are owned or controlled by their venture
investors (and hence excluded) and which are not. Nor it is practical to examine
the ownership structure of every venture capital firm that provides funding in
order to determine whether their ownership structure or the collective character
of their other investments breach the eligibility requirements.
It is also worth noting that firm ownership and control are by no means syn -
onymous.11 Ownership of a majority of outstanding voting shares is sufficient
to provide formal control. However, key personnel may still exert significant—
sometimes predominant—control over key decisions. Conversely, 51 percent
ownership is not necessarily required in order to exert effective control. However,
both because the SBA ruling focuses on 51 percent ownership and because any
statistical analysis must find ways to draw bright lines through murky questions,
this study assumes that the critical delineator for the purpose of access to SBIR
is 51 percent ownership.
After considerable discussion, and drawing on their extensive experiences,
the Committee agreed on two proxies for venture control of a firm 12:
9 It is important to note that there is considerable heterogeneity among venture capital firms
themselves. For a review of some of the differences, see William A. Sahlman, “The Structure and
Governance of Venture-capital Organizations,” Journal of Financial Economics, 27(2):473-521,
October 1990.
10There are also small biotechnology companies that are 51 percent owned by individuals but are
excluded based on affiliation of a venture capital company’s other portfolio companies.
11 Eugene Fama and Michael Jensen distinguished between ownership and control in their classic
1983 paper. See Eugene Fama and Michael Jensen, “Separation of Ownership and Control,” Journal
of Law and Economics, XXVI, 1983. The role of venture capital control of small innovative busi -
nesses is discussed in Paul Gompers and Josh Lerner, “The Venture Capital Revolution,” The Journal
of Economic Perspecties, 15(2):145-168, Spring 2001.
12 Committee members with extensive knowledge of venture capital include Linda Powers (Toucan
Capital Corporation), Michael Borrus (X/Seed Capital), Clark McFadden (Dewey & LeBoeuf, LLP),
and Pete Linsert (Columbia Biosciences Corporation).

OCR for page 22
2
STUDY METHODOLOGY
• Venture capital investments of $5 million or more; or
• At least two separate rounds of venture funding.
In both cases, it is reasonable to assume that 51 percent of company shares will
have passed into the hands of the new investors.13
Using these proxies, the Committee screened the original list of venture-
funded firms to develop a list of firms that it identified as having been excluded
from the SBIR program on the basis of the SBA ruling. This screened list is
referred to as “venture-funded firms” throughout this report. 14
Next, this list was adjusted to take into account other grounds for exclusion.
Most notably, this list includes firms that would be subject to exclusion from the
SBIR program on other grounds, regardless of their venture capital ownership
structure. This includes circumstances where the firm—
1. Had more than 500 employees.
2. Was purchased by a foreign firm, or by a U.S. firm not 51 percent
individual-owned.
After eliminating firms that were venture-funded and met criteria 1 or 2
above for venture control, but also met one or more of the criteria above, we were
left with a final list of venture-funded companies that were likely to be excluded
based on the SBA ruling.
Grouping the Firms
In order to provide the most relevant analysis, the Committee focused on
two sets of variables:
• Whether a firm was or was not venture-funded, as defined above.
• Whether a firm was among the top 200 most prolific winners of NIH
Phase II funding during the 1992-2002 period.15
This approach generated the matrix in Table 2-1.
13This assumption was later tested against companies known to be excluded on venture capital
grounds. We found that our criteria in these cases matched known outcomes with a high degree of
accuracy.
14 It is important to note that these categories of majority venture funded firms as opposed to
venture-funded firms are frequently not static. As often happens in the biotechnology industry, a
company may have some venture funding and be eligible for SBIR Phase I and Phase II awards, but
then receive venture capital additional funding and become ineligible for further SBIR funding.
15The numerical basis for whether a firm was among the top 200 most prolific winners was driven
by the need for more detailed data and the constraints imposed by limited study resources. “Prolific”
is defined as being among the winners of the most NIH Phase II SBIR awards in 1992-2002.

OCR for page 22
2 VENTURE FUNDING AND THE NIH SBIR PROGRAM
TABLE 2-1 Firms Funded by the NIH SBIR Program (1992-2002)
Top 200 Winners Other Winners Totals Percent
VC-Funded 35 150 185 12
Not VC-Funded 165 1,186 1,351 88
Total 200 1,336 1,536 100
SOURCE: U.S. Small Business Administration Tech-Net Database, VentureSource.
NOTE: Table 2-1 refers only to majority venture-funded firms that meet the two core venture capital
investment criteria for control—namely more than $5 million in investment or more than one round
of venture control funding. The Top 200 winners accounted for 42.3 percent of Phase II awards
during 1992-2002.
Majority venture-funded firms were clustered among the top 200 winners.
They accounted for a higher percentage of the top 200 winners (17.5 percent)
than they did of all NIH winners. Table 2-1 shows that overall, about 12 percent
of firms that won SBIR Phase II awards from 1992-2002 inclusive were also
venture capital-funded.
In addition, firms that were more prolific in winning SBIR awards were also
more successful in attracting significant venture capital funding: 35 (17.5 percent)
of the top 200 award winners were majority venture-funded, as against 11.2 per-
cent of the remaining 1,336 SBIR award winners.16
2.2 METHODOLOGY FOR MEASURING THE
IMPACT OF THE SBA RULING
The primary objective of this part of the assessment is to compare outcomes
for SBIR projects implemented by venture-funded and non-venture-funded firms
respectively.
Two kinds of outcomes are utilized:
• Project-leel outcomes identified by the NRC and NIH surveys. These
include sales and the attraction of additional funding as the two key metrics. 17
• Firm-leel outcomes, generated by cross-referencing firms with Phase
II awards against revenue and employment data extracted from the Hoover’s
database of small firms.
16 It
is important to note, as well, that while SBIR enhances the success of venture-backed firms,
the obverse is also true. It is also possible that the top 200 SBIR award winners had topics that were
more advanced in their development.
17Additional funding as used in the NRC and NIH surveys means funding from all sources, includ -
ing VCs, angel funding, and additional non-SBIR federal funding (a major component).

OCR for page 22
27
STUDY METHODOLOGY
Project-level Outcomes
Until recently, there were no available data on outcomes from SBIR pro -
grams through which to compare the performances of venture-funded and non-
venture-funded firms. Recent surveys by the NRC and the NIH, however, provide
at least initial indications of relative outcomes.18
The NIH survey generated 768 responses (1 per firm) and the NRC 496
responses from 368 firms.19 Together, the surveys generated responses cover-
ing at least one project from 861 firms that received Phase II awards during this
period.20
Firm-level Outcomes
While the NRC/NIH data provide important insights into outcomes from spe-
cific surveyed projects, it is also useful to generate a different perspective based
on the development of the firm, rather than an individual project.
Utilizing the Hoover’s database of small firms, we developed a dataset of
current revenue and employee data, which provides a useful proxy for the over-
all commercial success of the firm.21 Unfortunately, Hoover’s does not maintain
time-series data on individual firms, so our metrics were based on the size of
firm revenues and number of employees as of the most recent data available from
Hoover’s (in most cases, for 2006).
As each data point in this dataset had to be collected manually, it was not
cost effective to collect individual data on more than 1,200 firms. Accordingly,
while we collected data for all 183 venture-funded firms,22 we limited data col-
lection for non-venture-funded firms to an equivalent random sample: 35 firms
among the top 200 most prolific Phase II winners, and 148 firms from among
the pool of firms that were not among the most prolific winners and were not
venture-funded.
18 See National Institutes of Health, National Surey to Ealuate the NIH SBIR Program: Final
Report, op. cit. The NRC survey results for NIH are presented in National Research Council, An
Assessment of the SBIR Program at the National Institutes of Health, C. Wessner, op. cit.
19There is some overlap between the projects surveyed by the NIH and NRC. However, as project
ID data is not directly comparable, it is not possible to determine exactly the dimensions of that
overlap.
20 Not every firm responded to the survey questionnaires. The 861 firms are a subset of the 1,536
firms that won Phase II awards 1992-2002.
21The Hoover’s database of small firms is an extension of the well-known Dunn and Bradstreet
database. It is the most comprehensive source of information on small businesses in the United States.
Even so, it offers only a current snapshot (no historical data), and is not entirely comprehensive as it,
in turn, is based in part on survey data. Hoover’s Small Business Database can be accessed on line
at .
22Table 3-4 derives the total number of firms that were excluded or possibly excluded as a direct
impact of the SBA directive.

OCR for page 22
2 VENTURE FUNDING AND THE NIH SBIR PROGRAM
BOX 2-1
Downward Bias in the Estimation of the
Impact of the SBA Directive
Assessing the impact of the SBA exclusion of majority venture-funded firms is
difficult because obtaining data on the ownership structure of companies, espe-
cially private-held small firms, is hard to do. This assessment therefore relies on a
sample of firms that received Phase II SBIR awards during the period 1992-2002
(before the directive was issued). The use of such data is likely to yield a down-
ward bias in the estimate of the effects of the SBA exclusion for two reasons:
• Recent shift of venture capital to the biotechnology sector. The life sci-
ences and biotechnology were relatively embryonic industries during the sample
period (before 2002). More importantly, venture capital started shifting towards
biotechnology and life sciences after the Internet bubble burst in 2001.a Thus, the
number and proportion of small life science-based firms receiving venture funding
may be much higher than can be captured from data focused on the 1992-2002
period.
• Venture funding barred as biotechnology firms matured. Leading papers
in the economics of innovation literature point out that technological and product
development opportunities (and thus, commercialization of research) become more
prevalent as an industry matures.b Venture funding is focused mainly on product
development. This implies that it is likely that the SBA ruling may be excluding a
higher percentage of firms in this sector as the industry matures.
aDouglas P. Lee and Mark D. Dibner, “The Rise of Venture Capital and Biotechnology in the
US and Europe,” Nature Biotechnology, 23:672-676, 2005.
bOn product innovation and commercialization in a wide variety of industries, see Edwin
Mansfield, “Academic Research and Industrial Innovation: An Update of Empirical Findings,”
Research Policy, 26(7-8):773-776, April 1998.
2.3 CASE STUDY AND OTHER DATA
One final area of analysis concerns the impact of the ruling on the biotech
industry itself. As noted earlier, BIO and other groups have claimed that the
exclusion of venture funded companies from SBIR will result in a critical “gap”
in the funding flow, one that may prevent important discoveries from being
commercialized.
To substantiate these claims, BIO conducted telephone and Internet sur-
veys of its “emerging company” membership—defined as firms with fewer than
350 employees and no marketable products.23 The surveys suggest that a large
23 BIO appears to have contacted all 650 firms that make up its emerging company membership,
with a response rate of about 41 percent. BIO provided no additional background information about
the surveys. For a description of the surveys, see Appendix E of this report.

OCR for page 22
29
STUDY METHODOLOGY
majority of responding biotech companies would apply for NIH funding absent
the ruling, and that most would not use the funds for their lead product. Two
thirds of the proposed research was reported to be focused on preclinical or
discovery stages.
BIO also conducted six case studies designed to show that promising lines
of early-stage research have been abandoned or delayed as a result of the ruling.
These cases, however, do not provide counterfactual evidence based on products
that were funded prior to the SBA ruling, which would have been excluded by
the ruling.
These cases will be discussed in the context of the NRC’s own cases, which
included some venture-funded firms. An analysis of the evidence submitted by
BIO is included in Appendix E of this report.