Download:
pdf |
pdfFinal Report
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Jo Anne Goodnight
Office of Extramural Research
National Institutes of Health
Bethesda, MD
Susan Pucie
National Heart, Lung and Blood Institute
National Institutes of Health
Bethesda, MD
January 23, 2009
Stephanie Karsten
Lynne Firester
Georgine Pion
April Smith
Maura Kephart
Humanitas, Inc.
Silver Spring, MD
National Survey to Evaluate the NIH SBIR Program
Final Report
TABLE OF CONTENTS
ACKNOWLEDGEMENTS.....................................................................................x
EXECUTIVE SUMMARY .....................................................................................xi
1. INTRODUCTION
1.1 Background ..................................................................................................... 1-2
1.1.1 2002 National Survey to Evaluate the NIH SBIR Program .......................................1-2
1.1.2 Subsequent Performance Monitoring and Evaluation ..............................................1-3
1.2 Methodology for the 2008 NIH Survey .......................................................... 1-4
1.2.1
1.2.2
1.2.3
1.2.4
1.2.5
1.2.6
1.2.7
Questionnaire ............................................................................................................1-5
Sampling ...................................................................................................................1-5
Contact Database Update.........................................................................................1-7
Survey Implementation..............................................................................................1-8
Survey Response ......................................................................................................1-8
Data Cleaning and Coding ........................................................................................1-9
Analytic Strategies and Reporting Practices...........................................................1-10
1.3 Strengths and Limitations of the Evaluation.............................................. 1-11
2. CHARACTERISTICS OF SURVEY RESPONDENTS
2.1 Usability and Eligibility of Potential Respondent Small Businesses......... 2-1
2.2 Spokesperson Characteristics ..................................................................... 2-2
2.3 Awardee Small Businesses............................................................................ 2-3
2.3.1 Nonrespondent and Respondent Awardee Comparison ..........................................2-3
2.3.2 Awardee Small Business Respondents ....................................................................2-5
2.4 SBIR-Supported Projects and Products ....................................................... 2-8
3. STIMULATING TECHNOLOGICAL INNOVATION
3.1 NIH SBIR Awardees Stimulated Technological Innovation........................ 3-1
i
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
TABLE OF CONTENTS (continued)
3.2 Performance on NIH SBIR Product Development Targets.......................... 3-2
3.2.1 New or Improved Products in Health-Related Fields................................................3-2
3.2.2 Technical Articles and Related Activities in Support of the NIH Mission ..................3-2
3.2.3 Patents and Other Indicators of Stimulating Technological Innovation ....................3-6
3.3 Performance on Receipt of Additional SBIR Awards .................................. 3-9
3.4 Comparison with 1997-2001 Phase II Awardees ........................................ 3-10
4. INCREASED USE OF SMALL BUSINESS CONCERNS
4.1 NIH SBIR Awardees Met Federal Research and Development
Needs ................................................................................................................ 4-1
4.2 Performance on Contributions to Knowledge ............................................. 4-2
4.3 Performance on Dissemination of Information ............................................ 4-4
4.3.1 NIH Dissemination to SBIR Phase II Awardees.......................................................4-4
4.3.2 Awardee Dissemination of Health-Related Information ...........................................4-4
4.4 Satisfaction with the SBIR Application, Review, Award, and
Post-Award Processes .................................................................................. 4-9
4.5 Comparison of Two Awardee Cohorts’ Performance in Increasing the
Use of Small Business to Meet Federal R&D Needs................................. 4-10
5. COMMERCIALIZATION OF INNOVATIONS
5.1 NIH SBIR Awardees Commercialized Innovations ...................................... 5-1
5.2 Performance in Commercializing SBIR-Funded Products ......................... 5-2
5.2.1
5.2.2
5.2.3
5.2.4
5.2.5
5.2.6
5.2.7
Product Sales ...........................................................................................................5-2
Cumulative Sales .....................................................................................................5-5
FDA Approval of SBIR-Funded Products.................................................................5-9
Licensing of SBIR-Funded Projects .......................................................................5-12
Other Evidence of Commercialization....................................................................5-12
Receipt of Additional Non-SBIR Funding ..............................................................5-13
Summary of Commercialization Efforts .................................................................5-18
iii
J anuary 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
TABLE OF CONTENTS (continued)
5.3 Growth of Awardee Small Businesses ..................................................... 5-20
5.4 Comparison of Two Awardee Cohorts’ Performance in
Commercializing Innovations .................................................................. 5-23
6. CONCLUSIONS AND RECOMMENDATIONS
6.1 Awardees Met or Exceeded NIH Performance Targets .............................. 6-1
6.2 Enhancements to NIH SBIR Program Monitoring and Evaluation
Should be Considered ................................................................................... 6-3
6.2.1 Update the SBIR Program Evaluation Framework ..................................................6-3
6.2.2 Expand Program Monitoring Capabilities.................................................................6-4
iv
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
LIST OF EXHIBITS
Exhibit 1-1 NIH Institutes and Centers within the 2008 Survey Sample ........................ 1-6
Exhibit 2-1 Characteristics of Survey Respondents ....................................................... 2-3
Exhibit 2-2 Selected Characteristics of Nonrespondent and Respondent SBIR
Awardee Small Businesses...................................................................................... 2-4
Exhibit 2-3 Awardee Small Business Respondents by State ......................................... 2-5
Exhibit 2-4 Years in Operation at the Time of the SBIR Award for
Responding Small Businesses................................................................................. 2-6
Exhibit 2-5 Major Field of Business Reported by SBIR-Funded Awardees
for FYs 2002-2006 ................................................................................................... 2-7
Exhibit 2-6 Type of Planned Product for SBIR Funded Awardees................................. 2-9
Exhibit 2-7 Technological Innovations by Type of Planned Product ............................ 2-10
Exhibit 2-8 Current Project Status by Fiscal Year of Award......................................... 2-11
Exhibit 2-9 Current Project Status by Type of Planned Product .................................. 2-12
Exhibit 2-10 Current Project Status by NIH IC Sponsor............................................... 2-14
Exhibit 2-11 Reasons for Discontinued Projects.......................................................... 2-15
Exhibit 2-12 Discontinued Projects by Product Characteristic
and Awardee’s Receipt of Additional SBIR Awards ............................................... 2-16
Exhibit 3-1 Summary of Outcomes Associated with Stimulating Technological
Innovation................................................................................................................. 3-1
Exhibit 3-2 Summary of Publications, Presentations, and Awards
Associated with SBIR-Supported Projects ............................................................... 3-3
Exhibit 3-3 Awardees with Presentations, Publications, and Awards
by the Current Status of Their SBIR-Supported Project........................................... 3-4
Exhibit 3-4 Awardees with Publications, Presentations, and Awards
by the Type of SBIR-Supported Product.................................................................. 3-5
v
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
LIST OF EXHIBITS (continued)
Exhibit 3-5 Awardees’ Patents, Copyrights, and Trademarks........................................ 3-6
Exhibit 3-6 Awardees with One or More Patents, Copyrights, and Trademarks
by Type of SBIR-Supported Project ......................................................................... 3-7
Exhibit 3-7 Reported Patents, Pending Patent Applications, Copyrights, and
Trademarks by Status of SBIR-Funded Project ....................................................... 3-8
Exhibit 3-8 Additional Related Phase I and Phase II SBIR Awards ............................... 3-9
Exhibit 3-9 Comparison of NIH SBIR Awardees’ Performance
in Stimulating Technological Innovation ................................................................ 3-11
Exhibit 4-1 Summary of Outcomes Associated with
Using Small Business Concerns to Meet Federal Research and
Development Needs................................................................................................. 4-1
Exhibit 4-2 Medical, Societal, or Technological Outcomes of the
SBIR-Funded Projects ............................................................................................. 4-2
Exhibit 4-3 Target Population Groups that Use or Are Likely to Use the
SBIR-Supported Product.......................................................................................... 4-5
Exhibit 4-4 Target Populations Currently Using or Likely to Use SBIR-Funded
Products .................................................................................................................. 4-6
Exhibit 4-5 Anticipated Size of Most Important Target Populations
for Ongoing Projects ................................................................................................ 4-8
Exhibit 4-6 Awardee Satisfaction with the NIH Application, Review, Award,
and Post-Award Processes...................................................................................... 4-9
Exhibit 4-7 Comparison of NIH SBIR Awardees’ Performance in Increasing
Use of Small Business to Meet Federal R&D Needs ............................................. 4-11
Exhibit 5-1 Summary of Outcomes Associated with Commercializing Innovations ....... 5-1
Exhibit 5-2 Number and Percent of Awardees That Expected Sales by
Type of Product........................................................................................................ 5-3
vi
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
LIST OF EXHIBITS (continued)
Exhibit 5-3 Current Status of Sales by Fiscal Year of Phase II Award........................... 5-4
Exhibit 5-4 Realization of Sales upon Project Completion by Type of SBIR-Supported
Product..................................................................................................................... 5-5
Exhibit 5-5 Amount of Cumulative Sales for Completed SBIR-Funded Projects ........... 5-6
Exhibit 5-6 Percent of Companies that Achieved Cumulative Sales
by Type of Product ................................................................................................... 5-7
Exhibit 5-7 Estimated Cumulative Sales for Awardees Realizing Sales by Awarding
NIH Institute or Center ............................................................................................. 5-8
Exhibit 5-8 Summary of SBIR Projects’ Sales Performance for FYs 2002-2006
as of December 2007............................................................................................... 5-9
Exhibit 5-9 Percent of SBIR-Funded Products Requiring FDA Approval by Type
of Product............................................................................................................... 5-10
Exhibit 5-10 FDA Approval Status by Type of SBIR-Funded Product ......................... 5-11
Exhibit 5-11 Stage of the Approval Process for SBIR-Funded Projects
that Submitted Applications to the FDA ................................................................. 5-12
Exhibit 5-12 Percent of Awardees Engaged in Marketing Activities by
Current Project Status............................................................................................ 5-14
Exhibit 5-13 Percent of Awardees that Received Additional Non-SBIR
Funding by Type of SBIR-Supported Product........................................................ 5-15
Exhibit 5-14 Percent of Awardees that Reported Receipt of Additional
Non-SBIR Funding by Awarding NIH Institute or Center ....................................... 5-16
Exhibit 5-15 Percent of Awardees that Reported Receipt of Additional
Non-SBIR Funding by Funding Source and Most Important Funding
Source.................................................................................................................... 5-17
Exhibit 5-16 Number and Type of Commercialization-Related Outcomes
Achieved by Awardees........................................................................................... 5-19
Exhibit 5-17 Number of Full-Time Employees for Awardees ....................................... 5-21
Exhibit 5-18 Awardees’ Reported Growth-Related Activities ....................................... 5-22
vii
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
LIST OF EXHIBITS (continued)
Exhibit 5-19 Comparison of SBIR Awardees’ Performance in
Commercializing Innovations ................................................................................. 5-24
Exhibit 6-1 Summary of Performance for FYs 2002-2006 SBIR Phase II Awardees..... 6-2
viii
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
APPENDICES
A
Evaluation Framework
A1
Program Goals and Measures
A2
Map of Survey Items to Goals
B
Survey Instrument
C
Email Messages and Telephone Follow-up Briefing
C1
Contact Update Email
C2
Email Verification Message
C3
Advance Email
C4
First Email Cover Letter
C5
Thank You/Reminder Email
C6
Second Email Cover Letter
C7
Personal Jo Anne Goodnight Email
C8
Incomplete Email
C9
Promise to Complete Email
C10 Promise to Complete Last Call Email
C11 Final Notice for the SBIR Survey Email
C12 Telephone Follow-up Briefing
D
Response Rate – Final Disposition of Sample
D1 Response Rate
D2 Usability Rates for Remaining Institutes and Centers
E
Survey Sample
ix
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
ACKNOWLEDGEMENTS
Contract support for the survey was provided from NIH Evaluation Set-Aside funds. Humanitas, Inc.,
planned the survey methodology, implemented the questionnaire online, conducted the follow-up with
nonresponders, and cleaned and analyzed the survey data.
Guidance on survey topics and questions was provided by an internal NIH staff working group whose
membership included:
•
•
•
•
•
•
•
•
Kay Etzler, Office of the Director, Division of Special Programs
Frankie Giles, National Heart, Lung and Blood Institute
Jo Anne Goodnight, Office of the Director, Division of Special Programs
Margaret Grabb, Ph.D. National Institute of Mental Health
Rosemarie Hunziker, Ph.D. National Institute of Biomedical Imaging and Bioengineering
Matthew Portnoy, Ph.D., National Institute of General Medical Sciences
Susan Pucie, National Heart, Lung and Blood Institute
Michael Weingarten, National Cancer Institute
Finally, special thanks to the SBIR awardees who so willingly participated in the study to describe their
experiences with the SBIR program and their project outcomes.
x
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXECUTIVE SUMMARY
Established in 1982 through the Small Business Innovation Development Act, the Small Business
Innovation Research (SBIR) program offers qualified small business concerns the opportunity to propose
and develop innovative ideas with commercial potential while meeting the specific research and
development (R&D) needs of the Federal government. SBIR legislation currently requires agencies with
extramural R&D budgets that exceed $100 million to set aside 2.5 percent of their extramural R&D funds
for an SBIR program. Among the 11 Federal agencies that currently participate in the SBIR Program, the
National Institutes of Health (NIH) is the largest civilian agency and the second largest participating
Federal agency overall. Since the inception of the program in 1982, NIH has invested over $5 billion in
research support to small business concerns.
This report summarizes the findings from a 2008 survey of small businesses that competed successfully
for SBIR Phase II awards in fiscal years (FYs) 2002-2006. This new survey was a logical next step in the
NIH SBIR program evaluation and monitoring effort. A 2002 national survey to assess the outcomes of
the NIH SBIR program had described NIH SBIR program performance of Phase II awardees funded from
FYs 1992 through 2001. The 2008 survey included a new group of awardees. Both surveys addressed the
same major evaluation issues:
• To what degree has the NIH SBIR program stimulated technological innovation?
• Has the NIH SBIR program’s use of small business concerns satisfied Federal R&D needs?
• Has the NIH SBIR program increased private sector commercialization of innovations derived
from Federal R&D funding?
Survey Response
To leverage NIH expenditures to date and also to promote comparability of results, the design and
execution of the 2008 national survey to assess the outcomes of the NIH SBIR program was based on the
same guiding principles used for the 2002 survey.
The survey was administered to all 918 eligible awardee small businesses that received an NIH SBIR
Phase II award in FYs 2002-2006. There were 719 respondent companies, and the final response rate
was 78 percent.
Respondents and nonrespondents were similar in terms of the Institute/Center (IC) that sponsored the
Phase II award, the fiscal year of the award, and the geographic region in which the small business was
located. Coupled with the high response rate, these similarities reduce the potential threat of
nonrespondent bias to the survey results.
The largest percentage of respondents (32 percent) was located in the West, and the smallest percentage
(16 percent) in the Midwest. Twenty-eight percent were located in the Northeast, and the remaining 24
percent were in the South.
Over one-fourth of respondents (28 percent) received their Phase II award in FY 2006; between 17 and 19
percent of the companies received their award in each of the other calendar years covered by the survey,
FYs 2002-2006. The National Cancer Institute (NCI) supported 15 percent of all respondents, and the
National Heart, Lung and Blood Institute (NHLBI) supported 13 percent. The National Institute
xi
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
of General Medical Sciences (NIGMS); the National Institute of Diabetes, Digestive, and Kidney
Diseases (NIDDK); the National Institute on Aging (NIA); and the National Institute of Neurological
Disorders and Stroke (NINDS) each awarded funds to another 6 to 8 percent of the companies that
responded to the survey. The remaining awardee companies were funded by one of the other 16
participating ICs.
Overall, 41 percent of the respondents reported that their SBIR-funded product, process, or service was
currently in the development phase at the time of the survey. Another 20 percent said that the SBIRfunded product, process, or service was being commercialized, and 22 percent stated that it was in use by
the target population(s). Ten percent said that product, process, or service production was “on hold”,
awaiting the receipt of additional funding, or inactive for another reason. Eight percent of the awardee
companies reportedly had discontinued work on the project. (Numbers total more than 100 due to
rounding).
Summary of Findings
Three of the four program goals were the survey focus – stimulate technological innovation (Goal 1), use
small business concerns to meet R&D needs (Goal 2), and increase the commercialization of innovations
(Goal 4). The exhibit on the following page displays these three goals, related performance indices and
measures promulgated by the NIH SBIR program Evaluation Framework for each, and the 2008 survey
finding about the percentage of awardees that achieved each performance index.
xii
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Summary of 2008 Survey Findings:
Performance of NIH SBIR Phase II Awardees in FYs 2002-2006*
Finding
NIH SBIR Performance Goal
NIH SBIR Performance Index
NIH SBIR Performance Measure
1.0 Stimulate Technological Innovation
1.1 Whether or not sales have occurred, NIH SBIR awardees produce new or improved
products, processes, usages, and/or services in support of the NIH mission.
Percent of awardees developing new or improved products
Percent of awardees having published one or more technical articles on new or improved
SBIR supported products
Percent of awardees having obtained one or more patents relevant to the core technology
supported by the Phase II award
1.2 NIH SBIR awardees receive additional Phase I or Phase II awards that relate to the core
technology.
Percent of awardees receiving additional Phase I or Phase II awards that related to the
core technology
82%
53%
31%
58%
2.0 Use Small Business Concerns to Meet Federal R&D Needs
2.1 NIH awardees make contributions to knowledge in health promotion, disease prevention,
diagnosis, health care, and amelioration and cure of disease.
Percent of SBIR awardees that have contributed to increases in knowledge regarding
health promotion, disease prevention, diagnosis, health care, and amelioration and cure of
disease
2.2 NIH awardees are able to obtain and disseminate health-related information
2.3
Percent of awardees that have disseminated or plan to disseminate SBIR supported
technology and information among populations using and receiving health and health care
resources
NIH SBIR awardees express satisfaction with the usefulness of the NIH SBIR program
Percent of awardees, depending on the program component, that are completely or mostly
satisfied with the SBIR application, review, award, and post-administration of the program
82%
82%
74 – 90%
4.0 Increase the Commercialization of Innovations
4.1 Companies with NIH SBIR awards commercialize new or improved products, processes,
usages, and/or services in health-related fields.
Percent of SBIR-supported products, processes, usages, and/or services that yield sales
Percent of awardees that have successfully executed licensing arrangements for their
SBIR-supported product
25%
Percent of awardees reported commercializing the core technology or information
supported by their Phase II award
61%
Percent of awardees obtaining additional non-SBIR funding for their Phase II product
4.2 Companies with NIH SBIR awards grow their companies.
Percent of awardees that believed that the SBIR Program had had an impact on their
ability to grow their companies in terms of hiring additional personnel
*The survey assessed performance for Goals 1, 2, and 4 from the Evaluation Framework for the NIH
SBIR program.
xiii
33%
36%
82%
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Customer satisfaction responses indicated that nearly all SBIR awardees (96 percent) were satisfied with
their experiences in obtaining information about the Phase II program. Between 88 and 90 percent
expressed satisfaction with the application, award, and post-award administration components of the
Phase II program.
The 2008 survey instrument was essentially identical to that of the 2002 survey, so it was possible to
compare outcomes for respondents to both surveys. For such comparisons, the respondent cohort for the
2002 survey was restricted to awardees that received grant funding during the last five-year period
surveyed, FYs 1997 through 2001. This made the elapsed time from award receipt to survey
administration as comparable as possible because the 2008 survey cohort also received grant funding for a
five-year period, FYs 2002-2006.
Performance of the two cohorts did not significantly differ for the majority of outcomes. The earlier
cohort did have significantly higher percentages of awardees that published one technical article (68
percent versus 43 percent); made conference presentations (79 percent versus 73 percent); and were
granted at least one patent for an SBIR-supported product (39 percent versus 31 percent).
xiv
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
1. INTRODUCTION
The Small Business Innovation Research (SBIR) program, first authorized in 1982, occupies a key niche
in the nation’s scientific research and development arena. The program encourages small businesses to
explore their technological potential and provides the incentive to profit from commercializing products,
processes and services developed with SBIR support.
Through a competitive, three-phase award system, the program provides qualified small business
concerns with opportunities to propose and develop innovative ideas that meet the mission or specific
research and development (R&D) needs of the Federal government.
• Phase I. The objective of Phase I is to establish the technical merit, feasibility, and potential for
commercialization of the proposed research or R&D efforts, and to determine the quality of
performance of the small business awardee organization prior to providing further Federal
support in Phase II. Support under Phase I normally may not exceed $100,000 for total costs for a
period normally not to exceed six months.
• Phase II. The objective of Phase II is to continue the R&D efforts initiated in Phase I. Funding is
based on the results achieved in Phase I and the scientific and technical merit and commercial
potential of the project proposed in Phase II. Only Phase I awardees are eligible for a Phase II
award. Support for SBIR Phase II awards normally may not exceed $750,000 in total costs for a
period normally not to exceed two years.
• Phase III. The objective of Phase III, where appropriate, is for the small business concern to
pursue with non-SBIR/STTR funds the commercialization objectives resulting from the research
and R&D activities of Phases I and II. Phase III may involve follow-on non-SBIR funded R&D
or production contracts for products, processes or services intended for use by the U.S.
Government.
The National Institutes of Health (NIH) is the largest civilian agency and the second largest participating
agency of the 11 Federal agencies participating in the SBIR program. Since the inception of the program
in 1982, NIH has invested over $5 billion in research support to small business concerns. The NIH SBIR
program is intended to foster the agency’s mission to “uncover new knowledge that will lead to better
health for everyone,” particularly in translating research results into tangible products and services that
will improve human health.
In late 2006, NIH SBIR program administrators from a cross-section of the Institutes and Centers (ICs)
decided to sponsor a survey of Phase II awardees whose awards were granted in fiscal years (FYs) 20022006. This survey was planned as a next step in documenting NIH SBIR program outcomes. The first
step was the 2002 National Survey to Assess the Outcomes of the NIH SBIR Program, which described
NIH SBIR program performance from FYs 1992-2001. For both surveys, the major evaluation questions
addressed targeted performance related to three program goals:
• To what degree has the NIH SBIR program stimulated technological innovation?
• Has the NIH SBIR program’s use of small business concerns satisfied Federal R&D needs?
Page 1-1
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
• Has the NIH SBIR program increased private sector commercialization of innovations derived
from Federal R&D funding?
This report documents the 2008 national survey to assess the outcomes of the NIH SBIR program:
• Chapter 1 discusses background issues that influenced and prompted the study and also
summarizes the study methodology.
• Chapter 2 notes characteristics of potential and actual survey respondents.
• Chapters 3-5 describe study findings about each of the three key evaluation questions.
• Chapter 6 presents study conclusions and recommendations.
1.1
Background
Before 2003, there was little formal analysis of either the overall SBIR program or the one administered
by the NIH, with the exception of a few studies by the Government Accountability Office (GAO), the
Office of the Inspector General (OIG), and the Small Business Administration (SBA). These early
studies raised questions about program performance, prompting a series of recent evaluations of both the
Federal program and its NIH component.
1.1.1
2002 National Survey to Evaluate the NIH SBIR Program
Beginning in the late 1990s, NIH SBIR program administrators from a cross-section of ICs took a
heightened interest in the success of the NIH SBIR program. By 2001, they formalized consensus about
how to evaluate the program in a manner that would assess responsiveness to Congressional intent for the
SBIR program. They produced an Evaluation Framework document that reflects their agreement about
standards, indices, and measures for describing how well the NIH SBIR program is performing in
meeting all Congressional program goals. The Framework is, to their knowledge, the first such guide for
any SBIR program about why and how to evaluate overall performance.
The NIH administrators then decided to survey NIH awardees to learn about program performance. They
developed a draft survey instrument to query NIH awardees about performance of three of four program
goals. A focus group of nine principal investigators (PIs) who had won Phase I and Phase II SBIR
awards provided feedback about the questionnaire, the most survey appropriate respondents, the response
burden, the best metrics for measuring successful award outcomes, and motivations for participating in
the survey.
Based on the focus group findings, NIH revised the survey instrument and then sponsored an online
pretest to further assess its potential. The online pretest and subsequent telephone debriefings with nine
PIs whose small businesses had won Phase II awards prompted further refinements to planned
approaches. The final survey questionnaire used in 2002 incorporated those changes and clarifications. It
also conformed to the Paperwork Reduction Act standards to minimize respondent burden. Appendix A,
Evaluation Framework, shows both the framework and a map of survey questionnaire items to the NIH
SBIR program objectives.
In 2002, the NIH implemented the first National Survey to Assess the Outcomes of the NIH SBIR
Program. This survey was a census of all small business awardees that had been awarded NIH SBIR
Page 1-2
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Phase II grants in FYs 1992-2001. Each awardee response described experience on one Phase II award.
For the 32 percent of the businesses that received more than one Phase II award during the study period,
the survey focus was a single award chosen through random selection. Due to vigorous follow-up effort to
maximize survey participation, 758 of the 905 eligible awardee companies surveyed completed the
questionnaire, yielding an 85 percent response rate.
Survey results indicated that NIH SBIR awardees stimulate technological innovation and contribute
significantly to the body of scientific knowledge related to health issues. In addition, NIH SBIRsupported technologies generate substantial commercial activity. Respondents judged the NIH SBIR
program to be quite valuable, allowing them to further develop their businesses and pursue high-risk
ideas. The primary study recommendation was to continue building on this systematic assessment by
maintaining a performance monitoring system to document NIH SBIR program progress toward
achieving performance outcome targets. 1
1.1.2
Subsequent Performance Monitoring and Evaluation
Since 2003, the NIH SBIR/STTR program has been developing a Performance Outcomes and Data
System (PODS) to respond to the 2002 survey recommendation about monitoring SBIR performance.
PODS affords authorized NIH personnel easy access via the NIH Intranet to SBIR awardee information,
the most recently reported data measuring award outcomes, and user-defined lists and tables of awardee
information and outcome measures.
PODS contains data from the 2002 survey and from subsequent voluntary updates by survey respondents.
Seventy-four percent of the original survey respondents—or 568 awardees—have participated in at least
one of five voluntary update opportunities since the 2002 survey. Considering that about 280 of the
original 758 awardee respondents had either discontinued their projects or were no longer in business by
2007, the response is especially impressive.
NIH relies on PODS updates to document the continued achievements of SBIR awardees (FYs 19922001) over time:
• The number of awardees with FDA-approved projects increased 51 percent, from 59 in 2002 to
89 in March 2007.
• The estimated cumulative sales to date achieved by awardees’ commercialized projects increased
over 200 percent, from $821 million in 2002 to $1.95 billion in March 2007.
• The number of awardees receiving additional non-SBIR funding or capital increased 33 percent,
from 281 in 2002 to 375 in March 2007.
1
“National Survey to Evaluate the NIH SBIR Program”; available from
http://grants.nih.gov/grants/funding/sbir_report_2003_07.pdf; Internet; accessed 10 December 2008.
Page 1-3
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Building on the results of NIH’s 2002 survey, the National Research Council (NRC), an arm of the National Academy of Sciences, expanded the NIH assessment to include all agencies that participated in the
SBIR program. 2 This evaluation included an online survey of a probability sample of FY 1992 through
2001 awardees that incorporated many of the same questions as the 2002 NIH instrument; a survey of
Phase I awardees; case studies of awardee companies; and interviews with NIH staff, grantees, and other
stakeholders. 3 The NRC survey conclusions mirrored those of the 2002 NIH survey. NRC commended
the NIH SBIR program, both for its meaningful achievements and also for its evaluation efforts. The
report further recommended that the SBIR program receive more funding for administration, evaluation,
and testing of new initiatives to enhance commercialization and program operations.
1.2
Methodology for the 2008 NIH Survey
By 2006, the NIH decided to sponsor a second national survey of Phase II SBIR awardees. The purpose
of this initiative was to describe the performance of a second group of awardees—those whose NIH SBIR
awards were made from FYs 2002-2006. Learning about the performance of these awardees would be a
logical follow on to the 2002 survey. It could enable program sponsors to add contact and performance
data about a new cohort to PODS. Doing so could improve the NIH SBIR program’s ability to assess
performance of more recent program activities.
The 2008 survey implementation mirrored that of the 2002 NIH survey implementation.
• Use the Evaluation Framework to develop the questionnaire to ensure that the data include
metrics for answering all study questions.
• Conduct a census of awardees rather than a survey, if the sample universe for the study period is
not too large.
• Update the contact information before fielding the survey to maximize respondent participation.
• Construct the survey to facilitate response and minimize respondent burden.
• Use information technology for implementing the survey online to reduce respondent burden.
• Maximize the response rate to the survey to minimize nonresponse bias.
• Clean and code the survey data to produce an accurate data file.
The following sections discuss the implementation of these survey principles and refer to sections in the
appendices that contain additional information.
2
Charles W. Wessner, ed. An Assessment of the Small Business Innovation Research Program. (Washington, D.C.: National
Research Council, 2007).
3
The NRC survey sampled individual awards rather than awardee companies. Of the 1,127 awards for which the PI was asked to
complete the questionnaire, 496 responded, resulting in a response rate of 44 percent.
Page 1-4
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
1.2.1
Questionnaire
Appendix B, Survey Instrument, contains a copy of the survey questionnaire. With the exception of very
minor revisions to the wording of several response categories and the addition of several options, the
2002 and 2008 survey questionnaires are the same. The minor modifications were based on responses to
several items in the 2002 survey for which more than a few respondents supplied the same specific
answers in the “Other” response category. These same responses became new categories in the 2008
survey questionnaire.
To encourage response, the questionnaire design enabled respondents to answer all items within 15 to 30
minutes, including any time needed to retrieve information or consult other sources. The survey
questionnaire was as precise as possible, used consistent terminology, and continually referenced the
randomly selected award that was the focus of the survey. The intent was to have respondents answer all
questions in terms of the “referenced SBIR award” and the “product, process, or service” planned and
developed under the supported “project.” Since selecting a response makes it easier for most respondents
to answer a question, the questionnaire tended to provide close-ended response categories. Sometimes it
was appropriate also to provide respondents with the opportunity to choose an “Other (please specify)”
open-ended response to encourage their answering a question when they did not wish to select a closedend response. In keeping with accepted guidelines for online surveys, 4 the questionnaire prompted
respondents to answer positively or negatively to each response for lists of optional answers.
1.2.2
Sampling
Like the 2002 NIH survey, the 2008 National Survey to Evaluate the NIH SBIR Program is a census of
all Phase II NIH SBIR awardee recipients, funded during the study period. This sample includes awards
from 23 NIH ICs, as shown in Exhibit 1-1 on the following page.
4
Don A. Dillman. Mail and Internet Surveys: The Tailored Design Method (New York: John Wiley and Sons, 2000), 398-399.
Page 1-5
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 1-1
NIH Institutes and Centers within the 2008 Survey Sample
Awarding NIH Institute or Center
Abbreviation
NCI
NCCAM
NCMHD
NCRR
NEI
NHGRI
NHLBI
NIA
NIAAA
NIAID
NIAMS
NIBIB
NICHD
NIDA
NIDCD
NIDCR
NIDDK
NIEHS
NIGMS
NIMH
NINDS
NINR
NLM
Title
National Cancer Institute
National Center for Complementary and Alternative Medicine
National Center on Minority Health and Health Disparities
National Center for Research Resources
National Eye Institute
National Human Genome Research Institute
National Heart, Lung, and Blood Institute
National Institute on Aging
National Institute on Alcohol Abuse and Alcoholism
National Institute of Allergy and Infectious Diseases Extramural Activities
National Institute of Arthritis and Musculoskeletal and Skin Diseases
National Institute of Biomedical Imaging and Bioengineering
National Institute of Child Health and Human Development
National Institute on Drug Abuse
National Institute on Deafness and Other Communication Disorders
National Institute of Dental and Craniofacial Research
National Institute of Diabetes and Digestive and Kidney Diseases
National Institute of Environmental Health Sciences
National Institute of General Medical Sciences
National Institute of Mental Health
National Institute of Neurological Disorders and Stroke
National Institute of Nursing Research
National Library of Medicine
Just as in 2002, awardee respondents answered survey questions about one randomly selected single
award. This sampling approach helped minimize respondent burden and provided a single, consistent
focus for respondents’ assessments.
The sampling approach uses these definitions:
• Unit of Study. The unit of study for the survey is the award within the awardee.
• Award. The award is a single NIH SBIR Phase II grant awarded from FYs 2002-2006.
• Awardee. The awardee is the small business that received the grant.
• Project. The project is the R&D undertaken and supported by the award.
• Respondents. The eligible survey respondents for each awardee, in order of desirability, are: (1)
the Principle Investigator (PI) on the SBIR award application, (2) the replacement PI, if the
original one was no longer employed by the awardee, and (3) the business official who signed for
the awardee company on the award application, if there was no replacement PI.
• Study Period. The study period included in this evaluation is FYs 2002-2006.
Page 1-6
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
• Stratification. There was no stratification of the sample by funding institute or IC (an initial plan
prior to the 2002 survey) because of the decision to conduct a census. Post-sample stratification
using various criteria, such as grouping the smaller ICs, is part of the data analysis.
• Usable Sample Units. Usable sample units are existing, nonduplicate awardee small businesses
located within the U.S.
• Eligible Respondents. Eligible respondents are employed by the awardee small business, reside
within the U.S., and are capable of interviewing in English.
1.2.3
Contact Database Update
NIH databases of contact information for awardees and respondents were from one to six years old, so a
contact database update was the initial phase of the survey implementation. The purpose was to: (1)
confirm that the small business still existed, (2) learn the status of the business and resolve any name
changes resulting from mergers and acquisitions, (3) update the business’s email address and telephone
number, (4) determine the eligible respondent, and (5) update the respondent’s business email address and
telephone number.
An initial contact email message addressed to the PI on the SBIR award application asked the PI to update
information about the status of the awardee business and its contact information using an online form. If
the contact information was correct, the PI could merely reply to the email message with the word
“COMPLETE” in the subject or body of the email reply.
For nonresponders to the update request, contact methods included emails, telephone calls, web searches,
and research using databases available through the NIH eRA Commons. Contact activities attempted to
reach each nonresponder and obtain: (1) updated email contact information for the awardee business, (2)
the name and email address for the eligible respondent, and (3) when appropriate, information that the
awardee business was no longer in existence (and could therefore be classified as unusable sample).
Appendix C, Email Messages and Telephone Follow-up Briefing, contains copies of the update email
message, the online response form, and all other communications used during survey fielding and followup with nonrespondents.
The result of the update phase was a database of 1,037 unique awardee small businesses with current (or
presumed current) contact information for the 89 percent (928) of the awardees deemed usable sample
units, inasmuch as they were still operating and located in the U.S. Nearly all of these small businesses
(918) could be deemed eligible sample. Chapter 2 presents details about analyses of usability and
eligibility aspects of the quality of the survey sample data.
Appendix D, Response Rate—Final Disposition of Sample, describes the final disposition of sample. For
additional information about the response rate, sample disposition, and online calculator, please see this
appendix. It also includes a final disposition of the sample.
Appendix E, Survey Sample, contains more detailed information about the SBIR awardee database, the
selection procedure, and possible limitations to the database. While limitations conceivably could result
from the random procedure used to select the single award for awardees with multiple awards,
comparison confirms that the randomly selected awards are indeed representative of the universe of
awards.
Page 1-7
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
1.2.4
Survey Implementation
Online access and use of the Internet are now common for the respondent population, so the survey was
administered online only. (The 2002 survey had offered online, mail, and telephone response options.)
Online implementation minimized respondent burden—for most respondents, it is easier and less time
consuming to click on the appropriate answer than to check it on paper, or to type in an open-ended
response than to write it out. Online surveys typically elicit quicker response than do paper ones, and
they reduce data entry errors and costs associated with key-entering data.
The automated survey process included security and confidentiality safeguards. The survey was
implemented using Secure Socket Layer (SSL) encryption technology. The access process required
respondents to input their user name and unique password before they could begin the survey.
Respondents could save their responses, leave the online survey, and return to the survey later to enter
more responses.
The online implementation allowed each respondent to complete only a single survey. Submitting the
survey stored the survey responses in a secure database.
All survey communications were sent using personalized email messages. Using email allows
prospective respondents to receive communications nearly instantaneously and to respond to the survey
merely by clicking the embedded link to the online implementation. The communications included the
following.
•
•
•
•
•
•
•
•
•
Initial update email to update contact information
Email verification message
Advance email about the upcoming survey
First cover letter email
Reminder/thank you email
Second cover letter email to nonresponders
Thank-you for promising to respond email
Incomplete email to those who started the online survey but did not submit it
Last call email to nonresponders from NIH SBIR/STTR Program Coordinator
All but the first two email messages included the survey link, the user name and password, and telephone
numbers to call for additional information about the survey or for assistance with any technical problems.
1.2.5
Survey Response
As in 2002, the 2008 survey had a goal of achieving participation from 80 percent of the usable and
eligible awardees. This is an extremely high response rate, which is typically achieved only when
respondents believe the survey has high interest and importance or when respondents receive other valued
incentives. Past NIH experience indicated that the 2008 survey’s awardee spokespersons would view this
survey as important and worthy of participation.
In addition to the multiple email communications itemized in Section 1.2.4, a professional telephone
center promoted survey response. Supervisors and interviewers at the center received training about the
background and goals of this assessment and how to administer a brief telephone script approved by NIH.
Page 1-8
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Interviewers made up to nine attempts (staggered across days and time periods) to contact all
nonresponders, update awardee and respondent contact information, and elicit a promise to complete the
online survey. After the telephone center elicited a promise to complete, all those who agreed to
complete an online survey received new email messages containing the survey link, user name, and password.
An online response rate calculator, accessible at any time by authorized NIH personnel, monitored the
survey response rate on a live, continuous basis. In addition to showing the current response rate, it
tracked the number of usable and eligible potential respondents, the number of completed surveys, and
the status of various incomplete and promised surveys.
The 2008 survey achieved a 78 percent response rate (719 respondents from the pool of 918 eligible
awardees). This rate is relatively high but not as high as the 85 percent rate achieved in the 2002 survey.
The lower response rate may be due to two factors. First, interviewers who contacted nonresponders were
not told to administer the survey on the telephone either on the spot or at a convenient appointment time
for the survey recipient. (In the 2002 survey, interviewers were told to administer the survey, and 11
percent of respondents completed telephone interviews.) Second, online surveys are no longer the
novelty that they were in 2002.
1.2.6
Data Cleaning and Coding
Minimal cleaning is required for data from an online survey because the logic used by the implementation
program controls the skipping of questions, allows input of only displayed response codes, and checks the
ranges of uncategorized numeric data. Online implementation minimizes errors and typos that can occur
from the manual input of data from paper questionnaires.
Two survey coding schemes facilitated data analysis:
• Coding to categorize the narrative responses to open-ended questions
• Construction of subgroups of survey participants to facilitate comparing responses among key
subgroups, using statistical criteria to identify differences 5
The number of respondents for an individual IC often was too small (i.e., less than 30 awardees) to allow
precise estimates of outcomes. Constructing two clusters of the small ICs made it possible to compare
these ICs’ performance in certain areas of interest to project sponsors. Assignment of ICs to one or the
other cluster depended on the type of products that were supported by the Phase II award. One cluster
comprised ICs where more than half of the sponsored projects’ product development primarily relied on
biological and physical sciences research (e.g., drugs, biologics, and genomics). The second cluster
included the remaining ICs. ICs in the second cluster were those whose projects were being developed by
behavioral, social science, or statistical research techniques (e.g., educational materials, psychological
5
The appropriate use of statistical tests requires that the data are robust—that is, that they do not violate the test’s assumptions.
Often, these assumptions involve having subgroups of sufficient size to avoid confounding normal sampling variability with
significant differences between the groups.
Page 1-9
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
assessments and interventions, and statistical software); ICs whose projects were fairly evenly distributed
among both types also were included in this cluster.
IC groups constructed for the 2002 and 2008 surveys are not comparable. One reason is that they use
different grouping schemes. While the 2008 groups are constructed on the basis of project type, the 2002
survey groups were constructed on the basis of number of awards made by the IC. Another difference is
that the 2008 survey describes performance of ICs that either did not sponsor SBIR awards in 2002
(National Center on Minority Health and Health Disparities and National Institute of Biomedical Imaging
and Bioengineering) or had no eligible respondents from ICs whose awardees responded to the 2002
survey (National Library of Medicine).
1.2.7
Analytic Strategies and Reporting Practices
The analysis compared responses for two categories of respondents.
• Subgroups of special interest to NIH. Subgroups of interest included fiscal year of award, IC
that sponsored the award, type of SBIR-supported product, and current status of the Phase II
project. For comparisons that involved two categorical variables, chi square tests determined if
there was a relationship between the two variables (e.g., type of product and current project
status). 6 For outcomes that were interval-level variables (such as number of publications) t-tests
and analyses of variance determined if differences between subgroups were statistically
significant.
• 2002 and 2008 survey populations. In reporting attainment of performance indices and
measures, the narrative describes performance of the entire 2002 and 2008 survey respondent
populations. Comparisons of other types of performance take account of the fact that the 2002
survey describes a ten-year award period (FYs 1992-2001), while the 2008 survey describes a
five-year award period (FYs 2002-2006). To construct more equivalent groups in terms of the
award period being described, these comparisons describe (1) only the 2002 survey participants
whose awards were made in the last five-year period, FYs 1997-2001 and (2) all 2008 survey
respondents.
These general practices are used in reporting the survey results.
• Descriptive statistics. In reporting descriptive statistics summaries for a major index and
associated measure in the Evaluation Framework, the values for means, medians, and standard
deviations are rounded to the nearest tenth. Data are summarized for each major index and
associated measure in the Evaluation Framework. For categorical variables, percentages are
rounded to the nearest percentage point; this may result in the percentages for all categories of a
variable not summing to 100 percent.
6
The chi-square test statistic only indicates that there is a statistically significant relationship between two variables; it does not
provide information about the nature of the relationship. To determine this, we used adjusted residuals, which compare the
frequencies that were observed with the frequencies that would be expected if the two variables were not related.
Page 1-10
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
• Key comparisons. In reporting key comparisons of respondent subgroups, results of the relevant
statistical tests are reported (1) in the notes to graphs and tables that present the result of
statistical analyses, and (2) in the text itself, when comparing other variables of interest.
1.3
Strengths and Limitations of the Evaluation
The 2008 survey collected information about a set of awardees that had not yet been surveyed—those
small business awardees that received SBIR Phase II grants from FYs 2002-2006. It generated a written
report and a standard data set. These products add to the NIH SBIR program’s body of knowledge for a
new set of awardees. They provide information that can facilitate understanding of program performance
over time.
One strength of the study described in this report is the relatively high 78 percent response rate. Another
positive aspect of the study is the fact that the 2002 and 2008 surveys were essentially identical, thus,
allowing comparison of the two studies and their results.
Certain limitations exist, of course. One major limitation is that all data on outcomes are based solely on
information reported by the awardees themselves. Another limitation is that it was not possible to make
valid comparisons of FYs 1997-2001 and FYs 2002-2006 outcomes for outcomes of interest that could
only be answered with data from two questions that were revised in 2008.
• Question 18 queried awardees about the current status of their SBIR-funded project. Ten percent
of 2008 respondents selected a response option, “on hold,” to describe the current status of their
SBIR-supported project. This option was not available to 2002 respondents. Because current
project status was used in conjunction with other questions about key outcomes (developing new
or improved products; contributing to increases in health knowledge, research tools, and health
promotion; and disseminating SBIR-supported technology and information to populations using
and receiving health and health care resources), comparisons of these outcomes between the two
cohorts of awardees were not possible.
• Question 23 asked for the dollar range of total cumulative sales through December 2007 for the
product developed under their Phase II award. The response option in the 2002 survey
(“$5,000,000-$49,999,999”) was divided into two separate categories in the 2008 survey
(“$5,000,000-$24,999,999” and “$25,000,000-$49,999,999”). This revision precluded making
comparisons of the two survey cohorts for estimated total and average cumulative sales in those
categories; estimates are derived using the midpoints of response categories, and the response
categories are now different for the two cohorts.
One other limitation is that small businesses that were liquidated, had merged with another company, or
had been acquired by another business were defined as unusable and thus not included in the survey
sample. If these events had indeed occurred for these unusable awardees, performance on some outcomes
(such as commercialization and sales) may have been even stronger than that described in this report.
Page 1-11
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
2. CHARACTERISTICS OF SURVEY RESPONDENTS
This chapter describes the characteristics of survey respondents. It presents information about the
usability and eligibility of the pool of potential respondents, awardees who received NIH SBIR grants
from FYs 2002-2006. Then it discusses characteristics of the respondent spokespersons who completed
the survey questionnaire, the awardee small businesses with which they are associated, and the products
whose development was supported by an NIH SBIR Phase II award from FYs 2002-2006.
2.1
Usability and Eligibility of Potential Respondent Small Businesses
NIH databases provided contact information about a pool of potential respondents—1,037 small
businesses that were awarded a Phase II SBIR grant from FYs 2002-2006. Approximately 89 percent of
the entire sample of potential respondents was deemed both usable and eligible.
Usability. Pre-survey searching strategies determined that 933 awardee businesses were potentially
usable for the survey because they were still operating and located in the U.S. One hundred and nine of
these awardees were unusable for one of these reasons:
• Awardee business reportedly had been liquidated or dissolved (22 or 2 percent)
• Awardee business reportedly had merged with or been acquired by other companies (45 or 4
percent)
• Valid contact information about awardee businesses was not available to the survey team, even
after extensive tracking efforts (37 or 4 percent)
An additional five awardees completed the survey questionnaire and were deemed unusable because they
reported having merged with or been bought by another company.
The usable sample for the 2008 survey was 928, or 89 percent of the total available sample of 1,037
awardees. 7 The geographic locations of usable and unusable awardees did not differ significantly.
Unusable awardee companies were significantly more likely to have been awarded a Phase II award in the
earlier years of the survey period than were usable awardees. Whereas the usability rates were between
81 and 82 percent for FYs 2002 and 2003, they ranged from 91 to 97 percent for FYs 2003 through 2006
(chi square statistic=45.6, df=4, p < 0.0001).
Usability rates differed somewhat among awardees of different Institutes and Centers (ICs). Between 94
and 98 percent of Phase II awardees funded by the National Center for Research Resources (NCRR),
National Institute on Aging (NIA), National Institute of Child and Human Development (NICHD), and
the National Institute of Digestive, Diabetic, and Kidney Diseases (NIDDK) were usable. The same was
7
Being purchased by or merging with another company can be regarded as a successful outcome for a Phase II awardee. The
decision to designate these companies as unusable is primarily based on the substantial difficulty in obtaining recent contact
information for both the company and the PI. Exclusion of these companies, however, always introduces the issue of potential
bias in the survey results. Given the high usability rate in the 2008 study, the degree of bias is most likely extremely small.
Page 2-1
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
true for the cluster of ICs that included the National Institute on Alcohol and Alcoholism (NIAAA), the
National Institute on Drug Abuse (NIDA), the National Institute of Nursing Research (NINR), and the
National Center for Minority Health Disparities (NCMHD).
The National Heart, Lung, and Blood Institute (NHLBI) and the National Institute on Allergy and
Infectious Diseases (NIAID) had the lowest usability rates (83 percent). For the remaining ICs, usability
rates fell between 87 and 91 percent. 8
Eligibility. Of the 928 usable awardee businesses, nearly all (918) were determined to be eligible. The
ten companies that were ineligible were so designated because the Principal Investigator (PI) was
unavailable (either due to death or departure from the company), and no other individual with adequate
knowledge to complete the questionnaire could be identified. 9
2.2
Spokesperson Characteristics
Exhibit 2-1 on the next page displays selected characteristics of individual spokesperson respondents who
completed the survey. Approximately 80 percent of the individuals who completed the 2008
questionnaire were the original PIs listed on the Phase II SBIR award application. Another 11 percent
were individuals who later had been designated as the replacement PI, typically as a result of the initial PI
leaving the company. As such, a total of 91 percent of respondents were directly responsible for the work
under the award at some point during the award period. The remaining 9 percent of respondents were
either the business official listed on the SBIR application or another contact at the company; these
individuals agreed to complete the survey because the original PI could not be located.
Survey respondents often performed many roles in the awardee business. Approximately 36 percent
functioned in only one capacity—as owners (13 percent), part of the managerial team (13 percent),
employees (9 percent), and, in a few cases, shareholders (3 percent). The largest group of respondents
(40 percent) reported serving in multiple roles whereby they not only owned the company but also served
in a managerial role and were shareholders. Substantially smaller percentages held two roles.
8
See Appendix D, Response Rate—Final Disposition of Sample.
9
Statistical comparisons were not performed, given the small number of ineligible respondents.
Page 2-2
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-1
Characteristics of Survey Respondents
Characteristic
Percent
N
Respondent role
Initial PI
574
80
Replacement PI
82
11
Business official
33
5
Other company contact
27
4
Owner only
90
13
Owner and manager
27
4
9
1
281
40
Management only
89
13
Manager and shareholder
90
13
Employee only
61
9
Employee and shareholder
20
3
Shareholder only
19
3
Other (e.g., former employee)
15
2
Relationship to awardee company
Owner
Owner and shareholder
Owner, manager, and shareholder
Part of management
Employee
Note. Information on respondent role was provided by 716 awardees; the corresponding
figure for relationship to the awardee company was 701. The percentages for respondent
roles are column percentages and may not sum to 100 percent due to rounding.
Most respondents were either the original PI or a replacement PI. Respondents tended to
report multiple relationships within the awardee small business; most typical were owner, and
part of management.
2.3
Awardee Small Businesses
This section compares characteristics of awardee small businesses that were eligible for the survey. It
then provides additional detail about the characteristics of eligible awardees who responded.
2.3.1
Nonrespondent and Respondent Awardee Comparison
Approximately 78 percent of eligible respondent awardee companies participated in the 2008 survey.
Exhibit 2-2 on page 2-4 shows fiscal year of award, sponsoring IC, and region of the country for
respondent and nonrespondent awardees.
Page 2-3
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-2
Selected Characteristics of Nonrespondent and Respondent
SBIR Awardee Small Businesses
Characteristic
Nonrespondents
(N = 199)
Respondents
(N = 719)
N
%
N
%
2002
45
23
122
17
2003
26
13
131
18
2004
38
19
135
19
2005
42
21
130
18
2006
48
24
201
28
40
20
154
21
36
18
110
15
NIA
7
4
40
6
NIAAA, NIDA, NINR, and NCMHD
5
3
33
5
NIAID
21
11
59
8
NICHD
14
7
47
7
NIDDK
11
6
48
7
NIGMS
Fiscal Year of Award
Sponsoring IC
NBIB, NCRR, NHGRI, NIAMS, NIDCD,
NIDCR, NIEHS, NCCAM, and NLM
NCI
18
9
59
8
NIMH
9
5
33
5
NINDS
12
6
40
6
NHLBI
26
13
96
13
Midwest
31
16
114
16
Northeast
60
30
203
28
South
35
18
174
24
West
73
37
228
32
U.S. Region
Note. The percentages are column percentages and may not sum to 100 percent due to rounding. The full titles of the
sponsoring ICs are shown in Exhibit 1-1 on page 1-6.
Non-respondents and respondents were similar in terms of the IC that sponsored the Phase II award, the fiscal year of
the award, and the geographic region in which the small business was located.
As shown by Exhibit 2-2, over one-fourth of the 719 respondent awardees received their Phase II award
in FY 2006. Between 17 and 19 percent of the companies each received their award in one of the earlier
years included in the evaluation (FYs 2002-2006).
The NCI supported the largest number of awards during this five-year period, funding 110 companies or
15 percent of all respondents. The NHLBI supported the second largest number, making Phase II awards
Page 2-4
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
to 96 companies (13 percent). Six ICs each funded another 6 to 8 percent of the companies that
responded to the survey; these included NIAID, National Institute of General Medical Sciences (NIGMS),
NIDDK, NICHD, NIA, and National Institute of Neurological Disorders and Stroke (NINDS). The
remaining awardee companies were funded by other ICs.
The largest percentage of respondent companies (32 percent) was located in the West, and the smallest
percentage (16 percent) in the Midwest. Twenty-eight percent were in the Northeast, and the remaining
24 percent were in the South.
Respondents and nonrespondents were reasonably similar in terms of which IC sponsored the award.
They also were similar in terms of the year in which the award was made and region of the country. This
suggests that the threat of nonresponse bias is very small or nonexistent for survey responses that are
strongly related to one or more of these three characteristics.
2.3.2
Awardee Small Business Respondents
Exhibit 2-3 shows a detailed picture of the geographical distribution of responding awardee small
businesses, identifying the number of respondents by state. California by far had the heaviest
concentration of respondents (132), followed by Massachusetts (72) and Maryland (51). Twenty-five
percent of the states had between one and two awardees, and two states (Alaska and West Virginia)
received no Phase II awards from FYs 2002-2006 and thus had no awardees that participated in the
survey.
EXHIBIT 2-3
Awardee Small Business Respondents by State
0
1
The largest numbers of awardee respondents were in California, Massachusetts, and Maryland. Five states had
only one respondent. Alaska and West Virginia received no awards during FYs 2002-2006 and therefore had no
respondents to the 2008 survey.
Page 2-5
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Twenty-four percent of awardee small businesses were founded in 2000 or later, and more than one-third
(37 percent) began operations between 1995 and 1999. Sixteen percent were founded during the early
1990s, 12 percent between 1985 and 1989, and 7 percent between 1980 and 1984. Only a small
percentage (5 percent) were established prior to 1980.
Looking at this another way, a significant percentage of awardee companies were starting operations at
the time of the award. Exhibit 2-4 shows the distribution of awardee companies by years in operation at
time of award. Awardee respondents to the 2008 survey reported a range of years in operation at the time
of award from 1-2 to 26 or more years.
EXHIBIT 2-4
Years in Operation at the Time of the SBIR Award
for Responding Small Businesses
Percent of awardee companies
20
15
10
5
0
1-2
4
6
8
10
12
14
16
18
Years in operation at tim e of aw ard
20
22
24
26 or
m ore
Note. The total number of awardee businesses that provided this information was 711. Percentages may not
sum to 100 percent due to rounding.
Phase II awardee businesses tended to be “young” in terms of how long they had been in operation at the time of
award. Approximately 30 percent had been operating for five years or less.
Overall, 54 percent of awardee businesses had been in operation for eight years or less at the time of the
Phase II award.
There were no significant differences among awardees in the average age when examined by fiscal year
of the award or type of product. On the other hand, significant differences did appear for different types
of awardee businesses. Biotechnology, pharmaceutical, diagnostic, medical devices, and computer
software/hardware companies were significantly likely to be “younger” than either instrumentation
companies or companies in the other category (i.e., healthcare, engineering, chemical, and environmental
businesses).
Page 2-6
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Company age at the time of the award was similar among the ICs, with one exception. Small businesses
that were supported by the NIAAA and NIDA had been in operation significantly longer than those
supported by other ICs. This is primarily a function of the fact that a larger percentage of NIAAA and
NIDA awards were in businesses that specialized in medical education and health promotion—a group
that also tended to have been in existence longer (mean of 11.1 years at the time of the award) than other
types of businesses.
When asked about their major field of business, the 612 companies that responded to this question represented a varied group of companies (see Exhibit 2-5). Two areas were predominant—biotechnology (24
percent) and medical devices (21 percent). Between five and nine percent of all businesses were in each
of six fields, including medical education and health promotion, informatics, research, and statistics,
pharmaceuticals, diagnostics, computer hardware and software (HW/SW), and instrumentation. Much
smaller percentages were in as healthcare, chemicals, and environmental fields.
Percent of awardee companies
EXHIBIT 2-5
Major Field of Business Reported by SBIR-Funded Awardees
for FYs 2002-2006
40
35
30
25
24
21
20
15
10
9
9
8
5
7
6
5
3
2
1
4
1
B
io
te
ch
M
no
ed
lo
ic
gy
al
de
In
vi
st
ce
ru
s
m
en
Ph
t
at
ar
io
m
n
ac
eu
t ic
al
D
s
ia
C
g
M
om
no
ed
In
f
s
ic
pu
or
tic
al
m
te
s
ed
at
rH
ic
uc
W
s
at
/S
an
io
W
d
n/
re
he
s
ea
al
th
rc
pr
h
om
ot
io
En
n
gi
ne
er
in
H
g
ea
lth
ca
re
Ch
em
En
ic
al
vi
s
ro
nm
en
ta
l
O
th
er
0
Major field of business
Note. The total number of awardee businesses that provided this information was 612. Percentages
may not sum to 100 percent due to rounding.
By far, biotechnology and medical devices were the most common major fields of business for Phase II
SBIR-funded awardee respondents.
Page 2-7
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Of the 719 awardee companies that completed the survey, 107 (15 percent) did not report their major field
of business. Including the missing responses in the analysis, did not noticeably change the percentages or
relative ranking of the different major fields, although the amount of missing data suggests caution in
interpreting these percentages. It is not clear why 15 percent of respondents failed to answer the question
about major field of business. Because of the amount of missing data, this variable is not used in analyses
of responses to the three key study questions.
2.4
SBIR-Supported Projects and Products
The SBIR program supported a diverse group of projects and their product-related activities. 10 The
majority (70 percent) of the projects involved the development of a totally new product, and one-fifth (20
percent) focused on improving an existing product. Another 7 percent sought to develop a combination
of new products, and 3 percent intended to create a new use for an existing product. These percentages
did not differ by the year of the award or the sponsoring IC. They also were nearly identical to those
found in the 2002 survey.
As shown by Exhibit 2-6 on the next page, when asked about the specific type of product that was the
focus of the award, 22 percent of the awardees reported having received SBIR funding to develop medical
devices. Computer hardware and software, research tools, diagnostic materials and devices, and drugs
each comprised between 12 and 14 percent of the projects.
Educational materials were identified by approximately 7 percent of the small businesses, as were
measurement tools. Noticeably smaller percentages (2 to 3 percent) of the companies funded work was in
genomics, non-drug therapeutics, environmental tools, and chemicals and chemical processes.
10
For the ease of the reader, henceforth we use the term “product” throughout this report to represent the products, processes, and
services that were planned by awardees.
Page 2-8
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-6
Type of Planned Product for SBIR Funded Awardees
Percent of awardee companies
35
30
25
22
20
14
15
13
12
12
10
7
5
7
5
3
2
2
2
0
c
l
g
ic
al
ls
ol
ic
gi
ce
ru
ol
oo
ic
to
ut
lo
ria
D
vi
om
to
lt
e
m
e
o
e
h
n
i
a
t
t
e
t
e
a
B
ap
rc
l/ d
en
G
al
ch
en
er
ea
te
m
lm
ic
ria
g
h
m
s
e
a
u
t
e
d
u
e
n
r
n
p
e
at
R
dr
ro
io
M
ug
su
m
om
nvi
at
dr
ea
o
C
n
c
it c
N
E
M
u
s
on
N
Ed
no
g
ia
D
ce
vi
e
d
/S
W
H
r
W
Type of product, process, or service
Note. These percentages are based on a total of 716 awardees who provided information on type of product.
Percentages may not sum to 100 percent due to rounding. In addition, a very small percentage (0.1 percent) of
the awards planned a product other than the ones identified.
Phase II awards sponsored a diverse group of products. The most common were medical devices. Five types
of products—medical devices, computer hardware and software, research tools, diagnostic materials/devices,
and drugs—accounted for nearly three-quarters (73 percent) of all planned products.
Among the different types of planned products, there were differences in whether the project sought to
develop one or more new products or improve an existing product. As Exhibit 2-7 shows (on the next
page) projects that focused on drugs most often described their goal as developing a “totally new” product
(90 percent). This also was the case for companies working on biologics (77 percent), educational
materials (73 percent), and medical devices (71 percent). In fact, regardless of the type of technological
innovation, at least 50 percent or more of the companies labeled their product as “totally new,” with one
exception. Companies whose award was for a non-drug chemical or chemical process were more likely
to see their intended product as an improvement to an already existing product.
Page 2-9
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-7
Technological Innovations by Type of Planned Product
Drug
Medical device
Biologic
Type of product
Genomic
Diagnostic
Non-drug therapeutic
Research tool
Measurement tool
Educational materials
Computer HW/SW
Environmental, ergonomic, or assistive tool
Chemical or chemical process
0
10
20
30
40
50
60
70
80
90
100
Percent of awardee companies
Totally new product
Improvement to an existing product
Combination of products or other
New use for an existing product
Note. The total number of awardee businesses that provided this information was 715. For each type of product,
the percentages may not sum to 100 percent due to rounding. The relationship between the type of technological
innovation and the planned product was statistically significant (chi-square statistic=69.8, df = 33, p < 0.0002).
Most product types were described as “totally new” at least 50 percent of the time. The one exception was
products involving chemicals or chemical processes, which were most frequently described (46 percent) as
supporting improvements to an existing product.
Overall, 41 percent of the companies reported that their SBIR-funded product was currently in the
development phase at the time of the survey. Another 20 percent indicated that it was being
commercialized, and 22 percent stated that it was being used by the target population(s). Ten percent said
that the product was “on hold,” awaiting the receipt of additional funding or inactive for another reason.
Eight percent of the companies had actually discontinued work on the project. (Numbers total more than
100 percent due to rounding).
Not surprisingly, the time that had elapsed since initial receipt of SBIR Phase II funding was significantly
related to the current status of the project (see Exhibit 2-8 on the next page). Projects funded in FY 2006,
having been underway for only about two years, were most likely to still be in the development stage (58
percent) and least likely to be discontinued (1 percent). However, only 18 percent of the projects funded
in FY 2002, six years after the start of the award, were still under development, and 14 percent had been
discontinued. Similarly, the percentage of projects whose products were currently in use by the target
Page 2-10
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
population was reasonably small for FY 2006 awards (12 percent). This figure tripled for projects that
were first funded in FY 2002 (36 percent).
EXHIBIT 2-8
Current Project Status by Fiscal Year of Award
100
P ercent of projects
80
58
60
43
40
20
36
38
35
18 19
18
13 14
27
24
21
17
13
9
21 22
15
12
11
8
6
4
1
0
2002
2003
2004
2005
2006
Fiscal year of award
Under development
Being commercialized
In use by target population
On hold
Discontinued
Note. The total N of respondents was 716. For each fiscal year of the award, percentages may not sum to 100 percent due
to rounding. Fiscal year of the award and current project status were significantly related (chi-square statistic=99.4, df =16,
p < 0.001).
The longer the elapsed time since the start date of the award, the more likely it was that projects were under development.
Projects with relatively longer elapsed time, since the start date of the award were more likely to be in use by the target
population(s) than were projects funded in FY 2006.
Another factor affecting project status was the type of product being developed. As shown in Exhibit 2-9,
on the next page, awardees reported that the status of their SBIR product was either (1) under
development, (2) in the commercialization stage, (3) in use by the target population; (4) “on hold,” or (5)
discontinued. Environmental, ergonomic, and assistive tools, along with genomics, were significantly
more likely than other product types to be in use at the time of the survey. Drugs and medical devices
were significantly more likely to be under development.
Educational materials were most likely to be in the commercialization stage (41 percent) at the time of the
survey. Drugs (4 percent) and chemicals (8 percent) were least likely to be in the commercialization
stage. Chemical products were said to have been discontinued more often (25 percent) than other types of
products.
Page 2-11
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
100
80
71
65
51
60
40
20
0
Percent of projects
EXHIBIT 2-9
Current Project Status by Type of Planned Product
4
0
16
9
Drugs (n=77)
50
40
20
15
18
8 7
Medical devices
(n=160)
5
20 20
13
0
Biologics (n=40)
43
43
29
14 15 14
13
7
Genomics (n=15)
14
8
Diagnostics (n=86)
29
14
7 7
Non-drug
therapeutics (n=14)
7 7
Research tools
(n=97)
Type of planned product
Percent of projects
Under development
In commercialization stage
In use by the target population(s)
"On hold"
Discontinued
100
80
60
40
20
41
26 26 24
16
8
33
27 23 32
16
4
8
11
6
17
25
33
31
8
17
8
8
17
25
0
Measurement tools
(n=50)
Educational materials
(n=51)
Computer HW/SW
(n=101)
Environmental,
ergonomic, or
assistive tools (n=12)
Chemicals and
chemical processes
(n=12)
Type of planned product
Under development
In commercialization stage
In use by the target population(s)
"On hold"
Discontinued
Note. A total of 715 companies provided this information. For each type of product, percentages may not sum to 100 percent due to rounding. Current project status
and type of planned product were significantly associated (chi-square statistic=145, df = 27, p < 0.0001).
The type of product was significantly related to current project status. Environmental, ergonomic, and assistive tools, along with genomics, were significantly more likely
to be already in use at the time of the survey. Drugs and medical devices were significantly more likely to be under development.
Page 2-12
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Finally, differences in current project status surfaced, depending on the IC that sponsored the Phase II
award (see Exhibit 2-10 on the next page). The National Institute of Mental Health (NIMH), the NICHD,
and the cluster of ICs that included the NIAAA, the National Institute on Drug Abuse (NIDA), NINR,
and NCMHD were the most likely to have their products in use at the time of the survey. Nearly onethird of these ICs’ awardees reported that their SBIR-funded project had been completed and their
product made available to the respective target population(s). These percentages were significantly larger
than were those for projects sponsored by the NCI, NHLBI, NIAID, NIGMS, and NIDDK, which ranged
from 14 to 17 percent. Such differences might be expected: the ICs with lower percentages are more
likely to support projects to develop drugs and medical devices—two product types that take the longest
to reach the market because of such factors as the need for FDA approval.
The discontinued rates for Phase II projects were typically between 8 and 10 percent. A significantly
lower rate occurred for the NIGMS when compared with that for all other ICs, excluding the NIMH,
combined (chi-square statistic=5.33, df=1, p < 0.03). Although the NIMH also experienced a visibly
lower rate (3 percent), this difference was not significantly different.
Although the percentage discontinued SBIR-funded projects was only 8 percent overall, it is useful to
explore the extent to which these projects shared any distinctive characteristics from those of projects that
were still ongoing. Exhibit 2-11 on page 2-15 shows the various reasons for abandoning work on the
project for the 54 discontinued projects. As shown, 59 percent of the awardees reported that there was
insufficient funding for the planned work. The second most frequent reason cited by 44 percent of
awardees was that the business itself had shifted priorities. Between 24 and 30 percent cited problems
involving too high a level of risk, the non-competitive nature of the product, and a market demand that
was too small. Smaller percentages believed that the project was abandoned at least in part because of the
PI’s departure from the company, the inability to obtain FDA approval, or that products were licensed to
another company.
Page 2-13
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-10
Current Project Status by NIH IC Sponsor
Current Project Status
NIH IC Sponsor
Total
N
Under
development
%
N
Under
commercialization
%
N
In use
N
%
“On hold”
%
N
Discontinued
%
N
NBIB, NCRR,
NHGRI, NIAMS,
NIDCD, NIDCR,
NIEHS, NCCAM
and NLM
154
59
38
34
22
34
22
16
10
11
7
NCI
109
36
33
29
27
17
16
16
15
11
10
NHLBI
95
53
56
15
16
13
14
6
6
8
8
NIAID
59
32
54
3
5
10
17
9
15
5
8
NIGMS
58
20
35
18
31
14
24
6
10
0
0
NIDDK
48
22
46
10
21
8
17
4
8
4
8
NICHD
47
15
32
9
19
15
32
4
9
4
9
NIA
40
17
43
8
20
10
25
2
5
3
8
NINDS
40
16
40
4
10
11
28
6
15
3
8
NIMH
33
9
27
9
27
11
33
3
9
1
3
NIAAA, NIDA,
NINR, and
NCMHD
33
11
33
4
12
11
33
3
9
4
12
Note. The total number of respondents was 716. Percentages are row percentages and may not sum to 100 percent due to rounding. Sponsoring
IC and project status (under development, under commercialization, in use, and “on hold” or discontinued) were significantly associated (chi-square
statistic= 57.6, df = 33, p < 0.005). The full titles of the sponsoring ICs are shown in Exhibit 1-1 on page 1-6.
There were significant differences in the current project status of awardee companies, depending on the IC that funded the project. The NIMH, the
NICHD, and the cluster of ICs that included the NIAAA, NIDA, NINR, and NCMHD were the most likely to have their products in use at the time of
the survey. Compared to all other ICs, the NIGMS, NIMH, and NIA were significantly less likely to have any discontinued or “on hold” projects.
Page 2-14
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-11
Reasons for Discontinued Projects
Percent of discontinued projects
100
80
59
60
44
40
30
26
24
24
17
20
13
9
7
th
er
Li
ce
ns
ed
to
N
an
ot
o
FD
A
O
ap
pr
ov
he
al
rc
om
pa
ny
fa
ile
d
ef
t
Id
ea
PI
l
In
su
C
ffi
om
ci
en
pa
tf
ny
un
sh
di
ift
ng
ed
pr
Le
io
ve
ri
Pr
lo
tie
od
fr
s
uc
is
k
tw
to
as
M
o
ar
hi
n
ke
ot
gh
td
co
em
m
pe
an
tit
d
iv
w
e
as
to
o
sm
al
l
0
Reason for discontinuing project
Note. These percentages are based on a total of 54 respondents who provided reasons as to why their SBIR-funded
project was discontinued. Percentages do not sum to 100 percent, given that individuals could indicate multiple
reasons.
The most common reported reason for discontinuing work on an SBIR-funded project was insufficient funding (59
percent). A shift in the priorities of the company also was mentioned as a contributing factor by 44 percent of
respondents). Thirty percent indicated that the level of risk was too high and about one-quarter (24 and 26 percent)
believed the demand was too small or the product was not competitive.
Exhibit 2-12, on the next page shows that the relationship between product characteristics and
discontinuing work was only marginally significant (p < 0.06). Whereas 7 percent of companies
developing new products reported discontinuing work on the SBIR-funded project, this was true of 11
percent of companies that were working on improvements to or new uses for existing products.
The decision to discontinue a project was significantly related to receipt of additional SBIR awards. As
shown by Exhibit 2-12, companies that had no related SBIR awards were nearly twice as likely as those
that had competed successfully for these awards to have discontinued their project (10 versus 6 percent).
Page 2-15
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 2-12
Discontinued Projects by Product Characteristic
and Awardee’s Receipt of Additional SBIR Awards
Project was discontinued
Yes
No
%
N
N
%
Product characteristic
Developing one or more new products
36
7
510
93
Improving or finding new use for an existing product
18
11
146
89
Yes
23
6
385
94
No
30
10
261
90
Receipt of additional SBIR awards
Note. A total of 710 awardees provided information on both current project status and product characteristic.
The corresponding number for receipt of additional SBIR awards and project status was 699. Percentages are
row percentages. The relationship between product characteristic and project status was only marginally
significant (chi-square statistic=3.45, df=1, p < 0.06) whereas the relationship between receipt of additional
SBIR awards and project status was statistically significant (chi-square statistic=5.29, df=1, p < 0.02).
Companies that received additional SBIR Phase I or Phase II awards were significantly less likely to have
abandoned work on their Phase II project then were small businesses that had not competed successfully for
such additional awards.
Seldom did awardees attribute only one reason to their decision to abandon work on their product. Only
24 percent of this group cited only one factor, 33 percent identified two reasons, and 43 percent listed
three or more motivations. Certain rationales also tended to occur jointly. The belief that a product was
not sufficiently competitive was significantly related to also viewing the level of risk as too high (the
correlation between these two variables was 0.36, p < 0.008). Having insufficient funding also was
significantly correlated with the departure of the PI (the correlation was 0.27, p < 0.04). In both cases,
although these correlations were statistically significant, they were fairly modest in size.
Page 2-16
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
3. STIMULATING TECHNOLOGICAL INNOVATION
This chapter presents findings about attainment of the first NIH SBIR program goal, stimulating
technological innovation. It begins with a summary of findings about goal attainment. Then it presents
analyses of each related index and overall performance of SBIR awardees over time.
3.1
NIH SBIR Awardees Stimulated Technological Innovation
Exhibit 3-1 summarizes the survey findings about awardee outcomes for stimulating technological
innovation. Phase II awardees reported exceeding specified levels for both indices in the Evaluation
Framework—developing products in support of the NIH mission and receiving additional awards that
relate to the core technology.
EXHIBIT 3-1
Summary of Outcomes Associated with Stimulating Technological Innovation
Goal 1
Stimulate technological innovation
Index 1.1
Regardless of sales, 40 percent or more of
NIH SBIR awardees develop products in
support of the NIH mission
Measure 1.1.1
82 percent of the SBIR-supported
projects that responded produce new
or improved products in health-related
fields
Index 1.2
10 percent of NIH SBIR awardees receive
additional Phase I or Phase II awards that
relate to the core technology
Measure 1.2.1
57 percent of the responding awardees have
received additional Phase I or II awards that
are related to the core technology supported
by their Phase II award. This includes 1,615
awards.
Measure 1.1.2
53 percent of the SBIR supported
projects that responded have
published one or more technical
articles and have produced 1,397
technical publications
Measure 1.1.3
31 percent of the SBIR supported
projects that responded have been
granted one or more patents for a total
of 586 patents.
The next sections describe awardees’ reported performance in attaining each performance index
benchmark at the time of survey measurement.
Page 3-1
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
3.2
Performance on NIH SBIR Product Development Targets
Survey respondents exceeded the NIH SBIR performance target expressed by Index 1.1, which is
“Regardless of sales, 40 percent or more of NIH SBIR awardees develop products in support of the NIH
mission.” This section describes performance on the three related measures from the NIH SBIR program
Evaluation Framework.
3.2.1
New or Improved Products in Health-Related Fields
Measure 1.1.1 describes the number of SBIR-supported projects that produce new or improved products
in health-related fields, which is one indicator of alignment with the NIH mission. The number of
respondents who said they are developing and commercializing new or improved products in healthrelated fields is 587 (82 percent of 719 respondents). These included 290 projects with products under
development, 143 projects in the midst of the commercialization stage, and 154 projects that have made
their products available for commercial use.
3.2.2
Technical Articles and Related Activities in Support of the NIH Mission
Measure 1.1.2 describes the number of technical articles generated by NIH SBIR projects. This measure
provides important information about mission support because progress toward stimulating technological
innovation is accomplished when contributions are made to the body of scientific and technological
knowledge. These contributions include publishing articles in peer-reviewed scientific journals and
authoring or co-authoring articles that appear in industry or technical publications.
In addition to querying survey respondents about publications, the questionnaire also asked about other
indicators of contribution to the body of scientific and technological knowledge in support of the NIH
mission: conference presentations and honorary awards. Conference presentations can be precursors to
formal publications; they promote public awareness of the SBIR-supported product, and in turn increase
the product’s visibility and possibly its quality. Visibility and quality benefits also can accrue when
project personnel receive honorary awards that relate to their SBIR project activities.
As shown by Exhibit 3-2 (on the next page), slightly more than half (53 percent) of awardee respondents
indicated that they had generated one or more publications that were associated with the project, and nearly
three-quarters (73 percent) made one or more conference presentations. Thirteen percent received at least one
award.
Page 3-2
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 3-2
Summary of Presentations, Publications, and Awards
Associated with SBIR-Supported Projects
Conference
Presentations
Publications
Awards
Awardee companies
Number
519
375
95
Percent
73
53
13
Total
2,771
1,397
148
Mean
0.9
2.0
0.2
Standard deviation
7.4
4.9
0.7
Median
2.0
1.0
0.0
Items
Note. The total number of awardees who responded to survey queries about these
items was 709.
Nearly three-quarters (73 percent) of all awardees reported giving one or more
conference presentations. Fifty-three percent reported producing one or more
publications. Approximately 13 percent reported receiving at least one award.
The NIH SBIR program awards generated 1,397 publications, 2,771 conference presentations, and 148
awards. Median numbers per reporting awardee were 1, 2, and 0, respectively.
Whether awardees reported publications, presentations, and awards related to the SBIR-funded project
was significantly related to the current status of the project. Exhibit 3-3 on the following page shows that
larger percentages of companies whose products were being commercialized or in use by the target
population(s) indicated having one or more publications, presentations, and awards. Fifty-eight percent of
awardees who were in the midst of commercializing their product had published at least one technical
article. Sixty-nine percent of awardees whose product was commercially available and in use published
at least one technical article. The corresponding percentage was 46 percent for awardees who were still
involved in product development.
Regardless of project status, a sizable percentage of SBIR projects make contributions to the scientific
knowledge base. Noticeable levels of activity in authoring technical articles occurred even for projects
that were “on hold” or discontinued (45 and 42 percent, respectively).
Page 3-3
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Percent of awardee companies
EXHIBIT 3-3
Awardees with Presentations, Publications, and Awards
by the Current Status of their SBIR-Supported Project
100
82
84
80
70
69
63
58
55
60
46
45
42
40
18
21
20
11
6
6
0
Presentations
Publications
Awards
Type of contribution
Under developm ent
Com m ercialization stage
On hold
Discontinued
In use by target population(s)
Note. Current project status was significantly related to all three ways of contributing to the knowledge base. For
publications, the chi-square was 26.3 (df = 4, p < 0.001). For presentations, the chi-square statistic was 28.8 (df
= 4, p < 0.0001), and for awards, it was 19.1 (df = 4, p < 0.0007). The total number of awardees who responded
to survey queries about these outcomes was 709.
Awardees that were commercializing their product or had made it already available were significantly more likely
to have published one or more technical articles and to have received one or more awards than were awardees
whose projects were “on hold” or discontinued. Awardees whose projects were “on hold” or discontinued had
published at least one technical article (45 and 42 percent, respectively). Even higher percentages of these more
“inactive” projects reported making one or more conference presentations.
As also shown by Exhibit 3-3, a pattern similar to that observed for publications occurred with regard to
conference presentations and awards. The strongest performance was exhibited by awardees who had moved
past the product development stage and had ongoing or completed projects.
Overall, when compared to publications, the percent of awardees with at least one product-related conference
presentation was noticeably higher, and the percent with one or more awards was dramatically smaller,
regardless of the current status of the SBIR-funded project.
The elapsed time between the receipt of SBIR Phase II funds was significantly related to publications (chisquare statistic = 27.2, df = 4, p < 0.0001). Essentially, the more time that had elapsed, the more likely it was
for a company to have published one or more scientific articles. Whereas 68 percent of FY 2002 awardees
indicated one or more publications, the corresponding percentages for FYs 2003-2006 awardees were 60, 57,
44, and 42 percent, respectively. In contrast, there was no relationship between when the small business
received the SBIR Phase II award and whether it had since presented at a conference or received an honorary
award that was project-related.
Page 3-4
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Having one or more publications, presentations, or awards was not significantly related to the
Institute/Center (IC) groupings. Conference presentations and awards were not associated with the type
of product being sponsored by the Phase II award.
Publications and type of product were related. Exhibit 3-4 shows awardees that were working on non-drug
therapeutics were the most likely to have one or more publications (86 percent). Between 61 and 66 percent of
companies working on measurement and assessment tools, diagnostic materials and devices, and biologics,
and 53 to 58 percent of awardees developing new or improved drugs, genomics, research tools, and
environmental tools reported that they had produced one or more publications. Nearly half of the awardees
receiving SBIR support for medical devices, software and hardware, and chemical products reported having
one or more publications. Awardees planning on creating educational materials were the least likely to have
published, with 35 percent reporting having one or more publications related to the award.
EXHIBIT 3-4
Awardees with Publications, Presentations, and Awards
by Type of SBIR-Supported Product
0
10
20
30
Non-drug therapeutics (n=14)
Type of product
80
90
66
66
62
85
61
73
15
58
83
8
56
Drugs (n=75)
11
68
55
Research tools (n=96)
100
86
86
15
Diagnostics (n=86)
82
12
53
87
7
47
Medical devices (n=157)
19
46
Chem icals (n=13)
Softw are or hardw are (n=101)
70
10
Biologics (n=39)
Genom ics (n=15)
Percent
50
60
29
Measurem ent tools (n=50)
Environm ental tools (n=12)
40
31
46
10
67
69
69
35
Educational m aterials (n=51)
80
18
Publications
Presentations
Aw ards
Note. Type of product and publication activity were significantly related (chi-square statistic=24.1, df = 11, p < 0.01). For
presentations, the relationship with product type was only marginally significant (chi-square statistic=17.6, df = 11, p < 0.09). The
total number of awardees who responded to survey queries about these outcomes was 709.
Awardees with projects developing new or improving existing non-drug therapeutics were significantly more likely to have
published one or more technical articles that were related to their Phase II project than were awardees working on other types of
products. Awardees developing non-drug therapeutic products, chemicals or chemical processes also were significantly more
likely to have received recognition in terms of an award. Regardless of the type of product, large majorities of awardees (between
66 and 87 percent) had presented at one or more scientific conferences.
Page 3-5
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
3.2.3
Patents and Other Indicators of Stimulating Technological Innovation
Measure 1.1.3 quantifies patents, another indicator of the degree to which NIH SBIR projects stimulate
technological innovation in support of the agency’s mission. The survey provided information on other
indicators of technological innovation stimulus as well—copyrights and trademarks.
As shown by Exhibit 3-5, 31 percent of survey respondents received one or more patents associated with
their SBIR Phase II project. This figure is consistent with the earlier findings of the National Research
Council (NRC) (33 percent). 11 Another 41 percent had a pending patent application; 18 percent reported
having at least one copyright; and 27 percent reported having obtained at least one trademark.
EXHIBIT 3-5
Awardees’ Patents, Copyrights, and Trademarks
Patents
Pending Patent
Applications
Copyrights
Trademarks
Total
Awardee
companies
Number
217
289
127
191
491
Percent
31%
41%
18%
27%
69%
Total
586
614
517
306
2,023
Mean
Standard
deviation
Median
0.8
0.9
0.7
0.4
2.9
3.1
1.7
5.6
0.9
7.2
0.0
0.0
0.0
0.0
1.0
Items
Note. The Total column on the far right of this table reports the number and percent of awardees with one or more patents,
pending patent applications, copyrights, and trademarks. It also reports the sum of these four items and the mean and
median number per project. The number of awardees who responded to survey queries about these outcomes was 709.
Nearly one-third of awardees reported having obtained one or more patents for their SBIR-supported project. Whether
holding a patent or not, 41 percent had a patent application pending. Eighteen percent of the projects reported having one
or more copyrights, and 27 percent reported at least one trademark associated with their SBIR product. The majority (69
percent) had at least one patent, pending patent application, copyright, or trademark related to their SBIR project.
Across all 709 responding awardees, SBIR-funded work resulted in 586 patents, 614 pending patent
applications, 517 copyrights, and 306 trademarks. Although the average and median numbers of these
items per awardee are either 0 or less than 1, it is important to keep mind that patents, copyrights, and
trademarks are not relevant outcomes for all awardees. Failure to generate these outcomes may be due
partly to the current status of the project, or to the fact that some types of products may have no patent,
copyright, or trademark potential.
11
National Research Council, 2007. An Assessment of the Small Business Innovation Research Program. Charles W. Wessner,
ed. Washington, DC: National Academy Press.
Page 3-6
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Exhibit 3-6 shows that the type of SBIR-funded product and receipt of one or more patents, copyrights,
and trademarks are related. For example, whereas patents were granted to 48 percent of projects that
involved the development and improvement of drugs, very few such projects reported having obtained
trademarks (12 percent), and none had obtained a copyright. In contrast, no educational materials
products were given patents, but 69 percent had one or more copyrights—the largest percentage among
different product types for this outcome.
EXHIBIT 3-6
Awardees with One or More Patents, Copyrights, and Trademarks
by Type of SBIR-Supported Product
0
10
20
Non-drug therapeutics (n=14)
Typ e of pr oduc t
70
80
90
62
85
61
73
15
58
83
8
56
11
68
55
Research tools (n=96)
82
12
53
87
7
47
Medical devices (n=157)
100
86
86
15
Drugs (n=75)
19
46
Chemicals (n=13)
Educational materials (n=51)
60
66
66
Diagnostics (n=86)
Software or hardware (n=101)
Percent
50
10
Biologics (n=39)
Genomics (n=15)
40
29
Measurement tools (n=50)
Environmental tools (n=12)
30
31
46
10
67
69
69
35
80
18
Publications
Presentations
Awards
Note. Type of product and having one or more SBIR project-related patents were significantly associated (chi-square
statistic=76.4, df = 11, p < 0.0001). This also was true for copyrights and product type (chi-square statistic=180.9, df = 11,
p < 0.0001) as well as trademarks and product type (chi-square statistic=24.3, df = 11, p < 0.02. The total number of
awardees who responded to survey queries about each of these outcomes was 709.
The extent to which awardees had obtained at least one patent, copyright, or trademark was highly dependent on the type
of product supported by the award. Between 48 and 58 percent of environmental tools, biologics, and medical devices had
received at least one patent. Educational materials (69 percent) were most likely to have received copyrights. Non-drug
therapeutics (57 percent) were most likely to have received trademarks.
Page 3-7
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
As Exhibit 3-7 illustrates, the current status of the Phase II project was significantly related to all intellectual
property outcomes, with the exception of patents. The patterns of differences for pending patent applications,
copyrights, and trademarks were different. Copyrights were significantly more likely for projects where the
product was being commercialized or was actually available for use. Thirty-eight percent of these projects
reported receiving one or more copyrights related to the SBIR award, as compared to 9 percent of projects
where the product was still being developed, 4 percent of projects that were “on hold,” and 8 percent of
discontinued projects. The same was true for projects that had reached the commercialization stage; here, 26
percent indicated receipt of at least one copyright. The largest percentage of projects that possessed related
trademarks (38 percent) were those in the commercialization stage. The corresponding figures for products
under development, in use, “on hold”, or discontinued were 20, 22, 16, and 9 percent, respectively.
EXHIBIT 3-7
Reported Patents, Pending Patent Applications, Copyrights, and Trademarks
by Status of SBIR-Funded Project
100
P er cent of awardee companies
90
80
70
60
51
50
40
30
39 40
34
32
32
38
38
32
26
26
22
20
20
9
10
9
4
22
16
9
8
0
Patents
Pending Patents
Copyrights
Trademarks
Type of contribution
Under development
Commercialization stage
On hold
Discontinued
In use by target population(s)
Note. A total of 708 awardee small businesses provided this information. Current project status and having at
least one pending patent application were significantly related (chi square statistic=35.7, df=4, p < .0001, p <
0.03). The same was true for current project status and both copyrights (chi square statistic=77.3, df=4, p <
.0001) and trademarks (chi square statistic=44.5, df=4, p < .0001)
The current status of the project and having been granted at least one patent were unrelated. Projects whose
products were under development were significantly more likely to have at least one pending patent application.
Copyrights were significantly more likely for projects that were in the commercialization stage or whose products
were in use by the target population. Projects in the commercialization stage were significantly more likely to
have one or more trademarks.
Page 3-8
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Current status of project and receipt of one or more patents were unrelated. Fifty-one percent of projects that
were in the midst of product development reported pending patent applications. This percentage is
significantly higher than those for all other status categories.
The receipt of one or more patents was significantly related to the fiscal year of the award (chi-square
statistic=11.3, df=4, p < 0.03). Whereas 38 percent of awardees who received their Phase II award in FYs
2002 or 2003 reported having obtained one or more patents, this was true for 30 percent of FY 2004 awards,
24 percent of FY 2005 awards, and 25 percent of FY 2006 awards. Such results are not unexpected, given that
less time had elapsed for the more recently funded projects to have developed their product to the point where
a patent application is appropriate. In fact, the start date of the Phase II award and having one or more pending
patent applications also were significantly related (chi-square statistic=10.8, df=4, p < 0.03); here, the
percentage of pending applications was higher for more recently awarded projects (e.g., 48 percent of 2006
awardees versus 30 percent of FY 2002 awardees). In contrast, obtaining a copyright or getting a trademark
was not related to fiscal year of the award.
3.3
Performance on Receipt of Additional SBIR Awards
The second major index of degree of technological innovation identified by the Evaluation Framework
quantifies the number of awardees who received additional Phase I or Phase II SBIR awards related to the core
technology or product referenced in the survey. Exhibit 3-8 shows that 408 awardees (58 percent) who
responded to survey questions about this matter reported having competed successfully for one or more
additional Phase I or Phase II awards for products related to the supported project.
As also shown by Exhibit 3-8, a larger percentage of awardee companies obtained Phase I awards (58
percent) than Phase II awards (40 percent). This differential success is to be expected. Companies are
likely to have more than one idea or product under development at any given time, not all Phase I
awardees request Phase II funding, and not all Phase II funding requests are granted.
EXHIBIT 3-8
Additional Related Phase I and Phase II SBIR Awards
Phase I
Award
Phase II
Award
Either Phase I or
Phase II Award
Awardee Companies
Number with at least
one award
Percent
405
58
Phase I
Awards
283
408
40
58
Phase II
Awards
All Awards
Awards
Total
1,408
567
1,615
Mean
1.5
0.8
2.3
Standard deviation
1.4
1.4
3.4
Median
1.0
0.0
1.0
Note. A total of 699 awardee small businesses provided information on whether they had
received a Phase I or Phase II award, and 604 provided data on the actual number of
awards received.
Most awardees had successfully competed for one or more Phase I or Phase II awards
related to their SBIR-supported project.
Page 3-9
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Forty-two percent of awardee companies that competed successfully for additional SBIR awards also reported
success in obtaining non SBIR awards for their Phase II funded project. In contrast, only 28 percent of
awardees who had not competed successfully for additional SBIR awards reported successfully obtaining non
SBIR awards. This difference in performance was statistically significant (chi-square=14.1, df=1, p < 0.0002).
3.4
Comparison with 1997-2001 Phase II Awardees
Exhibit 3-9, on the next page, summarizes the performance of FYs 2002-2006 Phase II awardees on the
two indices for the goal of stimulating technological innovation. It also compares their performance with
that of the FYs 1997-2001 Phase II awardees who responded to the 2002 survey.
The FYs 2002-2006 Phase II awardees met or exceeded each of the two numerical indices identified by
the SBIR program. This also was true for FYs 1997-2001 awardees for the one index that could be
compared—receipt of additional SBIR Phase I or Phase II awards. 12
Further study is needed to determine reasons for the significant differences in performance that were
observed between the two cohorts:
• A decrease in the total number of technical articles authored by FYs 2002-2006 awardees as
compared to their FYs 1997-2001 counterparts (t-test statistic=2.69, df=544, p < 0.005).
• A decline for recent awardees in the percentage who obtained at least one patent for their SBIR
funded product (chi square statistic=9.7, df=1, p < 0.002).
• A decline in the percentage of FYs 2002-2006 awardees who published (chi square statistic=28.0,
df=1, p < 0.0001).
• A decline in the percentage of recent awardees with scientific conference presentations (chi
square statistic=5.4, df=1, p < 0.02), which also may be related to the lower percentage who
published.
As Ruhm and Link found for a sample of 1992-2001 Phase II awardees, involvement by faculty, graduate
students, and other academic personnel did distinguish awardees who commercialized their product from
those who did not. 13 This also may be true with regard to publications, given that academic institutions
are more likely to emphasize such outcomes. These types of variables could be explored in future studies.
12
As noted in Chapter 1, the question used to assess performance on the index involving the development of new or improved
products was not identical in the 2002 and 2008 surveys, preventing comparison between 1997-2001 and 2002-2006 awardees.
13
Albert N. Link and Christopher J. Ruhm. Bringing Science to Market: Commercializing from NIH SBIR Awards. (Working
Paper 14057). (Cambridge, MA: National Bureau of Economic Research, June 2008).
Page 3-10
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 3-9
Comparison of NIH SBIR Awardees’ Performance
in Stimulating Technological Innovation
Phase II Award Start
Performance Index
Performance Measure
1.1
Number of new or improved SBIR
supported products in health-related
fields
1.1.2
1.1.3
1.2.1
FYs 2002- 2006
Total
Percent
N
N
Whether or not sales have occurred, 40%
or more of NIH SBIR awardees produce
new or improved products, processes,
usages, and/or services in support of the
NIH mission
1.1.1
1.2
FYs 1997- 2001
Total
Percent
N
N
--
--
--
587
82
719
Number of technical articles on new or
improved SBIR-supported products
369
68
545
375
53
709
Number of patents for new or improved
SBIR-supported products
213
39
546
217
31
709
298
56
535
408
58
699
10% or more of NIH SBIR awardees receive
additional Phase I or Phase II awards that
relate to the core technology
Number and percent of awardees who
received additional related Phase I or
Phase II awards
Note. Performance indices and measures are posited by the Evaluation Framework (Appendix A). The FY 1997-2001 results are
based on analyses of this subset of NIH SBIR program 2002 survey data to allow comparisons for this report. Further study is
needed to determine reasons for apparent differences in performance. Differences in question wording between the 2002 and
2008 surveys did not allow comparison of performance for Performance Index 1.1.1.
Comparisons of performance for awardees that received SBIR awards between FYs 1997-2001 and those that received awards
between FYs 2002-2006 showed both similarities and differences in performance for performance indices 1.1 and 1.2 and
associated measures. For technical articles and patents, performance levels did differ. Significantly larger percentages of
awardees in the earlier cohort had published at least one technical article (68 percent) and obtained one or more patents (39
percent). The corresponding percentages for FY 2002-2006 awardees were 53 and 31 percent, respectively. Both cohorts far
exceeded the stated benchmark of 10 percent for receiving additional SBIR funding that was related to the core technology
supported by the Phase II award. Their level of performance also was quite similar—56 percent of FY 1997-2001 awardees and
58 percent of FY 2002-2006 awardees.
Page 3-11
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
4. INCREASED USE OF SMALL BUSINESS CONCERNS
This chapter presents findings about attainment of the second NIH SBIR program goal addressed by the
survey, using small business concerns to meet Federal research and development (R&D) needs. It opens
with a summary of findings about goal attainment. Then it presents analyses of each related index and
overall performance of SBIR awardees over time.
4.1
NIH SBIR Awardees Met Federal Research and Development Needs
Exhibit 4-1 displays the survey findings about awardee outcomes related to using small business concerns
to meet Federal R&D needs. Phase II awardees reported exceeding specified benchmarks for all three
related indices in the Evaluation Framework—contributing specified types of knowledge, obtaining and
disseminating health-related information, and expressing satisfaction with the usefulness of the NIH SBIR
program.
EXHIBIT 4-1
Summary of Outcomes Associated with
Using Small Business Concerns to Meet Federal Research and Development Needs
Goal 2
Increase the use of small business concerns to meet Federal R&D needs
Index 2.1
50 percent or more of NIH SBIR awardees make
contributions to knowledge in health promotion,
disease prevention, diagnosis, health care, and
amelioration and cure of disease
Index 2.2
50 percent or more of NIH SBIR awardees are able to
obtain and to disseminate health-related information
Measure 2.2.1
96 percent of awardee respondents give
high rankings to the usefulness of
outreach and informational services
Measure 2.1.1
82 percent of the responding awardees
contribute increases in health knowledge,
research tools, and education
Measure 2.2.2
82 percent of responding companies
anticipate or are disseminating SBIR-
supported technology and information to
populations using and receiving health
and health care resources
Index 2.3
50 percent or more of NIH SBIR awardees express
satisfaction with the usefulness of the NIH SBIR
program
Measure 2.3.1
74-90 percent of awardee respondents,
depending on the program component,
are completely or mostly satisfied with
the SBIR application, review, award, and
post-administration of the program.
Page 4-1
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Awardees’ reported performance in attaining each performance index at the time of survey measurement
is described in the following sections.
4.2
Performance on Contributions to Knowledge
Survey respondents reported having far exceeded the Index 2.1 benchmark that 50 percent or more of
awardees contribute to knowledge in health promotion, disease prevention, diagnosis, health care, and
amelioration and cure of disease. They provided strong evidence of having provided the increases in
health knowledge, research tools, and education specified by Measure 2.1.1.
To identify these contributions, the survey asked the awardees themselves to select which of 11 specific
medical, societal, and technological outcomes were associated with their funded project. This list was
developed by NIH personnel as those contributions most relevant to health promotion as well as
prevention, diagnosis, treatment, and cure of disease. Awardees were first asked to choose the outcomes
most relevant to their product; they were then to identify the one outcome that was most important.
Exhibit 4-2 lists the specific outcomes included in the 2008 survey instrument. As shown by the exhibit,
projects typically had multiple outcomes. In fact, only 11 percent of respondents identified only one of
the 11 outcomes listed; the median number of outcomes was four.
EXHIBIT 4-2
Medical, Societal, or Technological Outcomes of SBIR-Funded Projects
Treating disease or disability
Im proving research tools
62
18
Detecting disease or disability
Outcome
53
29
40
12
Preventing disease or disability
10
Diagnosing disease or disability
9
Reducing the cost of m edical care
5
Developing health inform ation for the general public
4
Developing inform ation for health professionals
4
Fostering new research collaborations
1
Training reesearch investigators
0
Other outcom es
36
62
25
39
54
28
8
0
38
10
11
20
30
40
50
60
70
80
90
100
Percent
Most im portant outcom e
Outcom es related to the project
Note. A total of 714 awardee small businesses provided information about whether each specific outcome applied to their SBIRfunded project. Data on the most important outcome were provided by 712 awardees.
SBIR-supported projects, in general, had multiple outcomes. Most outcomes were associated with health care, enhancing the
quality of research, or both. Respondents cited treating disease and disability and improving research tools as the most important
outcomes.
Page 4-2
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Most common were two outcomes associated with providing health care—reducing the cost of medical
care (62 percent) and treating disease or disability (53 percent)—and two aimed at enhancing the quality
of research—improving research tools (62 percent) and fostering new research collaborations (54
percent). The next most frequent outcomes all were targeted at health care issues; between 30 and 40
percent of awardee companies viewed disease prevention, detection, diagnosis and developing
information for health professionals as outcomes of their projects. Another research outcome—training
new investigators—was viewed as relevant by 28 percent of awardees.
When awardees were asked to identify the most important outcome, addressing healthcare and research
needs remained top priorities. Twenty-nine percent of SBIR-funded small businesses indicated that
treating disease and disability was the most important outcome associated with their project, and another
18 percent attached the same level of importance to improving research tools. Other outcomes were
deemed most important by smaller percentages of awardees. For example, reducing the cost of medical
care, although chosen as one outcome by 62 percent of respondents, was viewed as the most important
outcome by only 5 percent of the projects. Similarly, more than one half (54 percent) believed that
fostering research collaborations was an outcome of their SBIR award, but only a very small minority (1
percent) believed it to be the most important outcome. Overall, 92 percent believed that the most
important outcome for their SBIR-funded project was one of these healthcare or research outcomes.
It is interesting to note that significant differences repeatedly appeared in the relationship between the
current project status and both (1) whether a specific outcome was attributed to an SBIR-funded project,
and (2) which outcome was identified as most important. Projects still in the developmental stage were
more than twice as likely to identify treating disease as the most important outcome (45 percent) than
were completed projects and those being commercialized (20 percent for each) When compared to
completed projects, projects that were developing their products also were significantly less likely to
believe their most important outcome was the improvement of research tools (chi-square statistic=80.8,
df=10, p < 0.001). Whereas only 11 percent chose this as their primary outcome, this was true for 24
percent of products that already had been made available for use by the target population(s).
Such differences in perspectives might be expected, given that different types of products have different
timelines for product development. As reported in Chapter 2, current project status was related to type of
product, and certain types of products (e.g., specific types of research tools) may require less time to
develop and thus be available more quickly. In addition, views of outcomes most likely change as
products move further along the product development pathway. During the product development phase,
goals are broad, a host of ideas are continuously tested, and modifications are made. By the time a
product is ready for commercialization, however, developers have a much clearer idea about the product’s
capabilities and expected outcomes.
Performance assessment for this index focused on awardees that are still in the development phase,
commercializing their product, or completed their project and making it available to consumers. This was
the method used in assessing performance in the 2002 survey. A total of 587 projects (82 percent) fell into
these categories.
Technical articles and conference presentations also represent contributions to health-related knowledge.
However, the Evaluation Framework includes these types of contributions as a measure of the extent to
which SBIR awardees develop new and improved products consistent with the NIH mission (Goal 1)
rather than as a measure of contributions to knowledge (Goal 2). As reported in Chapter 1, sizable
percentages of awardees have published one or more technical articles and presented at scientific
conferences. This includes both projects in the developmental and commercialization stages, those that
have made their product available for use by the target populations(s), and projects that have been
Page 4-3
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
discontinued. Given that publications and presentations are not included as a measure for Goal 2, both
estimates of the percentage of projects that make contributions to health-related knowledge most likely
would increase if publications, presentations, and patents also were defined as relevant to include under
Measure 2.1.1.
4.3
Performance on Dissemination of Information
The second index for assessing performance on the overall goal of using small business concerns to meet
Federal R&D needs focuses on the dissemination of health-related information. The Evaluation
Framework describes dissemination as having two distinct aspects. First, the NIH should disseminate
information to their awardee companies. Second, Phase II awardees are responsible for disseminating
health-related information to populations using and receiving health care resources.
4.3.1
NIH Dissemination to SBIR Phase II Awardees
Measure 2.2.1 quantifies and describes awardee perception of the usefulness of the NIH SBIR/STTR
program office’s outreach and informational services. When asked if they were aware that they could
contact NIH staff for more information about the program or assistance with the application, review,
award, and award management, an overwhelming 96 percent of awardees responded affirmatively. There
were no significant differences in these perceptions among awardees with different award start dates,
different sponsoring Institutes/Centers (ICs), or whether they had received other related SBIR awards.
4.3.2
Awardee Dissemination of Health-Related Information
Measure 2.2.2 quantifies and describes the awardees who anticipate or are disseminating SBIR-supported
technology and information to populations using and receiving health and health care resources. The
2008 survey asked awardees to identify the potential or actual target populations for their product as well
as which target population was most important. Similar to the measure for contributions to knowledge,
the 17 specific target populations, as well as the four broader target groups under which these target
populations fell, were identified by NIH personnel.
The four broad target groups of key interest were: (1) hospital inpatients, outpatients, or staff; (2) other
healthcare practitioners, such as physicians, nurses, home care providers and emergency medical
technicians; (3) laboratories, and (4) other populations, including the general public, worksites, municipal
workers, and educational institutions. Awardees typically reported that their SBIR-supported product was
likely to be used by one or more major target population group.
Exhibit 4-3 on the next page shows which major target population groups were cited as using or likely to
use the SBIR-supported product. Sixty-five percent said that hospital inpatients, outpatients, or staff were
using or likely to use their product. Nearly two-thirds (66 percent) said SBIR-supported products were
used or likely to be used by other healthcare practitioners. Fifty-six percent were intended for use by
research and/or diagnostic laboratories. Nearly three-quarters (74 percent) were targeted to other
populations. As also shown by Exhibit 4-3, awardees reported that the most important target population
was least frequently other healthcare practitioners (15 percent) and most often the other three categories
(27 to 31 percent).
Page 4-4
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 4-3
Target Population Groups that Use or Are Likely to Use the SBIR-Supported Product
65
Tar get pop ulat ion group
Hospital outpatients,
inpatients, or personnel
31
56
Laboratories
27
66
Other healthcare
practitioners
15
74
Other populations
27
0
10
20
30
40
50
60
70
80
90
100
Percent of awardee companies
Most important target population
Target population
Note. A total of 658 awardee small businesses provided information about whether their SBIR-funded project was directed at
each of the four target population groups. Data on the most important target population group were reported by 651
awardees.
SBIR-supported products are directed at four broad target population groups: hospital outpatients, inpatients, or personnel;
laboratories; other healthcare practitioners; and other populations. Between 56 and 74 percent of awardees stated that their
product focused on each of these four different target population groups. When asked about their product’s most important
target population, 31 percent of awardees identified one in the hospital outpatients, inpatients, or personnel target group. The
corresponding percentages for laboratories, other healthcare practitioners, and other populations were 27, 15, and 27 percent,
respectively.
As shown by Exhibit 4-4 (on the next page), many awardees said that their SBIR projects target several
specific populations within the target population groups. The median number of specific populations selected
was five. Eight percent of awardees identified only one specific target population for their SBIR-supported
project.
Of the 16 specific target populations listed in the survey, all but five (research laboratories, outpatients,
medical practitioners, schools and universities, and municipal workers) were identified as targets by nearly
two-thirds or more of awardees. Worksites, emergency medical services, and homecare providers were cited
as a target population most frequently, and more than 80 percent of the time. Medical practitioners and
research laboratories were cited as a target population least frequently, and less than 50 percent of the time.
Page 4-5
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 4-4
Target Populations Currently Using or Likely to Use SBIR-Funded Products
Research laboratories
Outpatients
55
19
Medical practitioners
45
12
General public
65
11
Inpatients
T arget po pu lat ion
48
22
62
9
Diagnostic laboratories
5
Schools and universities
4
Other com paies or technologies
3
Hospital personnel
2
Other health services
2
Educators
2
Worksites
1
Em ergency m edical services
1
Hom ecare providers
1
Police, fire, and other m unicipal w orkers
0
Military m edical services
0
Other target population
68
58
67
68
70
71
87
87
83
58
66
88
7
0
10
20
30
40
50
60
70
80
90 100
Percent
Most im portant target population
A target population
Note. A total of 658 awardee small businesses provided information about whether each specific target population applied to their
SBIR-funded project. Data on the most important outcome was reported by 651 awardees.
Of the 16 specific target populations listed in the survey, all but five were identified as a target population by nearly two-thirds or
more of awardees. The two specific target populations that were identified as the most important by the largest percentages of
respondents were research laboratories and outpatients.
Page 4-6
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
For the hospital populations, outpatients, inpatients, or personnel category, awardees most frequently
identified outpatients as their most important target population (19 percent). For laboratories, awardees
most often stated that their most important population was research laboratories (22 percent). For other
healthcare practitioners, medical practitioners emerged as the key target population (12 percent). Finally,
when considering other populations, the general public was the key group, chosen by 11 percent of
respondents. Other specific populations were the most important target for smaller groups of awardee
businesses (1 to 9 percent).
Current project status was related to both whether awardees identified a target population group for their
SBIR-funded project and which segment was most important. Although no differences surfaced with
regard to research and diagnostic laboratories, they did appear for the other target population groups. The
primary disparities occurred between “on hold” projects and those at more active stages. Whereas 31
percent of the products in development, 25 percent of projects in the commercialization stage, and 30
percent of completed projects saw hospitals as a target population, this was true for only 14 percent of
projects that were “on hold.” Only with regard to other target populations did a different pattern emerge.
Here, 50 percent of projects in the midst of developing their products identified a specific target
population such as the general public, worksites, or schools and universities, compared to between 16 and
18 percent of projects at other stages.
When one considers only the most important specific target population, 41 percent of projects in the
developmental stage viewed hospitals as their most important consumers. However, only 17 percent of
completed projects, 23 percent of projects in the commercialization stage, and 34 percent of projects “on
hold” held similar opinions.
Exhibit 4-5, on the following page, presents SBIR awardees’ estimates of the anticipated size of the
primary target population for their products. All awardees estimate that the largest target population is
the general public; 72 percent of the companies that are gearing their product toward this group judge its
size as equal to or exceeding 500,000. The same percentage of companies whose most important target
population is other health service practitioners also estimate these groups to be of similar size, as do the
69 percent of projects with medical practitioners as their key constituency. Between 52 and 56 percent of
awardees who identified diagnostic labs, outpatients, and other populations also expect these groups to be
relatively large.
Multiplying the number of respondents in each size category by the midpoint of each category yields a
rough estimate of the size of the total population being served, for most size categories. Estimates for the
category of 500,000 or more used the start point of 500,000 because the category’s upper limit was not
specified. This yields 145.2 million as the approximate total size of all populations using and receiving
products relevant to health prevention, detection, diagnosis, and treatment as well as biomedical research.
This is clearly a “guesstimate” for several reasons. In addition to using the midpoint of each size
category, this figure does not take into account that several SBIR projects can have products that are
aimed at the same group(s) of individuals, and it relies on the information provided by respondents,
whose actual knowledge of the size of the target population may vary.
Page 4-7
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 4-5
Anticipated Size of Most Important Target Populations for Ongoing Projects
Anticipated Size of Population
Target Population
Group
Total
Under 10,000
%
N
Hospitals
Outpatients
Inpatients
Hospital personnel
Laboratories
Research laboratories
Diagnostic laboratories
Other healthcare
practitioners
Medical practitioners
Other health services
Other populations
General public
Educators
Schools and universities
Other companies or
technologies
Total
108
10
10
10,000 – 49,999
%
N
8
9
2
56
13
18
18
1
4
10
10
0
2
14
11
7
10
26
48
4
1
5
8
6
5
4
7
520
106
20
99
23
69
18
60
10
19
3
55
3
7
0
17
73
8
17
15
10
50
21
200,000 – 499,999
%
N
9
19
22
53
13
50,000 – 199,999
19
8
2
10
2
18
15
15
10
9
12
11
8
2
3
1
1
%
N
5
10
5
500,000 or more
%
N
61
22
56
42
46
9
4
0
8
8
0
3
5
3
22
13
12
13
52
6
1
9
6
38
13
69
72
4
2
2
7
20
11
43
1
7
27
72
10
37
56
243
47
15
3
6
3
14
59
11
39
8
6
Note. Due to small sample sizes, home care providers and emergency medical services were combined with other health services populations. For the same reason, worksites, police and
other municipal workers, and other companies also were included under Other Populations that are not the general public, educators, or educational institutions.
More than two-thirds of projects that identified the general public, medical practitioners or other health services professionals anticipated their target populations to total at least 500,000. This
also was true for 52 percent of projects whose major target population was diagnostic laboratories and 56 percent of projects with products primarily aimed at outpatients. Nearly threequarters (74 percent) of awardees working on core technologies for research laboratories, and 60 percent of awardees with projects aimed at educators, expected their most important target
population to be less than 50,000.
Page 4-8
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
4.4
Satisfaction with the SBIR Application, Review, Award, and Post-Award
Processes
Index 2.3 describes awardees’ satisfaction with selected aspects of the SBIR program. Measure 2.3.1
benchmarks degree of satisfaction with the NIH SBIR application, review, award, and post-administration
processes.
Exhibit 4-6 summarizes awardees’ judgments about the adequacy of the instructions for preparing
applications, the review and award processes, and post-award administration. For each, levels of
satisfaction were high. Between 88 and 90 percent were completely or mostly satisfied with the quality of
the instructions for preparing applications, the award process, and the administration after their grant had
been awarded.
EXHIBIT 4-6
Awardee Satisfaction with the NIH Application, Review, Award,
and Post-Award Processes
Percent of awardee companies
SBIR progra m component
Instructions for
preparing
applications
Review process
90
74
8
21
2
4
Aw ard process
88
10
2
Post-aw ard
adm inistration
89
8
3
Com pletely or m ostly satisfied
Mixed
Com pletely or m ostly dissatisfied
Note. A total of 658 awardee small businesses provided information about whether each specific outcome
applied to their SBIR-funded project. Data on the most important outcome was reported by 652 grantees.
The total numbers of awardees who rated each aspect ranged from 704 to 708. Respondents used a 5-point
rating scale where 1 was “completely satisfied”, 2 was “mostly satisfied”, 3 was “mixed”, 4 was “mostly
dissatisfied”, and 5 was “completely dissatisfied.” The numbers shown are percentages and may not sum to
100 percent due to rounding.
SBIR Phase II awardees expressed high levels of satisfaction with the application, review, award, and postaward administrative components of the program. Between 88 and 94 percent were completely or mostly
satisfied with all components except the review process, which completely or mostly satisfied 74 percent.
Page 4-9
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
The one component that generated a somewhat lower level of satisfaction was the review process, but
even here 74 percent were completely or mostly satisfied and only 4 percent were completely or mostly
dissatisfied. Twenty-one percent held more mixed views. This expression of ambivalence is not uncommon for the recipients of NIH funding in an environment that is increasingly competitive and where even
successful awardees often have to revise and resubmit their applications one or more times before
securing funding.
Satisfaction levels did not differ significantly by fiscal year of award or by IC grouping. Although overall
satisfaction levels remained high, they did depend somewhat, however, on whether awardees had won
other related SBIR awards. Whereas 92 percent of small businesses that had no additional SBIR funding
were satisfied with the award process, this was true for 86 percent of awardees who had won other SBIR
grants (chi-square statistic=7.5, df=2, p < 0.03).
Averaging the satisfaction ratings across the four components for each respondent yielded an index of
satisfaction for each respondent. The average rating was 1.75 (standard deviation = 0.60). Using a 5point rating scale that was used where 1 = “completely satisfied” and 2 = “mostly satisfied”, indicates that
awardees’ overall satisfaction level is slightly higher than “mostly satisfied.” This represents a very
positive assessment.
Awardees were asked whether the project funded by the award would have been pursued if SBIR support
had been unavailable. Only 12 percent of companies believed that the project would still have been undertaken, and 67 percent thought that it never would have been initiated. Twenty percent were uncertain
about what would have happened. Again, no significant differences in responses were found, depending
on the fiscal year of the award, the receipt of other related SBIR awards, IC grouping, or type of product
sponsored by the award.
Finally, respondents indicated how important SBIR support was or will be in the R&D of the planned
product. Again, the overwhelming majority (89 percent) believed it was very important, and 9 percent
judged it as important. Only 1 percent said that it was somewhat important, and less than 1 percent each
deemed SBIR support as not important or not very important. Again, there were no significant
differences in responses by type of product, fiscal year, IC grouping, or receipt of other SBIR awards.
4.5
Comparison of Two Awardee Cohorts’ Performance in Increasing the Use
of Small Business to Meet Federal R&D Needs
Exhibit 4-7, on the following page, presents data about awardees performance in increasing the use of
small business to meet Federal R&D needs. The exhibit describes performance for three indices and their
associated measures for the Phase II awardees in FYs 1997-2001 and FYs 2002-2006.
As shown by the exhibit, the FY 2002-2006 awardees exceeded the 50 percent benchmark for index 2.1—
making contributions to knowledge in health promotion, disease prevention, diagnosis, health care, and
amelioration and cure of disease. As was noted in Chapter 1, comparisons of these awardees’
performance with those of the earlier survey cohort could not be made for this index due to a difference in
question wording between the two survey instruments.
Page 4-10
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 4-7
Comparison of NIH SBIR Awardees’ Performance in
Increasing Use of Small Business to Meet Federal R&D Needs
Phase II Award Start
Performance Index
FYs 1997-2001
Performance Measure
Percent
Total N
--
--
--
567
82
719
Number and percent of awardee
companies that gave high rankings to
the usefulness of outreach and
informational services
528
94
559
674
96
705
Number of disseminations of SBIRsupported technology and information
among populations using and
receiving health and health care
resources
--
--
--
587
82
719
Number and percent of awardees who
have or anticipate disseminations of
SBIR-supported technology and
information among populations using
and receiving health and health care
resources
--
--
--
587
82
719
414-498
74-89
554-560
526-637
74-90
704-708
N
2.1
2.1.1
2.2
2.2.1
2.2.2
2.3
2.3.1
FYs 2002-2006
Percent
N
Total N
50 percent or more of NIH SBIR
awardees make contributions to
knowledge in health promotion,
disease prevention, diagnosis, health
care, and amelioration and cure of
disease
Number and percent of SBIRsupported contributions from ongoing
projects that yield increases in health
knowledge, research tools, and
education
50 percent or more of NIH SBIR
awardees are able to obtain and
disseminate health-related information
50 percent or more of NIH awardees
express satisfaction with the
usefulness of the NIH SBIR program
Numbers and percents of awardee
companies experiencing high levels of
satisfaction with the SBIR application,
review, award, and post-award
administration of the NIH SBIR
program
Note. The FYs 1997-2001 results are based on analyses of this subset of NIH SBIR program 2002 survey data to allow comparisons
for this report. Further study is needed to determine reasons for apparent differences in performance. The 2008 survey added a
response option that affected more than one outcome measure; therefore, it was not possible to compare performance in terms of the
percents of awardees that make health-related contributions or who have or anticipate making disseminations of SBIR-supported
technology and information.
Comparisons of performance between awardees for FYs 1997-2001 and FYs 2002-2006 were limited to two measures. For both
measures, the percentages were very similar. Between 94 and 96 percent of awardees gave high rankings to outreach and
informational service for measure 2.2.1. Between 74 and 90 percent of the two cohorts expressed high levels of satisfaction with the
SBIR application, review, award, and post-award administration of the program for measure 2.3.1.
Page 4-11
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
The SBIR program consistently outperformed the index for disseminating information to its awardees.
Ninety-six percent of FYs 2002-2006 awardees gave high marks to these efforts, similar to the 94 percent
obtained for the earlier cohort of respondents.
FYs 2002-2006 awardees also exceeded the third index regarding awardee satisfaction with the SBIR
program. From 74 to 90 percent expressed high levels of satisfaction with the application, review, award,
and post-award administration processes. These results were consistent with those for the earlier cohort.
Page 4-12
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
5. COMMERCIALIZATION OF INNOVATIONS
This chapter presents findings about attainment of the third NIH SBIR program goal addressed by the survey
and the fourth goal in the Evaluation Framework, commercializing innovations. It opens with a summary of
findings about goal attainment. Then it presents analyses of each related index and overall performance of
SBIR awardees over time.
5.1
NIH SBIR Awardees Commercialized Innovations
Exhibit 5-1 displays the survey findings about awardee outcomes related to commercializing innovations.
Phase II awardees reported attainment of benchmarks that bode well for success.
EXHIBIT 5-1
Summary of Outcomes Associated with Commercializing Innovations
Goal 4
Increase commercialization of innovations
Index 4.1
40 percent or more of NIH SBIR awardees
commercialize new or improved products in healthrelated fields
Measure 4.1.1
33 percent of the awardee respondents reported
that their SBIR-supported products yielded sales
Measure 4.1.2
The total dollar volume of sales of SBIR-funded
products for responding awardees was
$395.5 million
Index 4.2
40 percent of NIH SBIR awardees grow their
companies
Measure 4.2.1
The median number of years respondent
companies with NIH SBIR awards had existed was
8 years
Measure 4.2.2
The median number of full-time employees in
Phase II awardee companies that responded was
8 individuals
Measure 4.1.3
21 percent of the SBIR-funded products that
required FDA approval received this approval
Measure 4.1.4
26 percent of the SBIR-supported products
executed licensing arrangements
Measure 4.1.5
64 percent of the SBIR-funded projects showed
other evidence of commercialization
Measure 4.1.6
36 percent of the awardee respondents obtained
additional developmental funding for SBIRsupported innovations from other sources
Page 5-1
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Awardees’ reported performance in attaining each performance index at the time of survey measurement
is described in the following sections.
5.2
Performance in Commercializing SBIR-Funded Products
This section reports on reported awardee attainment of six related measures in the realms of sales, FDA
approval, licensing arrangements, other evidence of commercialization, and additional developmental
funding. In assessing performance for Index 4.1, commercialization is defined as accomplishing one or
more of these outcomes. By this definition, 76 percent of the FYs 2002-2006 awardees that provided the
necessary information commercialized new or improved products in health-related fields.
5.2.1
Product Sales
Overall, 581 (88 percent) of the awardees with ongoing projects anticipated sales upon completion of
their projects. This is roughly the same percentage as for the 2002 survey, when 85 percent of awardees
expected sales.
As shown by Exhibit 5-2 on the next page, large majorities of awardees, regardless of the type of SBIR
funded product, expected or had achieved sales. With only a few exceptions, whether sales were
expected did not dramatically differ among companies working on different types of products; the
percentage foreseeing sales ranged between 88 and 94 percent for all but three types of products. The
percentages expecting sales for companies working on drugs and those developing diagnostic materials or
devices were, however, significantly lower than companies with other types of projects. Here, 80 to 81
percent of these awardees expected sales (chi-square statistic = 16.0, df=1, p < 0.001). In addition, 67
percent of companies whose SBIR-supported project involved biologics foresaw sales, and this
percentage was significantly lower than those mentioned for any other type of product (p < 0.0001).
The survey asked awardees who anticipated sales upon completion of their projects whether their expectations had been realized. When project status is not taken into account, 33 percent or 189 SBIRsupported new or improved products yielded sales; 59 percent stated that sales were still anticipated; and
8 percent described other situations, in which sales were uncertain because the project was “on hold.”
Page 5-2
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-2
Number and Percent of Awardees that Expected Sales
by Type of Product
Total
N
Type of product
Expectations for Sales
N
Drugs
Percent
70
57
81
149
138
93
Biologics
40
27
68
Genomics
14
13
93
Research tools
91
84
92
Software or hardware
90
83
92
Educational materials
48
45
94
Diagnostic materials or devices
80
63
79
Measurement or assessment tools
46
40
87
Environmental tools
11
10
91
Chemicals
10
9
90
Non-drug therapeutics
13
11
85
662
581
88
Medical devices
Total
Note. Data are for projects that are under development, being commercialized, already in
use by the target population or on hold. No projects that had been discontinued expected
sales.
Overall, large majorities of awardees expected sales at the start of their SBIR-supported
projects. Awardees developing drugs, biologics, and diagnostics expected significantly lower
sales (68 to 81 percent) than did awardees working on other types of products (85 to 94
percent).
When the elapsed time between the start date of the Phase II award is considered, one gains more insight
regarding the potential of awardees’ sales expectations being fulfilled. As Exhibit 5-3 on the next page
shows, between 36 and 46 percent of awardees whose Phase II award began in FYs 2002-2005 reported
that sales had been realized, compared with 24 percent for awardees whose award began in FY 2006.
Page 5-3
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-3
Current Status of Sales by Fiscal Year of Phase II Award
Start date of
Phase II award
(fiscal year)
Status of Expected Sales
Total N
Realized
Anticipated
N (%)
N (%)
2002
92
42 (46)
50 (54)
2003
107
33 (31)
74 (69)
2004
103
35 (34)
68 (66)
2005
107
38 (36)
69 (64)
2006
171
41 (24)
130 (76)
Note. Data are for 580 ongoing projects that reported expected sales. Award date
and status of sales were significantly related (chi-square = 13.6, df = 4, p < 0.009).
Among awardee companies that anticipated sales, the elapsed time from the start
date of the Phase II award to survey administration was significantly related to
whether sales were realized. Awardees that received the Phase II SBIR award in
FY 2002 were the most likely to have realized their sales expectations; those that
received the award in FY 2006 were least likely to have realized sales.
The characteristics of SBIR-funded products were examined more closely for the ongoing projects that
started between FYs 2002-2006 that expected sales (see Exhibit 5-4 on the next page). The type of
product was significantly related to current sales status. Half or nearly half of the projects that produced
chemicals, environmental tools, or non-drug therapeutics, research tools, and measurement tool products
had already realized sales. However, this was true for only 22 to 25 percent of projects with medical
device, biologic, genomic, and diagnostic products. Only 5 percent of projects working on drug products
had achieved expected sales. These differences may well be a function of several factors that can increase
the time required to make a product commercially available, such as requiring FDA approval.
Page 5-4
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-4
Realization of Sales upon Project Completion by Type of SBIR-Supported Product
Ty pe of S BIR -su pport ed produc t
Environmental tools, chemicals, and non-drug
therapeutics
50
Resesarch tool
49
Measurement tools
48
44
Educatio nal materials
41
Computer HW/SW
Diagnostics
25
Biologics and genomics
25
Medical devices
22
5
Drugs
0
20
40
60
80
100
Percent
Note. Data are for 579 ongoing projects that reported whether expected sales had been realized. Status of sales was
significantly related to type of product (chi-square statistic=52.4, df=8, p < 0.0001)
As would be expected, there were significant differences between whether sales had been realized and the type of product
supported by the Phase II award. Projects involving the development or improvement of environmental tools, measurement
tools, and non-drug therapeutics; research tools; educational materials; and computer hardware and software were most likely
to have realized sales.
5.2.2
Cumulative Sales
Measure 4.1.2 indicates commercialization attainment levels by quantifying the total dollar volume of
sales of SBIR-supported products. Of the 189 awardees who realized sales, 184 provided information
about the amount of total cumulative sales through December 2007. As shown in Exhibit 5-5 on the next
page, 66 percent of awardees reported cumulative sales totaling less than $1 million. Sixteen percent
indicated sales of at least $1 million but less than $5 million. Seven percent reported sales of $5 million
or more; typically, these sales did not quite reach $25 million.
Page 5-5
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-5
Amount of Cumulative Sales for Completed SBIR-Funded Projects
Note. Data are for the 184 ongoing projects that reported realizing sales and provided information on the amount
of cumulative sales as of December 2007. Percents may not total 100 percent due to rounding.
The large majority (66 percent) of awardees indicated that their cumulative sales totaled less than $1 million.
Sixteen percent reported cumulative sales of at least $1 million but less than $5 million. Seven percent
generated sales of $5 million or more.
An estimate of cumulative total sales can be obtained by using the midpoint of each sales category and the
start point of the final unbounded category ($50 million or more). This yields an estimate of cumulative
total sales through December 2007 is $395.5 million, with an average amount per project of
approximately $2.15 million.
Certain types of products generated more sales dollars than others (see Exhibit 5-6 on the next page). The
total cumulative sales across all projects that focused on research tools nearly reached $100 million. For
medical devices, the corresponding figure was approximately $73 million. In contrast, educational
materials generated the lowest sales figures; 20 products produced a total of $4.8 million.
Page 5-6
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-6
Percent of Companies that Achieved Cumulative Sales
by Type of Product
Amount of cumulative sales through December 2007 (in thousands)
Type of product
Total
N
$50
or
less
$50$99
$100$499
$500$999
$1,000$4,499
$5,000$24,999
$25,000$49,999
$50,000
or More
%
%
Estimated
total
across all
products
Percent of Projects
%
%
%
%
%
%
3
0
0
0
33
33
33
0
0
$18,750
Medical devices
29
24
14
24
14
17
3
3
0
$73,075
Biologics and
genomics
10
30
0
30
10
10
10
10
0
$57,225
Research tools
40
10
10
38
5
28
10
0
0
$99,400
Software or
hardware
33
21
15
33
6
18
6
0
0
$53,350
Educational
materials
20
75
5
10
5
5
0
0
0
$4,800
Diagnostic materials
or devices
15
20
40
20
7
13
6
0
0
$8,175
Measurement or
assessment tools
19
26
16
37
5
11
5
0
0
$24,200
Other
15
27
7
40
13
7
0
0
0
$56,475
184
48
13
29
8
16
5
1
0
$395,450
Drugs
Total, all products
Note. Data are for the 184 ongoing projects that reported realizing sales and provided information on the amount of cumulative sales as of
December 2007. Percentages are row percentages and may not total to 100 percent due to rounding. The estimated total sales across each
product type was calculated, using the midpoint of each sales category; for the category “50,000 or more”, the bottom limit ($50 million) was
used.
Certain types of products generated more sales dollars than others. Most awardees reported estimated sales of $50,000 or less. The highest
estimated cumulative sales total was $99.4 million for companies that developed research tools. The next highest estimate—for projects that
were developing medical devices—was $73 million. Across all products, the cumulative sales totaled an estimated $395.5 million.
Page 5-7
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Exhibit 5-7 presents sales for each Institute/Center (IC) that reported sales figures. Comparing
cumulative sales among (1) different IC groupings and (2) new products and those improving or
identifying another use for an existing product indicated that there is no statistically significant
relationship between the two variables.
EXHIBIT 5-7
Estimated Cumulative Sales for Awardees Realizing Sales
by Awarding NIH Institute or Center
Awardees realizing
sales
% of all
awardees
funded
by this IC
Awarding NIH Institute or Center
N
NCI
NCRR
NCCAM
NEI
NHGRI
NHLBI
NIA
NIAAA
NIAID
NIAMS
NIBIB
NICHD
NIDA
NIDCD
NIDCR
NIDDK
NIEHD
NIGMS
NIMHI
NINDS
Total Sales
(in
thousands
of dollars)
Mean sales
(in
thousands
of dollars)
National Cancer Institute
National Center for Research Resources
National Center on Complementary and Alternative
Medicine
National Eye Institute
National Human Genome Research Institute
21
17
26
41
50,725
d
39,725
b
2,536.3
d
2,482.8
2
40
825
412.5
4
5
33
45
c
3,775
90,825
1,258.3
18,165.0
National Heart, Lung, and Blood Institute
National Institute on Aging
National Institute on Alcohol Abuse and Alcoholism
National Institute of Allergy and Infectious Diseases
National Institute of Arthritis, Musculoskeletal and Skin
Disease
13
13
5
13
17
38
56
30
7,900
a
6,900
400
a
20,900
607.7
a
575.0
80.0
a
1,741.7
1
7
75
--
3
33
625
208.3
16
38
8,325
520.3
3
21
3,100
1033.3
5
28
3,825
765.0
4
40
1,650
412.5
13
37
56,425
4,340.4
5
36
675
135.0
21
13
12
40
46
35
41,425
13,650
43,700
1,972.6
1,050.0
3,641.7
189
33
395,450
2,092.3
National Institute of Biomedical Imaging and
Bioengineering
National Institute of Child Health and Human
Development
National Institute on Drug Abuse
National Institute on Deafness and Communication
Disorders
National Institute of Dental and Craniofacial Research
National Institute on Diabetes, Digestive, and Kidney
Diseases
National Institute of Environmental and Health
Sciences
National Institute of General Medical Sciences
National Institute of Mental Health
National Institute of Neurological Disorders and Stroke
Total
b
c
Note. Data on realizing sales are for the 189 ongoing projects that provided this information. Information on cumulative sales as of December
2007 was provided by 184 of these respondents. Total sales and mean sales were estimated by using the midpoint of the categories shown in
Exhibit 5-6 (the start point was used for the final, unbounded category). None of the Phase II projects funded by the National Institute of
Nursing Research (NINR) and the National Center on Minority Health and Health Disparities (NCMHD) had yet realized sales.
a
Based on 12 respondents
b
Based on 20 respondents
c
Based on 3 respondents
d
Based on 16 respondents
The percent of awardees that realized sales, the estimated total cumulative sales, and the average sales for SBIR-funded projects varied
among ICs. NHGRI had the highest mean sales, an estimated $18 million. The four ICS with the next highest mean sales were NIDDK ($4.3
million), NINDS ($3.6 million, NCI ($2.5 million), and NCRR ($2.5 million).
Page 5-8
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Exhibit 5-8 summarizes the extent to which FYs 2002-2006 projects achieved sales.
EXHIBIT 5-8
Summary of SBIR Projects’ Sales Performance for FYs 2002-2006
as of December 2007
N of projects
Percent
Total N
Ongoing projects that expected sales
581
82
719
Ongoing projects that expected and had
realized sales
189
33
581
N of awardees
Estimated total cumulative sales to date
184
Estimated total
$395,450,000
Average per awardee
$2,092.300
Among the 719 respondents, 581, or 82, percent were ongoing and expected sales. Of this group, 189, or 33 percent,
stated that sales already had been realized. As of December 2007, 184 of the awardees who had realized sales
estimated total cumulative sales at $395.5 million, resulting in an average total of slightly more than $2 million in sales per
awardee.
5.2.3
FDA Approval of SBIR-Funded Products
Measure 4.1.3 quantifies the number of SBIR-funded products that have received FDA approval, a
paperwork intensive “must” for certain product types that delays the awardee’s ability to commercialize
in other ways, such as sales and marketing. Three hundred and nineteen respondents indicated that their
products required FDA approval. Exhibit 5-9, on the next page, shows that all projects that involved drug
development required FDA approval, as did substantial majorities of projects whose product was a
medical device (83 percent), a biologic (75 percent), or diagnostic material or device (62 percent). For
the remainder of products, FDA approval was less likely to be applicable.
Page 5-9
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-9
Percent of SBIR-Funded Products Requiring FDA Approval by Type of Product
100
100
90
83
75
80
Percen t
70
62
60
50
40
29
30
20
13
12
11
8
10
2
0
0
0
D
g
ru
ed
M
ce
vi
e
d
al
ic
B
c
gi
lo
o
i
D
tic
os
n
g
ia
nNo
ug
dr
th
tic
eu
p
a
er
l
ic
W
oo
/S
om
tt
n
W
n
e
e
H
G
er
em
ut
ur
p
s
ea
om
M
C
C
h
al
ic
em
Re
ch
ar
se
ol
to
n
io
at
c
u
Ed
m
al
l
ta
en
nm
iro
v
En
ls
ia
er
t
a
Type of product
Note. The number of awardees that indicated whether FDA approval was required and also identified the type of SBIR product
totaled 716. As would be expected, the type of product and the need for FDA approval were significantly related (chi-square
statistic=421.2, df=11, p < 0.0001).
All drug development products required FDA approval, as did 83 percent of medical device products, 75 percent of biologic products,
and 62 percent of diagnostic products.
As shown by Exhibit 5-10 (on the next page), among the 716 awardees, 319 (45 percent) indicated that
their projects needed FDA approval. Among the 319 respondents whose SBIR-funded projects required
FDA approval, slightly less than one-third (31 percent) had been submitted for FDA review. Of the 98
products that had been submitted, 68 percent had been approved. Most likely to receive approval were
diagnostics, measurement tools, non-drug therapeutics, and computer hardware and software, all of which
had approval rates of 75 percent or higher. Approval rates were somewhat lower (between 57 and 68
percent) for drugs, medical devices and biologics.
Page 5-10
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-10
FDA Approval Status by Type of SBIR-Funded Product
Type of product
Total N
N
Drug
Status of FDA Approval Process
Required FDA
approval
Submitted for FDA
review
N
%
%
Received FDA
approval
N
%
77
77
100
22
29
13
59
160
133
83
50
38
34
68
Biologic
40
30
75
7
23
4
57
Diagnostic
86
53
62
11
21
9
82
Non-drug therapeutic
14
4
29
1
25
1
100
Measurement tool
50
6
12
3
50
3
100
Genomic
15
2
13
3
75
67
68%
Medical device
Computer HW/SW
101
11
11
4
36
Chemical (non-drug)
13
1
8
0
0
Research tool
97
2
2
0
0
Educational materials
51
0
0
Environmental, ergonomic, or
assistive tool
12
0
0
716
319
45%
98
31%
All product types
Note. Three respondents provided no data on whether their product required FDA approval. The results are ordered from
highest to lowest in terms of the percent of SBIR-funded projects requiring FDA approval.
At least 75 percent of diagnostics, non-drug therapeutics, measurement tools, and software or hardware products that
required FDA approval had been approved.
As Exhibit 5-11 on the following page shows, 38 percent of the projects that had begun the FDA review
process had received approval for marketing. Another 32 percent had received approval for use in
clinical trials and had (1) Investigational New Drug (IND) status, (2) been granted an exemption from
IND, or (3) been judged a non significant risk. Only a small minority (2 percent) had completed the
process but not been approved.
Page 5-11
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-11
Stage of the FDA Approval Process for SBIR-Funded Projects
that Submitted Applications to the FDA
40
38
35
32
Percent
30
25
20
15
12
10
7
7
5
2
0
Obtained
approval
for
marketing
Obtained
approval
for use in
clinical
trials
Applied for Applied for
marketing clinical trial
approval
approval
Review
ongoing
Not
approved
Stage of the FDA approval process
Note. Data are for 98 projects that reported having submitted applications to the FDA.
For awardees that submitted applications to the FDA, 38 percent received approval for marketing, and 32
percent received approval for use in clinical trials. Only a small minority (2 percent) were not approved.
5.2.4
Licensing of SBIR-Funded Projects
Measure 4.1.4 quantifies the number of SBIR-supported products that executed licensing arrangements,
another indicator of commercialization. Among all projects, 26 percent (173 projects) reported that they
had done so. This percentage did not differ significantly by the start date of the Phase II award or by the
type of product, but significant differences did appear in terms of current project status (chi-square
statistic=40.4, df=8, p < 0.0001). As would be expected, discontinued projects and those that had been
placed “on hold” were the least likely to have licensed their products (13 and 14 percent, respectively). In
some cases, the failure to obtain a license was one of the reasons for not further pursuing
commercialization.
Awardee small businesses that were still developing their products also were less likely to have
successfully completed licensing negotiations (19 percent). Significantly larger percentages of projects
that were involved in commercialization (36 percent) or had made their products commercially available
(40 percent) had secured licenses for their products.
5.2.5
Other Evidence of Commercialization
Measure 4.1.5 describes and quantifies other evidence of commercialization—various types of marketing
activities, public stock offerings, and giving trade or commercial names to the SBIR-funded product.
Page 5-12
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Approximately 88 percent of companies (488) reported at least one ongoing or completed marketing
activity. Overall, about 23 percent of companies had completed their marketing plan, 20 percent had
completed hiring marketing staff, and 20 percent had completed test marketing activities. Eleven percent
reported that their publicity and advertising campaigns had been completed.
Exhibit 5-12 on the next page shows the status of marketing activities for awardees with ongoing projects
by current status of project. The shaded percentages identify their most typical level of involvement in
marketing activities, depending on the current status of their project. Between 64 and 66 percent of
awardees who were still developing their products had not yet become involved in most marketing efforts,
with the exception of preparing a marketing plan. In contrast, companies that were commercializing their
SBIR-funded projects were more likely to be in the midst of preparing marketing plans (60 percent),
publicizing and advertising their product (54 percent), and carrying out test marketing (50 percent). For
awardees who had already made their product available for use, the majority had completed all marketing
activities except ongoing publicity and advertising. When projects were “on hold,” little attention was
being paid to marketing activities.
Only 11 companies (2 percent) with ongoing projects reported a pubic offering of their stock. Nine of
these companies were listed on NASDAQ, one was on AMEX, and another was listed on another
exchange. Six of these 11 companies reported biotechnology as their major field of business, three were
pharmaceutical companies, one was primarily associated with diagnostic devices and materials, and one
was a medical devices company. Over half (6) of these companies had more than 50 employees.
Approximately 57 percent of awardees (366) provided a trade name for their SBIR-supported product.
5.2.6
Receipt of Additional Non-SBIR Funding
Measure 4.1.6 describes and quantifies awardees’ performance in obtaining additional developmental
funding for SBIR-supported innovations from non-SBIR funding sources. Approximately 36 percent of
companies received additional non-SBIR funding for their project. As would be expected, companies
whose projects were “on hold” or discontinued were significantly less likely to have obtained such nonSBIR support. Whereas 38 and 40 percent of products in the developmental and commercialization
stages and 44 percent of products in use reported the receipt of additional non-SBIR funding, this was
true for only 19 and 9 percent of “on hold” and discontinued projects, respectively (chi square
statistic=34.9, df=4, p < 0.0001).
Supplemental financial support also was more likely for certain types of products. As Exhibit 5-13 on
page 5-15 shows, between 47 and 55 percent of products that were biologics, environmental tools, drugs,
and genomics received additional non-SBIR funding. As a group, the percentage was significantly higher
than that for medical devices (43 percent had obtained additional non-SBIR funding). Between 31 and 34
percent of projects that involved diagnostic, research tool, and chemical products obtained such support as
did 29 and 27 percent of projects focused on non-drug therapeutics and computer hardware and software.
Again, the combined percentage for these products was significantly lower than for medical devices and
the previously mentioned cluster of products. The least likely to get additional funding were
measurement tools and educational materials.
Page 5-13
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-12
Percent of Awardees Engaged in Marketing Activities by Current Project Status
Project Status
Marketing Activity
Under development
Preparation of marketing
plan
Hiring of marketing staff
Publicity and advertising
Test marketing
Commercialization stage
Preparation of marketing
plan
Hiring of marketing staff
Publicity and advertising
Test marketing
In use by target
population(s)
Preparation of marketing
plan
Hiring of marketing staff
Publicity and advertising
Test marketing
“On hold”
Preparation of marketing
plan
Hiring of marketing staff
Publicity and advertising
Test marketing
Total
N
Status of involvement in marketing activities
Not yet
Planned
Ongoing
Complete
planned
%
%
%
%
N
N
N
N
274
98
36
54
20
99
36
23
8
237
250
246
155
164
157
65
66
64
45
51
32
19
20
13
23
30
4
10
12
2
14
5
13
6
2
5
133
15
11
9
7
80
60
29
22
113
128
122
31
27
19
27
21
16
29
24
24
26
19
20
27
69
61
24
54
50
26
8
18
23
6
15
140
8
6
5
4
42
30
85
61
122
141
123
16
10
7
13
7
6
7
2
2
6
1
2
36
79
31
30
56
25
63
50
83
52
36
68
57
32
56
7
12
13
23
5
9
47
52
49
43
40
39
92
77
80
1
4
3
2
8
6
2
7
5
4
14
10
1
1
2
2
2
4
Note. Percentages are row percentages and may not sum to 100 percent due to rounding. Shaded percentages refer to the current
status of individual marketing activities for the majority of projects in the status category.
Awardees’ involvement in marketing efforts depended on current project status. Awardees that were still developing their product or
had placed efforts “on hold”, typically had not yet planned marketing activities. In contrast, awardees that were in the
commercialization stage tended to have ongoing marketing activities. Awardees that had made their product available for use
typically were still running advertising and publicity campaigns, but had completed many other marketing activities.
Page 5-14
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-13
Percent of Awardees that Received Additional Non-SBIR Funding by Type
of SBIR-Supported Product
100
90
80
Perce nt
70
55
60
50
49
50
47
43
40
34
31
30
31
29
27
22
22
20
10
0
B
ol
to
l
ta
en
m
n
ro
vi
c
gi
lo
o
i
En
ug
Dr
al
ls
ic
ic
ol
ol
ic
W
ce
ic
st
m
to
to
ut
/S
vi
ria
o
e
o
m
t
e
e
h
n
W
e
d
at
g
en
ap
H
en
rc
al
ch
G
ia
er
ea
er
lm
em
g
ic
D
t
h
s
t
a
r
d
u
e
n
r
e
pu
R
su
ug
io
-d
M
ea
at
om
dr
on
c
M
C
N
on
du
N
E
Type of product
Note. Data are for 716 projects that provided information. Obtaining addition non-SBIR funding and type of product were
significantly related (chi square statistic=31.4, df = 11, p < 0.001)
Between 47 and 55 percent of awardees that were working on biologics, environmental tools, drugs, and genomics had
obtained additional non-SBIR funding for the core technology supported by the Phase II award. Slightly more than two-fifths (43
percent) of medical devices had been successful in securing this type of funding. For the remainder of product types, the
percentages with additional support ranged from 22 to 34 percent.
Page 5-15
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Of the 259 projects receiving non-SBIR support, the large majority (83 percent) attributed such support
partly to having received SBIR funding.
Significant differences also occurred among awarding NIH ICs (Exhibit 5-14). Awardees whose Phase II
award was sponsored by the NIAID were significantly more likely to obtain additional non-SBIR support
than all other Institutes or Centers except NIDDK. On the other hand, companies with awards from the
NIA were significantly less likely to report obtaining additional monies, with only one-fifth of each
indicating such support. This also was true for the cluster of ICs that included NIAAA, NIDA, NINR,
and NCMHD.
EXHIBIT 5-14
Percent of Awardees that Reported Receipt of Additional Non-SBIR Funding
by Awarding NIH Institute or Center
51
NIAID
46
NIDDK
41
Awar ding N IH Institute or C enter
NHLBI
38
NIGMS
35
NCI
32
NICHD
31
NIMH
25
NINDS
20
NIA
21
NIAAA, NIDA, NINR, and NCMHD
36
All other ICs
0
10
20
30
40
50
60
Percent
Note. Data are based on responses from 717 awardees. There were significant differences in the percentages of
awardees who received additional non-SBIR funding by IC (chi square statistic=19.4, df = 10, p < 0.0352). See Appendix
D for a full list of all other ICs and Exhibit 1-1 for the titles of all ICs.
Projects sponsored by different ICs differed significantly in terms of receipt of additional non-SBIR funding for their SBIRsupported project. Companies whose Phase II award was sponsored by the NIAID were significantly more likely to obtain
additional non-SBIR support than were those funded by all other ICs, except for the NIDDK.
Page 5-16
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Awardees reported different opinions about the importance to these additional funding streams. When
asked which source of funds had been most important, 18 percent identified monies from their own
company, and another 16 percent identified another private company (Exhibit 5-15). Another program
within the NIH or at another Federal agency was the most important additional source for 12 percent of
awardees. Thirteen percent perceived the monies that they had obtained from U.S. venture capital
companies as the most important additional funding source.
EXHIBIT 5-15
Percent of Awardees that Reported Receipt of Additional Non-SBIR Funding
by Funding Source and Most Important Funding Source
Aw ardee's ow n
com pany
Other private com pany
43
16
44
Non-SBIR Federal funds
Non-SBIR funding source
75
18
12
U.S. venture capital
27
13
Private individual
investor
8
Personal funds
8
State or local
governm ent funds
44
40
32
7
10
College or university
1
Foreign venture capital
institution
9
0
7
7
All other
0
10
20
30
40
50
60
70
80
Percent
Most im portant funding source
Funding source
Note. Percentages are based on responses from 257 awardees who received additional non-SBIR funding.
Awardees most often said that the most important non-SBIR funding source was their own company (18 percent), which also was
the most frequently cited non-SBIR funding source (75 percent). Sixteen percent of awardees cited other private companies as the
most important funding source, with 12 and 13 percent respectively citing non-SBIR Federal funds and U.S. venture capital most
important.
Page 5-17
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
Funding from the awardee company was least likely for projects “on hold” or discontinued (chi-square
statistic=7.30, df=3, p < 0.0629). Between 78 and 81 percent of companies whose product was being
developed, in the commercialization stage, or in use, reported investment by the business itself or
additional non-SBIR funding. This was true for only about half of companies with discontinued or
inactive projects. Although the percentages are much smaller, this pattern also occurred with regard to
foreign venture capital funding. No discontinued or inactive projects had this type of additional funding.
Only 3 percent of products in use had foreign venture capital monies, but 12 percent of companies that
were still developing or commercializing their product reported this source (chi-square statistic=6.97,
df=3, p < 0.0729). Additional investment from the individual PI was significantly more likely for projects
in the commercialization stage or in use (45 and 54 percent, respectively) as compared to projects that
were still in the developmental phase (33 percent) or inactive (11 percent) (chi-square statistic=15.1, df=3,
p < 0.0018). No statistically significant differences were found for the other sources.
5.2.7
Summary of Commercialization Efforts
As shown by Exhibit 5-1 and discussed throughout this chapter, the Evaluation Framework includes five
measures describing how NIH SBIR awardees commercialize their products—by realizing sales,
obtaining FDA approval, executing licensing arrangements, having ongoing or completed marketing
efforts, and obtaining additional non-SBIR funds for project support. Seventy-six percent of the FY 2002
through FY 2006 awardees reported commercializing new or improved products in health-related fields in
one or more of these ways.
Identifying the numbers and types of commercialization outcomes reported by awardees can be useful in
providing a deeper understanding of the nature of Phase II awardees’ commercialization. Exhibit 5-16 on
the next page provides this information for the 543 awardees who reported. Results should be interpreted
by keeping in mind that the total number of measures that any awardee can achieve depends on the
product being developed. For example, some products do not require FDA approval or may not need
licensing agreements.
As shown by Exhibit 5-16, at the time of the survey, approximately 39 percent of the respondents
reported having satisfied only one measure. The most frequently-cited accomplishment was marketing
efforts (21 percent).
Twenty-nine percent of the awardees mentioned progress in two measures of commercialization. The
most frequent mentions of two measures cited (1) realizing sales and marketing efforts (54 awardees, or
10 percent of the respondents), and (2) marketing activities and receipt of non-SBIR funds (9 percent).
Twenty-two percent reported progress on three measures. The most frequent mention of three measures
cited realizing sales, marketing efforts, and obtaining additional non-SBIR funds (40 awardees, or 7
percent of the respondents). Much smaller percentages of awardees had achieved four measures (9
percent) and all five measures (1 percent).
Page 5-18
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-16
Number and Type of Commercialization-Related Outcomes
Achieved by Awardees
Commercialization Measures
Realized
sales
Obtained FDA
approval
Executed
licensing
agreement(s)
Had ongoing or
completed
marketing efforts
Obtained
additional
non-SBIR
funds
◙
◙
◙
◙
◙
Achieved one of the five measures
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
Achieved two of the five measures
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
Achieved three of the five measures
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
◙
Achieved four of the five measures
◙
◙
◙
◙
Achieved all five measures
◙
%
N
7
9
24
115
56
1
2
4
21
10
211
39
1
54
2
1
6
3
24
16
50
<1
10
<1
<1
1
1
4
3
9
157
29
1
7
30
2
40
3
1
9
26
<1
1
6
<1
7
1
<1
2
5
119
22
2
27
10
2
9
<1
5
2
<1
2
50
9
6
1
6
1
Note. Percentages are based on responses from 543 awardees. Percentages may not total to 100 percent or to the percent for the
general category (e.g., achieved three measures) due to rounding.
Awardees were considered to have commercialized a new or improved product if they reported achieving at least one of the five
commercialization measures. Thirty-nine percent of the responding awardees had satisfied at least one measure—most typically
having ongoing or completed marketing efforts (21 percent). Twenty-nine percent of the responding awardees had achieved two of
the five measures, with the highest percentage (10 percent) having realized sales and having ongoing or completed marketing
efforts. Twenty-two percent of respondents realized three measures—the same two as the largest group in the two measure
category, and also obtaining additional non-SBIR funds. The 9 percent who achieved four of the five measures most typically
accomplished all but FDA approval (5 percent). Six awardees, or 1 percent of those reporting, achieved all five measures.
Page 5-19
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
5.3
Growth of Awardee Small Businesses
As described in Chapter 2, a significant percentage of awardee companies were in their early years of
operation at the time of the award. Approximately 30 percent had been operating for five or fewer years.
One-third (33 percent) had been in existence for 6-10 years, 16 percent for 11-15 years, and the remainder
(21 percent) for more than 15 years. The distribution of years in operation was skewed to the right. The
average years in existence was 10 years (standard deviation = 8), and the median was 8 years.
• One indicator of company size is the number of employees. As shown in Exhibit 5-17 on the
next page, 62 percent of awardee companies had 10 or fewer full-time employees. Twenty-three
percent employed 11 to 25 individuals, and 14 percent had between 26 and 100 employees. Only
a small minority (7 percent) had more than 100 employees. The distribution of full-time
employees was again skewed to the right, with a mean of 36 employees but a median that was
much smaller (8 employees).
Eighty-one percent of the respondents reported that granting of the SBIR award had allowed them to hire
additional personnel. Sixteen percent perceived the Phase II award to have no influence on staff hiring,
and 3 percent were uncertain.
Awardees also were asked to indicate whether their companies experienced various growth-related
activities because of the product developed with SBIR support. Exhibit 5-18, on page 5-22, shows the
percentage of awardee companies that responded positively to these questions. The most frequent event
was establishing a strategic partnership or alliance with another entity; nearly one-third (32 percent) of
awardees reported that such relationships had been developed. The second most frequent activity
involved executing a licensing agreement; here, slightly more than one-quarter (26 percent) of awardees
had either licensed the SBIR-funded product themselves or sold the licensing rights to another company.
In addition, about one-fifth (21 percent) of respondents reported private placement with angels, venture
capitalists, or relatives. Joint ventures and debt financing occurred for 15 and 17 percent of awardee
businesses, respectively.
Page 5-20
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-17
Number of Full-Time Employees for Awardees
100
90
80
70
Percent
60
50
40
32
30
23
18
20
10
14
7
7
0
1
2- 5
6 - 10
11 - 25
26 - 100
More than
100
Number of full-time employees
Note. Percentages are based on 711 awardees. For three companies that did not report this
information; data were extracted from Dunn and Bradstreet profiles available on www.manta.com.
Slightly more than three-fifths (62 percent) of Phase II awardee companies had 10 or fewer fulltime employees. The median number was eight.
Much lower percentages of awardees set up one or more spin-off companies (9 percent), merged their
company with another business (4 percent), or sold their company (2 percent). These respondents could
have been eliminated from the analysis because these outcomes made them technically ineligible to
respond, given the criteria used for inclusion in the survey target population. However, in order to remain
consistent with the analytic strategies used for the 2002 survey of Phase II awardees, these awardees were
included in the analyses of survey responses.
Only 2 percent of awardee small businesses reported a public offering of their stock. The stock
exchanges and ticker symbols for these 11 companies were:
•
•
•
New York Stock Exchange: CLRA
NASDAQ: BDSI, CASM, DCGN, IDIX, IMMC, MBRX, PANC, RPROS, RYND
Other: MIIS.OB
Page 5-21
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-18
Awardees’ Reported Growth-Related Activities
Strategic partnership or alliance
32
Growth-related activities
Executed licensing agreem ent
26
Private placem ent (angels, venture capital, or
relatives)
21
Joint venture (academ ic or com m ercial)
17
Debt financing
15
Set up one or m ore spin-off com panies
9
Merged com pany
4
Sold com pany
2
Public offering
2
0
5
10
15
20
25
30
35
Percent
Note. Depending on the specific activity, between 654 and 681 awardee companies provided information.
The most commonly reported growth-related activity was the development of strategic partnerships or alliances (32
percent). The next most frequently mentioned growth-related activities were successfully negotiating licensing
arrangements (26 percent) and private placements (21 percent).
The questionnaire asked awardees about the extent to which receiving an SBIR award had an impact on
other activities that are relevant to company growth. Eighty-six percent believed it had affected their
credibility or visibility for finding partners. Fifty-three percent responded that it had an impact on raising
additional capital.
Although less directly related to company growth, 94 percent said that the award had an effect on their
ability to pursue a high-risk idea or action that might otherwise not have been undertaken. This is a
strong testimonial to the role of the SBIR program in encouraging innovation.
Page 5-22
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
5.4
Comparison of Two Awardee Cohorts’ Performance in Commercializing
Innovations
Exhibit 5-19 on the next page shows the outcomes of FYs 1997-2001 and FYs 2002-2006 awardees on
the indices identified for commercialization. For the major index stating that companies with SBIR
awards commercialize new or improved products, 67 percent of the earlier cohort of awardees had some
involvement in commercialization as reported in 2002. The corresponding percentage for the later cohort
of awardees was 61 percent as reported in 2008. This difference was not statistically significant.
With respect to the other measures reported in Exhibit 4-7, the performances of each cohort were nearly
identical. The percentage of the later cohort of awardee companies that had executed licensing
arrangements was somewhat higher than the earlier cohort—25 versus 20 percent—but again, this was
not a significant difference.
In terms of growing their companies, the median numbers of years that awardees were in operation were
both 8 years. The reported median number of employees in the later cohort was 10, versus 8 employees
reported for the earlier survey cohort.
Page 5-23
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 5-19
Comparison of SBIR Awardees’ Performance in
Commercializing Innovations
Phase II Award Start
Performance Index
Performance Measure
FYs 1997- 2001
%
N
4.1
40 percent of companies with NIH
SBIR awards commercialize new or
improved products, processes, and or
a
services in health-related fields
4.1.1
Number and percent of SBIRsupported products that yield sales
(of those that expect sales)
4.1.2
Estimated dollar volume of sales of
SBIR-supported products (in millions)
4.1.3
4.1.4
4.1.5
4.1.6
4.2
Number and percent of SBIRsupported products receiving FDA
approval (of those that required
approval)
Number and percent of awardees
executing licensing arrangements
Number and percent of awardees
with other evidence of
commercialization
Number and percent of awardees
receiving additional non-SBIR
funding
40 percent of companies with NIH
b
SBIR awards grow their companies
FYs 2002- 2006
Total N
%
N
Total N
377
67
566
541
74
717
146
33
442
189
33
580
--
--
--
$395.5
NA
189
--
--
--
67
21
319
116
23
509
173
26
664
341
65
527
391
64
607
214
38
--
257
36
717
465
83
558
583
82
714
Median
Total N
Median
Total N
4.2.1
Median number of years companies with
NIH SBIR awards have existed (median)
7
557
8
711
4.2.2
Median number of full-time employees
10
557
8
711
Note. The FYs 1997-2001 results are based on analyses of a subset of the 2002 survey data. Total cumulative sales and the
percent obtaining FDA approval were not compared for the two cohorts, given that the response categories for the questions in
the 2008 survey were modified slightly from those listed in the 2002 questionnaire.
a
This percentage refers to the percent of awardee respondents who did one or more of the following: realized sales; obtained
FDA approval (if required); executed licensing arrangements; showed other evidence of commercialization (had
ongoing/completed marketing activities or a public stock offering); or received additional non-SBIR funding for their core
technology sponsored by the Phase II award.
b
This percentage refers to the percent of awardee respondents who stated that one of the SBIR awards related to their product
had on impact on their hiring of additional personnel.
Performance of the two cohorts was quite similar for measures related to both indices, Index 4.1, percent of companies that
commercialized, and Index 4.2, percent of companies with NIH-SBIR awards who reported growing their companies. The one
difference between the two cohorts was that a significantly higher percentage of FY 2002-2006 awardees reported some
evidence of commercialization (74 percent) compared to the earlier cohort (67 percent).
Page 5-24
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
6. CONCLUSIONS AND RECOMMENDATIONS
The 2008 national survey to assess the outcomes of the NIH SBIR program addresses three questions
about performance by NIH SBIR Phase II awardees from FYs 2002-2006:
• To what degree do small businesses supported through the NIH SBIR program stimulate
technological innovation?
• Has the NIH SBIR program’s use of small business concerns satisfied Federal research and
development (R&D) needs?
• Has the NIH SBIR program increased private sector commercialization of innovations derived
from Federal R&D funding?
The questions focus on how well NIH has done in attaining Congressionally-mandated SBIR program
goals 1, 2 and 4. Information from the 2008 survey supplements that obtained in the similar 2002
National Institutes of Health survey and the 2006 National Research Council assessment.
This chapter presents conclusions from the 2008 survey and offers recommendations for future NIH SBIR
program evaluation.
6.1
NIH SBIR Awardees Met or Exceeded NIH Performance Targets
Respondents to the 2008 national survey to assess the outcomes of the NIH SBIR program reported
exceeding all NIH targets related to the performance issues that were the study focus.
As shown by Exhibit 6-1 on the next page, at least 82 percent of awardees:
• Developed products that were consistent with the NIH mission
• Contributed to knowledge in fields of interest to NIH
• Demonstrated ability to obtain and disseminate health-related information to target populations
• Expressed satisfaction with the usefulness of the NIH SBIR program in terms of the application,
award, and post-award administration program
• Grew their companies
For 74 percent, there is evidence that commercialization has been initiated at some level. Eighty-two
percent of respondents reported that the SBIR program had had an impact on their hiring of additional
personnel. Fifty-eight percent of awardees received additional Phase I or Phase II awards that relate to
the core technology developed by the award described by their survey responses.
Page 6-1
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
EXHIBIT 6-1
Summary of Performance for FYs 2002-2006 SBIR Phase II Awardees*
NIH SBIR Performance Goal
NIH SBIR Performance Index
NIH Performance Index
Benchmark
Awardee
Performance
1.1 Regardless of sales SBIR awardees develop products
in support of the NIH’s mission
40%
82%
1.2 SBIR awardees received additional Phase I or Phase II
awards that relate to the core technology
10%
58%
2.1 SBIR awardees make contributions to knowledge in
health promotion, disease prevention, diagnosis, health
care, and amelioration and cure of disease
50%
82%
2.2 SBIR awardees are able to obtain and disseminate
health-related information
50%
82%
2.3 SBIR awardees express satisfaction with the
usefulness of the NIH SBIR program
50%
74 – 90%
4.1 SBIR awardees commercialize new or improved
products in health-related fields
40%
74%
4.2 SBIR awardees grow their companies
40%
82%
1. Stimulate Technological Innovation
2. Use Small Business Concerns to Meet Federal R&D
Needs
4. Increase Commercialization of Innovations
The 2008 national survey to assess the outcomes of the NIH SBIR program assessed performance for three of the
program’s four goals. Awardee performance exceeded performance benchmarks for all of the performance indices that
were examined.
*The survey assessed performance for Goals 1, 2, and 4 for the SBIR program.
In addition to exceeding these numerical benchmarks, substantial percentages of SBIR awardees achieved
several other related accomplishments of keen interest to Federal SBIR policy makers and practitioners.
These accomplishments include:
• Fifty-three percent published one or more technical articles about new or improved SBIRsupported products, and 31 percent obtained one or more patents relevant to the core technology
supported by the Phase II award.
• One-third (33 percent) of the projects realized sales, for a cumulative sales total of $395.5
million.
• One-quarter (25 percent) successfully executed licensing arrangements for their SBIR-supported
product.
• Over one-third (36 percent) obtained additional non-SBIR funding related to their Phase II core
technology.
Page 6-2
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
6.2 Enhancements to NIH SBIR Program Monitoring and Evaluation Should be
Considered
The NIH SBIR program has been in the vanguard of Federal programs in focusing scarce resources on
monitoring and evaluation—two activities strongly endorsed by recent SBIR surveys and both legislative
and executive branches of the Federal government. The 2008 survey implementation experience suggests
that it may now be timely to consider two types of enhancements—updating the NIH SBIR program
Evaluation Framework and expanding NIH SBIR program monitoring capabilities.
6.2.1 Update the SBIR Program Evaluation Framework
Evaluation frameworks can be valuable documents that accurately reflect up-to-date consensus about the
attributes of program success and how and when they should be measured. Frameworks provide a guide
about how to generate information that can be predicted to furnish credible evidence of program
performance to program policy makers and managers alike.
The NIH SBIR program Evaluation Framework was developed in 2001 by an NIH-wide committee of
SBIR administrators who wished to develop consensus on the attributes of a successful NIH SBIR
program. In 2006, its suitability for use in structuring assessment of 2008 survey results was reaffirmed
by a new NIH-wide committee. Revisiting the suitability of the NIH SBIR program Evaluation
Framework would be beneficial to confirm current consensus about performance aims, given the new
understanding of program performance generated by the NIH and NRC evaluations. At least these types
of reviews should be conducted.
• Performance measure suitability. Some current measures may not be the most suitable ones for
truly assessing attainment of a performance aim. Thus, this dimension of the measures should
also be examined.
• Performance measure focus. Assessing program performance should focus on measuring those
outcomes that indisputably relate to the effects of the SBIR award. Evaluation Framework
measures should be reviewed to ensure that they all meet this criterion.
• Performance benchmark level reasonableness. There are large disparities between some stated
performance index benchmark levels in the current Evaluation Framework and awardees’
performance. This can be interpreted as quite positive—the program more than exceeded its
expectations and selected the most talented and capable small businesses to support. However,
review of performance index benchmarks is warranted, given the much higher than required
levels of performance described by both the 2002 and 2008 surveys.
A committee of SBIR administrators from throughout the NIH could be convened to review the existing
framework prior to instituting additional performance assessments and data collection efforts. To broaden
the perspective of the group, it may be useful to ask professional evaluators—possibly including those
from other Federal agencies—to review and comment on the draft NIH Framework. The review could
address these issues:
• Does the current Evaluation Framework address all relevant performance questions? If not, what
should be added? What variables best measure success in those areas of performance?
Page 6-3
January 23, 2009
National Survey to Evaluate the NIH SBIR Program
Final Report
• Does the current Evaluation Framework address any performance questions that are no longer
key issues for the NIH? Which should be deleted?
• Do Evaluation Framework performance standards and indices accurately reflect current NIH
thinking about the degree of attainment that constitutes success? If not, how should they be
revised?
• Could alternate measurement approaches be more accurate in describing results?
6.2.2
Expand Program Monitoring Capabilities
The NIH SBIR program’s ability to evaluate and report on outcome attainment relies in part on its ability
to continuously monitor program performance. Several expansions to the program’s current monitoring
capabilities may well be worthwhile investments for NIH to consider as ways to leverage the investment
in the 2008 survey and other recent evaluations.
One priority issue is whether to continue the Program Outcomes Data System (PODS) beyond its current
end date of March 2009. PODS development was funded primarily by NIH evaluation funds, but now the
system must have other support for maintenance and ongoing operations. PODS provides ready access to
automated contact and outcome data from the 2002 survey, voluntary 2002 survey updates, and the 2008
survey. The NIH SBIR program has found PODS useful for monitoring and disseminating up-to-date
status and longitudinal data about awardees and award results. PODS is a system whose purpose and
architecture matches the need for regular monitoring and data collection on outcomes for all types of
Federal programs by providing up-to-date, usable, automated data.
If the decision is made to continue PODS, it may be useful to consider strategies for generating high
levels of participation in updates by SBIR awardees. Participation rates to update requests have been
lower than ideal, in part because Federal regulations dictate that participation must be voluntary. NIH
SBIR administrators may wish to explore additional options for improving their ability to collect a
standard data set for all awardees over time. They have already tested the option of having awardees
submit Phase II final progress reports online in a format that facilitates storing the data in PODS. Initial
attempts to do this indicate that awardees are enthusiastic about this submission method and that it may
well be a solution to ensuring participation by all awardees over time.
If monitoring capabilities are expanded, early on NIH OER will need to develop procedures for handling
multiple SBIR awards per awardee and analyzing their impact on product development. Both the 2002
and 2008 surveys randomly selected a single SBIR award and its product as the focus of each survey.
The subsequent updates followed up on just this single award per awardee. This is an idealized situation,
created to minimize respondent burden in the surveys.
Page 6-4
January 23, 2009
| File Type | application/pdf |
| File Modified | 0000-00-00 |
| File Created | 2017-09-26 |