3145-NEW I-CORPS, Part A 2017

National Science Foundation

The Office of Integrative Activities, Evaluation and Assessment Capabilities

Marilia Mochel, Project Manager

Manhattan Strategy Group

8120 Woodmont Avenue, Suite 850

Bethesda, MD 20814

301-828-1512

Email: [email protected]

Table of Contents

SUPPORTING STATEMENT 2

A. Justification 3

B. Collections of Information Employing Statistical Methods 16

SUPPORTING STATEMENT

Evaluation of the National Science Foundation’s Innovation Corps Teams Program

OMB Control Number 3145-NEW

Introduction

In fiscal year 2011, NSF created the I-Corps Teams Program to build a national innovation ecosystem by accelerating innovation among identified NSF-funded researchers. The I-Corps Teams Program provides training, mentoring, and a small grant to help project teams determine and advance the readiness of their technology products for transition to commercialization. By design, I-Corps Teams are composed of a principal investigator (PI), an entrepreneurial lead (EL), and a mentor.

As part of the program, the I-Corps Teams receive entrepreneurial training and ongoing support for the 6-month duration of the grant. This support facilitates each team’s entrepreneurial efforts. The grant requires I-Corps awardees to participate in an intensive immersion training on entrepreneurship (a 3-day opening workshop, 5 weeks of activities with online classes, and a 2-day final workshop). The training follows a structured approach to give team members hands-on experience in transferring knowledge into commercial products. NSF’s program manager for I-Corps monitors I-Corps Teams’ progress, as they are expected to hit milestones for the duration of the training and continue progress throughout the 6-month grant period. Additionally, NSF’s program manager monitors I-Corps Teams’ project outcomes after the end of the grant period, with longitudinal surveys conducted with I-Corps Teams at two intervals, Time 1, one year or more after the training, and Time 2, a similar interval after Time 1.

While NSF regularly collects data from its grantees through project annual reports, extant data on most NSF-funded projects and programs focus on the scientific aspects of research and not on commercialization prospects and results. Thus, the examination of program goals to help create a culture of innovation and entrepreneurship among academics and successful commercialization of products can only be accomplished with the new data collection proposed here, including the Time 2 survey. While Time 1 survey results are available for the earlier I-Corps cohorts, both changes in culture and outcomes from entrepreneurship take time so the Time 1 survey results by themselves are of limited value. Thus, from its conceptualization, NSF understood the great need for longitudinal survey responses, such as those at Time 2. In addition, at the conceptualization of the I-Corps Teams program, the independent evaluation of the program had not yet been foreseen. In 2013, NSF commissioned an independent evaluation of the I-Corps Teams program, which when completed advocated that a quasi-experimental design, with a comparison group of non-I-Corps PIs, would be a feasible.

A. Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

This ICR seeks revisions to a previously cleared survey instrument (OMB #3145-0238 – a monitoring collection of surveys, from which this will be pulled and aggregated with a new survey), the National Science Foundation (NSF) Engineering I-Corps Survey of Longitudinal Outcomes (Time 2). Specifically, this ICR proposes to:

1. modify, delete, and add survey instrument items;

2. extend surveyed groups to include a comparable group of NSF principal investigators who did not go through the I-Corps program; and,

3. add an additional data collection for a case study through semi-structured in-depth interviews, in person and over the phone, with select NSF I-Corps Teams and a comparable non-I-Corps group who pursued commercialization of NSF-funded research independently.

This ICR also covers revisions to the Time 1 longitudinal survey instrument to make it more suitable for the planned data collection for program monitoring and evaluation purposes at Time 2 for NSF grantee teams. As teams that persisted two to three years after the course were hypothesized to be farther along in their commercialization efforts than at Time 1, NSF realized the instrument needed to be revised in order to capture these milestones. For example, more specific questions were developed around private financing to better understand the type and amount of non-government financing received. Similarly, given the passage of time, NSF recognized that it was no longer appropriate to ask certain questions about the course’s influence on participants. As the instrument revisions include the removal of items that appear in the Time 1 survey that no longer are needed (i.e, short-term influence of course on career). The overall length/burden of the instrument remains unchanged.

The inclusion of non-I-Corps principal investigators to participate in the survey and of PIs and their collaborators in the case study interviews is part of the one-time evaluation study of the I-Corps Teams program, which will assess I-Corps outcomes and impacts. The evaluation seeks to understand the impact of the I-Corps Teams Program on team members and the academic culture. Specifically, the evaluation will identify a group of comparable non-I-Corps research teams using extant NSF data on PIs and compare entrepreneurial outcomes among PIs who participated in I-Corps to those who did not participate in I-Corps. The case study interviews will shed light into team dynamics as well as the challenges and opportunities PIs face when pursuing commercialization of their research.

NSF has engaged the Manhattan Strategy Group (MSG) to conduct the evaluation of the I-Corps Teams Program. The evaluation will cover I-Corps Teams starting in fiscal year 2011. The evaluation seeks to answer the following research questions:

1. Have I-Corps Teams participants become more entrepreneurially competent after their participation in I-Corps?

2. Has the I-Corps training program led to an increase in the translation of fundamental research into the commercial space more than for those who have not participated in the program?

3. Have viable technologies, processes, products, or services developed by I-Corps participants (either during the I-Corps award or after):

1. Reached the market?

2. Entered the process of translation?

3. Been applied to other developments?

4. Are there any other significant spillover effects not initially planned in the I-Corps logic model that can be considered unexpected positive outcomes of the program?

5. Can significant traits be identified as predictors or correlated to positive outcomes in research questions 1 through 4?

To address these evaluation questions, this ICR seeks approval for data collection for the following: (1) modification to the Time 2 Longitudinal Survey of Outcomes for NSF I-Corps grantee teams and for use with non-I-Corps PIs and (2) case study in-depth interviews of select I-Corps and non-I-Corps NSF grantees and their teams regarding their commercialization efforts and experiences. Details on each component of the data collection appear below.

1. I-Corps Longitudinal Survey of Outcomes (Time 2)

This ICR seeks to clear an omnibus survey instrument for I-Corps grantees and select non-I-Corps principal investigators. This omnibus instrument is based on a previously cleared instrument for a Time 1 longitudinal survey of NSF I-Corps grantees. It includes a new module directed to those with a focus on health and medical technology fields and changes in items to accommodate non-I-Corps principal investigators. The omnibus survey instrument, indicating respondent groups appears in Appendix A. Appendices B and C show the instruments as they read for I-Corps grantees and non-I-Corps PIs, respectively.

1. Survey of I-Corps Grantee Teams

The longitudinal survey of I-Corps (Time 2) will be administered to I-Corps PIs, ELs and mentors who completed the Time 1 survey and who at the Time 1 response consented to be contacted for the Time 2. Based on the timing of certain program outcomes, the Time 1 survey is administered an average of 13 months’ post-course completion and the Time 2 survey is administered an average of 36 months post-course completion. Pending unforeseen circumstances, the Time 2 survey will be administered according to this schedule for the duration of the I-Corps Teams program. Approximately 12 to 15 cohorts are expected to be included per administration.

2. Survey of Non-I-Corps Teams

The survey instrument for non-I-Corps grantees serves first, to identify non-I-Corps NSF-funded projects comparable to the I-Corps Teams Program projects, and second, to survey the comparable projects’ PIs on their efforts to commercialize their research and the outcomes of these efforts. Thus, it includes two modules: The first is a screener to identify comparable non-I-Corps PIs similar to I-Corps PIs in their potential and motivation to commercialize research. PIs who pass the screener will then be asked to complete a second module, a version of the Time 2 longitudinal survey of I-Corps PIs that replaces references to I-Corps with references to “your NSF project,” defined as the NSF project that spawned their commercialization efforts.

2. Case Study In-Depth Interview Protocol for I-Corps Teams and Comparable Non-I-Corps PIs

As part of the case study, we will interview PIs, close collaborators (mentors and ELs for I-Corps projects), graduate or postdoctoral students, colleagues, and technology transfer offices or similar at the home institutions that have been involved in research and/or commercialization efforts associated with the I-Corps or comparable non-I-Corps projects. The discussion guides for each respondent type appear in Appendix E.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The longitudinal survey of I-Corps Teams participants (Time 2) in not a new data collection. It had been cleared under OMB Approval No. 3145-0238. This ICR requests a revision of items in the survey.

The longitudinal survey enables NSF program officers to longitudinally monitor outputs and outcomes given the unique goals and purpose of the I-Corps program. This allows NSF to use appropriate and accurate evidence-based management of the program and to determine whether its specific goals are being met.

All I-Corps participants who completed the I-Corps training are asked to complete the longitudinal survey at Time 1 (approximately 1 year after completion of the training) for which OMB has already provided clearance (OMB Approval No. 3145-0238). Since the Time 2 instrument now differs from that used in Time 1, it is a partially new ICR. Its purposes are the same as those for the Time 1 longitudinal survey: to monitor program outputs and outcomes, but for an additional period of time, when further information regarding whether the goals of the program have been met is more likely to be available.

The one-time survey of non-I-Corps PIs is a new data collection. It uses an adapted version of the time longitudinal outcomes survey (Time 2) to enable a comparison between the outcomes of I-Corps participants and a comparable group of non-I-Corps participants pursuing commercialization without having received the I-Corps training. Similarly, the in-depth interviews with select NSF I-Corps Teams participants and non-Corps PIs are a new data collection.

NSF will use the results of the survey of non-I-Corps PIs and the in-depth interviews with I-Corps and comparable non-I-Corps teams to evaluate the outcomes and impact of the I-Corps Teams Program. This comparison relies on data from the longitudinal survey from both I-Corps participants (Time 2) and non-I-Corps participants. This will address the research questions by enabling the researchers to compare various project outcome metrics between I-Corps and non-I-Corps PIs on their commercialization experiences, business networks.

Additionally, the case study interviews will enable NSF to assess whether the program has achieved its goals related to the academic environment:

• Fostering a culture of innovation in academia;

• Disseminating entrepreneurial knowledge among academics;

• Producing commercialization outcomes attributable to the I-Corps program intervention; and

• Identifying other spillover effects of the program.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

NSF will conduct the survey using an online, Internet-accessed survey software. The Internet delivery system will allow respondents to take the survey at any time within the window of the data collection period. The online survey allows respondents to save responses and return to the survey later to finish, giving them convenience to choose the best time to complete the instrument. It also allows automated flow of the instrument based on skip patterns or questions dependent upon responses to previous questions. This is critical, as the screener can automatically assess whether the respondent should proceed to the second module of the survey, which will avoid the confusion associated with two separate surveys.

NSF will send an advance e-mail to all participants in the sample to inform them that they will be receiving the survey via e-mail from the evaluation team. The e-mail will detail the purpose of the survey and encourage participation. NSF will also send an e-mail midway through the data collection to remind participants to complete their surveys and thanking participants for their responses. The evaluation team will use e-mail for all invitations to complete the survey. We will email non-respondents weekly reminders for the duration of the data collection period. Completion rates will be tracked in real time.

Using e-mail for all communications keeps the burden on respondents to a minimum. Since all respondents have received NSF grants to support their research, and many will seek future support from NSF, their motivation to respond is high. Thus, we strive to achieve response rates of at least 80 percent. However, achieving such response rate level may not be possible. The Time 1 survey achieved an overall 66 percent response rate, across respondent groups, with a 94 percent team response rate (at least one response from one of the 3 I-Corps team members). The Time 2 response rate is dependent on participation in and agreement to be contacted for Time 2 at Time 1. We expect to have again a very high team response rate, and given previous survey participation and commitment, a response rate higher than at Time 1. If we do not reach the 80 percent team response rate, we will conduct a non-response bias analysis to assess whether respondent and non-respondents have different characteristics in areas that may impact the study’s findings.

For the interviews with I-Corps and non-I-Corps PIs and their teams, the evaluation team will conduct in-depth discussions, with participants’ consent by telephone or in-person and record the interviews, and produce transcripts of the recordings. The team will use the transcripts for accurate recall and efficient content analysis of the interviews.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

NSF has made every effort to identify duplication. NSF conducted a feasibility study of the I-Corps Teams Program in 2012–2014. The feasibility study found that no extant data sources could or would inform outcome assessments or evaluation of the I-Corps Teams Program.

The I-Corps Teams evaluation reviewed and will analyze extant administrative data on I-Corps teams to contribute to the evaluation. The survey instrument has revised questions on government funding (e.g., SBIR funding) to reduce the burden on respondents. Such information is publicly available using Federal agencies’ databases. In addition, the team will use data already collected by the training provider on I-Corps PIs to answer some of the research questions. The data currently collected on I-Corps PIs includes a pre-and a post-course survey, administered immediately before and after the I-Corps intensive workshop training. It also includes the longitudinal survey of I-Corps PIs that tracks team progress annually, the instrument for which this ICR seeks to modify. As a result, the only new data collection refers to the one-time survey of non-I-Corps PIs and the in-depth interviews.

An assessment of the commercialization experiences, business networks, and academic environment nurtured by the I-Corps Teams Program and comparable non-I-Corps projects cannot rely on existing data. For this reason, the evaluation is seeking approval for the case study of I-Corps and comparable non-I-Corps project teams.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

Some of the non-I-Corps PIs and I-Corps Teams participants surveyed and included in the in-depth interviews may have established small businesses in addition to continuing their academic research. About 40 percent of NSF I-Corps teams and 100% of NIH I-Corps teams have established businesses and because one of the outcomes of the I-Corps supports is precisely the creation of businesses, to fully monitor and evaluate program performance it is necessary to survey small businesses.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The data collection efforts proposed in this ICR will affect most respondents only once. Ten I-Corps and 10 non-I-Corps comparable PIs could receive the survey and be selected to take part on the case study. Each the two types of data collection will occur only once.

Without the data collection, NSF will not be able to assess whether the I-Corps Teams Program achieved it goals in promoting entrepreneurship and innovation among academics and successful research translation into commercial products. NSF would not have a clear understanding of program outcomes and impact. It would not be able to determine the effectiveness of the I-Corps Teams Program. Each data collection will occur only once, except for the longitudinal survey, which is administered twice to gather information on changes in outcomes over time. The I-Corps Teams program logic model estimate short-, medium and long-term outcomes (see Appendix F). These outcomes are not expected to unfold until well after each grantee award expires. This creates the need for the longitudinal data collection. A one-time administration of the longitudinal survey, previously approved by OMB and already administered to most I-Corps Teams, would fail to capture sufficient data to assess program outcomes.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

* requiring respondents to report information to the agency more often than quarterly;

* requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

* requiring respondents to submit more than an original and two copies of any document;

* requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;

* in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

* requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

* that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

* requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

None of the above special circumstances apply. While some questions ask respondents about patents, licenses, and revenues in an overall sense, they do not request details. Furthermore, none of these responses are required.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported. Consultation with representatives of those from whom information is to be obtained or those who must compile records should occur at least once every 3 years - even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

The 60-day notice to solicit public comments was published in the Federal Register on December 4, 2015. It was revised and republished on December 22, 2016 due to unforeseen delays in project as well as changes in the burden estimates. No consultation occurred with persons outside NSF, who are unlikely to be familiar with NSF grantees. A copy of each notice and the original notice that cleared the Time 2 longitudinal survey appear in Appendix G. No comments were received.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payments, gifts, or incentives will be provided in this data collection.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

Responses to this data collection are voluntary. We will ensure that respondents are fully informed about the purpose of this study and neither the names of respondents nor their affiliations will be included in any reports. We will maintain completed surveys and interview transcripts in a password-protected database. We will ensure that comments made through the survey and interviews will not be attributed to specific individuals in the report or any other publications resulting from this project.

As we conduct the in-depth interviews with PIs and their teams, we will ask if they are willing to have their experience highlighted in the evaluation report. With their informed consent, we may disclose their affiliation or describe their product/technology.

All researchers working with the data will take the following precautions to ensure the privacy of all data collected:

1. All staff on the project will be instructed in the privacy requirements of the study and will sign statements affirming their obligation to maintain privacy;

2. Data files for analysis will contain no personal identifiers for program participants;

3. Analysis and publication of study findings for the participant survey will be in terms of aggregated statistics only; and

4. Any quotations from responses to in-depth interviews used in public reporting will be reviewed and edited as needed to ensure that the identity of the respondent cannot be ascertained.

Reports prepared for the evaluation will summarize findings across the sample and will not associate survey responses with specific individuals. We will not share any personally identifiable information with anyone outside the study team.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

The survey instrument asks demographic information - gender, race, disability, military service, age on a voluntary basis. These questions are being asked to enable subgroup analysis to assess effectiveness of the program in promoting entrepreneurship in underserved groups.

12. Provide estimates of the hour burden of the collection of information. The statement should:

* Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices. * If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens.

* Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included under “Annual Cost to Federal Government.”

Exhibit A-1 below details the estimated time burden on individuals in the two data collection efforts presented in this ICR: (1) the survey of non-I-Corps PIs, and (2) the case study in-depth interviews. We estimate only a small portion of non-I-Corps surveyed will pass the screener survey (Module 1) since the module screens for PIs with research projects with potential for commercialization. Of the sample of 7,897 non-I-Corps PIs who will receive the survey invitation, we estimate 80% will complete the brief screener survey, and of those 6,318 PIs, 10% will have pursued the perceived commercialization potential of their projects for a total 631 responses for the second module. A total 2,000 I-Corps team members will receive an invitation to participate in the survey. Exhibit A-1 provides survey burden estimates using an expected 80 percent response rate.

Exhibit A-1 also shows the burden associated with the case study, which will cover 20 research teams, with approximately 8 members per team. We have estimated 8 hours of total burden on average for each project team. These may take the form of individual interviews or group interviews. Group interviews take into account burden on each participating individual.

Exhibit A-1. Estimated Hour and Annual Cost Response Burden

Data Collection Activities	Number of Respondents	Individual Burden (in minutes)	Annual Hourly Burden (in hours)	Cost
Survey of Non-I-Corps PIs
Respondents Screener	6318	5	527	$29,484.00
Respondents Full Survey	631	15	158	$8,834.00
I-Corps Teams
Respondents	1,600	15	400	$22,400.00
In-Depth Interviews
Respondents	160	60	160	$8,960.00
Total			1,245	$69,678.00

Hourly wages calculated using the annual salary of $116,802 (80 percentile of Associate Professor rank) for an hourly wage of $56. Based on Annual report on the economic status of the profession, 2015-16 Survey report Table 3 Retrieved from: https://www.aaup.org/sites/default/files/SurveyReportTablesMA16.pdf

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden already reflected on the burden worksheet).

* The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services component. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life of capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling and testing equipment; and record storage facilities.

* If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10), utilize the 60-day pre-OMB submission public comment process and use existing economic or regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate.

* Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices.

We do not anticipate any other costs to respondents or record keepers.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information.

Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

Exhibit A-2 presents the overall cost of this research to the Federal Government. The total costs are shown at $1,117,166.34. The overall costs to the Federal Government presented below cover costs for survey design, including sampling plan development and instrument development, data collection activities, data analysis, and reporting. The contract includes a base year only.

Exhibit A-2. Overall Cost to the Federal Government

Category	Costs Base Year
Study Design Including Instruments	$69,329.80
OMB Clearance	$31,740.84
Survey Data Collection/Analysis	$609,936.50
Case Studies Data Collection/Analysis	$270,012.52
Reporting	$136,146.68
Total	$1,117,166.34

15. Explain the reasons for any program changes or adjustments reported on the burden worksheet.

There is no change in the expected burden in the Time 1 longitudinal survey of I-Corps teams, which had previously been cleared. The remaining components of this ICR reported on the burden worksheet refer to the new data collections.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Exhibit A-3 shows the schedule of deliverables associated with the evaluation study, including the survey of I-Corps and non-I-Corps participants and the case study interviews.

Exhibit A-3. Deliverables and Due Dates

	Deliverables	Due Date
1	Draft Evaluation Report	18 months after award
2	Final Evaluation Report and Database	20 months after award

The data collection efforts included under this request will be administered by a third-party grantee or contractor that will deliver (1) analytical reports, (2) the raw data from the collections, or (3) both. Third parties are contractually forbidden from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF; i.e., NSF typically is the exclusive publisher of the information collected by the collections. Often it is only after seeing the quality of the information the collection delivers that NSF decides the format (raw or analytical) and manner (in its Online Document System or simply a page on the NSF website) in which to publish.

Additionally, the data collected from I-Corps grantees will not be used for publications as a stand-alone product, because the data will be used as an input to how NSF manages, documents, evaluates, and measures its performance as an agency. At this time, NSF plans to produce a summary or descriptive report every year after completion of the data collections.

Analysis of Survey Data

For all survey items, we will use descriptive statistics to summarize responses. We will use tests for significant differences (e.g., t-test, chi-square, and ANOVA) to compare responses between meaningful subgroups of respondents. Where appropriate, we will utilize factor analysis to explore and assess unobserved variables based on the correlation among observed variables.

With comparison groups, we will also model the program effects on outcomes of interest beyond descriptive statistics to draw more conclusive inferences about effects of the I-Corps program. Specifically, given the nested structure of data at the project and the individual levels, we may apply additional analysis methods, such as hierarchical linear models (HLM) or other mixed models to estimate program effects. Since project team members are nested within projects, the program outcomes are subject to the impact of factors at the project and individual levels. The HLM procedure allows formal partitioning of the variance of the outcome measures into two levels and estimates between-project variance as a proportion of the total variance. We may use the procedure to confirm that the model fit is good enough and the estimates generated from individual-level data using the ordinary least square (OLS) procedure are valid and precise. We may specify and estimate the cross-level effects (also known as interaction effects) between project characteristics and individual characteristics on the program outcomes: for example, the number of Small Business Innovation Research (SBIR) proposals submitted, the number and amount of non-governmental investments received, and the number of jobs created.

Formally, a simple mixed model would be, at level 1 (individual level):

(1)

where Y_ij denotes program outcomes for team member i in project j, β_oj is the expected error of individual i in project j, β_mj is the effect of a vector of individual-level covariates X_ij, e.g., a PI’s funding lineage, and r_ij is the outcome variance that is not accounted for by the model.

At level 2 (project level), we model the effects of individual-level covariates on program outcomes:

(2.1)

(2.2)

where 𝞬₀₀ is the mean outcome value for all team members; 𝞬₀₁ is the mean individual difference associated with W_j, project-level characteristics; depending on the specific individual-level covariates, 𝞬_m0 may represent the average effects of individual-level characteristics on program outcomes (e.g., average effects of a PI’s funding lineage on SBIR submission), and 𝞬_m1 may represent the average difference in such effects that is associated with project-level characteristics; and W_j; and µ_0j and µ_mj are the unique effects of project j on the mean program outcomes (β_0j) and the slope (β_mj), respectively, holding W_j constant.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

We will display OMB approval information on the data collection, including expiration date, at the beginning of all data collection instruments. The following statement will be attached to the data collection instrument:

“The OMB control number for this project is 3145-NEW. Public reporting burden for this collection of information is estimated to average 15 minutes per respondent, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspects of this collection of information, including suggestions for reducing this burden, to Suzanne H. Plimpton, Reports Clearance Officer, National Science Foundation, 4201 Wilson Boulevard, Suite 1265, Arlington, Virginia 22230 or send e-mail to [email protected].”

18. Explain each exception to the topics of the certification statement identified in “Certification for Paperwork Reduction Act Submissions.”

There are no exceptions to the certification statement.