2015-1-30-FINAL SUPPORTING STATEMENT REQUEST FOR NEW ENG IIP PROGRAM MONITORING DATA COLLECTIONS Part B

Section B

Introduction

B.1. Respondent Universe and Sampling Methods

The ENG IIP Program Monitoring Clearance’s goal is to count and describe the universe of NSF-funded ENG research and education projects in the IIP division. The statistical method employed in each collection is that of a census of all ENG-funded projects under the corresponding program/division/office for which the collection is being prepared. Data collection is expected to involve all awardees in the program.

The table below shows the total universe and sample size for each of the collections.

Table 4. Respondent Universe and Sample Size of ENG Program Monitoring Clearance Collections

Collection Title	Universe of Respondents	Sample Size
Grant Opportunities for Academic Liaison with Industry (GOALI)	200	200
Innovation Corps (I-Corps) Longitudinal Collection	800	800
Innovation Corps (I-Corps) Pre-Course Survey Questionnaire	150	150
Innovation Corps (I-Corps) Post-Course Survey Questionnaire	150	150
Partnerships For Innovation: Accelerating Innovation Research (PFI:AIR)	200	200
Partnerships For Innovation: building Innovation Capacity (PFI:BIC)	30	30
Small Business Innovation Research (SBIR)	1,100	1,100

B.2. Information Collection Procedures/Limitations of the Study

The data collections in this clearance are expected to use Web-based instruments but some could use interviews, either in person or by phone. Each respondent will provide answers once a year during the life of the award. Respondents post-award will be invited to report voluntarily up to four times over the course of 10 years after the award has expired.

ENG understands the limitations of the Program Monitoring Clearance, particularly in terms of using the data to determine program effectiveness. Data collected under this clearance are for monitoring purposes; evaluation studies are cleared under separate OMB requests. However, monitoring systems covered by this request will be explicitly identified as a source of data for independent program evaluations. ENG IIP Program Monitoring Clearance data are not used to determine the ultimate effectiveness of engineering research, but they are a key element in NSF-ENG’s efforts to manage its program portfolio, to report on agency activities and goals, and to lay the groundwork for future evaluations.

B.2.1. Statistical Methodology for Stratification and Sample Selection

Each of the collections in this clearance request is a census, in which the sample size is the universe. Details on the size of the universe in each collection are included in the burden estimate and in section B.1 above. A census approach to data collection is critical for monitoring of scientific research, particularly fundamental research, due to the uniqueness of each project. The merit review process for each program elicits unique and transformative projects in their contribution and methods. Each project asks a different research question and uses different experimental and theoretical approaches. As such, would be impractical to consider sampling methods that will yield a representative population of the universe of NSF funded research awards.

B.2.2. Estimation Procedure

Not applicable

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

Not applicable

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

Not applicable

B.2.5. Justification for Data Collection Cycles

In post-award monitoring systems, NSF-ENG endeavors to collect data on indicators of outcomes and impacts of investments in research that are unlikely to be realized during the course of the award. These data may include indicators such as publications, patents, and licensing activities, student career choices after participating in the funded research, and technologies developed from discoveries made by fundamental research, for example. In many cases, particularly in the case of fundamental research, the most important outcomes of research investments are not expected to be realized for several years after the award has ended, due to the inherent time lag in the transition from discovery to application of research findings. As such, we propose to collect data on these outcomes and impacts of our research investments for up to 10 years post-award. These collections for programs in IIP which are often focused on translation or commercialization of research findings, the important indicators are expected to appear sooner after the award ends. However, due to the burden on the PIs and our expectation that certain outcomes and impacts are more likely to occur at less frequent intervals post-award, in most cases we propose to collect data at 1-year, 3-year, and 5-year intervals post-award, with a 4^th data point collected at 10 years post-award for some programs.

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse 

All potential collections during the life of the award included in this clearance may become part of the reporting required of awardees for specific solicitations or programs, pending this corresponding OMB clearance. In those specific cases, a high response rate is expected. The pre and post survey questionnaires for the I-Corps program will be implemented before and after the teams of grantees undergo training. A high-response rate is also expected in this case.

For post-award monitoring, participation is entirely voluntary. Although there is no penalty for non-participation with data collection requests outside of the life of the award, many respondents are eager to communicate their achievements to NSF program staff in general, so we foresee no obstacles to achieving a high response rate even outside of the life of the award. The table below shows the expected response rates for each of the individual collections based on NSF’s experience with other monitoring systems.

The voluntary nature of the response will be clearly communicated to respondents in each instance.

Table 5. Expected Response Rates for ENG Program Monitoring Clearance Collections

Collection Title	Expected Response Rate
Grant Opportunities for Academic Liaison with Industry (GOALI)	80%
Innovation Corps (I-Corps) Longitudinal Collection	80%
Innovation Corps (I-Corps) Pre-Course Survey Questionnaire	95%
Innovation Corps (I-Corps) Post-Course Survey Questionnaire	95%
Partnerships For Innovation: Accelerating Innovation Research (PFI:AIR)	80%
Partnerships For Innovation: building Innovation Capacity (PFI:BIC)	80%
Small Business Innovation Research (SBIR)	80%

For web-based collection systems, a series of e-mail messages and phone calls, including introductory emails alerting the respondent to the data that will be collected will also be used to follow up with respondents.

B.4. Tests of Procedures or Methods

Several question items included in these collections use pretested questions, some of them have already received OMB clearance as a part of other clearance requests, or have been used extensively in well-established and nationally recognized surveys such as the Kauffman Innovation Survey [13] or by NCIIA. Other test methods used to improve the questions in the ENG IIP Program Monitoring Clearance include feedback from PIs, both as data are collected and during meetings and conferences; review by NSF staff; and testing performed by the data collection system developers. These monitoring collections are based on data collection methods currently used by other NSF groups, and many of the items and response categories follow formats that are already in place.

B.5. Names and Telephone Numbers of Individuals Consulted

The following individuals were consulted on the ENG Program Monitoring Clearance:

Table 6. Individuals Consulted on ENG IIP Program Monitoring Clearance request

Name	Title	NSF ENG Unit
George Antos	Program Director	CBET
Matt Carnavos	Program Analyst	CMMI
Joanne Culbertson	Program Manager for Integrative Activities	CMMI
Lindsay D’Ambrosio	Science Assistant	IIP
Shannon Dunphy	Science Assistant	OAD
Garie Fordyce	Program Manager	EFRI
Shannon Griswold	AAAS Science & Technology Policy Fellow	OAD
Barbara Kenny	Program Director	IIP
Alexandra Medina-Borja	Director, Program Evaluation & Assessment; Interim Head, Evaluation	OAD/ENG; OIIA/OD
Gracie Narcho	Staff Associate	IIP
Sarah Naylor	AAAS Science & Technology Policy Fellow	OAD/ENG
Joy Pauschke	Program Director	CMMI
Angela Shartrand	Director, Research and Evaluation	VentureWell
Laurie Stepanek	AAAS Science & Technology Policy Fellow	EEC
Bevlee Watford	Program Director/Cluster Leader	EEC
Grace Yuan	Associate Program Director	OAD

 

B.6. Contact Information for Individuals Responsible for Data Collections

Alexandra Medina-Borja

Program Officer in Evaluation & Assessment, Engineering Directorate

Interim Head of Evaluation & Assessment, Office of Integrated and International Activities

National Science Foundation, Arlington, VA 22230

703-292-7557

[email protected]

Bibliography

[1] National Science Foundation, “How We Work.”.

[2] National Science Foundation, “NSF at a Glance.”

[3] "About IIP" Available: http://www.nsf.gov/eng/iip/about.jsp. [Accessed: 02-December-2014].

[4] A. Medina-Borja and OAD, “NSF Evaluation and Assessment: A plan for the Directorate for Engineering,” May 2012.

[5] Public Law 111–352, vol. 31. 2011.

[6] United States Code, 2006 Edition, Supplement 5, Title 5 - GOVERNMENT ORGANIZATION AND EMPLOYEES, vol. 5. .

[7] United States Code, 2006 Edition, Supplement 5, Title 39 - POSTAL SERVICE, vol. 39. pp. 2801–2805.

[8] OMB, “Memorandum for the Senior Executive Service.” 14-Sep-2010.

[9] OMB-M-10-24, “Memorandum for Executive Departments and Agencies.” 25-Jun-2010.

[10] “NSF Strategic Plan for 2014 – 2018: Investing in Science, Engineering, and Education for the Nation’s Future.” National Science Foundation, Mar-2014.

[11] OMB M-10-32, “Memorandum for the Heads of Executive Departments and Agencies.” 29-Jul-2010.

[12] “Losing Focus: The Annual Report on the Economic Status of the Profession, 2013-14.” [Online]. Available: http://www.aaup.org/reports-publications/2013-14salarysurvey. [Accessed: 15-May-2014].

[13] “Kauffman Firm Survey - Annotated Questionnaire.” Ewing Marion Kauffman Foundation.