NASA Institutional Research Capacity Evaluation Survey and Interview Protocol

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number 2700-0159, expiration 09/30/2024

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION:

NASA Institutional Research Capacity (IRC) Evaluation Survey and Interview Protocol

2. TYPE OF COLLECTION:

þ	Attitude/Behavior Scale
o	Baseline Survey
o	Cognitive Interview Protocol
o	Consent Form
þ	Focus Group Protocol
o	Follow-up Survey
þ	Instructions
þ	Satisfaction Survey
o	Usability Protocol

GENERAL OVERVIEW: NASA Science, Technology, Engineering, and Mathematics (STEM) Engagement is comprised of a broad and diverse set of programs, projects, activities and products developed and implemented by HQ functional Offices, Mission Directorates and Centers. These investments are designed to attract, engage, and educate students, and to support educators, and educational institutions. NASA’s Office of STEM Engagement (OSTEM) delivers participatory, experiential learning and STEM challenge activities for young Americans and educators to learn and succeed. NASA STEM Engagement seeks to:

• Create unique opportunities for students and the public to contribute to NASA’s work in exploration and discovery.

• Build a diverse future STEM workforce by engaging students in authentic learning experiences with NASA people, content, and facilities.

• Strengthen public understanding by enabling powerful connections to NASA’s mission and work.

To achieve these goals, NASA makes vital investments toward building a future diverse STEM workforce across its portfolio of projects including the NASA Established Program to Stimulate Competitive Research (EPSCoR) program. The EPSCoR program is directed at those jurisdictions that have not participated equably in competitive aerospace and aerospace-related research activities. Twenty-five states, the Commonwealth of Puerto Rico, the U.S. Virgin Islands and Guam currently participate. Five federal agencies, including NASA, conduct EPSCoR programs. EPSCoR establishes partnerships with government, higher education and industry that are designed to effect lasting improvements in a state or region's research infrastructure, research and development (R&D) capacity and its national R&D competitiveness. The goal of NASA EPSCoR is to provide seed funding that will enable jurisdictions to develop an academic research enterprise directed toward long-term, self-sustaining, nationally competitive capabilities in aerospace and aerospace-related research.

The specific EPSCoR program objectives include:

• Contribute to and promote the development of research infrastructure in areas of strategic importance to the NASA mission.

• Improve the capabilities of the jurisdictions to gain support from sources outside the NASA EPSCoR program.

• Develop partnerships between NASA research assets, industry, and EPSCoR jurisdictions’ academic institutions.

• Contribute to the overall research infrastructure, science, and technology capabilities, higher education, and/or economic development of the jurisdiction.

INTRODUCTION AND PURPOSE: The purpose of this data collection effort is to support NASA Office of STEM Engagement and EPSCoR goals and evidence building directly related to the Learning Agenda through an Institutional Research Capacity (IRC) Outcome Assessment. The purpose of this pilot outcome assessment is to help NASA understand the outcomes of IRC building efforts from NASA EPSCoR grantees and awardees.

This data information collection request includes one survey instrument and an interview protocol that will be used to collect data from NASA EPSCoR stakeholders (i.e., scientist and researchers ). The goal of this evaluation is twofold: to collect stakeholders’ perceptions regarding institutional research capacity enabled from EPSCoR funding, and to collect cross-project metrics that contribute to NASA’s Learning Agenda and ability to answer associated learning questions. Specifically, this survey and interview protocol will address the NASA OSTEM Learning Question 1: How can NASA STEM Engagement develop cross-project metrics that support internal and external contributions to STEM Engagement goals and objectives?

3. RESEARCH DESIGN OVERVIEW: NASA’s work in STEM Engagement is focused on serving students. NASA EPSCoR provides support via competitive opportunities and awards to jurisdictions that have not participated equably in competitive aerospace and aerospace-related research activities. The proposed survey instrument and interview protocol to be used in this study have been created based on the findings of the FY 2022 Institutional Research Capacity Evaluation and the FY 2022 EPSCoR Historical Analysis. The FY22 Institutional Research Capacity Evaluation provided a baseline understanding of how NASA MUREP, Space Grant, and EPSCoR awardees (Principal Investigators, Co-Principal Investigators, and Activity Managers) engaged in evidence-based practices to build institutional research capacity through interviews with nine awardees. Across all three programs, awardees were focused on collaboration and building partnerships, utilizing funding for equipment and resources, and sustaining their research efforts. Additionally, awardees of Space Grant and MUREP developed new coursework and curriculum, created internships, fellowships, and scholarships, engaged students in research and hands-on experiences, and engaged faculty in professional development and training as methods for building research capacity.

The FY22 EPSCoR Historical Analysis analyzed the key metrics in the most recent Research Performance Progress Reports. This study reviewed 93 EPSCoR projects across the 28 jurisdictions. The metrics identified key metrics regarding products of interest to the EPSCoR program such as: publications and conference papers, talks, presentations, websites, new coursework, and inventions/patents and licenses. In addition, the study identified key metrics regarding participants and collaborating organizations amongst the EPSCoR projects (e.g., university research team members, NASA collaborators, industry collaborators, and new non-EPSCoR grant funding).

The survey instrument and interview protocol have been developed based on these findings to capture the broader perceptions regarding institutional research capacity enabled from EPSCoR funding. Expanded data collection through this survey will expand the data collection to a broader set of awardees, enabling a deeper understanding of how NASA STEM Engagement programs build the institutional research capacity of awarded institutions. The proposed instrument will used in ongoing program evaluation by NASA. This study falls under the category of a program evaluation, and will be guided by three evaluation questions for the approach and design of this study. Evaluation questions are presented in Figure 1 below.

Evaluation Question 1

How has participating in the EPSCoR project developed awardees' institutional research capacity?

Evaluation Question #2

In what ways do the metrics of the four components capture EPSCoR awardees’ institutional research capacity?

Figure 1. Evaluation Questions

The survey will be distributed to all of the Principal Investigators (PIs)/Directors of the NASA EPSCoR Program Jurisdictions (28) in April 2023 for administration to Co-PIs/Science PIs/Co-Science PI/Other Researchers (approximately 300). Although participation in the survey is voluntary, it will provide useful customer input to address programmatic decision making and assess IRC outcomes of NASA EPSCoR funding. Additionally, up to 50 interviews will be conducted virtually with respondents from the survey that volunteers to participate in an interview. Each interview will be approximately forty minutes long.

Construct survey item analysis. The survey instruments to be used in this study have been created based on the research questions proposed. Survey items have been created specifically for this evaluation. Survey questions were developed with the intent to provide NASA with information regarding how and to what extent NASA CONNECTS is:

• Increasing the reach of available NASA STEM resources and opportunities to its members

• Increasing the efficiency of use of NASA STEM resources and opportunities for teachers

• Including features to enhance collaboration between educators

• Providing relevant content

Interview Protocol. Interview questions have also been created based on the research questions and allow for deeper understanding of responses to survey questions.

4. TIMELINE: Data under this clearance will be collected in April 2023 - March 2024. This request includes one survey instrument and one interview protocol that will be used to collect data from Co-PIs/Science PIs/Co-Science PI/Other Researchers from the EPSCoR Program.

5. SAMPLING STRATEGY: The survey will be distributed to all of the Principal Investigators (PIs)/Directors of the NASA EPSCoR Program Jurisdictions (28) in April 2023 for administration to Co-PIs/Science PIs/Co-Science PI/Other Researchers (approximately 300). Although participation in the survey is voluntary, it will provide useful customer input to address programmatic decision making and assess IRC outcomes of NASA EPSCoR funding. Additionally, up to 50 interviews will be conducted virtually with respondents from the survey that volunteers to participate in an interview. Each interview will be approximately forty minutes long.

Table 1. Calculation chart to determine statistically relevant number of respondents

Data Collection Source	(N) Population Estimate	(A) Sampling Error +/- 5% (.05)	(Z) Confidence Level 95%/ Alpha 0.05	(P) *Variability (based on consistency of intervention administration) 50%	Base Sample Size	Response Rate	(n) Number of Respondents
NASA EPSCoR Co-PIs/Science PIs/Co-Science PI/Other Researchers (Survey)	300	N/A	N/A	N/A	300	N/A	300
NASA EPSCoR Co-PIs/Science PIs/Co-Science PI/Other Researchers (Focus Group)	50	N/A	N/A	N/A	50	N/A	50
TOTAL							350

6. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
NASA EPSCoR Co-PIs/Science PIs/Co-Science PI/Other Researchers (Survey)	300	1	25	125
NASA EPSCoR Co-PIs/Science PIs/Co-Science PI/Other Researchers (Focus Group)	50	1	40	33.33
TOTAL				158.33

7. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

8. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? Yes  No

– NOTE: First and Last Name are collected

2. If yes, will any information that is collected by included in records that are subject to the Privacy Act of 1974? Yes o No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

þYes o No

Published March 17, 2015, the Applicable System of Records Notice is NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,

Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

9. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? þYes o No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe The survey will be distributed to all of the Principal Investigators (PIs)/Directors of the NASA EPSCoR Program Jurisdictions (28) in April 2023 for administration to Co-PIs/Science PIs/Co-Science PI/Other Researchers (approximately 300). Although participation in the survey is voluntary, it will provide useful customer input to address programmatic decision making and assess IRC outcomes of NASA EPSCoR funding. Additionally, up to 50 interviews will be conducted virtually with respondents from the survey that volunteers to participate in an interview. Each interview will be approximately forty minutes long.

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? Not applicable.

10. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent: o Active þ Passive

6. How will the information be collected:

þ Web-based or other forms of Social Media (NOTE: Survey data will be collected via SurveyMonkey and focus groups will occur virtually using either Adobe Connect, Zoom, or Teams link)

o Telephone

o In-person

o Mail

o Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach.

7. Will interviewers or facilitators be used? þ Yes  No

11. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:

o Consent form

þ Instrument (attitude & behavior scales, and surveys)

þ Protocol script (Specify type: Script)

þ Instructions NOTE: Instructions are included in the instrument

o Other (Specify ________________)

12. GIFTS OR PAYMENT: o Yes þ No  If you answer yes to this question, please describe and provide a justification for amount.

ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $5,925. The cost is based on an annualized effort of 75 person-hours at the evaluator’s rate of $79/hour for administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 09/30/2024.

13. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Richard Gilmore

Title: Performance Assessment and Evaluation Program Manager, NASA

Office of STEM Engagement (OSTEM)

Email address or Phone number: [email protected]

Date: 12/14/2023

NASA Office of STEM Engagement 7