FY24 NASA Internship Program (Surveys and Focus Group Protocol MT Generic Clearance) SHORT FORM (12-11-23)

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number 2700-0159, expiration 9/30/2024

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION: NASA Internship Program Evaluation

2. TYPE OF COLLECTION:

þ	Attitude/Behavior Scale
o	Baseline Survey
o	Cognitive Interview Protocol
o	Consent Form
þ	Focus Group Protocol
þ	Follow-up Survey
þ	Instructions
o	Satisfaction Survey
o	Usability Protocol

GENERAL OVERVIEW: NASA Science, Technology, Engineering, and Mathematics (STEM) Engagement is comprised of a broad and diverse set of programs, projects, activities, and products developed and implemented by HQ functional Offices, Mission Directorates and Centers. These investments are designed to attract, engage, and educate students, and to support educators, and educational institutions. NASA’s Office of STEM Engagement (OSTEM) delivers participatory, experiential learning and STEM challenge activities for young Americans and educators to learn and succeed. NASA STEM Engagement seeks to:

• Create unique opportunities for students and the public to contribute to NASA’s work in exploration and discovery.

• Build a diverse future STEM workforce by engaging students in authentic learning experiences with NASA people, content, and facilities.

• Strengthen public understanding by enabling powerful connections to NASA’s mission and work.

To achieve these goals, NASA STEM Engagement strives to increase Intern engagement in NASA projects, enhance higher education, support underrepresented communities, strengthen online education, and boost NASA's contribution to informal education. The intended outcome is a generation prepared to code, calculate, design, and discover its way to a new era of American innovation.

The purpose of this study is to test and utilize two survey instruments, one that captures measurable (quantitative & qualitative) data on students’ outcomes one year after participating in a NASA STEM Internship and one that measures STEM identity and sense of belonging. The data collected through these two surveys and a set of focus groups will provide information to assess how and in what ways NASA Internships contribute to students’ planned educational pursuits and career placements/trajectories.

INTRODUCTION AND PURPOSE: NASA OSTEM has implemented several evaluation studies since 2019 that have led to this current study, including the FY21 Internships Outcome Evaluation, the FY22 Internships Longitudinal Evaluation, the FY22 Follow-On Internships Outcome Assessment, and the FY23 Internship Process and Retrospective studies. The findings from these studies demonstrated high levels of program satisfaction, gains in science and research-related outcomes among interns, and the effectiveness of the internship experience for maintaining and growing perceptions and aspirations regarding future STEM interests and intentions. The FY23 Internship Retrospective survey filled a gap in knowledge by developing an instrument to measure how the internship experience impacts interns’ future education and career one year after participation in a NASA STEM Internship.

This study will further test the psychometric properties of the Internship Retrospective Survey and test an instrument to understand interns’ sense of belonging and STEM identity, also identified by previous studies as needed. The study will provide information about the internship recruitment and application process, internship experience, and the impact of NASA OSTEM Internships on interns’ education and career plans one year after the completion of their internship. The study will further test and validate data collection instruments that can be used to assess the impact of the internship experience to support NASA OSTEM with evidence-based decision making and continual improvement.

RESEARCH DESIGN OVERVIEW: The evaluation study will answer four evaluation questions:

1. How are prospective interns recruited to NASA OSTEM Internship opportunities? How did the recruitment strategies impact prospective interns’ decision to apply?

2. What are the characteristics of the registrant, applicant, and intern pool in the NASA OSTEM Internship Program and how do they compare year over year?

3. What short-term impacts are experienced by interns during the internship experience and how does it vary across different internship categories, types, and locations (FT/PT, UG/G, length of internship, NASA Centers / Facilities, virtual / in-person)?

4. How did participation in the NASA OSTEM Internship program impact interns’ education, careers, and future planning one year after internship completion?

Methods to collect data include:

Retrospective Internship Survey: The Retrospective Internship Survey (see Appendix A) is a 28-item survey that asks interns to provide information about the impact of their internship experience on career, education, and future planning one year after completing the NASA OSTEM internship. The survey includes multiple choice, Likert scale, and free-response type questions. The online survey will be distributed through Survey Monkey.

All of the FY22 and FY23 interns will receive an email to complete the survey. The number of FY22 interns was approximately 2,220, with a similar number estimated for FY23. The estimated sample is 4,440, but the final sample size will depend on the number of interns who agree to participate in the study. Quantitative data will be summarized using descriptive statistics such as numbers of respondents, frequencies and proportions of responses, average responses by category (e.g., 1 = “Never Used” to 4 = “Used Every day”), and standard deviations. Qualitative data will be coded to identify themes.

Internship Experience Survey: The Internship Experience Survey (see Appendix B) will be piloted with the FY24 spring cohort of interns to better understand the short-term impact of the NASA OSTEM internship, specifically focused on STEM identity and STEM belonging. Survey items will be adapted from the existing NASA MUREP Outcome Assessment, which was validated in FY23. The online survey will be distributed through Survey Monkey. All interns in the FY24 spring cohort will receive an email to complete the survey. The number of interns in the FY23 spring cohort was approximately 500, and a similar number is expected to be invited to respond to this survey. The final number will depend on the size of the cohort and number of interns who agree to participate in the study. Quantitative data will be summarized using descriptive statistics such as numbers of respondents, frequencies and proportions of responses, average responses by category (e.g., 1 = “Never Used” to 4 = “Used Every day”), and standard deviations. Qualitative data will be coded to identify themes.

Intern Focus Groups: (see Appendix C) The focus groups will generate information to understand intern experiences (e.g., social experiences, onboarding, etc.); short-term outcomes (e.g., STEM identity and sense of belonging); and the effectiveness of recruitment strategies on decisions to apply to the NASA OSTEM Internship, specifically among individuals from groups underserved and underrepresented in STEM. All FY23 NASA OSTEM interns will be asked about their interest to participate in focus groups. A stratified sampling plan using NASA STEM Gateway data will be used to select interns who express an interest by institution type (i.e., MSI and non-MSI), internship session (fall, spring, and summer), and demographic data (i.e., gender, race, and ethnicity). The target sample will include approximately six to eight FY23 NASA OSTEM interns per focus group, with a goal of conducting no more than ten focus groups. From the execution of the FY23 Internship Process Evaluation and the FY23 Retrospective Internship Survey Pilot Study, the proposed number of focus groups (i.e., ten) will enable reaching saturation of answers to questions of interest. The goal of the sampling strategy is to create a group representative of the FY23 intern pool to develop generalizable findings that can validly inform changes in practices, programs, and policies. Focus groups will be conducted online using Microsoft Teams and will be recorded for analysis.

3. TIMELINE: The study will take place between November 2023 and October 2024.

• Retrospective Survey: November 2023 through February 2024 (FY22 all interns and FY23 fall cohort); May 2024 through June 2024 (FY23 spring cohort); and August 2024 through October 2024 (FY23 summer cohort)

• Internship Experience Survey: May 2024 through June 2024 (FY24 spring cohort)

• Focus Groups: February 2024 through June 2024 (FY23 interns)

SAMPLING STRATEGY: According to the NASA STEM Gateway, there were approximately 2,220 interns in FY 2022. The number of FY23 interns is estimated to be similar. The entire population of interns who participated in an internship during the timeframe (FY22 and FY23) will be invited to respond to the Internship Retrospective Survey. All FY23 interns will be invited to participate in focus groups. All interns in the spring FY24 cohort will be invited to respond to the Internship Experience Survey.  

4. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
Internship Retrospective Survey – FY22 and FY23 interns	4,400	1	10 (.167)	733.33
Internship Experience Survey – FY24 spring cohort interns	500	1	10 (.167)	83.33
Focus Groups – FY23 interns	80	1	45 (.75)	60
TOTAL				876.66

5. DATA CONFIDENTIALITY MEASURES: All information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

6. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? oYes þ No

2. If yes, will any information that is collected by included in records that are subject to the Privacy Act of 1974? oYes o No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

þYes o No

Published May 10, 2023, the Applicable System of Records Notice is NASA 10EDUA, NASA Education System Records- https://www.federalregister.gov/documents/2023/05/10/2023-09932/privacy-act-of-1974-system-of-records.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA Education System Records- https://www.federalregister.gov/documents/2023/05/10/2023-09932/privacy-act-of-1974-system-of-records

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1, Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

7. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? þYes o No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe? The entire population of interns who participated in an internship during the timeframe (FY22 and FY23) will be invited to respond to the Internship Retrospective Survey (4,440 or below). All FY23 interns will be invited to participate in focus groups (80 targeted participants). All spring interns in the FY24 cohort will be invited to respond to the Internship Experience Survey (500 or below).  

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? Not applicable.

8. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent: o Active þ Passive

6. How will the information be collected:

þ Web-based or other forms of Social Media

o Telephone

o In-person

o Mail

o Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach.

7. Will interviewers or facilitators be used? o Yes þ No

9. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:

o Consent form

þ Instrument (attitude & behavior scales, and surveys)

þ Protocol script (Specify type: Script)

þ Instructions NOTE: Instructions are included in the instrument

o Other (Specify ________________)

10. GIFTS OR PAYMENT: o Yes þ No  If you answer yes to this question, please describe and provide a justification for amount.

ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $7,900. The cost is based on an annualized effort of 100 person-hours at the evaluator’s rate of $79/hour for administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 09/30/2024.

11. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Richard Gilmore

Title: Performance Assessment and Evaluation Program Manager, NASA

Office of STEM Engagement (OSTEM)

Email address or Phone number: [email protected]

Date: 12/14/2023

NASA Office of STEM Engagement 6