NASA STEM Engagement Internship Applicants and Awardees Survey Supporting Statement

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number 2700-0159, expiration 06/30/2021

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION:

NASA Internship Applicants and Awardees Survey

2. TYPE OF COLLECTION:

	Attitude/Behavior Scale
	Baseline Survey
	Cognitive Interview Protocol
	Consent Form
	Focus Group Protocol
	Follow-up Survey
	Instructions
	Satisfaction Survey
	Usability Protocol

GENERAL OVERVIEW: NASA Missions and the Office of STEM Engagement (OSTEM) invest in excess of 70M annually in higher education student awards, internships and competitions, and other efforts to support STEM workforce development. These awards are intended to recruit and retain students in STEM disciplines of interest to NASA and the Aerospace industry. NASA’s Internship Program leverages NASA’s unique missions and programs to enhance and increase the capability, diversity, and size of the nation’s future science, technology, engineering and mathematics (STEM) workforce.

The critical need to increase the number and diversity of students interested in and equipped to pursue careers in STEM is well documented in a number of national reports (e.g., Bayer Corporation, 2013; National Science Board, 2018; PCAST, 2010). Workforce development programs serve to meet these needs by increasing the breadth, depth, and diversity of the nation’s STEM workforce. NASA has particular interest in accomplishing these goals. To ensure the ongoing progress of its missions and research, NASA relies on a steady pipeline of talent who possess robust STEM content knowledge, technical skills, and professional skills. To that end, NASA provides a variety of internship programs for high school, college, and graduate level students. NASA Internships are competitive awards to support educational opportunities that provide unique NASA-related research and operational experiences for high school, undergraduate, and graduate students, as well as educators. These opportunities serve students by integrating interns with career professionals emphasizing mentor-directed, degree-related tasks, while contributing to the operation of a NASA facility or the advancement of NASA’s missions. NASA Internships align to NASA’s STEM Engagement goals to 1) provide students with opportunities to contribute to NASA’s endeavors in exploration and discovery and 2) to build a diverse, skilled future STEM workforce. The internships contribute to the number and diversity of highly-qualified individuals entering the aerospace industry workforce by providing authentic experiential learning opportunities for students who have demonstrated interest and aptitude in NASA-related STEM areas. NASA internships provide authentic training to develop: (a) content knowledge, (b) technical skills, and (c) workplace-related professional skills.

To date, NASA has not attempted a comprehensive study to assess the effectiveness of these opportunities in achieving their stated goals. Although study designs have been proposed previously, NASA has not invested in a well-designed study due to a number of concerns including: a) burden on awardees, b) privacy implications in the collection of study data, 3) IT infrastructure challenges, and (4) available budget to support such work.

The survey for this information collection will obtain answers to the following two research questions:

• Do college graduates who participated in an on-site NASA internship (i.e. awardees) enter the NASA workforce, STEM workforce, or a STEM-focused post-graduate academic area at a higher rate than college graduates who did not participate in NASA opportunities (i.e. applicants)?

• Do college graduates from historically underrepresented groups who participated in an on-site NASA internship (i.e. awardees) enter the NASA workforce, STEM workforce, or a STEM-focused post-graduate academic area at a higher rate than college graduates who did notparticipate in NASA opportunities (i.e. applicants)?

3. INTRODUCTION AND PURPOSE: Obtaining a position in the STEM workforce or achieving an advanced degree in a STEM-related field is a multi-faceted outcome. Research typically investigates progression through the STEM pipeline through the lens of attrition or persistence (e.g., Maltese & Tai, 2011). The extant literature shows that there are a number of manipulable environmental factors influencing both attrition and persistence, including high school courses, STEM interest, classroom experiences, and college courses (e.g., Chen, 2013, National Academies of Sciences, Engineering, and Medicine, 2018; National Research Council, 2011). Additionally, a number of non-manipulable variables such as gender, race, and ethnicity are strong predictors of STEM career outcomes. Research on pipeline differences based on gender, race, or ethnicity highlight a disparity in STEM achievement rates by these factors (National Research Council, 2011; Wang & Degol, 2017). Despite these clear predictors, predicting STEM outcomes is a complex process. NASA has a keen interest in increasing the breadth, depth, and diversity of the nation’s STEM workforce to ensure the on-going progress of its missions and research.

This effort will yield data regarding the outcomes of NASA internship awardees and applicants, specifically in terms of their current roles, or lack thereof, in the STEM workforce. To that end, the pilot data collection will examine whether college graduates who participated in an on-site NASA internship ultimately enter the NASA workforce, STEM workforce, or a STEM-focused post-graduate academic area at higher rate than those who do not participate, by comparing these two groups on their responses to select questions of the National Survey of College Graduates (NSCG). Additionally, the pilot study is intended to establish reliable measures to assess outcome and impact of NASA STEM-based internships, fellowships, and scholarships on academic and workforce choices of awardees.

This pilot study is intended to validate the methodology and refine data collection based on its results to improve utility. The questions from the NSCG (OMB Control No. 3145-0141) are from an instrument that has been developed and validated by the U.S. Census Bureau and NSF. A shortened version of the NSCG has been adapted for the NASA Office of STEM Engagement. The questions are identical to those in the NSCG, however the questionnaire was modified to fit the contextual circumstances of the present study and minimize the response burden for participants by decreasing the number of question items. The 10 items selected for inclusion in the pilot study were not altered in any way. However, to test clarity and readability of these questions when presented in a different context, clarity and comprehensiveness questions will be included.

Hence, the goals of this cycle of pilot testing are as follows:

• Determine clarity, comprehensibility, and preliminary psychometric properties (e.g., validity, reliability) of the instrument. And, to explore individual item functioning, and to make any necessary adjustments in preparation for large-scale testing as the basis for more sophisticated statistical testing.

• Determine an accurate response burden for these instruments.

4. RESEARCH DESIGN OVERVIEW: NASA STEM Engagement is conducting a utilization-focused pilot study that is quasi-experimental in nature. This means that, like a true experiment, the study will measure the causal impact of an experimental intervention (i.e., the causal impact of the internship program) on a target population (i.e., students). However, unlike a true experiment, individuals are not randomly assigned to whether they receive the intervention or not. That is, we cannot randomly assign some individuals to participate in the internships program and randomly assign others to not participate. These groups already exist due to the awardee selection process. As such, the groups of applicants and awardees may differ from each other due to variables outside of our control. Furthermore, the lack of randomization contributes to an inability to make causal claims (Shadish et al., 2002).

A utilization-focused evaluation approach guides the design and execution of this investigation. Described by Patton (2015), utilization-focused evaluations are designed for utility – the actual use and application of evaluation findings. According to Patton (2015), when designing a utilization-focused evaluation “evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that is done, from beginning to end, will affect use. Use concerns how real people in the real world apply evaluation findings and experience the evaluation process” (p. 156). Patton’s description of utilization-focused evaluations strongly aligns to the Learning Agenda adopted by NASA’s Office of STEM Engagement.

The focus of this utilization-focused pilot study is to generate useful and actionable findings to design a Longitudinal Study of NASA’s engagement opportunities. The factors identified through this investigation may be used to inform decision-making and support program, activity, and policy improvements. To accomplish this action-oriented result, the pilot study consists of five separate studies with five unique methodologies. Each methodology will be explored individually to determine its ability to provide valuable information. The individual examinations will be followed by an overall evaluation comparing the different methodologies in terms of general effectiveness. The individual and comparative findings will take place before proceeding to a larger scale Longitudinal Study.

The methodology related to this information collection involves NASA STEM Engagement distributing a survey based on several items taken from the National Survey of College Graduates (NSCG) to the awardee and applicant groups. The NSCG (OMB Control No. 3145-0141) is a valid instrument provided by the United States Census Bureau and the National Science Foundation (NSF). Specifically awardee and applicants are asked to answer the questions A1, A7, A9, A10, A13, A19, A24, A29, A36, and A37 from the. Participants will also be asked additional questions about their participation in NASA and non-NASA out-of-school time (OST) programs and STEM classes prior to and after applying to the NASA Internship Program. Responses will be used to validate the survey for clarity, comprehensibility, and to determine psychometric properties with the respondent pool. Also, applicant responses to these items will be compared to those of awardees to determine if the two groups differ.

NASA STEM Engagement is pilot testing this survey. Despite the absence of a control group, this design can still yield strong causal effects when effort is made to satisfy requirements of quasi-experimentation such as identifying and reducing the plausibility of alternative explanations for the intervention- as- treatment effect (Shadish et al., 2002), identifying conceivable threats to internal validity, and statistically probing likelihood of treatment-outcome covariation (Mark & Reichardt, 2009).

Following this pilot phase of testing, NASA STEM Engagement has tentative research questions and hypotheses to test regarding the impact of STEM Challenge activities on larger samples of applicants and awardees from the NASA Internship Program. Thus, this work is integral to the iterative assessment and feedback process for NASA STEM Engagement Internship Program.

5. TIMELINE: Pilot testing of surveys will take place approximately April 2020 through August 2020.

6. SAMPLING STRATEGY: NASA STEM Engagement will administer the survey for testing to the full internship awardee population from 2014 and to a group of randomly selected student applicants according to the strategy described below.

The awardee population from FY 2014 consisted of 1704 student participants. All 1704 awardees will be requested to complete the survey. The responses of awardees will be compared to a matched sample of applicants. This matched sample comparison group will be a stratified random sample of 1704 internship applicants from 2014. Strata, or groups, will be defined based on the demographic characteristics of the awardees. The applicant comparison group will be matched to the awardee group based on these demographic characteristics and sampled in similar percentages.

Table 1. Calculation chart to determine statistically relevant number of respondents

Data Collection Source	(N) Population Estimate	(A) Sampling Error +/- 5% (.05)	(Z) Confidence Level 95%/ Alpha 0.05	(P) *Variability (based on consistency of intervention administration) 50%	Base Sample Size	Response Rate	(n) Number of Respondents
Internship Applicants	1704	N/A	N/A	N/A	1704	N/A	1704
Internship Awardees	1704	N/A	N/A	N/A	1704	N/A	1704
TOTAL							3408

7. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
Internship Applicants	1704	1	10	284
Internship Awardees	1704	1	10	284
TOTAL				568

8. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

9. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? Yes No

– NOTE: Insert (random identifier will be provided to potential respondents)

2. If yes, will any information that is collected by included in records that are subject to the Privacy Act of 1974? Yes  No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

Yes  No

Published March 17, 2015, the Applicable System of Records Notice is NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,

Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

10. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? Yes  No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe? The universe of potential respondents is 3408 participants. Participants include a random sample of 1704 internship awardees (college graduates who participated in an on-site NASA internship) from the year 2014. The comparison group will be a comparable sample of 1704 college graduates who were internship applicants in 2014.

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? Not applicable.

11. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent:  Active Passive

6. How will the information be collected:

 Web-based or other forms of Social Media (100%)

 Telephone

 In-person

 Mail

 Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach. The pilot survey will be administered via the web (100%).

7. Will interviewers or facilitators be used?  Yes  No

12. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:

 Consent form

 Instrument (attitude & behavior scales, and surveys)

 Protocol script (Specify type: Script)

 Instructions NOTE: Instructions are included in the instrument

 Other (Specify ________________)

13. GIFTS OR PAYMENT:  Yes  No  If you answer yes to this question, please describe and provide a justification for amount.

14. ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $3,000. The cost is based on an annualized effort of 40 person-hours at the evaluator’s rate of $75/hour for development and administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 06/30/2021.

15. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Richard Gilmore

Title: Educational Programs Specialist/Evaluation Manager, NASA GRC

Office of STEM Engagement

Email address or Phone number: [email protected]

Date: 1/14/2021

Bibliography

Bayer Corporation. (2013). U.S. STEM Workforce Shortage: Myth or Reality? Media, PA: International Communications Research.

Chen, X. (2013). STEM Attrition: College Students’ Paths Into and Out of STEM Fields (NCES 2014-001). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC.

Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: Examining the association of educational experiences with earned degrees in STEM among US students. Science Education, 95(5), 877-907.

Mark, M. M., & Reichardt, C. S. (2009). Quasi-experimentation. In L. Bickman, & D. J. Rog (Eds.), The SAGE handbook of applied social research methods (2nd ed., pp. 182-214). Thousand Oaks, CA: SAGE Publications, Inc.

National Academies of Sciences, Engineering, and Medicine (2018). English Learners in STEM Subjects: Transforming Classrooms, Schools, and Lives. Washington, DC: The National Academies Press. https://doi.org/10.17226/25182.

National Research Council. (2011). Successful K-12 STEM Education: Identifying Effective Approaches in Science, Technology, Engineering, and Mathematics. Committee on Highly Successful Science Programs for K-12 Science Education. Board on Science Education and Board on Testing and Assessment, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.

National Science Board. 2018. Science and Engineering Indicators 2018. NSB-2018-1. Alexandria, VA: National Science Foundation. Available at https://www.nsf.gov/statistics/indicators/.

Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4^th ed.). Los Angeles, CA: SAGE

President’s Council of Advisors on Science and Technology (PCAST; 2010). Prepare and Inspire: K-12 Science, Technology, Engineering, and Math (STEM) Education for America's Future. Education Digest: Essential Readings Condensed for Quick Review, 76(4), 42-46.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company.

Wang, M. T., & Degol, J. L. (2017). Gender gap in science, technology, engineering, and mathematics (STEM): Current knowledge, implications for practice, policy, and future directions. Educational Psychology Review, 29(1), 119-140.

NASA Office of STEM Engagement 8