NASA K-12 Student Outcome Surveys (NASA Elementary School Survey, NASA Middle School Survey, and NASA High School Survey)

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number 2700-0159, expiration 09/30/2024

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION:

NASA K-12 Student Outcome Surveys (NASA Elementary School Survey, NASA Middle School Survey, and NASA High School Survey)

2. TYPE OF COLLECTION:

þ	Attitude/Behavior Scale
o	Baseline Survey
o	Cognitive Interview Protocol
o	Consent Form
o	Focus Group Protocol
o	Follow-up Survey
þ	Instructions
þ	Satisfaction Survey
o	Usability Protocol

GENERAL OVERVIEW: NASA Science, Technology, Engineering, and Mathematics (STEM) Engagement is comprised of a broad and diverse set of programs, projects, activities and products developed and implemented by HQ functional Offices, Mission Directorates and Centers. These investments are designed to attract, engage, and educate students, and to support educators, and educational institutions. NASA’s Office of STEM Engagement (OSTEM) delivers participatory, experiential learning and STEM challenge activities for young Americans and educators to learn and succeed. NASA STEM Engagement seeks to:

• Create unique opportunities for students and the public to contribute to NASA’s work in exploration and discovery.

• Build a diverse future STEM workforce by engaging students in authentic learning experiences with NASA people, content, and facilities.

• Strengthen public understanding by enabling powerful connections to NASA’s mission and work.

To achieve these goals, NASA STEM Engagement strives to increase K-12 involvement in NASA projects, enhance higher education, support underrepresented communities, strengthen online education, and boost NASA's contribution to informal education. The intended outcome is a generation prepared to code, calculate, design, and discover its way to a new era of American innovation.

The focus of this study is to develop and pilot two grade-level appropriate instruments (high school instrument will contain demographic questions; elementary/middle-school instrument will not) for measuring student outcomes of participation in NASA K-12 STEM Engagement programming in grades 4-12. The surveys for this information collection are specific to determining the impact of the NASA STEM Engagement K-12 projects and activities on students (upper elementary grades 4 and 5; middle school grades 6 through 8; and high school grades 9-12). STEM Identity, STEM Self-Efficacy, STEM Interest, Sense of Belonging, and 21^st Century skills development are also measures of interest.

INTRODUCTION AND PURPOSE: The FY 2021 NASA STEM Engagement K-12 Comprehensive Evaluation study identified the current state of K-12 activities across NASA and proposed a common vision, goals, and objectives for K-12 projects and activities – grounded in a logic model and theory of change. Building upon this work, the NASA K-12 Student Outcome Assessment will examine the impact of NASA STEM Engagement K-12 activities on relevant student outcomes. The evaluation study process consisted of 1) instrument development, 2) strategic briefings, 3) pilot testing and 4) instrument utilization. Three surveys were developed, piloted, and validated for each of the following grade bands: upper elementary (grades 4-5); middle school (grades 6-8); and high school (grades 9-12) to be piloted and validated. The three surveys contained the same questions but used slightly different language based on reading-level analyses and cognitive interview feedback. However, upon reflection, the evaluation team determined that it would be best to use the language on the elementary study for all students from grades 4^th to 12^th. This is based on the knowledge that many elementary, middle, and high school students are reading below grade level (US Department of Education, 2013). Recent data have shown that currently only roughly half (57%) of 5^th graders nationwide are reading at grade level, but these percentages are lower and saw a more dramatic decrease over the past two years in lower elementary grades. In fact, nationwide, reading scores declined from 2019 to 2022 (Sparks, 2022; US Department of Education, 2022). In addition, opportunity gaps for many of the same groups who are underserved and underrepresented in STEM (such as Black students, Hispanic students, and students eligible for the free and reduced lunch program) are reflected in lower average reading scores (Kuhfeld et al. 2023; Sparks, 2022). Furthermore, survey fatigue can occur when participants feel burdened, in terms of effort, when completing a survey task (Sharp & Frankel, 1983). Thus, in an effort to a) make the survey more accessible for more students, and b) maintain consistency across grade levels for future student outcome comparisons by grade-level, and c) reduce survey fatigue, we suggest using the surveys with the lowest reading level and highest readability for all students (Marchland, 2017).

This study will provide evidence that can be used to: Explore, Describe, Classify and establish associations among variables (constructs) and the population of interest (NASA K-12 STEM engagement program participants in grades 4-12).

Our interest is to measure students’ immediate outcomes of participating in a NASA STEM Engagement K-12 project or activity. In addition, the psychometric properties of four new survey items addressing STEM identity, existing STEM interest, STEM capital/role models, and Sense of Belonging will be assessed. The high school survey will include three demographic questions which address ethnicity, race, and gender. Thus, the purpose for pilot testing is to reliably explain the ways in which participants in grades 4-5, 6-8 and 9-12 are impacted by participation in these activities, and additionally pilot and validate four additional survey items. Guided by current STEM education and measurement methodologies, it is the goal of this outcome assessment to provide information that becomes part of the iterative outcome assessment and feedback process for the portfolio of NASA STEM Engagement K-12 projects and activities.

Hence, the goals of this outcome assessment study is to 1) assess the STEM identity, self-efficacy, STEM Interest, Sense of Belonging, and 21^st century skills of students participating in NASA STEM engagement programs; 2) pilot and examine the psychometric properties of four additional survey items; and 3) assess the usefulness of the evaluation information for program managers to inform and improve their programs.

3. RESEARCH DESIGN OVERVIEW: NASA’s work in STEM Engagement is focused on serving students. It is recognized that providing support and resources to educators and educational institutions is vital to effectively engage students. The proposed instrument will used in ongoing program evaluation by NASA. This study falls under the category of a program evaluation, and will be guided by three evaluation questions for the approach and design of this study. Evaluation questions are presented in Figure 1 below.

Figure 1. Evaluation Questions

The two grade-level appropriate instruments (high school instrument will contain 3 demographic questions; elementary/middle-school instrument will not) will be developed as one instrument with skip logic in Survey Monkey online software, and a survey link will be distributed through email to ~500 NASA STEM Engagement program participants. Note that this sample size is nominal and will depend on the number of programs that agree to participate in the study. Quantitative and qualitative methods will be used to analyze survey data. Quantitative data will be summarized using descriptive statistics such as numbers of respondents, frequencies and proportions of responses, average response when responses categories are assigned to a Likert scale (e.g., 1 = “Never Used” to 4 = “Used Every day”), and standard deviations. Emergent coding will be used for the qualitative data to identify the most common themes in responses.

4. Construct survey item analysis. Rasch (1960, 1980) measurement was previously employed to assess the construct sections of the NASA Intern Survey in the Spring of 2021 (Sondergeld & Johnson, 2021) as well as the NASA STEM Engagement FY22 K-12 Student Outcome Assessment Final Report. Results showed all construct sections functioned well and could be used to form respective scales, or composite measures. Thus, items in different survey construct sections will be analyzed and an average scale score computed for the purpose of looking for significant differences in each construct (STEM Identity, STEM Self-Efficacy, STEM Interest, and 21^st Century Skills) by student participants.

5. TIMELINE: Testing of the two grade-level appropriate instruments (high school instrument will contain 3 demographic questions; elementary/middle-school instrument will not) as one instrument with skip logic is targeted to take place from April 2023 – August 2023 with student participants (grades 4-12) from NASA STEM Engagement K-12 projects and activities in coordination with project management. Note that this timeline is nominal and will depend on the number of programs that agree to participate in the study.

6. SAMPLING STRATEGY: For the outcome assessment, a purposeful sampling technique will be employed to select the NASA K-12 STEM Engagement activities from which student participants are recruited and data can be collected. Purposeful sampling will be used to select an appropriate sample for the purpose of developing findings that can be used to inform changes in practices, programs, and policies across different types of NASA STEM engagement programs (Patton, 2015). This sample will include different types of programs, program contexts, and student grade-levels. The universe of NASA K-12 participants (grades 4-12) for pilot testing is 600 or below. Items for the two grade-level appropriate instruments (high school instrument will contain 3 demographic questions; elementary/middle-school instrument will not) will be developed in Survey Monkey online software as one instrument with skip logic, and a survey link will be distributed through email to ~600 NASA K-12 participants (grades 4-12) and/or to the educator in the NASA STEM Engagement program to administer.

7. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
NASA K-12 Participants (grade 4-12)	600	1	20	200
TOTAL				200

8. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

9. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? oYes þ No

2. If yes, will any information that is collected by included in records that are subject to the Privacy Act of 1974? oYes o No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

þYes o No

Published March 17, 2015, the Applicable System of Records Notice is NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,

Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

10. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? þYes o No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe? For the outcome assessment, a purposeful sampling technique will be employed to select the NASA K-12 STEM Engagement activities from which student participants are recruited and data can be collected. Purposeful sampling will be used to select an appropriate sample for the purpose of developing findings that can be used to inform changes in practices, programs, and policies across different types of NASA STEM engagement programs (Patton, 2015). This sample will include different types of programs, program contexts, and student grade-levels. The universe of NASA K-12 participants (grades 4-12) for pilot testing is 600 or below. Items for the two grade-level appropriate instruments (high school instrument will contain 3 demographic questions; elementary/middle-school instrument will not) will be developed in Survey Monkey online software as one instrument with skip logic, and a survey link will be distributed through email to ~600 NASA K-12 participants (grades 4-12) and/or to the educator in the NASA STEM Engagement program to administer.

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? Not applicable.

11. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent: o Active þ Passive

6. How will the information be collected:

þ Web-based or other forms of Social Media

o Telephone

o In-person

o Mail

o Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach.

7. Will interviewers or facilitators be used? o Yes þ No

12. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:

o Consent form

þ Instrument (attitude & behavior scales, and surveys)

o Protocol script (Specify type: Script)

þ Instructions NOTE: Instructions are included in the instrument

o Other (Specify ________________)

13. GIFTS OR PAYMENT: o Yes þ No  If you answer yes to this question, please describe and provide a justification for amount.

ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $5,925. The cost is based on an annualized effort of 75 person-hours at the evaluator’s rate of $79/hour for administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 09/30/2024.

14. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Richard Gilmore

Title: Performance Assessment and Evaluation Program Manager, NASA

Office of STEM Engagement (OSTEM)

Email address or Phone number: [email protected]

Date: 12/14/2023

References

Kuhfeld, M., Lewis, K., & Peltier, T. (2023). Reading achievement declines during the covid-19 pandemic: Evidence from 5 million u.s. students in grades 3–8. Reading and Writing, 36(2), 245–261. https://doi.org/10.1007/s11145-022-10345-8

Marchland, L. (2017). What is readability and why should content editors care about it? Center for Plain Language.

Patton, M. Q. (2015). Sampling, Qualitative (Purposeful). The Blackwell Encyclopedia of Sociology. https://doi.org/10.1002/9781405165518.wbeoss012.pub2

Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests. (Copenhagen, Danish Institute for Educational Research), with foreward and afterword by B.D. Wright. The University of Chicago Press.

Sharp, L. M., & Frankel, J. (1983). Respondent burden: A test of some common assumptions. Public
Opinion Quarterly, 47(1), 36-53

Sondergeld, T. A., & Johnson, C. C. (2021). NASA intern study: Quantitative field study of intern survey.

1-18.

Sparks, S.D. (2022). More than 1 in 3 children who started school in the pandemic need “intensive”

reading help. Education Week.

U.S. Department of Education (2013). The Nation’s Report Card.

U.S. Department of Education (2022). The Nation’s Report Card.

NASA Office of STEM Engagement 8