fast track form

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number XXXXXXX, expiration XXXXXXX

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION:

NASA Student STEM Inventory (Grades 4-12)

2. TYPE OF COLLECTION:

þ	Attitude/Behavior Scale
o	Baseline Survey
o	Cognitive Interview Protocol
o	Consent Form
o	Focus Group Protocol
o	Follow-up Survey
þ	Instructions
o	Satisfaction Survey
o	Usability Protocol

GENERAL OVERVIEW: NASA Science, Technology, Engineering, and Mathematics (STEM) Engagement is comprised of a broad and diverse set of programs, projects, activities and products developed and implemented by HQ functional Offices, Mission Directorates and Centers. These investments are designed to attract, engage, and educate students, and to support educators, and educational institutions. NASA’s Office of STEM Engagement (OSTEM) delivers participatory, experiential learning and STEM challenge activities for young Americans and educators to learn and succeed. NASA STEM Engagement seeks to:

• Create unique opportunities for students and the public to contribute to NASA’s work in exploration and discovery.

• Build a diverse future STEM workforce by engaging students in authentic learning experiences with NASA people, content, and facilities.

• Strengthen public understanding by enabling powerful connections to NASA’s mission and work.

To achieve these goals, NASA STEM Engagement strives to increase K-12 involvement in NASA projects, support underrepresented communities, and boost NASA's contribution to informal education. The intended outcome is a generation prepared to code, calculate, design, and discover its way to a new era of American innovation.

INTRODUCTION AND PURPOSE: The FY21 NASA STEM Engagement K-12 Comprehensive Evaluation study identified the state of K-12 activities across NASA and proposed a common vision, goals, and objectives for K-12 projects and activities grounded in a logic model and theory of change. The FY22 study, NASA STEM Engagement K-12 Student Outcome Assessment and Instrument Development, created and validated a set of STEM student outcome surveys. A survey was developed, piloted, and validated for each of three grade bands: upper elementary (Grades 4-5); middle school (Grades 6-8); and high school (Grades 9-12) to measure STEM Identity, STEM Self-Efficacy, STEM interest, and 21st Century Skills. The three surveys contained the same questions but used slightly different language.

In the FY23 study, the three versions of the NASA Student STEM Inventory (Grades 4-12) were collapsed into one and the survey shortened in length. The FY23 K-12 Student Outcome Assessment Study collected information about survey psychometric properties, the utility of the information for program use, and student outcomes among NASA K-12 activity program participants in grades 4-12. Results supported the utility of the Inventory in measuring participant STEM attitudes and outcomes, suggesting the need to re-organize the organization of items in scales and making additional changes to specific items and formatting of the scale for delivery.

Building on these previous efforts, the FY24 NASA K-12 Student Outcome Assessment will test several additional modifications to the NASA Student STEM Inventory (Grades 4-12), examine the impact of NASA STEM Engagement K-12 activities on relevant student outcomes, and further study the utility of evaluation information for Activity Managers.

The surveys for this information collection are specific to determining the impact of the NASA STEM Engagement K-12 projects and activities on students (upper elementary Grades 4 and 5; middle school Grades 6 through 8; and high school Grades 9-12). STEM Identity, STEM Self-Efficacy, STEM Interest, Sense of Belonging, and 21^st Century skills development are also measures of interest.

3. RESEARCH DESIGN OVERVIEW: The evaluation will address three questions:

1. Participant Outcomes: To what extent do K-12 students who participate in NASA STEM Engagement Activities report positive STEM Outcomes?

2. NASA Student STEM Inventory (Grades 4-12): What are the psychometric properties of the NASA K-12 Student STEM Inventory?

3. Use of Findings: In what ways does the NASA Student STEM Inventory (Grades 4-12) provide useful information to NASA Activity Managers?

This PRA application addresses the NASA Student STEM Inventory (Grades 4-12).

The NASA Student STEM Inventory (Grades 4-12) is an online instrument for students in Grades 4-12. The Inventory includes 19 questions asked of all respondents, with three additional questions for high school students. The Inventory takes on average approximately 8 minutes to complete.

Students respond along a four-point scale (disagree a lot (1), disagree a little (2), agree a little (3), agree a lot (4)) to questions grouped in the categories of STEM identity (other people’s assessment of a person’s STEM competence), STEM interest (interest to learn and try STEM activities), STEM performance (engagement and leadership in STEM projects), and sense of belonging in the NASA activity. Respondents are asked about prior STEM interest, STEM role models in their lives, and career interest (open-ended responses). An item asks participants to write what they learned in the NASA activity. Based on student grade level, skip logic is used to ask middle and high school students three additional demographic questions (i.e., gender, race, and ethnicity) using the categories identified for federal data collection.

The NASA Student STEM Inventory (Grades 4-12) is deployed through Survey Monkey online software. A survey link will be created for each NASA activity that agrees to participate in the study. Activity Managers will share the link with participants.

Data Analysis EQ1: Participant Outcomes. Descriptive statistics will be calculated for the entire survey sample and for each participating activity (e.g., numbers of respondents, frequencies and proportions of responses, average response by Likert scale category, standard deviations). Results will be cross tabulated by program and activity descriptors (e.g., program type: STEM Program, STEM Challenge/Competition, Summer Institute/Camp) for constructs of interest. Differences in scores between program type will be analyzed for statistical significance (e.g., analysis of variance (ANOVA) using SPSS. Auto code features in NVivo qualitative analysis software (Version 12, 2018) will be used to identify themes in open-ended responses (e.g., prior interest in STEM, what was learned about NASA, career interest). Word clouds will illustrate frequency of responses.

Data Analysis EQ2: Psychometric Analysis. Confirmatory Factor Analysis (CFA), Rasch analysis, and reliability analysis will be used to test construct validity and overall scale reliability of the NASA K-12 Student Inventory. The FY23 K-12 Outcome Evaluation applied exploratory factor analysis techniques to the NASA Student STEM Inventory (Grades 4-12) to explore the relationships among the items. In this current study, confirmatory factor analysis (CFA) will be used to verify the factor structure of the Inventory. Confirmatory factor analysis is a multivariate statistical procedure that tests whether the hypothesized factor structure holds with a different set of participant responses.

Rasch techniques will also be used to evaluate the measurement functioning of the Inventory. Rasch model, sometimes called one-parameter model logistic, is an item analysis technique that measures the probability of endorsing an item as a function of ability. The Rasch modeling statistical procedure falls under Item Response Theory, which has several unique features that separate it from Classical Test Theory (CTT), two of the most important being group and test invariance.

4. TIMELINE: The FY24 NASA STEM Engagement K-12 Student Outcome Assessment study will be completed from November 2023 through October 2024. Table 1 provides a notional timeline of tasks and milestones aligned with the data collection process.

Table 1. FY24 K-12 Outcome Assessment Timeline

Task or Milestone	Timeframe Dates
Identify programs for participation in evaluation study	November 2023 – Sept 2024
Collect program information	November 2023 – Sept 2024
Collect NASA Student STEM Inventory (Grades 4-12) Data	November 2023 – Sept 2024
Analyze Survey Data	December 2023 – Sept 2024

SAMPLING STRATEGY:

NASA P&E Leadership and the evaluation team will work together to ensure that the student study sample includes NASA K-12 STEM Engagement activities with participants in elementary, middle, and high school grade bands (Grades 4-12). The estimated sample size is approximately 3,500 student participants.

Notional K-12 STEM Engagements that will be included in the FY24 K-12 Outcome Assessment are 1) Eclipse STEM Engagement opportunities, 2) NGS SPARX, and 3) 21st Century Community Learning Center STEM Engagement opportunities.

5. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
NASA Student STEM Inventory (Grades 4-12)	3500	1	8 mins (.13 hours)	466.67 hours
TOTAL				666.67

6. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

7. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? oYes þ No

2. If yes, will any information that is collected by included in records that are subject to the Privacy Act of 1974? oYes o No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

þYes o No

NOTE: Published March 17, 2015, the Applicable System of Records Notice is NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,

Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

8. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? þYes o No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe? The universe of NASA K-12 participants (Grades 4-12) for this outcome assessment is 5000 or below. The surveys are available via Survey Monkey online software, and survey links will be distributed through Activity Managers to their participants (Grades 4-12).

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? Not applicable.

9. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent: o Active þ Passive

6. How will the information be collected:

þ Web-based or other forms of Social Media

o Telephone

o In-person

o Mail

o Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach.

7. Will interviewers or facilitators be used? o Yes þ No

10. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:

o Consent form

þ Instrument (attitude & behavior scales, and surveys)

o Protocol script (Specify type: Script)

þ Instructions NOTE: Instructions are included in the instrument

o Other (Specify ________________)

11. GIFTS OR PAYMENT: o Yes þ No  If you answer yes to this question, please describe and provide a justification for amount.

ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $5,925. The cost is based on an annualized effort of 75 person-hours at the evaluator’s rate of $79/hour for administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 09/30/2024.

12. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Richard Gilmore

Title: Performance Assessment and Evaluation Program Manager, NASA

Office of STEM Engagement (OSTEM)

Email address or Phone number: [email protected]

Date: 9/11/2024

NASA Office of STEM Engagement 7