NASA CONNECTS Evaluation Survey & Focus Group - MT Generic Clearance

REQUEST FOR APPROVAL under the Generic Clearance for NASA STEM Engagement Performance Measurement and Evaluation, OMB Control Number 2700-0159, expiration 09/30/2024

_____________________________________________________________________________________

1. TITLE OF INFORMATION COLLECTION:

NASA CONNECTS Evaluation Survey and Focus Group Protocol

2. TYPE OF COLLECTION:

o	Attitude/Behavior Scale
o	Baseline Survey
o	Cognitive Interview Protocol
o	Consent Form
þ	Focus Group Protocol
o	Follow-up Survey
þ	Instructions
o	Satisfaction Survey
o	Usability Protocol

GENERAL OVERVIEW: NASA Science, Technology, Engineering, and Mathematics (STEM) Engagement is comprised of a broad and diverse set of programs, projects, activities and products developed and implemented by HQ functional Offices, Mission Directorates and Centers. These investments are designed to attract, engage, and educate students, and to support educators, and educational institutions. NASA’s Office of STEM Engagement (OSTEM) delivers participatory, experiential learning and STEM challenge activities for young Americans and educators to learn and succeed. NASA STEM Engagement seeks to:

• Create unique opportunities for students and the public to contribute to NASA’s work in exploration and discovery.

• Build a diverse future STEM workforce by engaging students in authentic learning experiences with NASA people, content, and facilities.

• Strengthen public understanding by enabling powerful connections to NASA’s mission and work.

To achieve these goals, NASA makes vital investments toward building a future diverse STEM workforce across its portfolio of projects including the NASA NextGen STEM (NGS) Connecting Our NASA Network of Educators for Collaborating Together in STEM (CONNECTS) community of practice. NASA STEM Engagement strives to increase K-12 involvement in NASA projects, enhance higher education, support underrepresented communities, strengthen online education, and boost NASA's contribution to informal education. The intended outcome is a generation prepared to code, calculate, design, and discover its way to a new era of American innovation.

NASA’s NGS project conceptualized and piloted the Connecting Our NASA Network of Educators for Collaborating Together in STEM (CONNECTS) community of practice in FY21. The overarching goal of CONNECTS is to inspire the next generation of explorers through authentic learning experiences. NASA aims to accomplish this goal through the enhancing and streamlining of collaborations, communications, engagements, and shared-knowledge practices of STEM community collaborators over time. The professional learning community is designed to facilitate collaboration amongst educators through discussion, networking, and the opportunity to share best practices. Educators can access NASA mission updates, content, resources (i.e., knowledge articles), lesson plans, activities, and educator professional development (PD) opportunities through CONNECTS.

INTRODUCTION AND PURPOSE: The purpose of this data collection effort is to support NASA Office of STEM Engagement and NGS goals and objectives and evidence building directly related to the Learning Agenda. The goal of this evaluation is to conduct a process evaluation that builds knowledge about NGS activities by understanding the efficiency, usefulness, and relevance of NASA CONNECTS to its end users (registered participants). This study will examine how and to what extent NASA CONNECTS: 1) reached U.S. educators, especially those working with underserved students; 2) displayed efficiency of use; 3) incorporated features to enhance educator collaboration; and 4) provided content relevant to educators and their students. In addition, the evaluation team will use existing literature to identify best practices to support the sustainability of the NASA CONNECTS: NGS Community of Practice. Specifically, the CONNECTS evaluation including survey data collection along with focus groups will address the FY22-FY23 NASA OSTEM Learning Question 3: How can NASA attract K-12 students, especially those underrepresented and underserved to STEM?

This data information collection request includes one survey instrument and a focus group protocol that will be used to collect data from registered participants (educators) of NASA CONNECTS. The data to be collected are not available elsewhere unless collected through this information collection. The survey instrument and focus group protocol will be used to gather information from registered participants (educators) of NASA CONNECTS and will provide data regarding their individual experiences to inform NASA of the efficiency, usefulness, and relevance of NASA CONNECTS to its end users (registered participants).

3. RESEARCH DESIGN OVERVIEW: NASA’s work in STEM Engagement is focused on serving students. It is recognized that providing support and resources to educators and educational institutions is vital to effectively engage students. The proposed instrument will used in ongoing program evaluation by NASA. This study falls under the category of a program evaluation, and will be guided by three evaluation questions for the approach and design of this study. Evaluation questions are presented in Figure 1 below.

Evaluation Question 1

How and to what extent is NASA CONNECTS reaching U.S. educators, especially those working with underserved students?

Evaluation Question #2

How and to what extent is NASA CONNECTS increasing the efficiency of use of NASA STEM resources and opportunities for teachers?

Evaluation Question #3

How and to what extent is NASA CONNECTS providing features to enhance educator communication?

Figure 1. Evaluation Questions

The survey will be distributed to all registered participants (educators) of NASA CONNECTS. The online NASA CONNECTS: NGS Community of Practice survey will be available to all (approximately 1,380) registered participants (educators) in April 2023. Although participation in the survey is voluntary, it will provide useful customer input to address programmatic decision making and assess progress of NGS programming toward its goals. Additionally, focus groups will be conducted virtually. Each focus group will be approximately one-hour long. All registered participants (educators) will be invited to participate in a focus group via the CONNECTS platform in order to reach a target of 50 total participants (educators) combined in up to six focus groups. Focus groups will be designed with grade band (K-12, 3-5, 6-8, and 9-12) and formal and informal education categories balanced.

Construct survey item analysis. The survey instruments to be used in this study have been created based on the research questions proposed. Survey items have been created specifically for this evaluation. Survey questions were developed with the intent to provide NASA with information regarding how and to what extent NASA CONNECTS is:

• Increasing the reach of available NASA STEM resources and opportunities to its members

• Increasing the efficiency of use of NASA STEM resources and opportunities for teachers

• Including features to enhance collaboration between educators

• Providing relevant content

Focus Group Protocol. Focus group questions have also been created based on the research questions and allow for deeper understanding of responses to survey questions.

4. TIMELINE: Data under this clearance will be collected in April 2023 - March 2024. This request includes one survey instrument and one focus group protocol that will be used to collect data from registered participants (educators) of NASA CONNECTS.

5. SAMPLING STRATEGY: The survey will be distributed to all registered participants (educators) of NASA CONNECTS (population of approximately 1,380 participants). NASA intends to use an online survey form through Survey Monkey to be posted on the NASA CONNECTS platform to automate its collection of data. Additionally, focus groups will be conducted virtually. Each focus group will be approximately one-hour long. All registered participants (educators) will be invited to participate in a focus group via the CONNECTS platform in order to reach a target of 50 total participants (educators) combined in up to six focus groups. Focus groups will be designed with grade band (K-12, 3-5, 6-8, and 9-12) and formal and informal education categories balanced.

Table 1. Calculation chart to determine statistically relevant number of respondents

Data Collection Source	(N) Population Estimate	(A) Sampling Error +/- 5% (.05)	(Z) Confidence Level 95%/ Alpha 0.05	(P) *Variability (based on consistency of intervention administration) 50%	Base Sample Size	Response Rate	(n) Number of Respondents
NASA CONNECTS Educators (Survey)	1380	N/A	N/A	N/A	1380	N/A	1380
NASA CONNECTS Educators (Focus Group)	1380	N/A	N/A	N/A	50	N/A	50
TOTAL							1430

6. BURDEN HOURS: Burden calculation is based on a respondent pool of individuals as follows:

Data Collection Source	Number of Respondents	Frequency of Response	Total minutes per Response	Total Response Burden in Hours
NASA CONNECTS Educators (Survey)	1380	1	20	460
NASA CONNECTS Educators (Focus Group)	50	1	60	50
TOTAL				510

7. DATA CONFIDENTIALITY MEASURES: Any information collected under the purview of this clearance will be maintained in accordance with the Privacy Act of 1974, the e-Government Act of 2002, the Federal Records Act, and as applicable, the Freedom of Information Act in order to protect respondents’ privacy and the confidentiality of the data collected.

8. PERSONALLY IDENTIFIABLE INFORMATION:

1. Is personally identifiable information (PII) collected? Yes  No

– NOTE: First and Last Name are not collected but demographic information is collected (e.g., gender, ethnicity, race, etc.)

2. If yes, will any information that is collected by included in records that are subject to the Privacy Act of 1974? Yes o No

3. If yes, has an up-to-date System of Records Notice (SORN) been published?

þYes o No

Published March 17, 2015, the Applicable System of Records Notice is NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html.

APPLICABLE RECORDS:

4. Applicable System of Records Notice: SORN: NASA 10EDUA, NASA STEM Engagement Program Evaluation System - http://www.nasa.gov/privacy/nasa_sorn_10EDUA.html

5. Completed surveys will be retained in accordance with NASA Records Retention Schedule 1,

Item 68D. Records will be destroyed or deleted when ten years old, or no longer needed, whichever is longer.

9. PARTICIPANT SELECTION APPROACH:

1. Does NASA STEM Engagement have a respondent sampling plan? þYes o No

If yes, please define the universe of potential respondents. If a sampling plan exists, please describe? The respondent universe for the methodological testing associated with the survey for the NASA CONNECTS: Next Gen STEM (NGS) Community of Practice Evaluation consists of individuals who are registered participants of NASA CONNECTS. These registered participants are both formal and informal educators. While the survey will be offered to all 1,380 registered educators, it is anticipated that 700 or fewer educators will respond. The survey is entered into the Survey Monkey software, and the survey link will be made available through the NASA CONNECTS platform. Additionally, all registered educators (1,380) will be invited to participate in a focus group via the CONNECTS platform in order to reach a target of 50 total participants (educators) combined in up to six focus groups. Focus groups will be designed with grade band (K-12, 3-5, 6-8, and 9-12) and formal and informal education categories balanced.

If no, how will NASA STEM Engagement identify the potential group of respondents and how will they be selected? Not applicable.

10. INSTRUMENT ADMINISTRATION STRATEGY

Describe the type of Consent: o Active þ Passive

6. How will the information be collected:

þ Web-based or other forms of Social Media (NOTE: Survey data will be collected via SurveyMonkey and focus groups will occur virtually using either Adobe Connect, Zoom, or Teams link)

o Telephone

o In-person

o Mail

o Other

If multiple approaches are used for a single instrument, state the projected percent of responses per approach.

7. Will interviewers or facilitators be used? þ Yes  No

11. DOCUMENTS/INSTRUMENTS ACCOMPANYING THIS REQUEST:

þ Consent form

þ Instrument (attitude & behavior scales, and surveys)

þ Protocol script (Specify type: Script)

þ Instructions NOTE: Instructions are included in the instrument

o Other (Specify ________________)

12. GIFTS OR PAYMENT: o Yes þ No  If you answer yes to this question, please describe and provide a justification for amount.

ANNUAL FEDERAL COST: The estimated annual cost to the Federal government is $5,925. The cost is based on an annualized effort of 75 person-hours at the evaluator’s rate of $79/hour for administering the survey instrument, collecting and analyzing responses, and editing the survey instrument for ultimate approval through the methodological testing generic clearance with OMB Control Number 2700-0159, exp. exp. 09/30/2024.

13. CERTIFICATION STATEMENT:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low burden for respondents and low cost for the Federal Government.

3. The collection is non-controversial and does raise issues of concern to other federal agencies.

4. The results will be made available to other federal agencies upon request, while maintaining confidentiality of the respondents.

5. The collection is targeted to the solicitation of information from respondents who have experience with the program or may have experience with the program in the future.

Name of Sponsor: Richard Gilmore

Title: Performance Assessment and Evaluation Program Manager, NASA

Office of STEM Engagement (OSTEM)

Email address or Phone number: [email protected]

Date: 12/14/2023

NASA Office of STEM Engagement 7