Phase 2 (Evaluation Surveys)_Supporting Statement B_(10-8-20)

SUPPORTING STATEMENT PART B

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

Section B must be completed if the information collected from covered members of the public will be used to statistical purposes.

If the collection owner does not feel this question applies, a possible response would be: “This information collection does not employ statistical methods.” If this is the case Section B can be deleted in your final submission of the supporting statement.

The information in this section pertains to the evaluation survey data collection only.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used.

All NASA STEM Gateway (Universal Registration and Data Management System) evaluation surveys will be administered as census surveys. That is, the surveys will be administered to the full universe of participants of NASA STEM engagement activities on a staggered implementation scheduled based on the type of engagement activity. NASA STEM engagement activity types include but are not limited to internships, challenges, educator professional development, and pre-college/college STEM engagement experiences. Internship sessions are conducted in the Fall, Spring and Summer while challenges, educator professional development, and pre-college/college STEM engagement experiences can be conducted at any time throughout the year. The size of the population for each instrument is estimated below in Table 2.

The following table displays the timing and expected response rate for each form.

Table 2. Potential Population Estimate for NASA STEM Gateway (Universal Registration and Data Management System) Survey Collection, by Form based on historical performance

Form	Target Population	Frequency	Population Estimate
Challenges and Internship Participant Survey	Students who participated in a NASA STEM Engagement activity	Ongoing; program/project managers will administer surveys at the conclusion of an STEM engagement activity – survey will be completed via a web-link to the survey (emailed to the participant after the activity’s completion).	5,500
Internship Applicants and Awardees Longitudinal Survey	Students who applied for or were selected for a NASA internship.	Ongoing; The Performance and Evaluation Team will administer surveys at the end of the internship cycle via a via a web-link to the survey (emailed to the participant).	5,500
Participant Satisfaction Surveys	Students or educators who participate in a NASA STEM Engagement activity	Ongoing; program/project managers will administer surveys at the conclusion of an activity – survey will be completed electronically	39,750
S-STEM Survey	Students (grades 3-8) who participate in a NASA STEM Engagement activity	Ongoing; program/project managers will administer surveys at the conclusion of an activity – survey will be completed electronically	5,500
Educator Feedback Form	K-12 informal and formal educators who participate in a NASA professional development activity	Ongoing; facilitator of professional development activity will administer surveys after an event for educators – survey will be completed electronically	12,250
Pre-Training Survey	K-12 informal and formal educators who participate in a NASA professional development activity	Ongoing; facilitator of professional development activity will administer surveys after an event for educators – survey will be completed electronically	5,500
Post-Training Survey	K-12 informal and formal educators who participate in a NASA professional development activity	Ongoing; facilitator of professional development activity will administer surveys after an event for educators – survey will be completed electronically	5,500
Parent Survey	Parents of K-12 students who participate in a NASA STEM Engagement activity	Ongoing; educators and/or team leads of pre-college STEM Engagement activities will administer surveys after an event for students – survey will be completed electronically	2,000

2. Describe the procedures for the collection of information.

This information collection utilizes web-based reports and surveys. Each respondent will be invited to complete a form(s) for each NASA-sponsored STEM engagement activity in which they participate. Surveys can be completed 1) survey urls can be emailed to participants or by 2) on-site after an activity (if terminals and internet connection are available).

The data collection is a census, so sampling and estimation procedures are not required. Use of data collection cycles less frequent than annual is not anticipated.

Table 3. Timing of Data Collection for NASA STEM Gateway (NASA Universal Registration and Data Management System) Surveys, by Form

Form	Target Population	Frequency
Challenges and Internship Participant Survey	Students who participated in a NASA STEM Engagement activity	Ongoing; program/project managers will administer surveys at the conclusion of an STEM engagement activity – survey will be completed via a web-link to the survey (emailed to the participant after the activity’s completion).
Internship Applicants and Awardees Longitudinal Survey	Students who applied for or were selected for a NASA internship.	Ongoing; The Performance and Evaluation Team will administer surveys at the end of the internship cycle via a via a web-link to the survey (emailed to the participant).
Customer Satisfaction Survey	Students or educators who participate in a NASA STEM Engagement activity	Ongoing; program/project managers will administer surveys at the conclusion of an activity – survey will be completed electronically
S-STEM Survey	Students (grades 3-8) who participate in a NASA STEM Engagement activity	Ongoing; program/project managers will administer surveys at the conclusion of an activity – survey will be completed electronically
Educator Feedback Form	K-12 informal and formal educators who participate in a NASA professional development activity.	Ongoing; facilitator of professional development activity will administer surveys after an event for educators – survey will be completed electronically
Pre-Training Survey	K-12 informal and formal educators who participate in a NASA professional development activity	Ongoing; facilitator of professional development activity will administer surveys after an event for educators – survey will be completed electronically
Post-Training Survey	K-12 informal and formal educators who participate in a NASA professional development activity	Ongoing; facilitator of professional development activity will administer surveys after an event for educators – survey will be completed electronically
Parent Survey	Parents of K-12 students who participate in a NASA STEM Engagement activity	Ongoing; educators and/or team leads of pre-college STEM Engagement activities will administer surveys after an event for students – survey will be completed electronically

3. Describe methods to maximize response rates and to deal with issues of non-response.

As described earlier, all evaluation surveys will be administered to the full universe of participants of NASA STEM engagement activities. Survey response rates can be calculated since NASA also collects (from project staff) participation counts by participant type (e.g., educators, K-12 students, higher education students). In order to maximize response rates, OSTEM staff will conduct a series of training sessions on the NASA STEM Gateway (Universal Registration and Data Management System) for NASA Center STEM Engagement staff to review evaluation survey administration processes and procedures. The OSTEM will also provide helpdesk services on the NASA STEM Gateway and data collection to OSTEM staff. The following steps will be recommended to Center staff:

• Provide opportunities during NASA STEM engagement activities to complete surveys;

• Utilize Marketing Cloud platform to develop participant journeys to send email reminders to encourage participants to fill out the follow-up surveys at home or wherever they have internet access; and

• Follow up with nonrespondents by re-sending surveys or contacting by phone.

Non-response bias analysis cannot be conducted at this time for subgroup response to surveys, since NASA OSTEM does not collect descriptive data on individual participants beyond participant type. If individual item response rates are more than 15 percent below the expected response rate, NASA will conduct an item non-response analysis to determine if the data are missing at random at the item level for at least the items in question. In those cases where the analyses indicate that the data are not missing at random, the amount of potential bias will inform the decision to publish data from individual items.¹

4. Describe any tests of procedures or methods to be undertaken.

The NASA STEM Gateway (Universal Registration and Data Management System) builds on four former systems that were used to monitor NASA programs. It was developed after extensive consultation with program managers, project administrators (i.e. Grant PIs), and feedback from project participants. NASA OSTEM has conducted Expert Review Panels to solicit advice from experts in the field of data collection and data management. NASA STEM Engagement and contract representatives have been working with the OSTEM Information Technology (IT) Team and the Performance & Evaluation (P&E) Team to design the new system.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Individuals Consulted on the Statistical Aspects of the Design

• Carla Johnson, Ph.D., Executive Director, Friday Institute

• Performance and Evaluation Contractor - Paragon TEC, Inc.

  • Anne Kotynski Gooding, Ph.D, Program Manager (216-433-3817)

  • Tara Strang, Ph.D, Evaluation Specialist

• Performance and Evaluation Contractor – Peerless Technologies, Inc

  • Jomill Wiley, Ph.D, Project Manager (216-433-2779)

  • Vanessa Mullins, Ph.D, Technical Manager

  • David Miller, Ph.D, Evaluation Specialist

Individuals Responsible for Data Collection and Analysis

Richard Gilmore, Performance Assessment and Evaluation Program Manager, NASA Office of STEM Engagement, 216-433-5493

1 Office of Management & Budget, Standards and Guidelines for Statistical Surveys. Retrieved May 26, 2020, at: http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf.

4