Phase 4 (Internship Outcome Assessment)_Supporting Statement B_(4-1-21)

SUPPORTING STATEMENT PART B

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

Section B must be completed if the information collected from covered members of the public will be used to statistical purposes.

If the collection owner does not feel this question applies, a possible response would be: “This information collection does not employ statistical methods.” If this is the case Section B can be deleted in your final submission of the supporting statement.

The information in this section pertains to the evaluation survey data collection only.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used.

The Internship Outcome Assessment Intern Survey will be administered as a census survey. That is, the survey will be administered to the full universe of participants in the NASA Internship Program on a staggered implementation scheduled. The NASA Internship Program conducts sessions in the Fall, Spring and Summer annually. The size of the population for the NASA Intern Survey is estimated below in Table 2.

The following table displays the timing and expected response rate for each form.

Table 2. Potential Population Estimate for Internship Outcome Assessment Intern Survey Collection based on historical performance

Form	Target Population	Frequency	Population Estimate
Intern Survey	Students who participated in a NASA Internship activity	Ongoing; program/project managers will administer surveys at the conclusion of an Internship session activity – survey will be completed via a web-link to the survey (emailed to the participant after the activity’s completion).	2,100

2. Describe the procedures for the collection of information.

This information collection utilizes a web-based survey (e.g., SurveyMonkey). Each respondent will be invited to complete a NASA Intern Survey for the Internship session in which they participate. Surveys will be emailed to participants so they can be completed via the survey url.

The data collection is a census, so sampling and estimation procedures are not required. Data collection cycles will line up with the Internship sessions conducted in Fall, Spring, and Summer on an annual basis.

Table 3. Timing of Data Collection for Internship Outcome Assessment Intern Survey

Form	Target Population	Frequency
Intern Survey	Students who participated in a NASA Internship activity	Ongoing; program/project managers will administer surveys at the conclusion of an Internship session activity – survey will be completed via a web-link to the survey (emailed to the participant after the activity’s completion).

3. Describe methods to maximize response rates and to deal with issues of non-response.

As described earlier, the NASA Intern Survey will be administered to the full universe of participants of the NASA Internship Program. Survey response rates can be calculated since NASA also collects (from project staff) participation counts by participant type (e.g., educators, K-12 students, higher education students). In order to maximize response rates, OSTEM staff will conduct a series of training sessions on the Internship Outcome Assessment for NASA Center STEM Engagement staff to review the NASA Intern Survey administration processes and procedures. The OSTEM Performance and Evaluation Team will also provide helpdesk services on data collection to OSTEM staff. The following steps will be recommended to Center staff:

• Provide opportunities during NASA Internship Program sessions to complete surveys;

• Utilize web-based platform to develop NASA Intern Survey along with processes to send email reminders to encourage participants to fill out the NASA Intern Survey; and

• Follow up with nonrespondents by re-sending surveys or contacting by phone.

Non-response bias analysis cannot be conducted at this time for subgroup response to surveys, since NASA OSTEM does not collect descriptive data on individual participants beyond participant type. If individual item response rates are more than 15 percent below the expected response rate, NASA will conduct an item non-response analysis to determine if the data are missing at random at the item level for at least the items in question. In those cases where the analyses indicate that the data are not missing at random, the amount of potential bias will inform the decision to publish data from individual items.¹

4. Describe any tests of procedures or methods to be undertaken.

The Internship Outcome Assessment plan was developed after extensive consultation with program managers, project administrators (i.e. Grant PIs), and feedback from project participants and support personnel. NASA OSTEM also conducted Expert Review Panel to solicit advice from external experts in the field of assessment, data collection and work-based learning programs to inform the Internship Outcome Assessment plan.

The NASA Intern Survey was field tested under the NASA Methodological Testing Generic Clearance process using quantitative methods including psychometric analysis; rasch item fit statistics; item and person reliability and separation; and rating scale analysis before finalizing the instrument for the Internship Outcome Assessment information collection.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Individuals Consulted on the Statistical Aspects of the Design

• Performance and Evaluation Contractor - Paragon TEC, Inc.

  • Vince Schiavone, Ph.D, Program Manager

  • Tara Strang, Ph.D, Evaluation Specialist

• Carla Johnson, Ph.D., Executive Director, Friday Institute at North Carolina State University

• Toni Sondergeld, Ph.D., Associate Professor, Drexel University

Individuals Responsible for Data Collection and Analysis

Richard Gilmore, Performance Assessment and Evaluation Program Manager, NASA Office of STEM Engagement, 216-433-5493

1 Office of Management & Budget, Standards and Guidelines for Statistical Surveys. Retrieved May 26, 2020, at: http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf.

3