NASA Summer of Innovation (SOI) Information Collection Results as of Dec.10.2015

December 10, 2015

NASA Office of Education (OE) Overview of Summer of Innovation (SoI) Information Collection Results

The following information summarizes the results of the NASA Summer of Innovation (SoI) information collection conducted in accordance with OMB Control No. 2700-0150. In fiscal year 2013 (FY2013), the SoI project operated through NASA Centers and SoI Awardees that were previously selected through a competitive process. NASA sought to provide its Awardees/Centers and their collaborators or partners with tools to offer high-quality STEM opportunities to middle school youth, while also allowing Awardees/Centers to leverage their own resources and expand their capacity to influence the educational trajectories of students underrepresented in the STEM fields. Parent survey, student survey and teacher focus group data along with camp observations, interviews and (not required to be cleared by the Paperwork Reduction Act) were collected as part of an implementation/outcome evaluation study based on the SoI project design and evaluation plan recommendations from nationally recognized evaluation experts. Evaluation efforts focused on investigating the SoI stand-alone model, which holds particular promise as a summer engagement model for middle school students that may be replicable across Federal government programs. Camps implementing the stand-alone model offered middle school students a minimum dosage of 30 hours of selected NASA SoI curricula, independent of other summer programs. The purpose of the implementation/outcome evaluation study was to assess quality of program implementation and student outcomes with the following primary objectives:

• Document process and supporting materials for performance data

• Identify the key implementation components of successful  SoI stand-alone activities

• Investigate student outcomes and

• Explore relationships of camp characteristics and student outcomes

Below are the implementation/outcome evaluation study research questions:

Implementation

1. What are the characteristics of SoI camps and their participants?

2. To what extent do SoI camps meet program quality expectations as defined by the PEAR Dimensions of Success (DoS) rubrics?¹

3. What supports and challenges do Awardees/Centers face in implementing SoI curricula? How do they handle these challenges?

4. What staff, materials, and NASA resources are necessary for successful SoI activities?

5. How early and to what extent must plans and preparation begin for successful program implementation?

Outcomes

6. Are SoI students’ levels of STEM interest and engagement similar at the start of SoI and in the fall?

7. Are there correlations between camp characteristics and program quality and the student attitudes and behaviors?

To answer the research questions, the evaluation simultaneously explored the implementation and outcomes of SoI. The evaluation used quantitative and qualitative data and analysis to examine implementation outcomes, estimate change in student behavior and attitudes over time through a comparison of baseline and follow-up survey results, and to describe any correlations between SoI program quality and student outcomes. The implementation evaluation used descriptive statistics and content analytic techniques to describe the approaches implemented at 11 SoI stand-alone camps during the summer of 2013. The outcome evaluation used a one-group, pre-post comparison design to assess whether there were observable changes in SoI students’ interest and engagement in science, how SoI student outcomes compared to other middle school students, and explored if and to what extent outcomes varied by camp quality and characteristics. The following information summarizes the keys findings and recommendations noted in the implementation/outcome evaluation study reports:

Key Findings

• SoI reached its target proportion of students from groups that are traditionally underrepresented in STEM

• Students in most of the observed camp classrooms experienced high quality out-of-school time STEM programming

• Self-reported student data suggested a statistically significant increase in both enthusiasm for science, and interest in out-of-school (OST) STEM activities following participation in SoI

• SoI students reported higher interest than their peers

• Teachers and students reported the hands-on activities as the most compelling aspect of SoI

• Providing teachers with ready-to-use materials and hands-on professional development was viewed as very helpful

Key Recommendations

• Continue to encourage parent and student engagement through outreach events

• Encourage camps to employ K-12 certified teachers and maintain educator-to-student ratios of 1:20 or higher

• Highlight STEM content and opportunity to learn about NASA, which are key reasons why students sign up for SoI

• Encourage camps to employ K-12 certified teachers and maintain educator-to-student ratios of 1:20 or higher

• Increase time and support available for hands-on and problem solving activities in SoI

• Continue to provide educators with access to hands-on curricula and materials as these are key to student engagement

• Continue use of the Dimensions of Success (DoS) to assess camp quality

For more detailed information regarding the SoI implementation/outcome evaluation study design, findings and recommendations, please view the final reports posted on the NASA’s Education Performance Assessment website (http://www.nasa.gov/offices/education/performance/index.html) under the NASA Education Performance Related Reports section.

Summer of Innovation Stand-Alone Implementation Report (2013) - http://www.nasa.gov/sites/default/files/soi_stand-alone_implementation2013.pdf

Summer of Innovation Stand-Alone Program Model Outcomes Report (2013) - http://www.nasa.gov/sites/default/files/soi_stand-alone_program_model_fy2013_outcome_report.pdf

Relationship between SoI and NASA STEM Challenges

The NASA STEM Challenges activity is the result of the redesign the Summer of Innovation (SoI) pilot into a sustainable model for STEM engagement across the Federal STEM Agencies in support of the Federal STEM Education 5-Year Strategic Plan. NASA applied its previous design work and evaluation findings to the design of a STEM Challenges pilot collaboration with the U.S. Department of Education (ED). The NASA STEM Challenges activity information collection (IC) under review modifies the SoI evaluation activities previously approved under OMB control number 2700-0150 to align with the new circumstances of the information collection. The information collection strategy was revised to collect the minimum amount of information required to (1) evaluate the program activity for improvement opportunities, (2) collect outcome data to access the activity model’s effectiveness in meeting objectives, and (3) eliminate information collected via parent surveys and teacher focus groups.

1 See http://www.pearweb.org/tools/dos.html for more information about the DoS.