updated NASA SOI OMB Part B 7 21 10 (2)

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART B

NASA Summer of Innovation Pilot

SURVEY DATA COLLECTION

National Aeronautics and Science Administration

July 2, 2010

Part B: Collection of Information Employing Statistical Methods

This study will use statistical methods to assess changes in teacher and student short-term outcomes in the following major areas covered in the survey instruments: STEM awareness, motivation, and attitudes. The analysis for this study will assess changes in outcomes from pre-program to post-program focusing on short-term outcomes. Statistical analyses will include descriptive statistics, correlations, analysis of variance, and multiple regressions where possible.

Future data collection and statistical analyses is contingent on the on-going support and funding for Summer of Innovation. If continued, the national evaluator will track students at the NASA Center partnerships yearly to continue surveying students on STEM awareness, motivation, and attitudes. Local evaluators will be responsible for collecting this data at their sites. The analysis will also focus on longer-term outcomes, such as student course taking patterns and interest in post-secondary education and careers. Because the Summer of Innovation is a 3-year project for middle school students, longer-term outcomes such as course taking patterns and interest in post-secondary education and careers, will be available for collection when the respondents are in high school.

B.1 Respondent Universe and Sampling Methods

The Summer of Innovation (SOI) pilot project will be implemented across three types of sites: Space Grant Consortiums, a Sub-award, and NASA Center Partnerships. Abt Associates, the national evaluator, will synthesize data across all Summer of Innovation sites to evaluate the efficacy of SOI on teacher and student outcomes. The NASA Centers Partnerships’ collaborative activities will involve students only; therefore, collection of information and analysis at these sites will focus exclusively on students. Local evaluators at the Space Grant Consortiums and at the Sub-award site will gather information from all students and teachers at their sites.

The purpose of the national evaluation of the Summer of Innovation pilot project is to generate lessons learned and hypotheses regarding impact that will be rigorously tested in years 2 and 3. Some of the Space Grant Consortiums and the Sub-Award sites consist of 1-6 specific collaborative activities, each serving up to 600 students. The collaborative activities may have programming that differs in terms of duration, content focus, and activities. Therefore, the plan is to survey the universe, or population, of teachers and students within the sites to develop an understanding of how efficacy relates to collaborative activity model.

At the NASA Center Partnership sites, NASA will purposively select a sample of about 6 collaborative activities from across 10 NASA Centers at which around 1,000 students will participate. Within the purposeful sampling framework (Patton, 1990), we will use criterion sampling, where we will a-priori develop rubrics, or criteria, to select Summer of Innovation sites. The criteria used to select SOI sites will include the specific content area in STEM (e.g. physics), duration, types of activities, and recruitment procedures. Of the sampled SOI collaborative activities, all student participants will be surveyed. Therefore, the only sampling that would take place across the three types of SOI implementations is a purposeful sample of about 6 SOI collaborative activities implemented at NASA Centers. The goal of the sampling approach is to obtain a sample of students with a variety of program experiences and characteristics. We believe that the purposeful sampling technique to identify similar SOI collaborative activities will produce a sufficient sample of students and teachers to answer our research questions regarding the overall efficacy and variation in outcomes.

Description of the Study Sample

Exhibit B.1 shows the total number of students and teachers included in the evaluation. The sample sizes are based on proposals and projections from NASA. Some components have activities for both teachers and students, while others focus only on student activities. Therefore, the expected sample size for students and teachers are listed separately.

Exhibit B.1 Description of Study Sample
Summer of Innovation Components	Student Population/ Sample Sizea	Teacher Population/ Sample Sizea
Space Grant Consortiums	6,095/ 6,095	673/ 673
External Sub Award	5,000/ 5,000	350/ 350
NASA Center Partnerships	10,000/ 1,000	n/a
Total	21,095/ 12,095	1,023/ 1,023
a Based on proposal submissions of the Space Grant Consortiums and handouts from the NASA SOI Planning Meeting on May 5, 2010. The NASA Centers, External sub-awards, and federal agency partnerships are projections made by NASA’s Office of Education. NASA Centers and Federal Agency Partners do not have a teacher-training component.

B.2 Information Collection Procedures

This request for clearance includes the teacher and student surveys across the three SOI site types. It is important to note that while all sites include students, only the Space Grant Consortiums and the External Sub Award will also have teacher training collaborative activities. The surveys will be implemented before the Summer of Innovation activities (baseline) at the Space Grant sites and immediately after the activities across all sites (post-implementation). These surveys will be administered by the local evaluators (Space Grant Consortiums and the Sub-Award) or site administrators at the NASA Center Partnerships using a paper-and-pencil format. The national evaluators at Abt Associates will provide technical assistance to all sites to ensure rigorous and systematic data collection procedures.

Exhibit B.2 Summary of Survey Data Collection Procedures
Survey Wave	Data Collection	Survey Mode
Baseline Student Survey NASA Centers External Sub-award Space Grant Consortiums	SOI Administrators Local Evaluator Local Evaluators	Paper and Pencil Paper and Pencil Paper and Pencil
Post-program Student Survey NASA Centers External Sub-award Space Grant Consortiums	SOI Administrators Local Evaluator Local Evaluators	Paper and Pencil Paper and Pencil Paper and Pencil

Baseline Survey: Procedures for Data Collection

The national evaluator will provide technical assistance to the local evaluators on data collection procedures. For the NASA Center Partnerships, the national evaluator will collect student survey data. At baseline, the local or national evaluators will obtain parent consent and student assent forms (see Appendix F and G), and teacher consent forms prior to data collection (see Appendix H). The consent forms will have information about the evaluation, the purpose of data collection, potential risk, and confidentiality assurances. Prior to any data collection, the national evaluator will obtain Institutional Review Board (IRB) approval. Parents and teachers who agree to participate (or grant permission for their child to participate) will return the following materials to the evaluators prior to data collection:

• Signed Parent Permission and Consent form

• Child assent form with child’s name signed or printed

• Completed student survey (baseline) sealed in manila privacy envelope

• Signed Teacher Permission and Consent form

• Completed teacher survey (baseline) sealed in manila privacy envelope

These materials will be part of the NASA SOI application process. To ensure high response rates and reduce burden on respondents, we propose other methods of obtaining consent and surveys. An on-line option is available, where parents, teachers, and students can provide written permission in an on-line form prior to completing an on-line web survey. Another option is telephone interviews, where parents, teachers, and students can provide verbal consent prior to completing a telephone interview. Parents, teachers, or students who decline to participate in the study will not need to complete the survey, but they can still take part in the Summer of Innovation activities. They will not be part of the analytic sample; therefore, they will not complete the baseline survey, nor will they receive post-test surveys in the future.

Follow-up Survey: Procedures for Data Collection

The follow-up surveys will be administered immediately after the Summer of Innovation activities in the summer. The purpose of the post-test surveys is to collect data on the same interests and attitudes about science surveyed at pre-test at a second point in time. For the post-test round of data collection, the evaluators will survey only those individuals who signed consent/assent forms.

The evaluators will administer surveys on site, as part of the program. If students or teachers were absent on the day of the survey administration, a paper and pencil survey data collection will occur shortly after the end of the program. As is true throughout the duration of the study, consent and parental permission to participate may be withdrawn at any time without penalty or change in assignment status.

B.3 Methods to Maximize Response Rates

The key for maximizing response rates among teachers and students is to provide opportunities for them to complete the survey while at the Summer of Innovation activities over the summer. Post summer program, the key is tracking students to update contact information. Several methods will be used to maximize response rates during the pre- and post-program surveys and to address non-response, such as:

• Providing opportunities to complete the survey while at the summer program;

• If respondents did not complete the survey during the summer program, providing respondents with a pre-paid, pre-addressed, Business Reply Envelope for ease of return mailing. Respondents can simply return the completed surveys via the U.S. Postal Service, rather than having to locate a FedEx or UPS drop-box;

• Designing a survey so that it take minimal effort (burden) and the reading level is appropriate for this age group;

• Following up with non-respondents via postcards and phone calls; and

• Offering non-respondents the option of completing the survey via internet or by telephone using Computer Assisted Telephone Interviewing (CATI) capabilities for the follow-up surveys.

We expect to achieve response rates of 80 percent or higher for both students and teachers at baseline and the first follow-up survey (post-test survey).

B.4 Test of Procedures

These procedures were tested and refined as follows. First, existing instruments with established psychometric characteristics were selected after an extensive literature review (e.g., Modified Attitudes Towards Science Inventory (mATSI), Weinburgh and Steele, 2000; Test of Science Related Attitudes (TOSRA), Fraser, 1981; and the Math and Science Interest Survey, Hulett, Williams, Twitty, Turner, Salamo, and Hobson, 2004). Second, experts in the field reviewed draft and final instruments for content validity and clarity. Finally, NASA Office of Education staff reviewed the instruments for final approval.

B.5 Individuals Consulted on Statistical Aspects of Design

The plans for statistical analyses for this study were primarily developed by Abt Associates, Inc. and the Education Development Center (EDC). The team is led by Ryoko Yamaguchi, Project Director; Alina Martinez, Principal Investigator; Amanda Parsad, Project Quality Advisor; and Kristen Neishi, Deputy Project Director. The surveys were developed by Jacqueline DeLisi, Daphne Minner, and Linda Hirsch at EDC. Hilary Rhodes will be replacing Dr. Yamaguchi as Project Director as of June 25, 2010. Contact information for these individuals is provided below.

Abt Associates, Inc.
Ryoko Yamaguchi	Project Director	301-634-1778	[email protected]
Hilary Rhodes	Project Director (6/25/10)	617-520-3516	[email protected]
Alina Martinez	Principal Investigator	617-349-2312	[email protected]
Amanda Parsad	Project Quality Advisor	301-634-1791	[email protected]
Kristen Neishi	Deputy Project Director	301-634-1759	[email protected]
Education Development Center
Jacqueline DeLisi	Survey Task Leader	617-969-5979	[email protected]
Daphne Minner	Survey Developer	617-969-5979	[email protected]
Linda Hirsch	Survey Developer	617-969-5979	[email protected]

National Aeronautics and Space Administration Part B: Collection of Information 1