Supporting Statement A - OMB - NASA SEMAA 1 5 08

Cambridge, MA

Lexington, MA

Hadley, MA

Bethesda, MD

Chicago, IL

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138

Supporting Statement for Paperwork Reduction Act Submission to OMB: Statement A

Evaluation of NASA’s Science, Engineering, Mathematics, and Aerospace Academy (SEMAA)

January 5, 2009

Prepared for

National Aeronautics and Space Administration

300 E Street SW

Washington, DC, 20546

Project Officer:

Bernice Alston

Prepared by

Alina Martinez, Carter Smith

Abt Associates Inc.

55 Wheeler St.

Cambridge, MA 02138

Clemencia Cosentino de Cohen

The Urban Institute

2100 M Street, NW

Washington, DC 20037

Project Director:

Alina Martinez

Supporting Statement A for

Evaluation of the NASA SEMAA Project

January 5, 2008

Name: Walter Kit, NASA Clearance Officer,

Address: NASA Headquarters

300 E Street, SW., JE0000

Washington, DC 20546

Telephone: (202) 358–1350

Fax:

Email: [email protected]

Table of contents

A.1 Circumstances Making the Collection of Information Necessary 3

A.2 Purpose and Use of the Information Collection 4

A.3 Use of Information Technology and Burden Reduction 7

A.4 Efforts to Identify Duplication and Use of Similar Information 8

A.5 Impact on Small Businesses or Other Small Entities 9

A.6 Consequences of Collecting the Information Less Frequently 9

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 9

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 9

A.9 Explanation of Any Payment of Gift to Respondents 10

A.10 Assurance of Confidentiality Provided to Respondents 10

A.11 Justification for Sensitive Questions 11

A.12 Estimates of Hour Burden Including Annualized Hourly Costs 11

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers 12

A.14 Annualized Cost to the Federal Government 12

A.15 Explanation for Program Changes or Adjustments 12

A.16 Plans for Tabulation and Publication and Project Time Schedule 12

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 13

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 13

List of Attachments:

• Attachment 1. Site Staff Interview Protocol

• Attachment 2. Student Survey

• Attachment 3. Parent Survey

• Attachment 4. Parent Permission and Consent Form

• Attachment 5. Child Assent Form

A.1 Circumstances Making the Collection of Information Necessary

The National Aeronautics and Space Administration (NASA), requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, clearance for NASA to conduct data collection efforts for the evaluation of the NASA Science, Engineering, Mathematics, and Aerospace Academy (SEMAA) project. Current authorization for NASA’s research and information dissemination activities is contained in the National Aeronautics and Space Act of 1958, as amended.

The SEMAA project was established in 1993 as a joint venture between the Glenn Research Center and Cuyahoga Community College and was expanded to other sites throughout the nation. SEMAA is designed to increase the participation and retention of histori­cally underserved and underrepresented K-12 youth in the areas of Science, Technology, Engineering, and Mathematics (STEM). The goals of SEMAA are to:

• Inspire a more diverse student population to pursue careers in STEM-related fields;

• Engage students, parents/adult family members and teachers by incorporating emerging technologies.

• Educate students utilizing rigorous STEM curricula, designed and implemented as only NASA can.

SEMAA has three components:

• Curriculum enhancement activities aligned to national science, math and technology standards to provide Kindergarten through-12^th grade students up to 441 hours of exposure to inquiry-based studies in STEM fields related to NASA’s four mission directorates;

• An Aerospace Education Laboratory with an Advanced Flight Simulator, laboratory-grade wind tunnel, short-wave receiver and handheld GPS technology to provide middle- and high-school students with hands-on engagement in real-world challenges related to aeronautics and micro-gravity research situations;

• A Family Café providing parents/family caregivers with information on parenting and STEM-education information and opportunities available for their children

SEMAA activities typically occur on Saturday mornings in three-hour sessions for five to eight consecutive weeks (K-2^nd grade attend for 5 weeks; 3-12^th grade attend for eight). For each grade level, SEMAA offers a distinct, age-appropriate set of curricular enhancements, with an emphasis on hands-on projects and exposure to current technology. Three eight-week sessions are offered each academic year; during the summer, a one-week “bridge” sessions is offered for students to transition from the previous grade level SEMAA activities to those anticipated for the following grade level. In a given academic year, a student may enroll in one eight-week session and one summer session. Students may enroll at any grade level; there are no prerequisites.

Although the “Saturday model” is the original and most common SEMAA model, some sites offer an after-school or in-school program instead. In school SEMAA models typically expand the SEMAA curriculum for integration into the existing science curricula in the schools. After-school models are sometimes used in areas where families have difficulty attending on Saturdays (e.g., families with transportation barriers).

There are currently 14 SEMAA sites in operation (as of FY 2009). In the past three years alone, over 200,000 students, parents/adult family members and teachers have been involved in the SEMAA project. In 2007, SEMAA was recognized as one of the top 18 programs in the Innovations in the American Government Awards competition by the Ash Institute for Democratic Governance and Innovation at Harvard University’s John F. Kennedy School of Government.

Despite its recognition as an innovative program, a recent report by the National Research Council¹ emphasized the need for a rigorous evaluation of SEMAA. This request for clearance will allow a rigorous evaluation of SEMAA to be conducted. The evaluation of this program will help NASA assess the extent to which the program is realizing its goals.

A.2 Purpose and Use of the Information Collection

The purpose of this study is to collect data that supports the evaluation of the SEMAA project. This evaluation consists of two modules: (a) an impact evaluation and (b) an implementation evaluation. The purpose of the data collected for the impact evaluation is to address the following research questions:

What impact does SEMAA have on students and parents? Specifically:

• Does SEMAA enhance student interest and engagement in STEM learning?

• Does SEMAA enhance students’ likelihood to pursue STEM coursework and career goals?

• Does SEMAA enhance parental support for their child’s engagement in STEM learning?

• Does SEMAA enhance parental support for their child’s pursuit of STEM coursework and career opportunities?

The purpose of the data collected for the implementation evaluation is to address the following research questions:

To what extent are variations in SEMAA implementation models associated with differences in the outcomes for SEMAA students and parents? Specifically:

• Are some SEMAA characteristics associated with better outcomes for students and parents?

• Are some SEMAA characteristics more sustainable in the long term than other characteristics?

• How can variation in SEMAA implementation be used to improve the SEMAA project as a whole?

The proposed impact evaluation is a multi-site randomized control trial (RCT). Student applicants to SEMAA at each of up to 8 participating sites will be randomly assigned to a treatment (participation in an eight-week SEMAA curricular enhancement program in the Fall of 2009) or control (delayed participation in SEMAA until the Winter or Spring of 2010 session) condition. Parents of student applicants will be assigned to the same condition as their children; those parents in the treatment group will be eligible, but not required, to attend the Family Café component of the eight-week SEMAA program. Both prior participants and new applicants to SEMAA will be randomly assigned to enrollment in the Fall 2009.

In addition, all student applicants of the same household who are applying for the Fall 2009 SEMAA session will be assigned to the same experimental condition.

• Prior to random assignment, one sibling will be selected at random to be the “target” student for purposes of the evaluation.

• The target student will be randomly assigned to treatment or control status;

• his/her sibling will be assigned by default to this same condition.

The purposes of this random assignment procedure are twofold:

• to reduce burden on families;

• to reduce the likelihood of contamination of the control group.

Burden: If two children in the same household were assigned to different conditions, many parents might incur extra transportation or childcare costs: while the parent accompanied one child to the Saturday SEMAA session, another adult could be needed to care for the non-SEMAA sibling.

Contamination: If one sibling were assigned to treatment and the other to control, it is possible—perhaps even likely—that a parent would have to bring the control child along to the SEMAA site during Saturday sessions. This control child could thus be directly exposed to SEMAA. Alternatively, indirect exposure could occur even if a parent were able to provide separate care during SEMAA sessions for each sibling. For example, if the SEMAA-enrolled (treatment group) child brought home projects, activities, or materials, the control child conceivably would have access to these materials. This access would interfere with the intended assignment of the control child (embargoed exposure to SEMAA) during the Fall 2009 session.

The proposed data to be collected for the impact evaluation consists of a Student survey (see Attachment 2) and a Parent survey (see Attachment 3), each administered twice. To provide pre-test or baseline data, all study participants will be asked to complete the survey prior to the start of the eight week SEMAA session. At the end of the eight-week SEMAA session, both treatment group and control group participants will be asked to complete the survey a second time to provide follow-up (post-test) data. Data will be used in impact analyses of the difference between treatment and control group members’ post-test minus pre-test responses.

Exhibit A.1 summarizes the key outcomes and associated indicators to be assessed with the student and parent surveys. Drafts of the proposed survey instruments themselves are included as Attachments 2 and 3.

Exhibit A.1: Outcomes and indicators for the impact evaluation of SEMAA
Outcome	Sample indicators
Student interest in science	Desire to learn about science and math, scientists Participation in science clubs, fairs, camps, competitions Interest in college major in science or math field Planned coursework in science and math Career goals in science and math fields
Student self-efficacy in science	Confidence in own ability to do well in science/math courses Anxiety about science/math problem-solving Persistence with problem solving
Student support for NASA and STEM research and development	Student attitude about the value of STEM research to society Student support for R&D in STEM fields
Student achievement in STEM coursework	Student self-reported course grade in prior year science and mathematics courses Student self-reported course grade in current academic year science and mathematics courses
Parent support for student’s education in science	Desire for student to learn about science, math, scientists Support for student’s participation in informal science activities Support for student majoring in a STEM field in college Support for student enrolling in STEM coursework in school Support for student pursuing STEM career goals
Parent perception of student’s ability in science	Confidence in student’s ability to do well in science/math courses Perception of student’s level of anxiety Perception of student’s persistence in problem solving
Parent support for NASA and STEM research and development	Parental attitude about the value of STEM research to society Parental support for R&D in STEM fields
Parent report of student achievement in STEM coursework	Parent-reported course grade for student’s prior year science and mathematics courses Parent-reported course grade for student’s current academic year science and mathematics courses

The proposed implementation evaluation will capitalize on existing data about local site implementation from extant sources including SEMAA program performance and participant data (e.g., NEEIS), other program documents (e.g., project proposals) maintained by NASA, site-specific documents, background data, and websites. However, to fill gaps in the extant data, structured interviews with Site Directors at each of the 14 active sites will be necessary. The data will be used to examine how program characteristics are related to cross-site variation in such factors as student and parent attendance, program experiences, and enthusiasm for STEM learning. The proposed interview protocol is included as Attachment 1.

Information collected during the evaluation will be used in multiple ways. First, the data will provide NASA evidence concerning SEMAA’s effectiveness at meeting its intended goals, namely to encourage students, particularly those from underserved and underrepresented groups to pursue STEM educational and career opportunities. In addition, the results of this study will provide evidence about program effectiveness and the relative merits of additional funding for SEMAA sites from federal or non-federal sources. The combined results of the impact and implementation evaluations will assist NASA in determining whether there is particular variation among sites that helps explain the relative effectiveness of individual sites.

A.3 Use of Information Technology and Burden Reduction

The available use of automated information technology is limited in this study. The study will rely on baseline data and follow-up data gathered from self-administered surveys of applicants to the local sites. Baseline surveys will be completed during the application process and follow-up surveys will be sent to participants’ homes because access to technology may be limited in the homes. To minimize burden, the design process for the study identified and reviewed the information that could be obtained through extant data sources. Only data not available through other sources will be collected. Steps to reduce the burden on respondents will be taken.

Director Interviews. A thorough review of project and program documents will be carried out to avoid collecting data already available and reduce burden on participants. These documents include project proposals, annual reports, and SEMAA program summary performance and participant data from the NEEIS database. This information was used to guide the development of the Director Interview Protocol included in Attachment 1.

In addition, the interview protocol was modeled after similar protocols used in prior studies and with known administration times. Interviews will be conducted by telephone and restricted to one hour to reduce burden on participants

Student survey. To reduce burden on respondents, surveys have been developed from existing protocols with known administration times (Modified Attitudes Towards Science Inventory, Weinburgh and Steele, 2000; and the Math and Science Interest Survey, Hulett, Williams, Twitty, Turner, Salamo, and Hobson, 2004). In addition, pilot testing of surveys has been completed to screen out items that are confusing, unclear, or time-consuming to respondents. These measures help limit the burden on respondents, both in terms of time needed to complete the survey, readability, and adequacy of response choices offered.

To the extent possible, the questions will be in a multiple-choice format, with the choices carefully selected to be applicable to most respondents (See Attachment 2 for the student survey).

Parent survey. To reduce burden on respondents, surveys have been developed from existing protocols with known administration times (Modified Attitudes Towards Science Inventory, Weinburgh and Steele, 2000; and the Math and Science Interest Survey, Hulett, Williams, Twitty, Turner, Salamo, and Hobson, 2004).. In addition, pilot testing of surveys has been completed to screen out items that are confusing, unclear, or time-consuming to respondents. These measures help limit the burden on respondents, both in terms of time needed to complete the survey, readability, and adequacy of response choices offered.

To the extent possible, the questions will be in a multiple-choice format, with the choices carefully selected to be applicable to most respondents (See Attachment 3 for the parent survey).

A.4 Efforts to Identify Duplication and Use of Similar Information

The proposed data collection is one element of a larger evaluation, which will also utilize program monitoring data that has been collected via the NASA Education Evaluation and Information System (NEEIS). As part of the study design a thorough assessment of available data sources was conducted to investigate what information is contained within the extant sources and what new data would need to be collected. The alternatives to primary data collection were carefully explored through a combination of literature review, discussions with NASA personnel knowledgeable about the extant data sources, and direct inspection and analysis of both data collection forms and data from the NASA Education Evaluation and Information System (NEEIS). This comprehensive review revealed that:

1. Extant data sources do not include information collected from non-participants in SEMAA, a key control group needed for the proposed effort; NEEIS does not contain data needed for the proposed effort. For example, NEEIS data are collected only from students and parents who participate in SEMAA and not from a comparison group. NEEIS participant feedback data are collected only once, after participation in SEMAA, preventing any investigation of change in key outcomes before and after exposure to SEMAA.

2. Extant data sources lack an adequate identification system to link data collected from the respondent at multiple points in time (e.g., the same student may provided responses to a SEMAA feedback survey after participation in sixth and seventh grades, but this respondent has two different and non-linked identification numbers in NEEIS);

3. In addition, the extant data do not provide sufficient information on key outcomes necessary for the conduct of the proposed study of SEMAA, including, for example, measures of:

• student self-efficacy in STEM (i.e., belief in own ability to do well in STEM coursework);

• student achievement in formal STEM coursework;

• students’ career goals and career outcomes

• parental attitudes towards STEM education for their children;

• parental belief in children’s ability to do well in STEM coursework;

• parental support for children’s pursuit of STEM coursework and career opportunities

4. There are gaps in the information available about the specific implementation practices of the local sites.

A.5 Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this study.

A.6 Consequences of Collecting the Information Less Frequently

Failure to collect the information proposed in this request will prevent NASA from assessing the degree to which the SEMAA program has led to improvements in targeted student and parent outcomes or from learning systematically whether project approaches are associated with positive impacts.

Respondents will be contacted only twice to collect information for this study. Collecting the data less frequently would undermine the study design. The study design depends critically on two comparisons: (1) a comparison of post-test to pre-test student and parent responses; and (2) a comparison of treatment and control group at both time points The inclusion of pretest data also provides a measure of change in key outcomes prior to and after exposure (or non-exposure) to SEMAA.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

The project will fully comply with the guidelines of 5 CFR 1320.5. No special circumstances apply to this data collection.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

A copy of the 60-day Federal Register Notice is provided with this application.

Published October 23, 2008, volume 73, number 206 and page 36211

[Describe comments here, if any received, and describe actions taken.]

Consultation on the study design was conducted by the research firms contracted by NASA to prepare an evaluation plan and conduct the study. In addition, the evaluation plan was developed in consultation with the NASA SEMAA office and local sites. In addition, the proposed data collection instruments were pilot tested with respondents drawn from the target populations. Respondents were asked to comment on the clarity and content of the questions. The duration of the data collections were recorded to help with an accurate estimation of time burden.

A.9 Explanation of Any Payment of Gift to Respondents

No payment or gift will be provided to respondents to the pre-test survey. As an incentive to complete and return the post-test survey, families who complete and return the post-test student survey and the post-test parent survey will receive a $20 gift card redeemable as cash at major retailers. The U.S. Department of Education has recently obtained empirical support for the positive effect of cash incentives on response rates. In an experimental study of the impact of financial incentives on survey response rates for teachers participating in the Reading First Impact Study, the response rate for teachers offered a cash incentive (of $15 or $30) was 76 percent, whereas the response rate for teachers not offered an incentive was 64 percent, a difference that was statistically significant (p < .01).² Other studies have found that the use of incentives can improve response completeness.³ In addition, the Council of Professional Associations on Federal Statistics recommended to OMB that incentives be considered to ensure cooperation from members of a control or comparison group where the absence of adequate response rates for the group would potentially compromise the findings (CPAFS, 1993). These results suggest that the moderate cash incentive offered to families in the current study will help ensure that the study achieve acceptable response rates, particularly because the study requires the cooperation of a control group of students and parents.

A.10 Assurance of Confidentiality Provided to Respondents

The contractors conducting the study will be required to adhere to the following procedures:

• Access to the electronic files shall be controlled by user ID and by group membership. All paper files (such as hand-written interview notes, completed surveys) shall be stored in locked cabinets. All electronic and paper files shall be destroyed two years after the end of the contract.

• Names and other identifiable information shall be redacted in all primary data (interview notes, survey results) and replaced with identifier numbers. A separate file shall be created that links interviewee names to the identifier numbers.

• All data shall be reported in aggregate and will not contain any identifying information (such as respondent’s name, address, or affiliation with an identified SEMAA site).

• Respondents will be provided with the following statement of confidentiality: “The answers you and your child provide on the surveys will be kept confidential to the extent permissible by law. The Privacy Act of 1974, as amended, protects the information you and your child provide for this study. Our reports do not include your name, your child’s name, or any other information that could be used to identify you or your child. We will not share information that could identify you or your child with NASA or SEMAA staff. ” Respondents will also be told that participation in the study is voluntary and that there will be no consequences to non-participation.

Prior to any data collection from students and parents, these individuals will be advised of the purpose and use of the data collection, and the fact that participation is voluntary (see the Parental Permission and Consent Form, Attachment 4, and the Child Assent Form, Attachment 5).

A.11 Justification for Sensitive Questions

Data collection instruments will not include any sensitive questions. (see Attachments 1 and 2)

A.12 Estimates of Hour Burden Including Annualized Hourly Costs

Estimates for the hour burden are based on time estimates provided by developers of the originals surveys that were adapted for the current evaluation, and similar interviews conducted on similar evaluations.

A.12–1. Estimates of Hour Burden

Type of Respondents	Number of Respondents	Frequency of Response	Average Time per Response	Annual Hour Burden
Site staff	50	1	1	50
Students	990	2	.50	990
Parents	990	2	.50	990
Totals	2030			2030

A.12-2. Annualized Cost To Respondents

Type of Respondents	Number of Respondents	Frequency of Response	Average Time per Respondents	Hourly Wage Rate	Respondent Cost
Site Staff	50	1	1	35.00	1,750
Students	990	2	.50	7.00	6,930
Parents	990	2	.50	20.00	19,800
Totals	2030				28,480

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers

There are no annualized capital/startup or ongoing operation and maintenance costs involved in collecting the information. Other than the opportunity costs represented by the time to complete the surveys, there are no direct monetary costs to respondents.

A.14 Annualized Cost to the Federal Government

The estimated cost to the Federal Government for the data collection activities included in this request for approval is $724,350. This cost estimate includes: instrument development and pretesting; staff training; site recruitment; implementing random assignment; collecting data; editing, key entry, and data processing.

A.15 Explanation for Program Changes or Adjustments

This is a new collection of information.

A.16 Plans for Tabulation and Publication and Project Time Schedule

The data collection will be conducted by an outside contractor firm that will work with the Project Officer.

To test the impact of SEMAA on students and parents, we will use regression models that account for the clustering of respondents within sites (multi-level modeling) and key covariates. For each outcome of interest (e.g., student interest in pursuing future science coursework at post-test) we will test the difference between the treatment (SEMAA participation) and control groups (embargoed SEMAA participation) controlling for key covariates such as pretest score on the outcome measure (baseline interest in pursuing future science coursework at pre-test), extent of involvement in non-SEMAA informal science activities, number of years’ prior participation in SEMAA, and key demographic variables such as ethnicity and parent’s highest level of education obtained. The statistical significance of impact estimates will be tested using F-tests and an alpha level of .05.

For the implementation module, we will employ simple descriptive statistics—such as counts, ranges, and frequencies—will be employed, and statistical tests, such as ² test or t-test, to test for differences between groups will be employed. The analyses of the interviews will include simple frequencies as well as descriptive summaries of emergent themes.

An analytic evaluation report will be prepared based on findings from the student and parent surveys and the telephone interviews with key informants at the site.

A.16-1 Project Time Schedule

Activity	Schedule
Schedule interviews	1 months after OMB approval
Conduct interviews	immediately after OMB approval
Recruit survey respondents	4-5 months after OMB approval
Implement survey	5-6 months after OMB approval
Analyze data	8 months after OMB approval
Report findings	9 months after OMB approval

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

The data collection instruments will display the expiration date.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are sought.

1 National Research Council (2008) NASA's Elementary and Secondary Education Program: Review and Critique. Committee for the Review and Evaluation of NASA's Precollege Education Program, Helen R. Quinn, Heidi A. Schweingruber, and Michael A. Feder, Editors, Board on Science Education, Center for Education, Division of Behavioral and Social Science and Education. Washington, DC: The National Academies Press.

2 Gamse, B. & Jacob, R. (2005). Memorandum to Tracy Rimdzius, Institute for Education Sciences, U.S. Department of Education (August 2, 2005). Reading First Impact Study: Incentive Substudy (Revised). Cambridge, MA: Abt Associates, Inc.

3 See Baumgartner et al, 1998; Berk et al, 1987; James & Bolstein, 1990; Singer et al, 1998

2