NASA SoI OMB Package # 1 Part A 4-18-11

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART A

NASA Summer of Innovation FY2011

PARENT CONSENT FORM/SURVEY & AWARDEE PLANNING DATA COLLECTION

National Aeronautics and Science Administration

Revised April 5, 2011

Part A: Justification

A.1 Explanation of Circumstances That Make Collection of Data Necessary

The National Aeronautics and Space Administration (NASA) Office of Education, requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, an emergency clearance for NASA to collect parental consent forms and awardee data about their plans for implementation as part of the formative evaluation of NASA’s Summer of Innovation (SoI) Project FY2011.

In 2010, NASA’s Office of Education launched the SoI pilot, a NASA-infused summer experience for middle school students who underperform, are underrepresented, and underserved in science, technology, engineering, and mathematics (STEM) fields. The SoI pilot utilized a multi-faceted approach to reach and engage middle school students in STEM learning with NASA content and experiences. The topics ranged broadly and included activities concerning robotics, rocketry, engineering design, meteorology, space science, and climate science. Evaluation data were collected from various sources during the pilot to produce lessons learned regarding program design, implementation, and program evaluation. The pilot evaluation produced valuable insight into the program and was used to redesign SoI for this year. However, it was limited in its ability to generate hypotheses about promising practices; in most cases the evaluation of the pilot was not able to field baseline surveys, necessary for assessing change in the program’s outcomes of interest.

Drawing heavily upon the lessons learned identified in the evaluation of last summer’s SoI pilot, NASA modified its approach to focus on expanding the capacity of community and school-based organizations to engage youth in STEM learning activities. In FY2011, NASA will implement a three-tiered solicitation and award structure that is designed to provide selected awardees with different levels of funding and access to NASA staff, facilities, and technology; these NASA resources will support the awardees’ efforts to provide 4^th through 9^th grade students with high-quality, inquiry-based content learning experiences during the summer and throughout the following school year. For the formative evaluation of SoI, NASA wants to collect data from two of the three tiers (the national awards and NASA centers) where it is investing the majority of SoI project funds. It also wants to be able to collect the baseline data that will inform the program’s development and refinement and allow it to generate hypotheses regarding promising practices.

NASA has revised SoI’s expected outcomes to better reflect the nature and objectives of the new SoI model (see Appendix A for the revised logic model).¹ Given that the activities are short in duration, SoI 2011 has shifted the focus of the program from attempting to impact student achievement directly to inspiring and engaging middle school students in NASA STEM content. To accomplish this goal, NASA has set programming requirements as follows: national awardees are required to provide 40 hours of student STEM activities over the summer and an additional 25 hours by March 2012, while NASA center partnerships must provide 20 hours of student STEM activities during the summer and an additional two STEM activities by March 2012.

Expected outcomes for students in FY011 include increased interest in STEM topics and STEM careers, as well as increased participation in informal STEM activities in the short term; ultimately, the program aims to increase the overall number of students pursuing STEM degrees and related careers and, more specifically, increase the proportion of underrepresented students who pursue these paths. To support and produce these student outcomes, NASA seeks to build the sustainability of the awardees’ programs by supporting the development of partnerships with formal and informal STEM institutions so that they can eventually operate high-quality STEM programs independently at scale, even as SoI funding diminishes in subsequent years. NASA also aims to increase classroom teachers’ access to and use of NASA content and resources so that over the long term, they have better understanding of NASA content, increased confidence in teaching NASA topics, and improved ability to teach NASA topics.

This clearance request pertains to the portions of the data collection that need to be conducted before June 2011 for the formative evaluation. These include a parental consent form and a short associated survey (see Appendix B), a question-by-question justification for each survey item (Appendix C), as well the awardee planning form, awardee focus group consent script, and awardee focus group protocol (Appendices D, E, and F). As suggested by the lessons learned from last year’s pilot evaluation, NASA wants the parent consent forms to be included in the student registration process that will begin shortly after awards are announced on April 18, 2011; last year’s pilot experience made it clear that obtaining parent consent after they had registered their students is both logistically difficult and can raise suspicion regarding the evaluation’s intent. Furthermore, planning information needs to be collected from awardees prior to the start of the summer activities, as recollection of plans typically become increasingly unreliable once the sites begin implementation. Plus, SoI’s kick-off meeting in May provides an especially opportune time to conduct focus groups when all awardee PIs are present in Cleveland, Ohio.

As described in more detail below, the parent consent form will fulfill multiple purposes: it will solicit consent for student participation in the student surveys; it will inform the sampling frame from which the national evaluators can select representative groups of students to complete surveys; and it will allow NASA to fulfill its internal monitoring requirements regarding participant demographics. The awardee reporting forms and focus groups can be used to gather the baseline information required to characterize SoI’s participants. The awardee tools will be used to collect the planning data that help construct the sampling frame and serve as a baseline for understanding the fidelity of the implementation and discern lessons learned. These instruments will allow NASA to capture awardees’ plans for its program models and implementation strategies for later comparison with actual implementation. While we expect to capture some of this information from awardees’ proposals, we anticipate that awardees will finalize and operationalize their plans after winning the funding. Some key details may not be included in the proposals and practical considerations may necessitate some deviation from their proposals. As a result, the national evaluation team needs to verify and collect the planning data closer to the time of the activities’ initiation. A separate request will be made for subsequent data collection activities that are scheduled to begin after the awardees have been identified in the spring.

The national evaluation is an important opportunity to collect information needed to refine the program’s design .However, the evaluation is not intended to address questions of program impact on students or teachers. It will explore whether NASA’s requirements, as now defined, are feasible and appropriate, and continue to generate lessons learned for future implementations of SoI and NASA’s education activities more broadly. It will help NASA identify the promising practices and models that missing baseline data during the pilot limited. Finally, it will help NASA understand which practices might be ready for more rigorous examination in a future summative study.

A.2 How the Information Will Be Collected, by Whom, and For What Purpose

How Information Will Be Collected and by Whom

Data will be collected at the awardees and NASA center program sites using a parent consent form and an associated short survey. Planning data will be collected from the national awardee sites using a planning form and focus groups with program administrators.

NASA would like the SoI awardees and NASA centers to include a parent consent form and an associated parent survey (Appendix B) in the registration materials that parents must return to enroll students in the SoI activities. The first page of the form provides information regarding the purpose of the evaluation, describing what participation entails and how the national evaluators will protect students’ privacy. It also includes contact information for parents who have questions about the study and their students’ rights as participants. On the second page, parents are asked to indicate whether they provide permission for their student to participate. Finally, the last page of the consent form is a brief survey that asks parents to report student demographic characteristics as well as the reasons for enrolling their child in the program. Parents will be expected to return the consent form to the SoI sites, which would then send these forms to the national evaluator for safe-keeping and data entry. Participation in SoI programs will not be conditional on providing consent. We anticipate that enrollment will begin shortly after the awards are announced on April 18, 2011.

To collect planning data, once the awards have been announced, NASA will distribute electronic planning forms to awardees. The national evaluator will pre-populate these forms based on the sites’ proposal; sites will be required to review these forms to ensure their accuracy, fill in any missing data, and submit prior to the kick-off meeting, scheduled for May 2011. The responses will inform the focus groups during SoI’s kick-off meeting so that the national evaluator may use the focus group time most effectively.

For What Purpose

The purpose of this data collection effort is to collect descriptive information on the students and SoI grantees that is essential to inform the national evaluation. The goal of the national evaluation is formative, that is, to gather data that will inform NASA’s continued development of the program as well as to assess whether evidence supports the progression to a more rigorous summative, impact evaluation. Accordingly, the study will focus on SoI’s implementation and associated outcomes to identify promising practices. The implementation work will develop a description of the awardee models and their local objectives prior to implementation, revealing how the planned methods are linked to desired outcomes; the data will enable us to assess the fidelity of implementation and generate lessons learned to improve future activities.

Exhibit 1 on the following page outlines the research questions for the SoI national evaluation, data sources, and outcome measures; those that will be addressed with data that need to be collected before June 2011 (and the items for which this request seeks emergency clearance) are bolded. .

Exhibit 1: National Evaluation Research Questions
Research Questions	SoI Tier of Interest	Data Sources	Outcomes
1. Who participates in SoI FY2011?	National awardees and NASA Centers	Parent consent forms/surveys forms	Participant demographic information
2. Does student interest in science change significantly between the baseline and follow-up surveys? If so, are these changes larger at some awardees/NASA Centers than others? If a change is detected, does science interest change significantly between the summer follow-up survey and the school year follow-up survey? If so, are these changes larger at some awardees/NASA Centers than others?	National awardees and NASA Centers	Student surveys	Overall interest in science, career interest in science, leisure interest in science
3. Does comfort in teaching NASA topics and access/use of NASA resources change between baseline and summer follow-up surveys? If so, are these changes larger at some awardees than others? If a change is detected, does comfort in teaching NASA topics and access/use of NASA content and resources change between the summer follow-up survey and the school year follow-up surveys? If so, are these changes larger at some awardees than others?	National awardees only	Teacher surveys	Access and use of NASA content and resources; comfort in teaching NASA topics
4. How do awardees plan and implement their summer and school-year activities? What are the similarities and differences across the approaches? Are there any apparent relationships between the approaches and desired outcomes?	National awardees only	Planning forms; Implementation forms; focus groups; site visits	Program scheduling, activities, duration, content, delivery methods, participants
5. What supports and challenges do awardees face in implementing their SoI programs? How do they negotiate these challenges?	National awardees only	Focus groups; Site visits	Implementation challenges and successes
6. How are awardees preparing to operate independently of SoI funding?	National awardees only	Focus groups; Site visits	Sustainability planning

The parent consent form and associated survey addresses the first research question. This instrument also supports the national evaluation’s data collection by soliciting consent for student participation in the student surveys, required to administer surveys and assess student outcomes. The parent consents are also part of the registration process. Furthermore, they will be used by the national evaluator to investigate variance in student outcomes according to different demographic and socio-economic groups. All the parent survey data, however, will be used by NASA for internal monitoring and compliance purposes.

The fourth research question will be answered using data from the awardee planning forms and focus groups. These instruments will collect the awardees’ plans for implementation and will also inform the construction of the sampling frame and the sampling strategy, serving as a means for exploring the extent to which awardees plans can be implemented with fidelity, given the reality of the context in which implementation takes place. Not only will the data be vital in understanding the context in which any change in key outcomes is identified, but will also serve as a resource to support additional research on STEM learning as it relates to informal and K-12 education by academic researchers and others interested in STEM engagement.

A.3 Use of Improved Information Technology to Reduce Burden

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. Therefore, the parent survey and form were designed to only include questions not available elsewhere. The national evaluator will provide training and support to all sites to assist in obtaining systematic and consistent data.

The national evaluator designed the survey to require minimal effort. The survey was also designed to be easy to read with straight-forward questions and minimal skip-patterns. All parent survey data will be collected on paper distributed by the sites. The national evaluator’s electronic mail address and toll-free telephone number will be included on the parent consent form for participants who have questions. The sites will return the consent forms to the national evaluator. Taken together, these procedures are all designed to minimize the burden on respondents.

A.4 Efforts to Identify and Avoid Duplication

This effort will inform the national evaluation efforts to explore students’ interest in science as it relates to SoI activities; as such, there is no similar evaluation being conducted and there is no alternative source for collecting the information. The national evaluator will pre-populate implementation reporting forms using information included in the awardees’ proposal so that sites only need to review the report for accuracy and fill in any missing data. NASA will identify a single point of contact for the awardees who will ensure that duplicative data collection is avoided.

A.5 Efforts to Minimize Burden on Small Business or Other Entities

No small businesses will be involved as respondents. The primary survey entities for data collection efforts described in this package are parents and awardees. Burden is minimized for all respondents by requesting only the minimum information required to meet study objectives. All primary data collection will be coordinated by the site administrators in partnerships with the national evaluator, so as to reduce the burden on the SoI awardees and NASA centers.

A.6 Consequences of Less-Frequent Data Collection

If the proposed parent survey data were not collected, NASA would not fulfill NASA’s compliance need to ascertain the demographic characteristics of the SoI participants. Thus, by not administering surveys, federal resources would be allocated and program decisions would be made in the absence of evidence about who benefitted from the SoI efforts. If the proposed awardee planning data were not collected, NASA would not understand how the program models were intended to work and what would be required to replicate the models, should they be associated with promising outcomes.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

There are no special circumstances associated with this data collection.

A.8 Federal Register Comments and Persons Consulted Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, NASA published a notice in the Federal Register announcing the agency’s intention to request an OMB review of data collection activities. The notice was published on January 10, 2011, in volume 76, number 6, page 1461, and provided a 30-day period for public comments. To date, no comments have been received.

The parent survey instruments were developed by the national evaluators, Abt Associates, Inc. and staff from the Education Development Center (EDC), comprising: Ricky Takai, Principal Investigator; Hilary Rhodes, Project Director; Kristen Neishi, Deputy Project Director; and Melissa Velez, Survey Analysis Task Manager; and Jacqueline DeLisi, Abigail Levy and Yueming Jia at EDC. Feedback on the surveys was solicited from staff at NASA’s Office of Education.

A.9 Payments to Respondents

There will be no payments to respondents.

A.10 Assurance of Confidentiality

Every effort will be made to maintain the privacy of respondents, using several procedural and control measures to protect the data from unauthorized use. Collected data will not be released with individual identifiers, and results will be presented only in aggregated form. A statement to this effect will be included on the first page of each survey and will be read to awardees prior to conducting a focus group (See Appendices B & E for draft versions of the parent consent form and focus group consent script). Respondents will be assured that all information identifying them will be kept private.

The procedures to protect data during information collection, data processing, and analysis activities are as follows:

• All respondents included in the study sample will be informed that the information they provide will be used only for the purpose of this research. Individuals will not be cited as sources of information in prepared reports.

• Hard-copy data collection forms will be delivered to a locked area at the contractor’s office for receipt and processing. The contractor will maintain restricted access to all data preparation areas (i.e., receipt, coding, and data entry). All data files on multi-user systems will be under the control of a database manager, with access limited to project staff on a “need-to-know” basis only.

• Individual identifying information will be maintained separately from completed data collection forms and from computerized data files used for analysis.

The national evaluation team will also have the data collection protocols and surveys reviewed by Abt’s Institutional Review Board (IRB), a process which has already been initiated and will be completed once the awardees have been selected. Prior to their use, Abt’s IRB will approve the data collection instruments, including student and parent surveys, the parent consent form, the focus group protocol, the consent script, and the planning form. The IRB will assure that the data collection protocols and procedures, including consent forms, abide by strict procedures to maintain privacy.

A.11 Questions of a Sensitive Nature

The questions included on the data collection instruments for this study do not involve sensitive topics and respondents may skip items if they so wish.

A.12 Estimates of Respondent Burden

Exhibit 2 presents estimates of the reporting burden for the surveys and the implementation plan reporting: we estimate that the annualized response burden is 3,333 hours for parents and 25 hours for awardees for a total of 3,358 hours. This estimate assumes that it will take about five minutes for parents to read the consent script and answer the demographic questions. As the form will be included in registration materials, we assume that all parents registering students will also review and return the documents. Estimates for the hour burden are based on time requirements from similar surveys conducted on comparable evaluations.

Awardee PIs will be asked to participate in focus groups scheduled for 2 hours during the SoI kick-off meeting. Given the time it took awardees to complete planning forms last year, and in similar evaluations, we assume these reports will require 30 minutes or fewer. For the estimate of burden, we assume that all awardees will participate in the focus groups and complete a planning form.

Although not included in this package as its data collection will take place after June 2011, the national evaluation will also request that students and teachers to complete surveys. In order to provide some sense of total burden, however, we executed power calculations under some broad assumptions that would allow us to detect an overall change in student interest. If the student and teachers surveys take approximately 10 minutes each, as we assume given the time requirements from similar teachers surveys conducted on comparable evaluations and the average time (times ranged between 3 and 15, taking an average of 8 minutes) it took seven students to finish the pilot surveys, the student and teacher surveys generate an additional 4,275 hours in annualized response burden. Furthermore, we anticipate that the site visits and implementation reporting will generate an additional 125 hours of PI or their staff’s time for a total of approximately 7,758 respondent hours for the entire evaluation. These assumptions will be updated once NASA has selected the awards and the sites have submitted planning forms.

A.13 Estimates of the Cost Burden to Respondents

We estimate that the annualized cost burden is $79,767 for parents and $925 for awardees, for a total of $80,692.The cost burden for parents is estimated using the 2009 U.S median income and the assumption of 40-hour work weeks across the year. The cost burden for students is estimated using the federal minimum wage. For the cost burden to PIs, we assumed that this year’s PIs will be similar to the ones who participated in the pilot, several of whom were associate professors at baccalaureate institutions. According to the American Association of University Professors 2009/10 survey, these individuals earn an average of $83,700 a year nationally. There is no annualized capital/startup or ongoing operation and maintenance costs associated with collecting the data. Other than their time to complete the surveys, which is estimated in Exhibit 2, there are no direct monetary costs to respondents.

Again, although not included in this package, we have calculated some preliminary estimates of the cost burden to respondents of the teacher surveys ($8,185), student surveys ($28,058), and implementation reporting and follow-up ($3,316) for a grand total of $120,251. These estimates will be updated once NASA has selected the awards, the sites have submitted planning forms, and our sampling strategy finalized.

Exhibit 2. Estimates of Annualized Burden Hours and Cost for Data Collected Before Jun 2011
Data Collection Sources	Number of Respondents a	Frequency of Response	Total Minutes per Respondent	Response Burden in Hours	Estimated Cost Per Hour b	Total Burden (Costs)
Parent consent form & survey	40,000	1	5	3,333	$23.93	$79,767
Awardee planning focus group	10 c	1	120	20	$40.24	$805
Awardee planning form	10 d	1	30	5	$23.93	$120
Total	40,020			3,358		$80,692
Notes: a Number of parents filling out consent form based on total number of students NASA expects to reach (25,000 at the national awardees and 15,000 at the NASA centers). b Estimated cost per hour for parents is calculated based on the national median income of $49,777 (~23.93 per hour, assuming a 40 hour work week) for 2009 according to the Current Population Survey (http://www.census.gov/prod/2010pubs/p60-238.pdf, retrieved on March 9, 2011). Estimated cost per hour for awardees is calculated based on assumption that, as last year, PIs will likely be associate professors, whose national average annual salary is $83,700 (~$40.24 per hour, assuming a 40 hour work week) for 2009-2010 at Baccalaureate institutions, as calculated using American Association of University Professors survey results (http://chronicle.com/article/Searchable-Database-AAUP/64231/, retrieved on March 4, 2011). c NASA intends to fund 10 NASA Centers and 10 national awards. d Assumes that evaluation coordinator – not the PI – completes the planning forms.

A.14 Estimates of Annualized Government Costs

This information collection activity has been developed in the performance of the Contract Number: NNH08CD70Z (Task Order NNH11CC54D). Under this contract, the evaluation’s plans will cost approximately $40,795 to update SoI’s pilot survey instruments. The awardees’ evaluation coordinators will collect the survey data.

A.15 Changes in Hour Burden

This is a new collection of information.

A.16 Time Schedule, Publication, and Analysis Plan

The schedule shown in Exhibit 3 displays the sequence of activities required to conduct the information collection activities and includes key dates for activities related to data collection, analysis, and reporting. As noted in the Exhibit, two evaluation reports based on findings from the surveys and implementation data will be prepared; one following the completion of summer activities (Fall 2011) and one after the completion of the school-year activities (Summer 2012).

Exhibit 3. SoI Schedule
Activities and Deliverables	Responsible Party	Date
Development & refinement of instruments	National evaluator	January –April 2011
Parent consent form & survey collection	National evaluator & site administrators	May – June 2011
Student survey data collection a	National evaluator & site administrators	July – August 2011; March 2012 b
SoI kick-off meeting and planning focus groups	NASA & national evaluator	May 2011
SoI planning forms submission	Awardees	May – June 2011
Teacher survey data collection a	National evaluator & site administrators	June – August 2011; March 2012 b
SoI site visits a	National evaluator	July – August 2011
SoI implementation forms submission a	Awardees	July – August 2011
Data analysis of baseline/follow-up student and teacher surveys, implementation data	National evaluator	Fall 2011
SoI “Lessons Learned” meeting and implementation focus groups a	NASA & national evaluator	Fall 2011
Expert panel review meeting	NASA & national evaluator	Fall 2011
National Report #1	National evaluator	Fall 2011
School-year check-in with PIs b	National evaluator	Winter 2011
Post school-year PI focus groups b	National evaluator	Spring 2012
Data analysis of post-school year student and teacher survey, implementation data	National evaluator	Summer 2012
National Report #2	National evaluator	Summer 2012

^a Data collection activity will be included in a subsequent emergency clearance package to cover this summer’s data collection efforts, which will be submitted once awardees have been identified.

^b Data collection activity will be included in a subsequent clearance package pertaining to the school-year data collection efforts, which will be submitted before school-year activities begin as emergency clearance likely will not cover the school-year data collection efforts.

The national evaluator will provide cross-site analyses as well as awardee reports to describe the similarities and differences across the awardees’ models and explore whether overall any change in student short-term outcomes at either the NASA centers or at the national awardees is discernable. Data will be analyzed separately for NASA centers and awardees. Given the descriptive nature of the information sought, we will generally rely on simple descriptive statistics—such as counts, ranges, and frequency—for the analyses of the data.

Although not included in this package, the national evaluation will use means and standard deviations to describe both central tendency and variation for survey items using continuous scales. Frequency distributions and percentages will be used to summarize answers given on ordinal scales. We also plan to calculate differences across baseline, post-summer, and post school year surveys to examine whether there is discernable change over time on short-term outcomes overall. We are considering the use of statistical tests, such as ² tests, McNemar’s Test, or paired t-tests depending on the distribution of the outcome variables, to test for differences between baseline and follow-up periods. If an overall change is detected, we will identify the awardees where variation existed and then use the survey data in conjunction with the implementation data to explore for associations and generate hypotheses.

A.17 Display of Expiration Date for OMB Approval

NASA is not requesting a waiver for the display of the OMB approval number and expiration date on the data collection instruments.

A.18 Exceptions to Certification Statement

This submission does not require an exception to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

1 This package is the first of two that will be submitted for SoI. It focuses on the data collection efforts prior to June 2011, and thus does not include the data collection necessary to measure student and teacher outcomes (as described in the logic model). Two packages are necessary so that NASA may collect key baseline data – including parent consent and awardee plans for implementation- prior to the start of this year’s activities and also have sufficient time to review grant proposals, make awards, and use the proposal information to inform the evaluation’s sampling and data collection strategies.

National Aeronautics and Space Administration Part A: Justification 1