PEP OMB clearance request

American Institutes for Research®

Evaluation of the Carol M. White Physical Education Program -
OMB Clearance Request

December 15, 2010

Prepared By:

American Institutes for Research^®

1000 Thomas Jefferson St., NW

Washington, DC 20007

Prepared for:

U.S. Department of Education

Policy and Program Studies Service

Office of Planning, Evaluation, and Policy Development

Introduction and overview

The U.S. Department of Education (ED) is conducting an evaluation of the Carol M. White Physical Education Program (PEP). The American Institutes for Research (AIR) is under contract with ED to conduct the evaluation. The purpose of this evaluation is twofold: a) to describe the key design features of PEP grantees’ projects and their implementation and b) to examine changes in student outcomes pertaining to physical activity level, fitness level, and nutrition intake as well as the extent to which changes in student outcomes are associated with features of program design and implementation. The study will survey project administrators of PEP grants. In addition, the evaluation will analyze changes in student outcomes based on data for GPRA measures provided by PEP grantees. This OMB request is for the clearance for the survey instruments and for requesting PEP grantees to provide raw data that they will be collecting to report on GPRA measures but have not been required to report to ED.

This document contains a description of the evaluation of the PEP and provides context on the data collection instruments for which we are seeking clearance. Parts A and B of the supporting statement for the Paperwork Reduction Act Submission and the instruments for which we are seeking clearance are contained in separate files.

Background

In establishing the PEP, Congress acknowledged the critical need to improve physical education programs for K-12 students, in order to help them make progress toward meeting state standards for physical education. Since its inception in 2001, as a major federal funding source for physical education, the PEP has supported a variety of projects that encourage fitness and healthy lifestyle choices among K-12 students. Recently, in response to the current Administration’s call for action to prevent and decrease childhood overweight and obesity, the PEP underwent substantive changes to strengthen and enhance the program’s principal objectives so that they align more closely with current best practices and research related to improving children’s health and fitness. Because of the recent changes and enhancement, it is important to document how grantees are implementing the new PEP and assess the extent to which the new program priorities and requirements produce desired student outcomes as intended.

Overview of Study Design

ED has put forth four sets of evaluation questions that pertain to basic program implementation, partnerships, data use, and student outcomes, respectively.

• Evaluation questions about basic implementation

1. What does the planning, partnership, and implementation process look like during the life cycle of a grant?

2. How do grantees interpret the elements (of the six) they choose to implement?

3. What gaps are identified by the grantees based on the School Health Index (SHI) needs assessment? Do grantees put in place specific plans to address the identified gaps?

4. To what degree do grantees change school physical activity and food policies?

5. For grantees that use grant-related funds to improve their physical education or nutrition instruction curricula, how do they use CDC’s Physical Education Curriculum Analysis Tool (PECAT) and Health Education Curriculum Analysis Tool (HECAT) to inform curricula changes?

6. How do local wellness policies influence the work of the grantees? If a grantee does not have a local wellness policy, does it adopt one?

7. Do grantees encounter unforeseen problems when implementing the new requirements?

8. What lessons are learned by the grantees that could inform future ED policy regarding the PEP program?

9. What are grantees’ priorities for funding? What are PEP funds spent on? What is cost per unit (e.g., school, student served).

• Evaluation questions about partnerships

1. How many grantees develop partnerships with supporting community entities?

2. What is the average number of partners for a grantee?

3. How do grantees work with partners?

4. What roles do partners play?

5. How do the activities planned by grantees with partnerships differ from grants without partnerships?

• Evaluation questions about data use

1. How many grantees integrate BMI in their program?

2. How many grantees inform parents about the result of their child’s BMI assessment?

3. How do grantees use data, such as BMI and other performance measures, for improvement purposes?

• Evaluation questions about student outcomes

1. How do the physical activity level, fitness level, and nutritional intake of students served by the grantees change over time?

2. To what extent are features of program design and implementation associated with changes in student outcomes?

To answer ED’s evaluation questions, AIR proposes a two-phase research design, drawing on survey data to be collected from administrators at PEP projects and student outcome data pertinent to physical activity levels, fitness, and nutrition intake. Below we present a graphic representation of AIR’s proposed design for the overall evaluation (Exhibit 1).

• Phase 1: Program Design and Implementation (PDI) Surveys

AIR will survey the project administrators at all 2010 grantees in order to gather rich information about the design and implementation of the PEP program. Two waves of PDI surveys will be administered to both local education agency (LEA) grantees and community-based organization (CBO) grantees. The Year 1 PDI survey will gather information about the design and implementation of the new awards under the revised program. The Year 3 PDI survey will focus on grantees’ reflections on accomplishment of program goals, challenges in implementing the program, and lessons learned. Together, the Year 1 and Year 3 PDI surveys will collect detailed information about the planning, operations, activities, challenges, and perceived outcomes of the PEP program as implemented by the FY 2010 cohort of grantees. Such information will provide valuable feedback to ED on the new PEP program and inform future improvement of the program.

• Phase 2: Analyses of Changes in Student Outcomes

AIR will analyze student outcome data that will be collected by the PEP grantees as part of the program. AIR will use these data to examine changes in key student outcomes and the associations between changes in student outcomes and features of program design and implementation.

Exhibit 1. Schematic Representation of AIR’s Two-Phase Research Design

Below we elaborate on each of the two proposed data collection phases and the instruments to be employed during each phase, as well as the data analysis procedures. Each of the data collection instruments proposed by AIR to collect the program design and implementation data is contained in a separate document.

Data Collection and Instruments

Data gathered as part of AIR’s two-phase research design will serve as documentation of the implementation process of PEP projects and will be used to examine changes in student outcomes pertaining to physical activity level, fitness level, and nutrition intake, as well as the extent to which changes in student outcomes are associated with features of program design and implementation. All instruments developed by AIR have been thoroughly reviewed by the evaluation’s technical work group (TWG) for content and suitability and will be field-tested by a small number of PEP project administrators.

Administration of the Program Design and Implementation (PDI) Surveys

During Phase I of the evaluation, AIR will administer the two waves of PDI surveys to all of the 2010 PEP grantees. The first wave of the surveys will be administered in Spring 2011 (Year 1) and the second wave in Winter 2012/2013 (Year 3). The Year 1 PDI survey will gather information about the design and implementation of the revised grant program, such as areas of focus, number of partnerships, and usage of BMI data. The Year 3 PDI survey will focus on grantees’ reflections on accomplishment of program goals, challenges encountered in implementing the program, lessons learned, and sustainability of program activities post grant funding.

For each wave of the PDI survey, two forms have been developed: one for LEA grantees and the other for CBO grantees. That is, four PDI surveys will be administered. The surveys will be completed by key personnel (e.g., program directors) of each PEP grantee. The surveys will include both close-ended and open-ended questions in order to allow respondents the opportunity to elaborate on their responses. Both survey forms will be available on-line and incorporate skip patterns that are based on the characteristics of grantees (e. g., with partner or not, or used BMI or not).

Together, the Year 1 and Year 3 PDI surveys address ED’s questions with regard to basic implementation, partnership, and data use by collecting detailed information about the projects’ planning, operations, activities, challenges, and perceived outcomes. Through these two surveys (each with two forms: LEA and CBO), we will be able to assess whether the program as implemented appropriately reflects the new direction of the PEP and how the new program priorities and requirements are addressed by the grantees. Such information will provide valuable feedback to ED on the new PEP program and inform future improvement of the program. Therefore, we request clearance for the four PDI surveys.

Collection of Data on Student Outcomes

To address the two research questions about student outcomes and gain preliminary information on the effectiveness of the PEP in producing changes in desired student outcomes, AIR will request de-identified, student-level outcome data on the GPRA measures collected by each of the 2010 grantees. Following ED guidance, all PEP grantees will collect student-level data on the GPRA measures and report the GPRA measures in the aggregate form to ED as part of their grant performance reports. The GPRA measures are summarized below (see Table 1).

Table 1. GPRA Measures Required by the PEP Mandate

Performance Measures	Indicators	Required assessment tools	Grade levels
Student physical activity levels	Percentage of students who engage in 60 minutes of daily physical activity	Pedometers	K-12
		3-Day Physical Activity Recall (3DPAR)	5-12
Student health-related fitness levels	Percentage of students who achieve age-appropriate cardiovascular fitness levels	20-meter shuttle run	Middle-high school
		Not specified	K-elementary
Student nutrition	Percentage of students who consume fruit two or more times per day and vegetables three or more times per day	Questions from the Youth Risk Behavior Survey	High school
		Other assessment tools selected by grantees	Elementary-middle

Data on the GPRA measures aggregated to the grantee level will be available to AIR at the end of the first and second grant year (i.e., 2011 and 2012). In addition, AIR will also request the grantees to provide the raw student-level GPRA data that they have collected but have not been required to report to ED at the end of the first and second grant years. Because all the 2010 PEP grantees will be collecting student-level outcome data in order to report their performance to ED, AIR’s request for raw student-level data will not impose additional data collection burden on the grantees. However, given that ED only requires the grantees to report aggregated data (as opposed to raw student-level data) on GPRA measures, additional burden related to data reporting will be involved. Thus, we seek clearance for requesting grantees to make available to us the raw student-level GPRA data that they will have collected.

Data Analyses

Analyses of PDI Survey Data

First, AIR will analyze the years 1 and 3 program design and implementation data to understand how grantees with different design features operate at early and late stages of the program respectively. In addition, we will combine and analyze the two waves of PDI survey data longitudinally, to document continuity and change in program implementation over time. We will conduct descriptive analysis to obtain sums, means, percentages, and frequencies of relevant program variables. Particularly, we will also inspect systematic variations based on key features of PEP projects (e.g., types of grantees, inclusion of agreement partners, grade levels served, and student demographics). Furthermore, the PDI data collected by AIR will be merged with the GPRA outcome data to examine the relationship between key design and implementation features of PEP projects and changes in key student outcomes.

Analyses of Student Outcome Data

If the PEP is successful in achieving its goals, the expectation is that there will be a detectable improvement in the relevant outcomes of students served by PEP grantees (e.g., physical activity level, fitness level, and healthy nutritional habits). In the absence of a comparison group, we will not be able to attribute improvement in student outcomes, if any, exclusively to the PEP. Nevertheless, assessing changes in student outcomes in grantee sites will still offer useful preliminary evidence for the effectiveness of the PEP.

To assess changes in student outcomes at the grantee level, we will combine the baseline GPRA data and GPRA data collected at eight follow-up time points over the course of the first two years of the grant period, as well as the program implementation data collected through the PDI surveys. We will conduct two-level multilevel modeling analyses to assess changes in student outcomes over time at the grantee level and the extent to which such changes are associated with program characteristics, such as types of grantees (i.e., LEA vs. CBO), existence of partners, grade levels served, and the extent of modifications in implementation.

Summary

The above discussion outlines AIR’s research design for evaluating the Carol M. White PEP program. To address evaluation questions related to basic implementation, partnerships, data use, and student outcomes, AIR has proposed a two-phase design that includes PDI surveys and analyses of student outcomes, which will provide rich information on key design and implementation features of the program, changes in key student outcomes, and the relationship between program features and changes in student outcomes. Findings from this study will provide valuable feedback to both ED and grantees regarding the performance of the PEP, and will inform future improvements to the program.

Notice of Trademark: “AMERICAN INSTITUTES FOR RESEARCH” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or registered trademarks of their respective owners.

1000 Thomas Jefferson Street, NW | Washington, DC 20007‑3835